Console [trevis-41vm7] log opened at 2017-04-02 03:12:23 UTC. Console [trevis-41vm7] joined by on pts/53 at 04-03 15:55. Console [trevis-41vm7] disconnected from at 04-03 16:16. Console [trevis-41vm7] connected to at 04-03 16:16. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005034] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008203] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011690] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014092] Mount-cache hash table entries: 4096 [ 0.015301] Initializing cgroup subsys memory [ 0.016011] Initializing cgroup subsys devices [ 0.018004] Initializing cgroup subsys freezer [ 0.019005] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.025004] Initializing cgroup subsys net_prio [ 0.026111] Disabled fast string operations [ 0.027019] mce: CPU supports 10 MCE banks [ 0.028055] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] tlb_flushall_shift: 6 [ 0.032174] Freeing SMP alternatives: 28k freed [ 0.037936] ACPI: Core revision 20130517 [ 0.039853] ACPI: All ACPI Tables successfully acquired [ 0.041025] ftrace: allocating 25811 entries in 101 pages [ 0.049274] smpboot: Max logical packages: 2 [ 0.050005] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.052004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.053252] Enabling x2apic [ 0.054000] Enabled x2apic [ 0.054004] Switched APIC routing to physical x2apic. [ 0.056550] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.057003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.060025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.062605] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.316049] Brought up 2 CPUs [ 0.316049] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.318006] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.319614] devtmpfs: initialized [ 0.322952] EVM: security.selinux [ 0.323005] EVM: security.ima [ 0.324000] EVM: security.capability [ 0.325763] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.326010] pinctrl core: initialized pinctrl subsystem [ 0.327164] NET: Registered protocol family 16 [ 0.328301] ACPI: bus type PCI registered [ 0.329006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.330096] PCI: Using configuration type 1 for base access [ 0.334087] ACPI: Added _OSI(Module Device) [ 0.335013] ACPI: Added _OSI(Processor Device) [ 0.336006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.337008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.339694] ACPI: Interpreter enabled [ 0.340013] ACPI: (supports S0 S5) [ 0.341000] ACPI: Using IOAPIC for interrupt routing [ 0.342028] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.345655] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.347010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.348021] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.349250] acpiphp: Slot [1] registered [ 0.350031] acpiphp: Slot [2] registered [ 0.351022] acpiphp: Slot [3] registered [ 0.352022] acpiphp: Slot [4] registered [ 0.353022] acpiphp: Slot [5] registered [ 0.354022] acpiphp: Slot [6] registered [ 0.355021] acpiphp: Slot [7] registered [ 0.356022] acpiphp: Slot [8] registered [ 0.357033] acpiphp: Slot [9] registered [ 0.358022] acpiphp: Slot [10] registered [ 0.359022] acpiphp: Slot [11] registered [ 0.360021] acpiphp: Slot [12] registered [ 0.361021] acpiphp: Slot [13] registered [ 0.362022] acpiphp: Slot [14] registered [ 0.363022] acpiphp: Slot [15] registered [ 0.364029] acpiphp: Slot [16] registered [ 0.365029] acpiphp: Slot [17] registered [ 0.366022] acpiphp: Slot [18] registered [ 0.367022] acpiphp: Slot [19] registered [ 0.368022] acpiphp: Slot [20] registered [ 0.369021] acpiphp: Slot [21] registered [ 0.370025] acpiphp: Slot [22] registered [ 0.371031] acpiphp: Slot [23] registered [ 0.372022] acpiphp: Slot [24] registered [ 0.373022] acpiphp: Slot [25] registered [ 0.374022] acpiphp: Slot [26] registered [ 0.375029] acpiphp: Slot [27] registered [ 0.376022] acpiphp: Slot [28] registered [ 0.377021] acpiphp: Slot [29] registered [ 0.378029] acpiphp: Slot [30] registered [ 0.379022] acpiphp: Slot [31] registered [ 0.380253] PCI host bridge to bus 0000:00 [ 0.381007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.382007] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.383005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.385006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.386977] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390805] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.391013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.408700] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.410076] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.411765] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.413153] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.414817] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.416221] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.417286] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.418015] vgaarb: loaded [ 0.419005] vgaarb: bridge control possible 0000:00:02.0 [ 0.420222] SCSI subsystem initialized [ 0.421054] ACPI: bus type USB registered [ 0.422041] usbcore: registered new interface driver usbfs [ 0.423018] usbcore: registered new interface driver hub [ 0.424057] usbcore: registered new device driver usb [ 0.425219] PCI: Using ACPI for IRQ routing [ 0.426350] NetLabel: Initializing [ 0.427007] NetLabel: domain hash size = 128 [ 0.428004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.429053] NetLabel: unlabeled traffic allowed by default [ 0.430164] Switched to clocksource kvm-clock [ 0.440473] pnp: PnP ACPI init [ 0.441532] ACPI: bus type PNP registered [ 0.443178] pnp: PnP ACPI: found 5 devices [ 0.444321] ACPI: bus type PNP unregistered [ 0.454055] NET: Registered protocol family 2 [ 0.455637] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.457879] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.459554] TCP: Hash tables configured (established 16384 bind 16384) [ 0.461101] TCP: reno registered [ 0.462165] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.463601] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.465170] NET: Registered protocol family 1 [ 0.466489] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.467932] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.469366] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.471112] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.472849] Unpacking initramfs... [ 4.609840] Freeing initrd memory: 42360k freed [ 4.629561] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.631194] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.633036] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.634545] Initialise system trusted keyring [ 4.635826] audit: initializing netlink socket (disabled) [ 4.637211] type=2000 audit(1491236220.824:1): initialized [ 4.658188] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.659728] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.662275] zpool: loaded [ 4.663347] zbud: loaded [ 4.664602] VFS: Disk quotas dquot_6.5.2 [ 4.665843] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.667571] msgmni has been set to 3357 [ 4.668841] Key type big_key registered [ 4.671550] NET: Registered protocol family 38 [ 4.673878] Key type asymmetric registered [ 4.676085] Asymmetric key parser 'x509' registered [ 4.678577] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.683157] io scheduler noop registered [ 4.685448] io scheduler deadline registered (default) [ 4.687930] io scheduler cfq registered [ 4.689919] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.691330] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.693019] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.695300] ACPI: Power Button [PWRF] [ 4.696629] GHES: HEST is not enabled! [ 4.697873] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.720645] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.722446] Non-volatile memory driver v1.3 [ 4.723693] Linux agpgart interface v0.103 [ 4.725043] crash memory driver: version 1.1 [ 4.726362] rdac: device handler registered [ 4.727659] hp_sw: device handler registered [ 4.728908] emc: device handler registered [ 4.730124] alua: device handler registered [ 4.731394] libphy: Fixed MDIO Bus: probed [ 4.732653] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.734179] ehci-pci: EHCI PCI platform driver [ 4.735447] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.736923] ohci-pci: OHCI PCI platform driver [ 4.738203] uhci_hcd: USB Universal Host Controller Interface driver [ 4.739906] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.741299] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.743584] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.744968] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.746487] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.748085] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.750351] usb usb1: Product: UHCI Host Controller [ 4.751698] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.753344] usb usb1: SerialNumber: 0000:00:01.2 [ 4.754864] hub 1-0:1.0: USB hub found [ 4.756088] hub 1-0:1.0: 2 ports detected [ 4.757444] usbcore: registered new interface driver usbserial [ 4.758916] usbcore: registered new interface driver usbserial_generic [ 4.760516] usbserial: USB Serial support registered for generic [ 4.762176] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.765252] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.766552] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.767968] mousedev: PS/2 mouse device common for all mice [ 4.769682] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.772218] rtc_cmos 00:00: RTC can wake from S4 [ 4.772799] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.773032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.779251] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.780869] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.782418] cpuidle: using governor menu [ 4.783805] hidraw: raw HID events driver (C) Jiri Kosina [ 4.785400] usbcore: registered new interface driver usbhid [ 4.786858] usbhid: USB HID core driver [ 4.788112] drop_monitor: Initializing network drop monitor service [ 4.789722] TCP: cubic registered [ 4.790873] Initializing XFRM netlink socket [ 4.792222] NET: Registered protocol family 10 [ 4.793899] NET: Registered protocol family 17 [ 4.795423] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.797232] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.798799] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.801571] Loading compiled-in X.509 certificates [ 4.802957] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.805565] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.808711] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.811357] registered taskstats version 1 [ 4.815310] Key type trusted registered [ 4.818708] Key type encrypted registered [ 4.822092] IMA: No TPM chip found, activating TPM-bypass! [ 4.823881] rtc_cmos 00:00: setting system clock to 2017-04-03 16:16:59 UTC (1491236219) [ 4.827253] Freeing unused kernel memory: 1680k freed [ 4.833340] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.852481] systemd[1]: Detected virtualization kvm. [ 4.853826] systemd[1]: Detected architecture x86-64. [ 4.855100] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.857214] systemd[1]: No hostname configured. [ 4.858560] systemd[1]: Set hostname to . [ 4.859929] systemd[1]: Initializing machine ID from KVM UUID. [ 4.886488] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Created slice Root Slice. [ 4.890043] systemd[1]: Created slice Root Slice. [ 4.891289] systemd[1]: Starting Root Slice. [ OK ] Listening on Journal Socket. [ 4.893169] systemd[1]: Listening on Journal Socket. [ 4.894443] systemd[1]: Starting Journal Socket. [ OK ] Reached target Encrypted Volumes. [ 4.896316] systemd[1]: Reached target Encrypted Volumes. [ 4.897615] systemd[1]: Starting Encrypted Volumes. [ OK ] Reached target Timers. [ 4.899440] systemd[1]: Reached target Timers. [ 4.900625] systemd[1]: Starting Timers. [ OK ] Reached target Swap. [ 4.902349] systemd[1]: Reached target Swap. [ 4.903513] systemd[1]: Starting Swap. [ OK ] Created slice System Slice. [ 4.905307] systemd[1]: Created slice System Slice. [ 4.906556] systemd[1]: Starting System Slice. [ 4.908403] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Slices. [ 4.913132] systemd[1]: Reached target Slices. [ 4.914375] systemd[1]: Starting Slices. [ OK ] Reached target Local File Systems. [ 4.916187] systemd[1]: Reached target Local File Systems. [ 4.917527] systemd[1]: Starting Local File Systems. [ OK ] Listening on udev Kernel Socket. [ 4.919464] systemd[1]: Listening on udev Kernel Socket. [ 4.920771] systemd[1]: Starting udev Kernel Socket. [ 4.922780] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 4.926460] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ 4.930129] systemd[1]: Listening on udev Control Socket. [ 4.931455] systemd[1]: Starting udev Control Socket. [ 4.933164] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.936797] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 4.939911] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ 4.946072] systemd[1]: Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ 4.948187] systemd[1]: Started Apply Kernel Variables. [ 4.950186] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ 4.953825] systemd[1]: Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ 5.003094] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.077936] dracut-cmdline[87]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.210253] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.214939] Loading iSCSI transport class v2.0-870. [ 5.219483] iscsi: registered transport (tcp) [ 5.253283] FDC 0 is a S82078B [ 5.258351] iBFT detected. [ 5.259424] iBFT has incorrect checksum (0x54)! [ 5.267160] md: raid0 personality registered for level 0 [ 5.271086] md: raid1 personality registered for level 1 [ 5.274400] async_tx: api initialized (async) [ 5.276533] xor: automatically using best checksumming function: [ 5.288007] avx : 22148.000 MB/sec [ 5.308006] raid6: sse2x1 gen() 7925 MB/s [ 5.326006] raid6: sse2x2 gen() 9976 MB/s [ 5.344009] raid6: sse2x4 gen() 11546 MB/s [ 5.345243] raid6: using algorithm sse2x4 gen() (11546 MB/s) [ 5.346641] raid6: using ssse3x2 recovery algorithm [ 5.352403] md: raid6 personality registered for level 6 [ 5.353840] md: raid5 personality registered for level 5 [ 5.355207] md: raid4 personality registered for level 4 [ 5.361370] md: raid10 personality registered for level 10 [ 5.364701] md: linear personality registered for level -1 [ 5.369905] device-mapper: uevent: version 1.0.3 [ 5.371295] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.385173] device-mapper: multipath: version 1.11.0 loaded [ 5.388567] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.419767] RPC: Registered named UNIX socket transport module. [ 5.421236] RPC: Registered udp transport module. [ 5.422534] RPC: Registered tcp transport module. [ 5.423821] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... [ 5.629852] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 5.631501] tsc: Refined TSC clocksource calibration: 2593.658 MHz Starting Show Plymouth Boot Screen... [ OK [ 5.633927] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 ] Reached target System Initialization. [ 5.643143] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 5.645478] [drm] Initialized drm 1.1.0 20060810 [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ 5.653133] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.655296] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 5.656778] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.663958] AES CTR mode by8 optimization enabled [ 5.667231] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 5.672851] scsi host0: ata_piix [ 5.682497] 8021q: 802.1Q VLAN Support v1.8 [ 5.685413] scsi host1: ata_piix [ 5.686514] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.687980] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ OK ] Reached target Sockets. [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 5.697698] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.699150] [TTM] Initializing pool allocator [ 5.700570] [TTM] Initializing DMA pool allocator [ 5.702281] [drm] fb mappable at 0xF0000000 [ 5.711523] [drm] vram aper at 0xF0000000 [ 5.712663] [drm] size 33554432 %G[ 5.718457] [drm] fb depth is 24 [ 5.718458] [drm] pitch is 3072 [ OK ] Started Device-Mapper Multipath Device Controller. Starting Open-iSCSI... [ 5.724487] fbcon: cirrusdrmfb (fb0) is primary device [ 5.732943] vda: vda1 vda2 vda4 [ 5.754282] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 5.759149] Console: switching to colour frame buffer device 128x48 [ 5.776879] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK [ 5.780656] alg: No test for crc32 (crc32-pclmul) ] Started Open-iSCSI. %G Starting dracut initqueue hook... [ 5.784090] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 5.858898] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.584671] dracut-initqueue[614]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.585163] dracut-initqueue[614]: Dload Upload Total Spent Left Speed [ 8.686153] dracut-initqueue[614]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14602 100 14602 0 0 139k 0 --:--:-- --:--:-- --:--:-- 141k [ 9.216227] dracut-initqueue[614]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.217129] dracut-initqueue[614]: Dload Upload Total Spent Left Speed [ 12.670044] dracut-initqueue[614]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 23 317M 23 74.5M 0 0 94.9M 0 0:00:03 --:--:-- 0:00:03 94.9M 51 317M 51 163M 0 0 91.5M 0 0:00:03 0:00:01 0:00:02 91.5M 80 317M 80 254M 0 0 91.2M 0 0:00:03 0:00:02 0:00:01 91.2M 100 317M 100 317M 0 0 91.8M 0 0:00:03 0:00:03 --:--:-- 91.8M [ 12.784995] loop: module loaded [ OK ] Started dracut initqueue hook. Starting dracut pre-mount hook... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. Starting dracut mount hook... [ 12.950901] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 15.008588] systemd-journald[93]: Received SIGTERM from PID 1 (systemd). [ 15.570676] type=1403 audit(1491236230.246:2): policy loaded auid=4294967295 ses=4294967295 [ 15.576952] systemd[1]: Successfully loaded SELinux policy in 277.103ms. [ 15.683760] systemd[1]: Relabelled /dev and /run in 19.358ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Listening on LVM2 metadata daemon socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Reached target Slices. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on udev Control Socket. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Created slice system-anaconda\x2dtmux.slice. Mounting Debug File System... [ OK ] Stopped Journal Service. Starting Journal Service... Mounting POSIX Message Queue File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Mounting Temporary Directory... [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... Starting Rebuild Hardware Database... Starting Load/Save Random Seed... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Mounted Temporary Directory. [ OK ] Started Load/Save Random Seed. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Configure read-only root support. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... Starting udev Wait for Complete Device Initialization... [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... %G%G[ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting firewalld - dynamic firewall daemon... Starting Wait for Plymouth Boot Screen to Quit... Starting pre-anaconda logging service... Starting Dump dmesg to /var/log/dmesg... Starting Terminate Plymouth Boot Screen... [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting Service enabling compressing RAM with zRam... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... Starting Login Service... [ OK ] Started Dump dmesg to /var/log/dmesg. %G (B                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (BStarting installer, one moment... [anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (B anaconda 21.48.22.93-1 for CentOS Linux 7 started. * installation log files are stored in /tmp during the installation * shell is available on TTY2 * when reporting a bug add logs from /tmp as separate text/plain attachments16:17:17 Running pre-installation scripts 16:17:18 Not asking for VNC because of an automated install16:17:18 Not asking for VNC because text mode was explicitly asked for in kickstartStarting automated install..........................Checking software selectionGenerating updated storage configuration Checking storage configuration... ================================================================================ ================================================================================Installation 1) [x] Language settings 2) [x] Time settings  (English (United States)) (Etc/UTC timezone)  3) [x] Installation source 4) [x] Software selection  (http://10.9.0.10/cblr/links/Ce (Custom software selected)  ntOS-7.3-x86_64) 6) [x] Kdump  5) [x] Installation Destination (Kdump is enabled)  (Custom partitioning selected) 8) [ ] User creation  7) [x] Network configuration (No user will be created)  (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped System Logging Service. [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped target Swap. Stopping Load/Save Random Seed... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting Configuration File System... Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/dev/shm... Unmounting /mnt/sysimage/sys/fs/selinux... Unmounting Temporary Directory... Unmounting /mnt/sysimage/proc... Unmounting /mnt/sysimage/run... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... [ OK ] Unmounted Temporary Directory. [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. [ OK ] Unmounted /mnt/sysimage/run. [ OK ] Unmounted /mnt/sysimage/proc. [ OK ] Unmounted /mnt/sysimage/dev/shm. [ OK ] Unmounted Configuration File System. [ OK ] Failed unmounting /mnt/sysimage/dev/pts. Unmounting /mnt/sysimage/dev... Unmounting /mnt/sysimage/sys... [ OK ] Unmounted /mnt/sysimage/dev. [ OK ] Unmounted /mnt/sysimage/sys. Unmounting /mnt/sysimage... [ OK ] Started Restore /run/initramfs. [ OK ] Unmounted /mnt/sysimage. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 1177.946593] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=33048c39-eb62-4771-ae0f-bb41f1134c20 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fc5000-0x367dafff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=33048c39-eb62-4771-ae0f-bb41f1134c20 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694520k/2097140k available (6765k kernel code, 400k absent, 402220k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005029] Security Framework initialized [ 0.006009] SELinux: Initializing. [ 0.008195] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011444] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013491] Mount-cache hash table entries: 4096 [ 0.014271] Initializing cgroup subsys memory [ 0.016013] Initializing cgroup subsys devices [ 0.017006] Initializing cgroup subsys freezer [ 0.018005] Initializing cgroup subsys net_cls [ 0.019005] Initializing cgroup subsys blkio [ 0.020005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.024005] Initializing cgroup subsys net_prio [ 0.025106] Disabled fast string operations [ 0.027019] mce: CPU supports 10 MCE banks [ 0.028056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] tlb_flushall_shift: 6 [ 0.032180] Freeing SMP alternatives: 28k freed [ 0.037497] ACPI: Core revision 20130517 [ 0.039514] ACPI: All ACPI Tables successfully acquired [ 0.040693] ftrace: allocating 25813 entries in 101 pages [ 0.049201] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.051004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.052004] smpboot: Max logical packages: 2 [ 0.053245] Enabling x2apic [ 0.054000] Enabled x2apic [ 0.054005] Switched APIC routing to physical x2apic. [ 0.056521] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.057003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.060025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.062479] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.065042] Brought up 2 CPUs [ 0.065032] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.067005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.068512] devtmpfs: initialized [ 0.070881] EVM: security.selinux [ 0.071005] EVM: security.ima [ 0.072000] EVM: security.capability [ 0.072772] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.073028] pinctrl core: initialized pinctrl subsystem [ 0.074149] NET: Registered protocol family 16 [ 0.075273] ACPI: bus type PCI registered [ 0.076006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.077088] PCI: Using configuration type 1 for base access [ 0.081080] ACPI: Added _OSI(Module Device) [ 0.082013] ACPI: Added _OSI(Processor Device) [ 0.083006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.084005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.086682] ACPI: Interpreter enabled [ 0.087014] ACPI: (supports S0 S5) [ 0.088005] ACPI: Using IOAPIC for interrupt routing [ 0.089017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.092584] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.093010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.094009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.095018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.096245] acpiphp: Slot [1] registered [ 0.097034] acpiphp: Slot [2] registered [ 0.098022] acpiphp: Slot [3] registered [ 0.099022] acpiphp: Slot [4] registered [ 0.100022] acpiphp: Slot [5] registered [ 0.101022] acpiphp: Slot [6] registered [ 0.102022] acpiphp: Slot [7] registered [ 0.103021] acpiphp: Slot [8] registered [ 0.104029] acpiphp: Slot [9] registered [ 0.105022] acpiphp: Slot [10] registered [ 0.106022] acpiphp: Slot [11] registered [ 0.107022] acpiphp: Slot [12] registered [ 0.108022] acpiphp: Slot [13] registered [ 0.109021] acpiphp: Slot [14] registered [ 0.110021] acpiphp: Slot [15] registered [ 0.111029] acpiphp: Slot [16] registered [ 0.112029] acpiphp: Slot [17] registered [ 0.113022] acpiphp: Slot [18] registered [ 0.114022] acpiphp: Slot [19] registered [ 0.115022] acpiphp: Slot [20] registered [ 0.116022] acpiphp: Slot [21] registered [ 0.117022] acpiphp: Slot [22] registered [ 0.118029] acpiphp: Slot [23] registered [ 0.119022] acpiphp: Slot [24] registered [ 0.120022] acpiphp: Slot [25] registered [ 0.121021] acpiphp: Slot [26] registered [ 0.122021] acpiphp: Slot [27] registered [ 0.123029] acpiphp: Slot [28] registered [ 0.124022] acpiphp: Slot [29] registered [ 0.125022] acpiphp: Slot [30] registered [ 0.126029] acpiphp: Slot [31] registered [ 0.127250] PCI host bridge to bus 0000:00 [ 0.128007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.129006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.130006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.131006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.132006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.133956] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.134006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.135005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.136005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.137796] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.138013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.155607] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.157076] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.158744] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.160069] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.161716] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.163186] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.164276] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.165006] vgaarb: loaded [ 0.165984] vgaarb: bridge control possible 0000:00:02.0 [ 0.166205] SCSI subsystem initialized [ 0.167052] ACPI: bus type USB registered [ 0.168037] usbcore: registered new interface driver usbfs [ 0.169017] usbcore: registered new interface driver hub [ 0.170048] usbcore: registered new device driver usb [ 0.171186] PCI: Using ACPI for IRQ routing [ 0.172346] NetLabel: Initializing [ 0.173007] NetLabel: domain hash size = 128 [ 0.174004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.175040] NetLabel: unlabeled traffic allowed by default [ 0.176138] Switched to clocksource kvm-clock [ 0.185493] pnp: PnP ACPI init [ 0.186549] ACPI: bus type PNP registered [ 0.188163] pnp: PnP ACPI: found 5 devices [ 0.189302] ACPI: bus type PNP unregistered [ 0.198953] NET: Registered protocol family 2 [ 0.200494] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.202694] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.204258] TCP: Hash tables configured (established 16384 bind 16384) [ 0.205752] TCP: reno registered [ 0.206800] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.208190] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.209737] NET: Registered protocol family 1 [ 0.211043] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.212499] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.213927] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.215651] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.217379] Unpacking initramfs... [ 0.616937] Freeing initrd memory: 24664k freed [ 0.628843] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.630493] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.632539] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.634069] Initialise system trusted keyring [ 0.635387] audit: initializing netlink socket (disabled) [ 0.636793] type=2000 audit(1491237423.470:1): initialized [ 0.657889] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.659438] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.662118] zpool: loaded [ 0.663195] zbud: loaded [ 0.664485] VFS: Disk quotas dquot_6.5.2 [ 0.665761] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.667510] msgmni has been set to 3357 [ 0.668813] Key type big_key registered [ 0.671567] NET: Registered protocol family 38 [ 0.672872] Key type asymmetric registered [ 0.674137] Asymmetric key parser 'x509' registered [ 0.675526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.677910] io scheduler noop registered [ 0.679167] io scheduler deadline registered (default) [ 0.680574] io scheduler cfq registered [ 0.681899] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.683332] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.685077] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.687468] ACPI: Power Button [PWRF] [ 0.688840] GHES: HEST is not enabled! [ 0.690148] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.713002] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.714920] Non-volatile memory driver v1.3 [ 0.716202] Linux agpgart interface v0.103 [ 0.717577] crash memory driver: version 1.1 [ 0.718930] rdac: device handler registered [ 0.720281] hp_sw: device handler registered [ 0.737258] emc: device handler registered [ 0.738500] alua: device handler registered [ 0.739770] libphy: Fixed MDIO Bus: probed [ 0.741053] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.742567] ehci-pci: EHCI PCI platform driver [ 0.743843] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.745311] ohci-pci: OHCI PCI platform driver [ 0.746580] uhci_hcd: USB Universal Host Controller Interface driver [ 0.748320] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.749715] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.752023] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.753426] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.754946] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.756536] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.758844] usb usb1: Product: UHCI Host Controller [ 0.760237] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.762591] usb usb1: SerialNumber: 0000:00:01.2 [ 0.764064] hub 1-0:1.0: USB hub found [ 0.765300] hub 1-0:1.0: 2 ports detected [ 0.766683] usbcore: registered new interface driver usbserial [ 0.768204] usbcore: registered new interface driver usbserial_generic [ 0.769740] usbserial: USB Serial support registered for generic [ 0.771310] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.774336] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.775623] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.777069] mousedev: PS/2 mouse device common for all mice [ 0.778767] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.781803] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.784111] rtc_cmos 00:00: RTC can wake from S4 [ 0.784524] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.784657] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.784725] cpuidle: using governor menu [ 0.784882] hidraw: raw HID events driver (C) Jiri Kosina [ 0.785076] usbcore: registered new interface driver usbhid [ 0.785080] usbhid: USB HID core driver [ 0.785150] drop_monitor: Initializing network drop monitor service [ 0.785259] TCP: cubic registered [ 0.785263] Initializing XFRM netlink socket [ 0.785455] NET: Registered protocol family 10 [ 0.787224] NET: Registered protocol family 17 [ 0.787130] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.787897] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.787943] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.788147] Loading compiled-in X.509 certificates [ 0.788206] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.788234] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.788897] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.788940] registered taskstats version 1 [ 0.792869] Key type trusted registered [ 0.796069] Key type encrypted registered [ 0.799211] IMA: No TPM chip found, activating TPM-bypass! [ 0.800513] rtc_cmos 00:00: setting system clock to 2017-04-03 16:37:02 UTC (1491237422) [ 0.828355] Freeing unused kernel memory: 1680k freed [ 0.834868] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.839137] systemd[1]: Detected virtualization kvm. [ 0.839410] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.843278] systemd[1]: Detected architecture x86-64. [ 0.844716] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.846962] systemd[1]: Set hostname to . [ OK ] Reached target Timers. [ 0.881410] systemd[1]: Reached target Timers. [ 0.882670] systemd[1]: Starting Timers. [ OK ] Reached target Local File Systems. [ 0.884504] systemd[1]: Reached target Local File Systems. [ 0.885826] systemd[1]: Starting Local File Systems. [ OK ] Reached target Swap. [ 0.887636] systemd[1]: Reached target Swap. [ 0.888846] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 0.890649] systemd[1]: Created slice Root Slice. [ 0.891910] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Control Socket. [ 0.893736] systemd[1]: Listening on udev Control Socket. [ 0.895080] systemd[1]: Starting udev Control Socket. [ OK ] Listening on Journal Socket. [ 0.896971] systemd[1]: Listening on Journal Socket. [ 0.898331] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Kernel Socket. [ 0.900367] systemd[1]: Listening on udev Kernel Socket. [ 0.901696] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 0.903522] systemd[1]: Reached target Sockets. [ 0.905170] systemd[1]: Starting Sockets. [ OK ] Created slice System Slice. [ 0.906963] systemd[1]: Created slice System Slice. [ 0.908235] systemd[1]: Starting System Slice. [ 0.909883] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.912303] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 0.915937] systemd[1]: Starting Journal Service... Starting Journal Service... [ OK ] Reached target Slices. [ 0.997460] systemd[1]: Reached target Slices. [ 0.998699] systemd[1]: Starting Slices. [ 1.000267] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 1.002789] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Started Journal Service. [ 1.008753] systemd[1]: Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. Starting dracut initqueue hook... [ 1.103833] FDC 0 is a S82078B [ 1.108854] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.114096] scsi host0: ata_piix [ 1.123209] scsi host1: ata_piix [ 1.124429] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.124538] [drm] Initialized drm 1.1.0 20060810 [ 1.127287] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.130083] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.134462] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.137185] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.137345] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.137931] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.137953] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.156213] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.157648] [TTM] Initializing pool allocator [ 1.159394] [TTM] Initializing DMA pool allocator [ 1.161161] [drm] fb mappable at 0xF0000000 [ 1.162328] [drm] vram aper at 0xF0000000 [ 1.163462] [drm] size 33554432 [ 1.164488] [drm] fb depth is 24 [ 1.165575] [drm] pitch is 3072 %G[ 1.166945] fbcon: cirrusdrmfb (fb0) is primary device [ 1.187548] vda: vda1 vda2 [ 1.214015] Console: switching to colour frame buffer device 128x48 [ 1.227457] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.233043] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 %G[ OK ] Found device /dev/disk/by-uuid/33048c39-eb62-4771-ae0f-bb41f1134c20. Starting File System Check on /dev/...9-eb62-4771-ae0f-bb41f1134c20... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d...c39-eb62-4771-ae0f-bb41f1134c20. Mounting /sysroot... [ 1.368810] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.370547] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Plymouth switch root service. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 1.625532] systemd-journald[97]: Received SIGTERM from PID 1 (systemd). [ 1.631031] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ 2.003597] SELinux: Disabled at runtime. [ 2.041029] type=1404 audit(1491237423.741:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.069877] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.071326] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Reached target Encrypted Volumes. [ OK ] Created slice system-getty.slice. Mounting Debug File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped Journal Service. Starting Journal Service... Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Remote File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Delayed Shutdown Socket. Mounting Huge Pages File System... [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Mounted POSIX Message Queue File System. [ 3.694990] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Remount Root and Kernel File Systems. Starting Flush Journal to Persistent Storage... Starting Rebuild Hardware Database... Starting Configure read-only root support... Starting Load/Save Random Seed... [ OK ] Started LVM2 metadata daemon. Startin[ 3.710276] systemd-journald[344]: Received request to flush runtime journal from PID 1 g LVM2 metadata daemon... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Configure read-only root support. [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ OK ] Found device /dev/disk/by-uuid/ee77eaa7-e723-47fc-8698-c6133eda0ea8. Activating swap /dev/disk/by-uuid/e...7-e723-47fc-8698-c6133eda0ea8... [ OK ] Found device /dev/ttyS0. %G%G[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... [ 5.466805] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 5.469451] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Starting Rebuild Journal Catalog... Starting Mark the need to relabel after reboot... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ 5.558164] AES CTR mode by8 optimization enabled [ 5.561230] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ OK ] Started Mark the need to relabel after reboot. [ 5.568209] alg: No test for crc32 (crc32-pclmul) [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Update is Completed. [ 5.723800] intel_rapl: no valid rapl domains found in package 0 [ OK ] Started Security Auditing Service. [ 5.730378] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS Starting Update UTMP about System Boot/Shutdown... [ OK ] Activated swap /dev/disk/by-uuid/ee77eaa7-e723-47fc-8698-c6133eda0ea8. [ 5.737710] ppdev: user-space parallel port driver [ OK ] Reached target Swap. [ 5.766553] intel_powerclamp: No package C-state available [ 5.772561] intel_powerclamp: No package C-state available[ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. Starting OpenSSH Server Key Generation... Starting Login Service... Starting Dump dmesg to /var/log/dmesg... Starting Authorization Manager... Starting Permit User Sessions... [ OK ] Started D-Bus System Message Bus. [ 5.932795] type=1305 audit(1491237427.632:3): audit_pid=415 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... Starting Network Manager... Starting NTP client/server... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started Permit User Sessions. Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Terminate Plymouth Boot Screen... [ OK ] Started Login Service. [ OK ] Started OpenSSH Server Key Generation. [ 12.938256] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 12.939112] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-03T16:37:21+00:00] WARN: ***************************************** [2017-04-03T16:37:21+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-03T16:37:21+00:00] WARN: ***************************************** [2017-04-03T16:37:26+00:00] INFO: Forking chef instance to converge... [2017-04-03T16:37:26+00:00] INFO: *** Chef 12.4.3 *** [2017-04-03T16:37:26+00:00] INFO: Chef-client pid: 4675 [2017-04-03T16:37:28+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-03T16:37:28+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-03T16:37:28+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-03T16:37:28+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-03T16:37:28+00:00] INFO: Running start handlers [2017-04-03T16:37:28+00:00] INFO: Start handlers complete. [2017-04-03T16:37:29+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-03T16:37:29+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-03T16:37:29+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-03T16:37:29+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-03T16:37:29+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170403163729.582818 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170403163729.585138 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170403163729.588968 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170403163729.592780 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170403163729.596568 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170403163729.599771 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-03T16:37:29+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170403163729.601471 [2017-04-03T16:37:29+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-03T16:37:29+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-03T16:37:29+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170403163729.609774 [2017-04-03T16:37:29+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-03T16:37:29+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-03T16:37:39+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-03T16:37:40+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170403163740.057604 [2017-04-03T16:37:40+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-03T16:37:40+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-03T16:37:40+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-03T16:38:33+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-03T16:38:33+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-03T16:38:33+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-03T16:38:49+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-03T16:38:49+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-03T16:38:49+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-03T16:38:49+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-03T16:38:49+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-03T16:38:49+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-03T16:38:49+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-03T16:38:49+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-03T16:38:50+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-03T16:38:50+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170403163850.177991 [2017-04-03T16:38:50+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-03T16:38:50+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-03T16:38:50+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-03T16:38:50+00:00] INFO: service[sssd] restarted [2017-04-03T16:38:50+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-03T16:38:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:38:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:38:50+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-03T16:38:50+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-03T16:38:51+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-03T16:38:51+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-03T16:38:51+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-03T16:38:51+00:00] INFO: service[ntpd] enabled [2017-04-03T16:38:51+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-03T16:38:51+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170403163851.684529 [2017-04-03T16:38:51+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-03T16:38:51+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-03T16:38:51+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-03T16:38:51+00:00] INFO: service[ntpd] restarted [2017-04-03T16:38:51+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-03T16:38:51+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170403163851.986575 [2017-04-03T16:38:51+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-03T16:38:51+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-03T16:38:51+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-03T16:38:52+00:00] INFO: service[ntpd] restarted [2017-04-03T16:38:52+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-03T16:38:52+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-03T16:38:52+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-03T16:38:52+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-03T16:38:52+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-03T16:38:52+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170403163852.802242 [2017-04-03T16:38:52+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-03T16:38:52+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-03T16:38:52+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-03T16:38:52+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170403163852.809891 [2017-04-03T16:38:52+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-03T16:38:52+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-03T16:38:52+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-03T16:38:53+00:00] INFO: service[autofs] restarted [2017-04-03T16:38:53+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-03T16:38:53+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170403163853.134371 [2017-04-03T16:38:53+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-03T16:38:53+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-03T16:38:53+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-03T16:38:53+00:00] INFO: service[autofs] restarted [2017-04-03T16:38:53+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-03T16:38:53+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-03T16:38:53+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-03T16:38:53+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-03T16:38:54+00:00] INFO: service[autofs] restarted [2017-04-03T16:38:54+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-03T16:38:54+00:00] INFO: service[autofs] enabled [2017-04-03T16:38:54+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-03T16:38:55+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-03T16:38:55+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-03T16:38:55+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-03T16:38:55+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-03T16:38:55+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170403163855.457865 [2017-04-03T16:38:55+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-03T16:38:55+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-03T16:38:55+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170403163855.465732 [2017-04-03T16:38:55+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-03T16:38:55+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-03T16:38:55+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-03T16:38:55+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-03T16:38:55+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-03T16:38:55+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170403163855.475400 [2017-04-03T16:38:55+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-03T16:38:55+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-03T16:38:55+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-03T16:38:57+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-03T16:38:57+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-03T16:38:57+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-03T16:38:57+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-03T16:38:58+00:00] INFO: service[munge] restarted [2017-04-03T16:38:58+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-03T16:38:58+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-03T16:38:58+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-03T16:38:58+00:00] INFO: service[munge] restarted [2017-04-03T16:38:58+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-03T16:38:59+00:00] INFO: service[munge] enabled [2017-04-03T16:38:59+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-03T16:38:59+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-03T16:38:59+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-03T16:39:03+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-03T16:39:03+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-03T16:39:03+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-03T16:39:05+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-03T16:39:05+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170403163905.632239 [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-03T16:39:05+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170403163905.636195 [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-03T16:39:05+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-03T16:39:05+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-03T16:39:05+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:05+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-03T16:39:05+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-03T16:39:05+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:05+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-03T16:39:05+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-03T16:39:06+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-03T16:39:06+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-03T16:39:06+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:06+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-03T16:39:06+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-03T16:39:06+00:00] INFO: service[xinetd] started [2017-04-03T16:39:06+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-03T16:39:06+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-03T16:39:07+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-03T16:39:07+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-03T16:39:07+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-03T16:39:07+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-03T16:39:07+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170403163907.784550 [2017-04-03T16:39:07+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-03T16:39:07+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-03T16:39:07+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-03T16:39:07+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-03T16:39:07+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-03T16:39:07+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-03T16:39:07+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-03T16:39:07+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-03T16:39:07+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-03T16:39:07+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-03T16:39:07+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-03T16:39:07+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-03T16:39:07+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-03T16:39:07+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-03T16:39:07+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-03T16:39:07+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-03T16:39:07+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:07+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-03T16:39:07+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-03T16:39:07+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-03T16:39:09+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-03T16:39:09+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-03T16:39:10+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-03T16:39:12+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-03T16:39:12+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-03T16:39:12+00:00] INFO: service[rsh.socket] enabled [2017-04-03T16:39:12+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-03T16:39:12+00:00] INFO: service[rsh.socket] started [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-03T16:39:12+00:00] INFO: service[rlogin.socket] enabled [2017-04-03T16:39:12+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-03T16:39:12+00:00] INFO: service[rlogin.socket] started [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-03T16:39:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-03T16:39:12+00:00] INFO: execute[Guard resource] ran successfully [2017-04-03T16:39:12+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-03T16:39:12+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-03T16:39:12+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-03T16:39:12+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-03T16:39:12+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-03T16:39:13+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-03T16:39:13+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-03T16:39:13+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-03T16:39:13+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-03T16:39:13+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-03T16:39:13+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-03T16:39:17+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-03T16:39:17+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-03T16:39:17+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-03T16:39:17+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-03T16:39:17+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-03T16:39:17+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-03T16:39:19+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-03T16:39:19+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-03T16:39:20+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.el7 from addon-epel7-x86_64 repository [2017-04-03T16:39:23+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.el7 [2017-04-03T16:39:23+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-03T16:39:23+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-03T16:39:23+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-03T16:39:24+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-03T16:39:24+00:00] INFO: service[rdma] enabled [2017-04-03T16:39:24+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 141.850027] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 141.860942] Loading iSCSI transport class v2.0-870. [ 141.865382] iscsi: registered transport (iser) [ 141.937088] RPC: Registered named UNIX socket transport module. [ 141.937616] RPC: Registered udp transport module. [ 141.938077] RPC: Registered tcp transport module. [ 141.938485] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 141.941068] RPC: Registered rdma transport module. [ 141.941483] RPC: Registered rdma backchannel transport module. [2017-04-03T16:39:25+00:00] INFO: service[rdma] started [2017-04-03T16:39:25+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-03T16:39:25+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-03T16:39:25+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 142.085166] FS-Cache: Loaded [ 142.117970] FS-Cache: Netfs 'nfs' registered for caching [ 142.121460] Key type dns_resolver registered [ 142.130402] NFS: Registering the id_resolver key type [ 142.130919] Key type id_resolver registered [ 142.131316] Key type id_legacy registered [2017-04-03T16:39:25+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-03T16:39:25+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-03T16:39:25+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170403163925.361405 [2017-04-03T16:39:25+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-03T16:39:25+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-03T16:39:25+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-03T16:40:06+00:00] INFO: service[kdump] restarted [2017-04-03T16:40:06+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-03T16:40:06+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-03T16:40:06+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-03T16:40:06+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-03T16:40:06+00:00] INFO: group[quota_usr] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-03T16:40:06+00:00] INFO: user[quota_usr] created [2017-04-03T16:40:06+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-03T16:40:06+00:00] INFO: group[quota_2usr] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-03T16:40:06+00:00] INFO: user[quota_2usr] created [2017-04-03T16:40:06+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-03T16:40:06+00:00] INFO: group[sanityusr] created [2017-04-03T16:40:06+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-03T16:40:06+00:00] INFO: user[sanityusr] created [2017-04-03T16:40:06+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-03T16:40:06+00:00] INFO: group[sanityusr1] created [2017-04-03T16:40:06+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-03T16:40:06+00:00] INFO: user[sanityusr1] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_1] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_2] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_3] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_4] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_5] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_6] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_7] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_8] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_9] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_10] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-03T16:40:06+00:00] INFO: user[quota15_11] created [2017-04-03T16:40:06+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_12] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_13] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_14] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_15] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_16] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_17] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_18] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_19] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_20] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_21] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_22] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_23] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_24] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_25] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_26] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_27] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_28] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_29] created [2017-04-03T16:40:07+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-03T16:40:07+00:00] INFO: user[quota15_30] created [2017-04-03T16:40:07+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-03T16:40:07+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-03T16:40:07+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-03T16:40:07+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-03T16:40:07+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-03T16:40:07+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-03T16:40:07+00:00] INFO: bash[set_hostid] ran successfully [2017-04-03T16:40:07+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-03T16:40:07+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-03T16:40:07+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-03T16:40:07+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-03T16:40:07+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-03T16:40:07+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-03T16:40:07+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-03T16:40:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-03T16:40:07+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-03T16:40:07+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-03T16:40:07+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-03T16:40:07+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-03T16:40:07+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-03T16:40:08+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-03T16:40:08+00:00] INFO: user[mpiuser] created [2017-04-03T16:40:08+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-03T16:40:08+00:00] INFO: group[mpiuser] created [2017-04-03T16:40:08+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-03T16:40:08+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-03T16:40:08+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-03T16:40:08+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-03T16:40:08+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-03T16:40:08+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-03T16:40:08+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-03T16:40:08+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-03T16:40:08+00:00] INFO: service[postfix] restarted [2017-04-03T16:40:08+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-03T16:40:08+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-03T16:40:08+00:00] INFO: execute[newaliases] ran successfully [2017-04-03T16:40:08+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-03T16:40:08+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-03T16:40:09+00:00] INFO: service[sshd] restarted [2017-04-03T16:40:09+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-03T16:40:09+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-03T16:40:09+00:00] INFO: service[rsyslog] restarted [2017-04-03T16:40:09+00:00] INFO: Chef Run complete in 160.484354333 seconds [2017-04-03T16:40:09+00:00] INFO: Skipping removal of unused files from the cache [2017-04-03T16:40:09+00:00] INFO: Running report handlers [2017-04-03T16:40:09+00:00] INFO: Creating JSON run report [2017-04-03T16:40:09+00:00] INFO: Report handlers complete [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Unmounted /scratch. [ OK ] Stopped Automounts filesystems on demand. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... Stopping System Security Services Daemon... [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped target Swap. Stopping Update UTMP about System Boot/Shutdown... Stopping Load/Save Random Seed... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ 3681.519195] type=1305 audit(1491241104.695:1096): audit_pid=0 old=415 auid=4294967295 ses=4294967295 res=1 [ OK ] Stopped Security Auditing Servic[ 3681.520365] type=1130 audit(1491241104.697:1097): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' e. [ 3681.522740] type=1131 audit(1491241104.699:1098): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Create Volatile Files and Directories. [ 3681.528264] type=1130 audit(1491241104.705:1099): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3681.531948] type=1131 audit(1491241104.708:1100): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. [ 3681.538181] type=1130 audit(1491241104.715:1101): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3681.538192] type=1131 audit(1491241104.715:1102): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Configure read-only root support. [ 3681.550701] type=1130 audit(1491241104.727:1103): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3681.550712] type=1131 audit(1491241104.727:1104): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Configure read-only root support... Unmounting /run/user/0... [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Reached target Shutdown. [ 3681.581839] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 3681.586483] systemd-journald[344]: Received SIGTERM from PID 1 (systemd-shutdow). [ 3681.595331] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 3681.600578] systemd-shutdown[1]: Unmounting file systems. [ 3681.690953] EXT4-fs (vda1): re-mounted. Opts: (null) [ 3681.710474] EXT4-fs (vda1): re-mounted. Opts: (null) [ 3681.712678] systemd-shutdown[1]: All filesystems unmounted. [ 3681.714831] systemd-shutdown[1]: Deactivating swaps. [ 3681.717054] systemd-shutdown[1]: All swaps deactivated. [ 3681.719328] systemd-shutdown[1]: Detaching loop devices. [ 3681.722109] systemd-shutdown[1]: All loop devices detached. [ 3681.724226] systemd-shutdown[1]: Detaching DM devices. [ 3681.726355] systemd-shutdown[1]: All DM devices detached. [ 3681.729641] systemd-shutdown[1]: Rebooting. [ 3681.732654] Restarting system. [ 3681.735075] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=33048c39-eb62-4771-ae0f-bb41f1134c20 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fc5000-0x367dafff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=33048c39-eb62-4771-ae0f-bb41f1134c20 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694520k/2097140k available (6765k kernel code, 400k absent, 402220k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004006] pid_max: default: 32768 minimum: 301 [ 0.005036] Security Framework initialized [ 0.007010] SELinux: Initializing. [ 0.008208] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011882] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013521] Mount-cache hash table entries: 4096 [ 0.015354] Initializing cgroup subsys memory [ 0.016015] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.019009] Initializing cgroup subsys net_cls [ 0.020006] Initializing cgroup subsys blkio [ 0.021006] Initializing cgroup subsys perf_event [ 0.022007] Initializing cgroup subsys hugetlb [ 0.023009] Initializing cgroup subsys pids [ 0.024006] Initializing cgroup subsys net_prio [ 0.025133] Disabled fast string operations [ 0.026021] mce: CPU supports 10 MCE banks [ 0.028059] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030005] tlb_flushall_shift: 6 [ 0.032152] Freeing SMP alternatives: 28k freed [ 0.037622] ACPI: Core revision 20130517 [ 0.039773] ACPI: All ACPI Tables successfully acquired [ 0.041018] ftrace: allocating 25813 entries in 101 pages [ 0.049348] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.051005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.052006] smpboot: Max logical packages: 2 [ 0.054257] Enabling x2apic [ 0.055000] Enabled x2apic [ 0.055007] Switched APIC routing to physical x2apic. [ 0.058000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.059003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.062037] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.064731] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.068086] Brought up 2 CPUs [ 0.068067] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.070007] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.072023] devtmpfs: initialized [ 0.075013] EVM: security.selinux [ 0.076005] EVM: security.ima [ 0.077005] EVM: security.capability [ 0.079046] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.080009] pinctrl core: initialized pinctrl subsystem [ 0.081190] NET: Registered protocol family 16 [ 0.082332] ACPI: bus type PCI registered [ 0.083007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.084099] PCI: Using configuration type 1 for base access [ 0.088101] ACPI: Added _OSI(Module Device) [ 0.089014] ACPI: Added _OSI(Processor Device) [ 0.090007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.091015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.093785] ACPI: Interpreter enabled [ 0.094015] ACPI: (supports S0 S5) [ 0.095006] ACPI: Using IOAPIC for interrupt routing [ 0.096023] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.099686] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.100013] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.101012] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.102026] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.103271] acpiphp: Slot [1] registered [ 0.104036] acpiphp: Slot [2] registered [ 0.105024] acpiphp: Slot [3] registered [ 0.106024] acpiphp: Slot [4] registered [ 0.107024] acpiphp: Slot [5] registered [ 0.108024] acpiphp: Slot [6] registered [ 0.109023] acpiphp: Slot [7] registered [ 0.110023] acpiphp: Slot [8] registered [ 0.111035] acpiphp: Slot [9] registered [ 0.112024] acpiphp: Slot [10] registered [ 0.113023] acpiphp: Slot [11] registered [ 0.114029] acpiphp: Slot [12] registered [ 0.115031] acpiphp: Slot [13] registered [ 0.116023] acpiphp: Slot [14] registered [ 0.117023] acpiphp: Slot [15] registered [ 0.118035] acpiphp: Slot [16] registered [ 0.119035] acpiphp: Slot [17] registered [ 0.120023] acpiphp: Slot [18] registered [ 0.121023] acpiphp: Slot [19] registered [ 0.122023] acpiphp: Slot [20] registered [ 0.123023] acpiphp: Slot [21] registered [ 0.124026] acpiphp: Slot [22] registered [ 0.125034] acpiphp: Slot [23] registered [ 0.126024] acpiphp: Slot [24] registered [ 0.127024] acpiphp: Slot [25] registered [ 0.128024] acpiphp: Slot [26] registered [ 0.129024] acpiphp: Slot [27] registered [ 0.130036] acpiphp: Slot [28] registered [ 0.131024] acpiphp: Slot [29] registered [ 0.132023] acpiphp: Slot [30] registered [ 0.133031] acpiphp: Slot [31] registered [ 0.134260] PCI host bridge to bus 0000:00 [ 0.135008] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.136007] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.137007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.138008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.139007] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.141050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.142006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.143006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.144006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.145830] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.146014] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.167774] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.169079] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.170762] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.172073] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.173739] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.175253] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.176350] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.177014] vgaarb: loaded [ 0.178000] vgaarb: bridge control possible 0000:00:02.0 [ 0.179257] SCSI subsystem initialized [ 0.180057] ACPI: bus type USB registered [ 0.181044] usbcore: registered new interface driver usbfs [ 0.182019] usbcore: registered new interface driver hub [ 0.183070] usbcore: registered new device driver usb [ 0.184219] PCI: Using ACPI for IRQ routing [ 0.185406] NetLabel: Initializing [ 0.186008] NetLabel: domain hash size = 128 [ 0.187005] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.188068] NetLabel: unlabeled traffic allowed by default [ 0.189165] Switched to clocksource kvm-clock [ 0.201330] pnp: PnP ACPI init [ 0.202444] ACPI: bus type PNP registered [ 0.204191] pnp: PnP ACPI: found 5 devices [ 0.205366] ACPI: bus type PNP unregistered [ 0.215339] NET: Registered protocol family 2 [ 0.217910] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.221256] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.223907] TCP: Hash tables configured (established 16384 bind 16384) [ 0.226441] TCP: reno registered [ 0.227608] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.229089] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.230756] NET: Registered protocol family 1 [ 0.232060] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.233512] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.234950] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.236763] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.238571] Unpacking initramfs... [ 0.645501] Freeing initrd memory: 24664k freed [ 0.654048] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.655831] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.657702] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.659227] Initialise system trusted keyring [ 0.660513] audit: initializing netlink socket (disabled) [ 0.661910] type=2000 audit(1491241135.490:1): initialized [ 0.684196] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.685720] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.688345] zpool: loaded [ 0.689411] zbud: loaded [ 0.690680] VFS: Disk quotas dquot_6.5.2 [ 0.691925] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.693649] msgmni has been set to 3357 [ 0.694914] Key type big_key registered [ 0.697423] NET: Registered protocol family 38 [ 0.698713] Key type asymmetric registered [ 0.699932] Asymmetric key parser 'x509' registered [ 0.701286] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.703729] io scheduler noop registered [ 0.704951] io scheduler deadline registered (default) [ 0.706326] io scheduler cfq registered [ 0.707649] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.709058] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.710763] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.713040] ACPI: Power Button [PWRF] [ 0.714412] GHES: HEST is not enabled! [ 0.715697] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.738556] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.740453] Non-volatile memory driver v1.3 [ 0.741715] Linux agpgart interface v0.103 [ 0.743084] crash memory driver: version 1.1 [ 0.744405] rdac: device handler registered [ 0.745712] hp_sw: device handler registered [ 0.762585] emc: device handler registered [ 0.763814] alua: device handler registered [ 0.765085] libphy: Fixed MDIO Bus: probed [ 0.766351] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.767866] ehci-pci: EHCI PCI platform driver [ 0.769139] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.770614] ohci-pci: OHCI PCI platform driver [ 0.771879] uhci_hcd: USB Universal Host Controller Interface driver [ 0.773614] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.775037] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.777324] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.778726] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.780254] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.781870] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.784176] usb usb1: Product: UHCI Host Controller [ 0.785549] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.787930] usb usb1: SerialNumber: 0000:00:01.2 [ 0.789388] hub 1-0:1.0: USB hub found [ 0.790659] hub 1-0:1.0: 2 ports detected [ 0.792081] usbcore: registered new interface driver usbserial [ 0.793606] usbcore: registered new interface driver usbserial_generic [ 0.795180] usbserial: USB Serial support registered for generic [ 0.796754] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.799783] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.801100] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.802562] mousedev: PS/2 mouse device common for all mice [ 0.804244] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.807252] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.809794] rtc_cmos 00:00: RTC can wake from S4 [ 0.810040] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.813954] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.815573] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.817114] cpuidle: using governor menu [ 0.818506] hidraw: raw HID events driver (C) Jiri Kosina [ 0.820037] usbcore: registered new interface driver usbhid [ 0.821531] usbhid: USB HID core driver [ 0.822813] drop_monitor: Initializing network drop monitor service [ 0.824432] TCP: cubic registered [ 0.825605] Initializing XFRM netlink socket [ 0.826993] NET: Registered protocol family 10 [ 0.828670] NET: Registered protocol family 17 [ 0.830256] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.831822] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.833387] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.836133] Loading compiled-in X.509 certificates [ 0.837563] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.840194] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.843374] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.846047] registered taskstats version 1 [ 0.850054] Key type trusted registered [ 0.853378] Key type encrypted registered [ 0.856666] IMA: No TPM chip found, activating TPM-bypass! [ 0.858452] rtc_cmos 00:00: setting system clock to 2017-04-03 17:38:56 UTC (1491241136) [ 0.861802] Freeing unused kernel memory: 1680k freed [ 0.868497] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.872690] systemd[1]: Detected virtualization kvm. [ 0.873977] systemd[1]: Detected architecture x86-64. [ 0.875318] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.877484] systemd[1]: Set hostname to . [ OK ] Created slice Root Slice. [ 0.912720] systemd[1]: Created slice Root Slice. [ 0.914023] systemd[1]: Starting Root Slice. [ OK ] Listening on Journal Socket. [ 0.915861] systemd[1]: Listening on Journal Socket. [ 0.917180] systemd[1]: Starting Journal Socket. [ OK ] Reached target Local File Systems. [ 0.919068] systemd[1]: Reached target Local File Systems. [ 0.920386] systemd[1]: Starting Local File Systems. [ OK ] Reached target Swap. [ 0.922252] systemd[1]: Reached target Swap. [ 0.923416] systemd[1]: Starting Swap. [ OK ] Listening on udev Control Socket. [ 0.925309] systemd[1]: Listening on udev Control Socket. [ 0.926615] systemd[1]: Starting udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ 0.928523] systemd[1]: Listening on udev Kernel Socket. [ 0.929820] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 0.931650] systemd[1]: Reached target Sockets. [ 0.932865] systemd[1]: Starting Sockets. [ OK ] Created slice System Slice. [ 0.934668] systemd[1]: Created slice System Slice. [ 0.935948] systemd[1]: Starting System Slice. [ 0.937703] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.941323] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.946494] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.950208] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ 0.952587] systemd[1]: Reached target Slices. [ 0.955444] systemd[1]: Starting Slices. [ OK ] Reached target Timers. [ 0.957199] systemd[1]: Reached target Timers. [ 0.959037] systemd[1]: Starting Timers. [ 0.960845] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ 1.045393] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ 1.138457] FDC 0 is a S82078B [ 1.139879] scsi host0: ata_piix [ 1.141144] scsi host1: ata_piix [ 1.142224] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.143668] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.149654] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.157824] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.161635] [drm] Initialized drm 1.1.0 20060810 [ 1.161763] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.161810] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.162238] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.173989] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.180434] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.189939] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.191494] [TTM] Initializing pool allocator [ 1.192824] [TTM] Initializing DMA pool allocator [ 1.194511] [drm] fb mappable at 0xF0000000 [ 1.216229] [drm] vram aper at 0xF0000000 [ 1.217492] [drm] size 33554432 [ 1.218685] [drm] fb depth is 24 [ 1.219856] [drm] pitch is 3072 [ 1.221354] vda: vda1 vda2 vda4 %G[ OK ] Started dracut initqueue hook. [ 1.295362] fbcon: cirrusdrmfb (fb0) is primary device [ 1.323791] Console: switching to colour frame buffer device 128x48 [ 1.334987] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Found device /dev/disk/by-uuid/33048c39-eb62-4771-ae0f-bb41f1134c20. Starting File System Check on /dev/...9-eb62-4771-ae0f-bb41f1134c20... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. %G[ 1.341030] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started File System Check on /dev/d...c39-eb62-4771-ae0f-bb41f1134c20. Mounting /sysroot... [ 1.411801] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.413346] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 1.655101] tsc: Refined TSC clocksource calibration: 2593.641 MHz [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.937084] systemd-journald[92]: Received SIGTERM from PID 1 (systemd). [ 2.244237] SELinux: Disabled at runtime. [ 2.282032] type=1404 audit(1491241137.923:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.292960] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.294417] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Stopped Journal Service. Starting Journal Service... Mounting Debug File System... [ OK ] Listening on LVM2 metadata daemon socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Reached target Slices. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Reached target Encrypted Volumes. Starting Read and set NIS domainname from /etc/sysconfig/network... Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on udev Control Socket. Mounting NFSD configuration filesystem... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ 3.083823] RPC: Registered named UNIX socket transport module. [ 3.085298] RPC: Registered udp transport module. [ 3.085299] RPC: Registered tcp transport module. [ 3.085299] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.089111] EXT4-fs (vda1): re-mounted. Opts: (null) [ 3.117535] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 3.132198] systemd-journald[337]: Received request to flush runtime journal from PID 1 Starting Create list of required st... nodes for the current kernel... [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. Starting Create Static Device Nodes in /dev... [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting udev Kernel Device Manager... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Load/Save Random Seed... Starting udev Coldplug all Devices... [ OK ] Mounted NFSD configuration filesystem. [ OK ] Started Load/Save Random Seed. [ OK ] Started Flush Journal to Persistent Storage. [ 3.185168] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 3.212218] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Started udev Kernel Device Manager. [ OK ] Started Configure read-only root support. [ OK ] Started udev Coldplug all Devices. [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/ee77eaa7-e723-47fc-8698-c6133eda0ea8. Activating swap /dev/disk/by-uuid/e...7-e723-47fc-8698-c6133eda0ea8... [ 3.235659] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 3.256951] AES CTR mode by8 optimization enabled [ 3.261905] ppdev: user-space parallel port driver [ OK ] Activated swap /dev/disk/by-uuid/ee77eaa7-e723-47fc-8698-c6133eda0ea8. [ OK ] Reached target Swap. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Preprocess NFS configuration... Starting Tell Plymouth To Write Out Runtime Data... %G Starting Import network configuration from initramfs... %G[ OK ] Started Preprocess NFS configuration. [ 3.308040] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 3.312759] alg: No test for crc32 (crc32-pclmul) [ 3.337174] intel_rapl: no valid rapl domains found in package 0 [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... Mounting RPC Pipe File System... [ OK ] Mounted RPC Pipe File System. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ 3.367142] intel_powerclamp: No package C-state available[ 3.380878] type=1305 audit(1491241139.021:3): audit_pid=477 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ 3.500514] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 3.511565] Loading iSCSI transport class v2.0-870. [ 3.516283] iscsi: registered transport (iser) [ 3.537093] RPC: Registered rdma transport module. [ 3.538625] RPC: Registered rdma backchannel transport module. Starting D-Bus System Message Bus... Starting Network Manager... Starting Dump dmesg to /var/log/dmesg... Starting Authorization Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting NTP client/server... Starting GSSAPI Proxy Daemon... Starting System Security Services Daemon... [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started NTP client/server. [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ OK ] Started Authorization Manager. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ 4.541955] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.544656] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Started Hostname Service. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting Notify NFS peers of a restart... Starting Postfix Mail Transport Agent... Starting MUNGE authentication service... Starting Xinetd A Powerful Replacement For Inetd... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting System Logging Service... [ OK ] Reached target Network is Online. Mounting /scratch... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started MUNGE authentication service. [ OK ] Started System Logging Service. [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ 10.851140] FS-Cache: Loaded [ 10.865982] FS-Cache: Netfs 'nfs' registered for caching [ 10.884947] Key type dns_resolver registered [ OK ] Started OpenSSH server daemon. [ 10.907304] NFS: Registering the id_resolver key type [ 10.908690] Key type id_resolver registered [ 10.910874] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Crash recovery kernel arming... Starting Permit User Sessions... [ OK ] Started Permit User Sessions. Starting Terminate Plymouth Boot Screen... [ OK ] Started Job spooling tools. Starting Job spooling tools... [ OK ] Started Command Scheduler. Starting Command Scheduler... Starting Wait for Plymouth Boot Screen to Quit... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [ 334.015901] device-mapper: uevent: version 1.0.3 [ 334.016747] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 437.757570] libcfs: loading out-of-tree module taints kernel. [ 437.758343] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 437.760965] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 437.764246] alg: No test for adler32 (adler32-zlib) [ 437.764772] alg: No test for crc32 (crc32-table) [ 438.784173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 438.804705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 438.910176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 438.975733] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 439.013546] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 439.102049] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 439.474252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 439.618956] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 441.477575] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 441.724361] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 441.999633] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [ 442.566326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 453.729883] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 453.729883] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 453.729883] if [ $running -ne $mpts ]; then [ 453.729883] echo $(hostname) env are INSANE!; [ 453.729883] exit 1; [ 453.729883] fi [ 454.024802] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 454.024802] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 454.024802] if [ $running -ne $mpts ]; then [ 454.024802] echo $(hostname) env are INSANE!; [ 454.024802] exit 1; [ 454.024802] fi [ 454.935969] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 455.193895] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 455.437371] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 455.812480] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 455.975143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 456.002107] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [ 456.019761] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 456.020354] LNet: Accept all, port 7988 [ 456.030371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.052991] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 456.180086] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 456.228969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 456.257207] Lustre: lustre-MDT0000: new disk, initializing [ 456.304423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.318207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.532667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 456.788523] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 457.313369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 457.316802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 457.478370] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 457.490484] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 457.629530] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 457.883628] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 458.132849] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 458.389499] Lustre: DEBUG MARKER: sync; sync; sync [ 461.289230] Lustre: lustre-MDT0000: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 463.681807] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 463.950224] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 465.373327] Lustre: MGS: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 466.540945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 466.690324] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 474.027664] Lustre: lustre-MDT0000: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 474.030736] Lustre: Skipped 1 previous similar message [ 475.050542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 475.211533] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 482.460785] Lustre: lustre-MDT0000: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 483.524794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 483.679348] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 490.929330] Lustre: lustre-MDT0000: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 491.883769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 492.034168] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 500.582801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 500.746155] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 508.023395] Lustre: lustre-MDT0000: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 508.026562] Lustre: Skipped 1 previous similar message [ 509.159947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 509.319137] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 517.610242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 517.776961] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 525.304055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 525.459004] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 527.190499] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 527.500360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 527.652313] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 527.782372] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 528.067868] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 538.764637] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 539.391945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-03\/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.429740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-03\/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.470250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-03\/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.500537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-03\/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.511983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-03\/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.654236] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-03/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.757406] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-03/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.827062] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-03/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.857633] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-03/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 539.871593] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-03/lustre-master-el7-x86_64--basic--1_2_1__482__-70343242510440-155454 [ 540.547225] Lustre: DEBUG MARKER: uname -n [ 541.107064] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 541.764413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 541.790039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 541.815924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 541.817862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 541.846114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 541.999961] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 542.060841] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 542.180814] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 542.189331] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 542.210218] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 542.226260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || /usr/sbin/lctl lustre_build_version 2>/dev/null || /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 542.590667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.54_76_ge2ce727 [ 542.746673] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.54_76_ge2ce727 [ 542.876053] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 543.182707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.54_76_ge2ce727 [ 543.332879] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.54_76_ge2ce727 [ 543.763485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.54_76_ge2ce727 [ 543.918820] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.54_76_ge2ce727 [ 544.108776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity ============----- Mon Apr 3 17:47:59 UTC 2017 [ 544.268041] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Mon Apr 3 17:47:59 UTC 2017 [ 544.579159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration -o sec ============================================================= 17:47:59 \(1491241679\) [ 544.735441] Lustre: DEBUG MARKER: == sanity test complete, duration -o sec ============================================================= 17:47:59 (1491241679) [ 545.540857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 545.559234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 545.707039] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 545.718844] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 547.910217] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 547.910217] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 547.910217] if [ $running -ne $mpts ]; then [ 547.910217] echo $(hostname) env are INSANE!; [ 547.910217] exit 1; [ 547.910217] fi [ 548.213623] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 548.213623] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 548.213623] if [ $running -ne $mpts ]; then [ 548.213623] echo $(hostname) env are INSANE!; [ 548.213623] exit 1; [ 548.213623] fi [ 549.134462] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 549.404920] Lustre: DEBUG MARKER: cat /proc/mounts [ 553.339972] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 553.642630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 553.789351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 553.919966] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 554.244252] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 554.778463] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 554.812138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 554.836526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 554.898811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 554.990917] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.092177] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.104703] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.271915] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.522585] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 555.792605] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 556.210915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 76 407 42a 42b 42c 42d 45 68b [ 556.359218] Lustre: DEBUG MARKER: excepting tests: 76 407 42a 42b 42c 42d 45 68b [ 556.539376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o [ 556.691855] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o [ 556.992626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0a: touch\; rm ========================================================================= 17:48:12 \(1491241692\) [ 557.146664] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 17:48:12 (1491241692) [ 557.456148] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 558.156233] Lustre: DEBUG MARKER: rc=0; [ 558.156233] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 558.156233] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 558.156233] echo $(hostname -s): $val; [ 558.156233] rc=$val; [ 558.156233] fi; [ 558.156233] exit $rc [ 558.732739] Lustre: DEBUG MARKER: dmesg [ 559.103944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0b: chmod 0755 \/mnt\/lustre ============================================================ 17:48:14 \(1491241694\) [ 559.465954] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 17:48:14 (1491241694) [ 559.649552] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 560.246984] Lustre: DEBUG MARKER: rc=0; [ 560.246984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 560.246984] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 560.246984] echo $(hostname -s): $val; [ 560.246984] rc=$val; [ 560.246984] fi; [ 560.246984] exit $rc [ 560.835852] Lustre: DEBUG MARKER: dmesg [ 561.198957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0c: check import proc ================================================================= 17:48:16 \(1491241696\) [ 561.353381] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 17:48:16 (1491241696) [ 561.535726] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 562.143415] Lustre: DEBUG MARKER: rc=0; [ 562.143415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 562.143415] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 562.143415] echo $(hostname -s): $val; [ 562.143415] rc=$val; [ 562.143415] fi; [ 562.143415] exit $rc [ 562.728367] Lustre: DEBUG MARKER: dmesg [ 563.087985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 1: mkdir\; remkdir\; rmdir ============================================================== 17:48:18 \(1491241698\) [ 563.243266] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 17:48:18 (1491241698) [ 563.488959] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 564.083394] Lustre: DEBUG MARKER: rc=0; [ 564.083394] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 564.083394] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 564.083394] echo $(hostname -s): $val; [ 564.083394] rc=$val; [ 564.083394] fi; [ 564.083394] exit $rc [ 564.667671] Lustre: DEBUG MARKER: dmesg [ 565.160480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 2: mkdir\; touch\; rmdir\; check file ==================================================== 17:48:20 \(1491241700\) [ 565.323385] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 17:48:20 (1491241700) [ 565.534653] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 566.162315] Lustre: DEBUG MARKER: rc=0; [ 566.162315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 566.162315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 566.162315] echo $(hostname -s): $val; [ 566.162315] rc=$val; [ 566.162315] fi; [ 566.162315] exit $rc [ 566.738771] Lustre: DEBUG MARKER: dmesg [ 567.107139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 3: mkdir\; touch\; rmdir\; check dir ===================================================== 17:48:22 \(1491241702\) [ 567.258531] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 17:48:22 (1491241702) [ 567.462557] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 568.057716] Lustre: DEBUG MARKER: rc=0; [ 568.057716] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 568.057716] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 568.057716] echo $(hostname -s): $val; [ 568.057716] rc=$val; [ 568.057716] fi; [ 568.057716] exit $rc [ 568.644761] Lustre: DEBUG MARKER: dmesg [ 569.014561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 4: mkdir\; touch dir\/file\; rmdir\; checkdir \(expect error\) ============================== 17:48:24 \(1491241704\) [ 569.163628] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 17:48:24 (1491241704) [ 569.354785] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 569.967228] Lustre: DEBUG MARKER: rc=0; [ 569.967228] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 569.967228] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 569.967228] echo $(hostname -s): $val; [ 569.967228] rc=$val; [ 569.967228] fi; [ 569.967228] exit $rc [ 570.582048] Lustre: DEBUG MARKER: dmesg [ 570.955297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 5: mkdir ...\/d5 ...\/d5\/d2\; chmod ...\/d5\/d2 ============================================ 17:48:26 \(1491241706\) [ 571.112835] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 17:48:26 (1491241706) [ 571.312949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 571.906936] Lustre: DEBUG MARKER: rc=0; [ 571.906936] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 571.906936] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 571.906936] echo $(hostname -s): $val; [ 571.906936] rc=$val; [ 571.906936] fi; [ 571.906936] exit $rc [ 572.485676] Lustre: DEBUG MARKER: dmesg [ 572.847669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6a: touch f6a\; chmod f6a\; runas -u 500 -g 500 chmod f6a \(should return error\) ========= 17:48:27 \(1491241707\) [ 573.004206] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 17:48:27 (1491241707) [ 573.199805] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 573.795602] Lustre: DEBUG MARKER: rc=0; [ 573.795602] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 573.795602] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 573.795602] echo $(hostname -s): $val; [ 573.795602] rc=$val; [ 573.795602] fi; [ 573.795602] exit $rc [ 574.378675] Lustre: DEBUG MARKER: dmesg [ 574.756860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6c: touch f6c\; chown f6c\; runas -u 500 -g 500 chown f6c \(should return error\) ========= 17:48:29 \(1491241709\) [ 574.915075] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 17:48:29 (1491241709) [ 575.113976] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 575.743744] Lustre: DEBUG MARKER: rc=0; [ 575.743744] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 575.743744] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 575.743744] echo $(hostname -s): $val; [ 575.743744] rc=$val; [ 575.743744] fi; [ 575.743744] exit $rc [ 576.337094] Lustre: DEBUG MARKER: dmesg [ 576.698185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6e: touch f6e\; chgrp f6e\; runas -u 500 -g 500 chgrp f6e \(should return error\) ========= 17:48:31 \(1491241711\) [ 576.869712] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ========= 17:48:31 (1491241711) [ 577.145546] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 577.763054] Lustre: DEBUG MARKER: rc=0; [ 577.763054] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 577.763054] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 577.763054] echo $(hostname -s): $val; [ 577.763054] rc=$val; [ 577.763054] fi; [ 577.763054] exit $rc [ 578.345457] Lustre: DEBUG MARKER: dmesg [ 578.713064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 17:48:33 \(1491241713\) [ 578.867243] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 17:48:33 (1491241713) [ 579.074484] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 579.699392] Lustre: DEBUG MARKER: rc=0; [ 579.699392] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 579.699392] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 579.699392] echo $(hostname -s): $val; [ 579.699392] rc=$val; [ 579.699392] fi; [ 579.699392] exit $rc [ 580.281176] Lustre: DEBUG MARKER: dmesg [ 580.637828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 ...\/f6h \(should return error\) ================ 17:48:35 \(1491241715\) [ 580.789153] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) ================ 17:48:35 (1491241715) [ 580.984946] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 581.578017] Lustre: DEBUG MARKER: rc=0; [ 581.578017] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 581.578017] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 581.578017] echo $(hostname -s): $val; [ 581.578017] rc=$val; [ 581.578017] fi; [ 581.578017] exit $rc [ 582.160460] Lustre: DEBUG MARKER: dmesg [ 582.528751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7a: mkdir ...\/d7\; mcreate ...\/d7\/f\; chmod ...\/d7\/f ==================================== 17:48:37 \(1491241717\) [ 582.693567] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 17:48:37 (1491241717) [ 582.929975] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 583.520721] Lustre: DEBUG MARKER: rc=0; [ 583.520721] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 583.520721] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 583.520721] echo $(hostname -s): $val; [ 583.520721] rc=$val; [ 583.520721] fi; [ 583.520721] exit $rc [ 584.105207] Lustre: DEBUG MARKER: dmesg [ 584.464658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7b: mkdir ...\/d7\; mcreate d7\/f2\; echo foo \> d7\/f2 ===================================== 17:48:39 \(1491241719\) [ 584.615658] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 17:48:39 (1491241719) [ 584.806816] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 585.410460] Lustre: DEBUG MARKER: rc=0; [ 585.410460] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 585.410460] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 585.410460] echo $(hostname -s): $val; [ 585.410460] rc=$val; [ 585.410460] fi; [ 585.410460] exit $rc [ 585.997790] Lustre: DEBUG MARKER: dmesg [ 586.359239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 8: mkdir ...\/d8\; touch ...\/d8\/f\; chmod ...\/d8\/f ======================================= 17:48:41 \(1491241721\) [ 586.517155] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 17:48:41 (1491241721) [ 586.710895] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 587.307791] Lustre: DEBUG MARKER: rc=0; [ 587.307791] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 587.307791] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 587.307791] echo $(hostname -s): $val; [ 587.307791] rc=$val; [ 587.307791] fi; [ 587.307791] exit $rc [ 587.890211] Lustre: DEBUG MARKER: dmesg [ 588.262019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 9: mkdir ...\/d9 ...\/d9\/d2 ...\/d9\/d2\/d3 ================================================ 17:48:43 \(1491241723\) [ 588.418987] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 17:48:43 (1491241723) [ 588.605346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 589.217795] Lustre: DEBUG MARKER: rc=0; [ 589.217795] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 589.217795] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 589.217795] echo $(hostname -s): $val; [ 589.217795] rc=$val; [ 589.217795] fi; [ 589.217795] exit $rc [ 589.800282] Lustre: DEBUG MARKER: dmesg [ 590.172818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 10: mkdir ...\/d10 ...\/d10\/d2\; touch ...\/d10\/d2\/f ====================================== 17:48:45 \(1491241725\) [ 590.326746] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 17:48:45 (1491241725) [ 590.522893] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 591.119384] Lustre: DEBUG MARKER: rc=0; [ 591.119384] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 591.119384] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 591.119384] echo $(hostname -s): $val; [ 591.119384] rc=$val; [ 591.119384] fi; [ 591.119384] exit $rc [ 591.711234] Lustre: DEBUG MARKER: dmesg [ 592.094260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 11: mkdir ...\/d11 d11\/d2\; chmod ...\/d11\/d2 ============================================ 17:48:47 \(1491241727\) [ 592.257126] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 17:48:47 (1491241727) [ 592.470171] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 593.097429] Lustre: DEBUG MARKER: rc=0; [ 593.097429] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 593.097429] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 593.097429] echo $(hostname -s): $val; [ 593.097429] rc=$val; [ 593.097429] fi; [ 593.097429] exit $rc [ 593.696681] Lustre: DEBUG MARKER: dmesg [ 594.104797] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 12: touch ...\/d12\/f\; chmod ...\/d12\/f ...\/d12\/f ======================================== 17:48:49 \(1491241729\) [ 594.256481] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 17:48:49 (1491241729) [ 594.447828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 595.062674] Lustre: DEBUG MARKER: rc=0; [ 595.062674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 595.062674] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 595.062674] echo $(hostname -s): $val; [ 595.062674] rc=$val; [ 595.062674] fi; [ 595.062674] exit $rc [ 595.652103] Lustre: DEBUG MARKER: dmesg [ 596.014998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 13: creat ...\/d13\/f\; dd ...\/d13\/f\; \> ...\/d13\/f ======================================== 17:48:51 \(1491241731\) [ 596.174724] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 17:48:51 (1491241731) [ 596.409942] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 597.016492] Lustre: DEBUG MARKER: rc=0; [ 597.016492] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 597.016492] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 597.016492] echo $(hostname -s): $val; [ 597.016492] rc=$val; [ 597.016492] fi; [ 597.016492] exit $rc [ 597.600125] Lustre: DEBUG MARKER: dmesg [ 597.978924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 14: touch ...\/d14\/f\; rm ...\/d14\/f\; rm ...\/d14\/f ======================================= 17:48:53 \(1491241733\) [ 598.131066] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 17:48:53 (1491241733) [ 598.326052] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 598.925901] Lustre: DEBUG MARKER: rc=0; [ 598.925901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 598.925901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 598.925901] echo $(hostname -s): $val; [ 598.925901] rc=$val; [ 598.925901] fi; [ 598.925901] exit $rc [ 599.531891] Lustre: DEBUG MARKER: dmesg [ 599.910244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 15: touch ...\/d15\/f\; mv ...\/d15\/f ...\/d15\/f2 ========================================== 17:48:55 \(1491241735\) [ 600.070430] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 17:48:55 (1491241735) [ 600.262748] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 600.877657] Lustre: DEBUG MARKER: rc=0; [ 600.877657] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 600.877657] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 600.877657] echo $(hostname -s): $val; [ 600.877657] rc=$val; [ 600.877657] fi; [ 600.877657] exit $rc [ 601.452836] Lustre: DEBUG MARKER: dmesg [ 601.838670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 16: touch ...\/d16\/f\; rm -rf ...\/d16\/f ================================================= 17:48:56 \(1491241736\) [ 601.994269] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 17:48:56 (1491241736) [ 602.184481] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 602.783259] Lustre: DEBUG MARKER: rc=0; [ 602.783259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 602.783259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 602.783259] echo $(hostname -s): $val; [ 602.783259] rc=$val; [ 602.783259] fi; [ 602.783259] exit $rc [ 603.355033] Lustre: DEBUG MARKER: dmesg [ 603.741041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17a: symlinks: create, remove \(real\) ================================================== 17:48:58 \(1491241738\) [ 603.902732] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 17:48:58 (1491241738) [ 604.108204] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 604.698641] Lustre: DEBUG MARKER: rc=0; [ 604.698641] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 604.698641] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 604.698641] echo $(hostname -s): $val; [ 604.698641] rc=$val; [ 604.698641] fi; [ 604.698641] exit $rc [ 605.276859] Lustre: DEBUG MARKER: dmesg [ 605.647579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17b: symlinks: create, remove \(dangling\) ============================================== 17:49:00 \(1491241740\) [ 605.817180] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 17:49:00 (1491241740) [ 606.021801] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 606.615927] Lustre: DEBUG MARKER: rc=0; [ 606.615927] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 606.615927] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 606.615927] echo $(hostname -s): $val; [ 606.615927] rc=$val; [ 606.615927] fi; [ 606.615927] exit $rc [ 607.203853] Lustre: DEBUG MARKER: dmesg [ 607.571609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17c: symlinks: open dangling \(should return error\) ==================================== 17:49:02 \(1491241742\) [ 607.732039] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 17:49:02 (1491241742) [ 607.918750] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 608.511819] Lustre: DEBUG MARKER: rc=0; [ 608.511819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 608.511819] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 608.511819] echo $(hostname -s): $val; [ 608.511819] rc=$val; [ 608.511819] fi; [ 608.511819] exit $rc [ 609.088711] Lustre: DEBUG MARKER: dmesg [ 609.470962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17d: symlinks: create dangling ======================================================== 17:49:04 \(1491241744\) [ 609.633244] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 17:49:04 (1491241744) [ 609.837361] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 610.454398] Lustre: DEBUG MARKER: rc=0; [ 610.454398] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 610.454398] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 610.454398] echo $(hostname -s): $val; [ 610.454398] rc=$val; [ 610.454398] fi; [ 610.454398] exit $rc [ 611.037577] Lustre: DEBUG MARKER: dmesg [ 611.404307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17e: symlinks: create recursive symlink \(should return error\) ========================= 17:49:06 \(1491241746\) [ 611.556670] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 17:49:06 (1491241746) [ 611.766502] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 612.366697] Lustre: DEBUG MARKER: rc=0; [ 612.366697] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 612.366697] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 612.366697] echo $(hostname -s): $val; [ 612.366697] rc=$val; [ 612.366697] fi; [ 612.366697] exit $rc [ 612.951030] Lustre: DEBUG MARKER: dmesg [ 613.319705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17f: symlinks: long and very long symlink name ======================================== 17:49:08 \(1491241748\) [ 613.478508] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 17:49:08 (1491241748) [ 613.680548] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 614.287804] Lustre: DEBUG MARKER: rc=0; [ 614.287804] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 614.287804] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 614.287804] echo $(hostname -s): $val; [ 614.287804] rc=$val; [ 614.287804] fi; [ 614.287804] exit $rc [ 614.864305] Lustre: DEBUG MARKER: dmesg [ 615.247289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 17:49:10 \(1491241750\) [ 615.403903] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 17:49:10 (1491241750) [ 615.543404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 615.543404] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 615.543404] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 615.812257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 615.812257] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 615.812257] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 616.085332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 616.085332] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 616.085332] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 616.614914] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 617.209474] Lustre: DEBUG MARKER: rc=0; [ 617.209474] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 617.209474] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 617.209474] echo $(hostname -s): $val; [ 617.209474] rc=$val; [ 617.209474] fi; [ 617.209474] exit $rc [ 617.791688] Lustre: DEBUG MARKER: dmesg [ 618.164482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17h: create objects: lov_free_memmd\(\) doesn\'t lbug ==================================== 17:49:13 \(1491241753\) [ 618.323085] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 17:49:13 (1491241753) [ 618.478752] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000141 [ 618.787040] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 619.380291] Lustre: DEBUG MARKER: rc=0; [ 619.380291] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 619.380291] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 619.380291] echo $(hostname -s): $val; [ 619.380291] rc=$val; [ 619.380291] fi; [ 619.380291] exit $rc [ 619.977326] Lustre: DEBUG MARKER: dmesg [ 620.351633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17i: don\'t panic on short symlink ===================================================== 17:49:15 \(1491241755\) [ 620.513429] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 17:49:15 (1491241755) [ 620.664845] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000143 [ 620.798216] Lustre: *** cfs_fail_loc=143, val=0*** [ 620.973624] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 621.575834] Lustre: DEBUG MARKER: rc=0; [ 621.575834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 621.575834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 621.575834] echo $(hostname -s): $val; [ 621.575834] rc=$val; [ 621.575834] fi; [ 621.575834] exit $rc [ 622.159762] Lustre: DEBUG MARKER: dmesg [ 622.524140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 17:49:17 \(1491241757\) [ 622.678963] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 17:49:17 (1491241757) [ 622.998627] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 623.604614] Lustre: DEBUG MARKER: rc=0; [ 623.604614] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 623.604614] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 623.604614] echo $(hostname -s): $val; [ 623.604614] rc=$val; [ 623.604614] fi; [ 623.604614] exit $rc [ 624.200845] Lustre: DEBUG MARKER: dmesg [ 624.574623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17l: Ensure lgetxattr\'s returned xattr size is consistent ============================= 17:49:19 \(1491241759\) [ 624.739964] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 17:49:19 (1491241759) [ 624.983960] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 625.599157] Lustre: DEBUG MARKER: rc=0; [ 625.599157] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 625.599157] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 625.599157] echo $(hostname -s): $val; [ 625.599157] rc=$val; [ 625.599157] fi; [ 625.599157] exit $rc [ 626.179117] Lustre: DEBUG MARKER: dmesg [ 626.548554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17m: run e2fsck against MDT which contains short\/long symlink ========================= 17:49:21 \(1491241761\) [ 626.700138] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 17:49:21 (1491241761) [ 626.846382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 626.846382] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 626.846382] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 627.112664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 627.112664] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 627.112664] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 629.772600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 634.039281] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 634.294941] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 634.422773] Lustre: Failing over lustre-MDT0000 [ 636.606376] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.610592] LustreError: Skipped 6 previous similar messages [ 639.021870] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.496090] Lustre: 20980:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491241770/real 1491241770] req@ffff880062d66a00 x1563680142533280/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491241776 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 640.599612] Lustre: server umount lustre-MDT0000 complete [ 640.727772] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 640.981828] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [ 641.493171] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 641.753358] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 642.006719] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 642.259124] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 642.415913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.433084] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 642.436740] Lustre: Skipped 5 previous similar messages [ 642.539110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.692470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 642.961845] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 643.463481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 643.473545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 643.651874] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 643.673797] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 643.816178] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 644.071654] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 644.327494] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 644.586962] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 644.842324] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 649.010025] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 649.454605] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 649.661394] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 650.302719] Lustre: DEBUG MARKER: rc=0; [ 650.302719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 650.302719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 650.302719] echo $(hostname -s): $val; [ 650.302719] rc=$val; [ 650.302719] fi; [ 650.302719] exit $rc [ 650.901461] Lustre: DEBUG MARKER: dmesg [ 651.281664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17n: run e2fsck against master\/slave MDT which contains remote dir ==================== 17:49:46 \(1491241786\) [ 651.448820] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 17:49:46 (1491241786) [ 651.594892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 651.594892] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 651.594892] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 651.869356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 651.869356] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 651.869356] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 652.202707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_17n needs \>= 2 MDTs [ 652.377569] Lustre: DEBUG MARKER: SKIP: sanity test_17n needs >= 2 MDTs [ 652.554067] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 653.151981] Lustre: DEBUG MARKER: rc=0; [ 653.151981] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 653.151981] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 653.151981] echo $(hostname -s): $val; [ 653.151981] rc=$val; [ 653.151981] fi; [ 653.151981] exit $rc [ 653.741466] Lustre: DEBUG MARKER: dmesg [ 654.137490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17o: stat file with incompat LMA feature ============================================== 17:49:49 \(1491241789\) [ 654.295079] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 17:49:49 (1491241789) [ 654.443868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 654.443868] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 654.443868] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 654.739083] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 655.003554] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 655.132243] Lustre: Failing over lustre-MDT0000 [ 658.617613] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.621970] LustreError: Skipped 4 previous similar messages [ 661.186076] Lustre: 23396:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491241790/real 1491241790] req@ffff880062ee6a00 x1563680142534720/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491241796 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 661.349166] Lustre: server umount lustre-MDT0000 complete [ 661.476327] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 671.777365] Lustre: DEBUG MARKER: hostname [ 672.079933] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 672.329849] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 672.580914] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 672.736639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.802878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.949116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 673.203308] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 673.686888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 673.693319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 673.852468] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 673.855367] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 673.999027] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 674.252882] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 674.507403] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 674.767845] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 675.542232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.593636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.783138] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 675.789092] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 681.040311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 681.454507] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 682.076641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 682.076642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 682.285318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 682.290538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 683.629993] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x194 [ 683.764719] Lustre: *** cfs_fail_loc=194, val=0*** [ 683.767197] Lustre: 23795:0:(osd_handler.c:752:osd_check_lma()) lustre-MDT0000: unsupported incompat LMA feature(s) 0x0 for fid = [0x200000401:0x658:0x0], ino = 1255 [ 683.896298] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 684.200068] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 684.816350] Lustre: DEBUG MARKER: rc=0; [ 684.816350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 684.816350] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 684.816350] echo $(hostname -s): $val; [ 684.816350] rc=$val; [ 684.816350] fi; [ 684.816350] exit $rc [ 685.423746] Lustre: DEBUG MARKER: dmesg [ 685.817246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 18: touch ...\/f \; ls ... ============================================================== 17:50:20 \(1491241820\) [ 685.985582] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 17:50:20 (1491241820) [ 686.174663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 686.792833] Lustre: DEBUG MARKER: rc=0; [ 686.792833] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 686.792833] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 686.792833] echo $(hostname -s): $val; [ 686.792833] rc=$val; [ 686.792833] fi; [ 686.792833] exit $rc [ 687.393478] Lustre: DEBUG MARKER: dmesg [ 687.765219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19a: touch ...\/f19 \; ls -l ... \; rm ...\/f19 =========================================== 17:50:22 \(1491241822\) [ 687.923404] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 17:50:22 (1491241822) [ 688.156046] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 688.766217] Lustre: DEBUG MARKER: rc=0; [ 688.766217] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 688.766217] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 688.766217] echo $(hostname -s): $val; [ 688.766217] rc=$val; [ 688.766217] fi; [ 688.766217] exit $rc [ 689.354799] Lustre: DEBUG MARKER: dmesg [ 689.737637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19b: ls -l ...\/f19 \(should return error\) ============================================== 17:50:24 \(1491241824\) [ 689.911722] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 17:50:24 (1491241824) [ 690.090088] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 690.686332] Lustre: DEBUG MARKER: rc=0; [ 690.686332] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 690.686332] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 690.686332] echo $(hostname -s): $val; [ 690.686332] rc=$val; [ 690.686332] fi; [ 690.686332] exit $rc [ 691.264664] Lustre: DEBUG MARKER: dmesg [ 691.646731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19c: runas -u 500 -g 500 touch ...\/f19 \(should return error\) ========================== 17:50:26 \(1491241826\) [ 691.798141] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 17:50:26 (1491241826) [ 691.989068] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 692.607969] Lustre: DEBUG MARKER: rc=0; [ 692.607969] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 692.607969] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 692.607969] echo $(hostname -s): $val; [ 692.607969] rc=$val; [ 692.607969] fi; [ 692.607969] exit $rc [ 693.214811] Lustre: DEBUG MARKER: dmesg [ 693.593592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19d: cat ...\/f19 \(should return error\) ================================================ 17:50:28 \(1491241828\) [ 693.749387] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 17:50:28 (1491241828) [ 693.931410] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 694.543469] Lustre: DEBUG MARKER: rc=0; [ 694.543469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 694.543469] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 694.543469] echo $(hostname -s): $val; [ 694.543469] rc=$val; [ 694.543469] fi; [ 694.543469] exit $rc [ 695.137568] Lustre: DEBUG MARKER: dmesg [ 695.516066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 20: touch ...\/f \; ls -l ... =========================================================== 17:50:30 \(1491241830\) [ 695.673495] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 17:50:30 (1491241830) [ 695.859642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 done [ 696.018934] Lustre: DEBUG MARKER: 1 done [ 696.209491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 done [ 696.366476] Lustre: DEBUG MARKER: 2 done [ 696.564718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 done [ 696.724279] Lustre: DEBUG MARKER: 3 done [ 696.916754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 697.517809] Lustre: DEBUG MARKER: rc=0; [ 697.517809] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 697.517809] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 697.517809] echo $(hostname -s): $val; [ 697.517809] rc=$val; [ 697.517809] fi; [ 697.517809] exit $rc [ 698.104217] Lustre: DEBUG MARKER: dmesg [ 698.495688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 21: write to dangling link ============================================================ 17:50:33 \(1491241833\) [ 698.656330] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 17:50:33 (1491241833) [ 698.861891] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 699.471619] Lustre: DEBUG MARKER: rc=0; [ 699.471619] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 699.471619] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 699.471619] echo $(hostname -s): $val; [ 699.471619] rc=$val; [ 699.471619] fi; [ 699.471619] exit $rc [ 700.068626] Lustre: DEBUG MARKER: dmesg [ 700.439359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 22: unpack tar archive as non-root user =============================================== 17:50:35 \(1491241835\) [ 700.599177] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 17:50:35 (1491241835) [ 700.994206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 701.631890] Lustre: DEBUG MARKER: rc=0; [ 701.631890] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 701.631890] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 701.631890] echo $(hostname -s): $val; [ 701.631890] rc=$val; [ 701.631890] fi; [ 701.631890] exit $rc [ 702.216601] Lustre: DEBUG MARKER: dmesg [ 702.606814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23a: O_CREAT\|O_EXCL in subdir ========================================================= 17:50:37 \(1491241837\) [ 702.759744] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 17:50:37 (1491241837) [ 702.996974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 703.597981] Lustre: DEBUG MARKER: rc=0; [ 703.597981] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 703.597981] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 703.597981] echo $(hostname -s): $val; [ 703.597981] rc=$val; [ 703.597981] fi; [ 703.597981] exit $rc [ 704.186788] Lustre: DEBUG MARKER: dmesg [ 704.567840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23b: O_APPEND check =================================================================== 17:50:39 \(1491241839\) [ 704.727524] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 17:50:39 (1491241839) [ 704.932849] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 705.531252] Lustre: DEBUG MARKER: rc=0; [ 705.531252] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 705.531252] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 705.531252] echo $(hostname -s): $val; [ 705.531252] rc=$val; [ 705.531252] fi; [ 705.531252] exit $rc [ 706.134297] Lustre: DEBUG MARKER: dmesg [ 706.526239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24a: rename file to non-existent target =============================================== 17:50:41 \(1491241841\) [ 706.690612] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 17:50:41 (1491241841) [ 706.887501] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 707.504949] Lustre: DEBUG MARKER: rc=0; [ 707.504949] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 707.504949] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 707.504949] echo $(hostname -s): $val; [ 707.504949] rc=$val; [ 707.504949] fi; [ 707.504949] exit $rc [ 708.093319] Lustre: DEBUG MARKER: dmesg [ 708.469896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24b: rename file to existing target =================================================== 17:50:43 \(1491241843\) [ 708.628196] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 17:50:43 (1491241843) [ 708.831460] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 709.442075] Lustre: DEBUG MARKER: rc=0; [ 709.442075] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 709.442075] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 709.442075] echo $(hostname -s): $val; [ 709.442075] rc=$val; [ 709.442075] fi; [ 709.442075] exit $rc [ 710.053051] Lustre: DEBUG MARKER: dmesg [ 710.435660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24c: rename directory to non-existent target ========================================== 17:50:45 \(1491241845\) [ 710.600802] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 17:50:45 (1491241845) [ 710.805451] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 711.409575] Lustre: DEBUG MARKER: rc=0; [ 711.409575] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 711.409575] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 711.409575] echo $(hostname -s): $val; [ 711.409575] rc=$val; [ 711.409575] fi; [ 711.409575] exit $rc [ 711.999360] Lustre: DEBUG MARKER: dmesg [ 712.378061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24d: rename directory to existing target ============================================== 17:50:47 \(1491241847\) [ 712.547056] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 17:50:47 (1491241847) [ 712.788048] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 713.399131] Lustre: DEBUG MARKER: rc=0; [ 713.399131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 713.399131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 713.399131] echo $(hostname -s): $val; [ 713.399131] rc=$val; [ 713.399131] fi; [ 713.399131] exit $rc [ 713.986786] Lustre: DEBUG MARKER: dmesg [ 714.355320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24e: touch ...\/R5a\/f\; rename ...\/R5a\/f ...\/R5b\/g ====================================== 17:50:49 \(1491241849\) [ 714.548765] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 17:50:49 (1491241849) [ 714.762295] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 715.375813] Lustre: DEBUG MARKER: rc=0; [ 715.375813] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 715.375813] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 715.375813] echo $(hostname -s): $val; [ 715.375813] rc=$val; [ 715.375813] fi; [ 715.375813] exit $rc [ 715.971568] Lustre: DEBUG MARKER: dmesg [ 716.349735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24f: touch ...\/R6a\/f R6b\/g\; mv ...\/R6a\/f ...\/R6b\/g ==================================== 17:50:51 \(1491241851\) [ 716.505195] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 17:50:51 (1491241851) [ 716.720218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 717.333092] Lustre: DEBUG MARKER: rc=0; [ 717.333092] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 717.333092] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 717.333092] echo $(hostname -s): $val; [ 717.333092] rc=$val; [ 717.333092] fi; [ 717.333092] exit $rc [ 717.937327] Lustre: DEBUG MARKER: dmesg [ 718.312590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24g: mkdir ...\/R7{a,b}\/d\; mv ...\/R7a\/d ...\/R7b\/e ====================================== 17:50:53 \(1491241853\) [ 718.468672] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 17:50:53 (1491241853) [ 718.673291] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 719.289432] Lustre: DEBUG MARKER: rc=0; [ 719.289432] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 719.289432] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 719.289432] echo $(hostname -s): $val; [ 719.289432] rc=$val; [ 719.289432] fi; [ 719.289432] exit $rc [ 719.877839] Lustre: DEBUG MARKER: dmesg [ 720.264335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24h: mkdir ...\/R8{a,b}\/{d,e}\; rename ...\/R8a\/d ...\/R8b\/e ============================== 17:50:55 \(1491241855\) [ 720.417058] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 17:50:55 (1491241855) [ 720.614170] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 721.215885] Lustre: DEBUG MARKER: rc=0; [ 721.215885] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 721.215885] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 721.215885] echo $(hostname -s): $val; [ 721.215885] rc=$val; [ 721.215885] fi; [ 721.215885] exit $rc [ 721.805647] Lustre: DEBUG MARKER: dmesg [ 722.183191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24i: rename file to dir error: touch f \; mkdir a \; rename f a ========================= 17:50:57 \(1491241857\) [ 722.336125] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 17:50:57 (1491241857) [ 722.528770] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 723.133543] Lustre: DEBUG MARKER: rc=0; [ 723.133543] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 723.133543] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 723.133543] echo $(hostname -s): $val; [ 723.133543] rc=$val; [ 723.133543] fi; [ 723.133543] exit $rc [ 723.731451] Lustre: DEBUG MARKER: dmesg [ 724.138996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24j: source does not exist ============================================================ 17:50:59 \(1491241859\) [ 724.291368] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 17:50:59 (1491241859) [ 724.483901] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 725.080923] Lustre: DEBUG MARKER: rc=0; [ 725.080923] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 725.080923] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 725.080923] echo $(hostname -s): $val; [ 725.080923] rc=$val; [ 725.080923] fi; [ 725.080923] exit $rc [ 725.666786] Lustre: DEBUG MARKER: dmesg [ 726.058521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24k: touch ...\/R11a\/f\; mv ...\/R11a\/f ...\/R11a\/d ======================================= 17:51:01 \(1491241861\) [ 726.214500] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 17:51:01 (1491241861) [ 726.417865] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 727.022525] Lustre: DEBUG MARKER: rc=0; [ 727.022525] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 727.022525] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 727.022525] echo $(hostname -s): $val; [ 727.022525] rc=$val; [ 727.022525] fi; [ 727.022525] exit $rc [ 727.620567] Lustre: DEBUG MARKER: dmesg [ 727.997911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24l: Renaming a file to itself ======================================================== 17:51:03 \(1491241863\) [ 728.162977] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 17:51:03 (1491241863) [ 728.408301] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 729.015322] Lustre: DEBUG MARKER: rc=0; [ 729.015322] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 729.015322] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 729.015322] echo $(hostname -s): $val; [ 729.015322] rc=$val; [ 729.015322] fi; [ 729.015322] exit $rc [ 729.612206] Lustre: DEBUG MARKER: dmesg [ 730.000829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24m: Renaming a file to a hard link to itself ========================================= 17:51:05 \(1491241865\) [ 730.162869] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 17:51:05 (1491241865) [ 730.359792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 730.971471] Lustre: DEBUG MARKER: rc=0; [ 730.971471] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 730.971471] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 730.971471] echo $(hostname -s): $val; [ 730.971471] rc=$val; [ 730.971471] fi; [ 730.971471] exit $rc [ 731.565842] Lustre: DEBUG MARKER: dmesg [ 731.949446] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24n: Statting the old file after renaming \(Posix rename 2\) ============================ 17:51:07 \(1491241867\) [ 732.097893] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 17:51:07 (1491241867) [ 732.295580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 732.896721] Lustre: DEBUG MARKER: rc=0; [ 732.896721] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 732.896721] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 732.896721] echo $(hostname -s): $val; [ 732.896721] rc=$val; [ 732.896721] fi; [ 732.896721] exit $rc [ 733.475525] Lustre: DEBUG MARKER: dmesg [ 733.877822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24o: rename of files during htree split =============================================== 17:51:08 \(1491241868\) [ 734.042316] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 17:51:08 (1491241868) [ 776.304433] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 776.908795] Lustre: DEBUG MARKER: rc=0; [ 776.908795] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 776.908795] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 776.908795] echo $(hostname -s): $val; [ 776.908795] rc=$val; [ 776.908795] fi; [ 776.908795] exit $rc [ 777.516065] Lustre: DEBUG MARKER: dmesg [ 777.916942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24p: mkdir ...\/R12{a,b}\; rename ...\/R12a ...\/R12b ===================================== 17:51:53 \(1491241913\) [ 778.081087] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 17:51:53 (1491241913) [ 778.293481] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 778.895035] Lustre: DEBUG MARKER: rc=0; [ 778.895035] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 778.895035] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 778.895035] echo $(hostname -s): $val; [ 778.895035] rc=$val; [ 778.895035] fi; [ 778.895035] exit $rc [ 779.485730] Lustre: DEBUG MARKER: dmesg [ 779.870171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24q: mkdir ...\/R13{a,b}\; open R13b rename R13a R13b =================================== 17:51:54 \(1491241914\) [ 780.022783] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 17:51:54 (1491241914) [ 780.237958] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 780.851894] Lustre: DEBUG MARKER: rc=0; [ 780.851894] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 780.851894] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 780.851894] echo $(hostname -s): $val; [ 780.851894] rc=$val; [ 780.851894] fi; [ 780.851894] exit $rc [ 781.447634] Lustre: DEBUG MARKER: dmesg [ 781.837048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24r: mkdir ...\/R14a\/b\; rename ...\/R14a ...\/R14a\/b ===================================== 17:51:56 \(1491241916\) [ 782.009653] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 17:51:56 (1491241916) [ 782.215790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 782.850725] Lustre: DEBUG MARKER: rc=0; [ 782.850725] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 782.850725] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 782.850725] echo $(hostname -s): $val; [ 782.850725] rc=$val; [ 782.850725] fi; [ 782.850725] exit $rc [ 783.450333] Lustre: DEBUG MARKER: dmesg [ 783.825203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24s: mkdir ...\/R15a\/b\/c\; rename ...\/R15a ...\/R15a\/b\/c ================================= 17:51:58 \(1491241918\) [ 783.975693] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 17:51:58 (1491241918) [ 784.171525] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 784.774860] Lustre: DEBUG MARKER: rc=0; [ 784.774860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 784.774860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 784.774860] echo $(hostname -s): $val; [ 784.774860] rc=$val; [ 784.774860] fi; [ 784.774860] exit $rc [ 785.363716] Lustre: DEBUG MARKER: dmesg [ 785.740949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24t: mkdir ...\/R16a\/b\/c\; rename ...\/R16a\/b\/c ...\/R16a ================================= 17:52:00 \(1491241920\) [ 785.895717] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 17:52:00 (1491241920) [ 786.091626] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 786.697926] Lustre: DEBUG MARKER: rc=0; [ 786.697926] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 786.697926] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 786.697926] echo $(hostname -s): $val; [ 786.697926] rc=$val; [ 786.697926] fi; [ 786.697926] exit $rc [ 787.282308] Lustre: DEBUG MARKER: dmesg [ 787.697969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24u: create stripe file =============================================================== 17:52:02 \(1491241922\) [ 787.855391] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 17:52:02 (1491241922) [ 788.101461] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 788.704845] Lustre: DEBUG MARKER: rc=0; [ 788.704845] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 788.704845] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 788.704845] echo $(hostname -s): $val; [ 788.704845] rc=$val; [ 788.704845] fi; [ 788.704845] exit $rc [ 789.295519] Lustre: DEBUG MARKER: dmesg [ 789.698902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24v: list directory with large files \(handle hash collision, bug: 17560\) ============== 17:52:04 \(1491241924\) [ 789.867841] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 17:52:04 (1491241924) [ 1015.362034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1016.682433] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1017.280358] Lustre: DEBUG MARKER: rc=0; [ 1017.280358] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1017.280358] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1017.280358] echo $(hostname -s): $val; [ 1017.280358] rc=$val; [ 1017.280358] fi; [ 1017.280358] exit $rc [ 1017.871764] Lustre: DEBUG MARKER: dmesg [ 1018.341999] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24w: Reading a file larger than 4Gb =================================================== 17:55:53 \(1491242153\) [ 1018.509694] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 17:55:53 (1491242153) [ 1018.813812] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1019.423307] Lustre: DEBUG MARKER: rc=0; [ 1019.423307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1019.423307] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1019.423307] echo $(hostname -s): $val; [ 1019.423307] rc=$val; [ 1019.423307] fi; [ 1019.423307] exit $rc [ 1020.003857] Lustre: DEBUG MARKER: dmesg [ 1020.389303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24x: cross MDT rename\/link ============================================================ 17:55:55 \(1491242155\) [ 1020.556360] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 17:55:55 (1491242155) [ 1020.754317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24x needs \>= 2 MDTs [ 1020.923504] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 1021.125103] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1021.746270] Lustre: DEBUG MARKER: rc=0; [ 1021.746270] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1021.746270] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1021.746270] echo $(hostname -s): $val; [ 1021.746270] rc=$val; [ 1021.746270] fi; [ 1021.746270] exit $rc [ 1022.333234] Lustre: DEBUG MARKER: dmesg [ 1022.716192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24y: rename\/link on the same dir should succeed ======================================= 17:55:57 \(1491242157\) [ 1022.876095] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 17:55:57 (1491242157) [ 1023.056563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24y needs \>= 2 MDTs [ 1023.207217] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 1023.398427] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1024.005466] Lustre: DEBUG MARKER: rc=0; [ 1024.005466] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1024.005466] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1024.005466] echo $(hostname -s): $val; [ 1024.005466] rc=$val; [ 1024.005466] fi; [ 1024.005466] exit $rc [ 1024.589625] Lustre: DEBUG MARKER: dmesg [ 1024.971430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24A: readdir\(\) returns correct number of entries. ===================================== 17:56:00 \(1491242160\) [ 1025.127218] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 17:56:00 (1491242160) [ 1035.227513] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1035.835834] Lustre: DEBUG MARKER: rc=0; [ 1035.835834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1035.835834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1035.835834] echo $(hostname -s): $val; [ 1035.835834] rc=$val; [ 1035.835834] fi; [ 1035.835834] exit $rc [ 1036.429893] Lustre: DEBUG MARKER: dmesg [ 1036.832997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24B: readdir for striped dir return correct number of entries ========================= 17:56:11 \(1491242171\) [ 1036.996325] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 17:56:11 (1491242171) [ 1037.183969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24B needs \>= 2 MDTs [ 1037.338605] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 1037.535380] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1038.172133] Lustre: DEBUG MARKER: rc=0; [ 1038.172133] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1038.172133] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1038.172133] echo $(hostname -s): $val; [ 1038.172133] rc=$val; [ 1038.172133] fi; [ 1038.172133] exit $rc [ 1038.763646] Lustre: DEBUG MARKER: dmesg [ 1039.156221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24C: check .. in striped dir ========================================================== 17:56:14 \(1491242174\) [ 1039.306779] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 17:56:14 (1491242174) [ 1039.489770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24C needs \>= 2 MDTs [ 1039.647197] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 1039.835996] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1040.446096] Lustre: DEBUG MARKER: rc=0; [ 1040.446096] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1040.446096] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1040.446096] echo $(hostname -s): $val; [ 1040.446096] rc=$val; [ 1040.446096] fi; [ 1040.446096] exit $rc [ 1041.023237] Lustre: DEBUG MARKER: dmesg [ 1041.441334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24D skipping SLOW test 24D [ 1041.592374] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [ 1041.792580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24E: cross MDT rename\/link ============================================================ 17:56:16 \(1491242176\) [ 1041.945729] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 17:56:16 (1491242176) [ 1042.133186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24E needs \>= 4 MDTs [ 1042.288158] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 1042.471321] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1043.085324] Lustre: DEBUG MARKER: rc=0; [ 1043.085324] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1043.085324] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1043.085324] echo $(hostname -s): $val; [ 1043.085324] rc=$val; [ 1043.085324] fi; [ 1043.085324] exit $rc [ 1043.691020] Lustre: DEBUG MARKER: dmesg [ 1044.073338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25a: create file in symlinked directory =============================================== 17:56:19 \(1491242179\) [ 1044.229679] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 17:56:19 (1491242179) [ 1044.426251] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1045.040758] Lustre: DEBUG MARKER: rc=0; [ 1045.040758] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1045.040758] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1045.040758] echo $(hostname -s): $val; [ 1045.040758] rc=$val; [ 1045.040758] fi; [ 1045.040758] exit $rc [ 1045.631315] Lustre: DEBUG MARKER: dmesg [ 1046.022382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25b: lookup file in symlinked directory =============================================== 17:56:21 \(1491242181\) [ 1046.178804] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 17:56:21 (1491242181) [ 1046.367236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1046.983707] Lustre: DEBUG MARKER: rc=0; [ 1046.983707] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1046.983707] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1046.983707] echo $(hostname -s): $val; [ 1046.983707] rc=$val; [ 1046.983707] fi; [ 1046.983707] exit $rc [ 1047.564683] Lustre: DEBUG MARKER: dmesg [ 1047.978609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26a: multiple component symlink ======================================================= 17:56:23 \(1491242183\) [ 1048.140617] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 17:56:23 (1491242183) [ 1048.340458] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1048.945167] Lustre: DEBUG MARKER: rc=0; [ 1048.945167] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1048.945167] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1048.945167] echo $(hostname -s): $val; [ 1048.945167] rc=$val; [ 1048.945167] fi; [ 1048.945167] exit $rc [ 1049.528786] Lustre: DEBUG MARKER: dmesg [ 1049.925368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26b: multiple component symlink at end of lookup ====================================== 17:56:25 \(1491242185\) [ 1050.083055] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 17:56:25 (1491242185) [ 1050.286934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1050.902913] Lustre: DEBUG MARKER: rc=0; [ 1050.902913] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1050.902913] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1050.902913] echo $(hostname -s): $val; [ 1050.902913] rc=$val; [ 1050.902913] fi; [ 1050.902913] exit $rc [ 1051.491094] Lustre: DEBUG MARKER: dmesg [ 1051.866829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26c: chain of symlinks ================================================================ 17:56:26 \(1491242186\) [ 1052.027294] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 17:56:26 (1491242186) [ 1052.239359] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1052.852490] Lustre: DEBUG MARKER: rc=0; [ 1052.852490] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1052.852490] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1052.852490] echo $(hostname -s): $val; [ 1052.852490] rc=$val; [ 1052.852490] fi; [ 1052.852490] exit $rc [ 1053.475149] Lustre: DEBUG MARKER: dmesg [ 1053.905035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26d: create multiple component recursive symlink ====================================== 17:56:29 \(1491242189\) [ 1054.064604] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 17:56:29 (1491242189) [ 1054.249661] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1054.863777] Lustre: DEBUG MARKER: rc=0; [ 1054.863777] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1054.863777] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1054.863777] echo $(hostname -s): $val; [ 1054.863777] rc=$val; [ 1054.863777] fi; [ 1054.863777] exit $rc [ 1055.453119] Lustre: DEBUG MARKER: dmesg [ 1055.829874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26e: unlink multiple component recursive symlink ====================================== 17:56:30 \(1491242190\) [ 1055.994983] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 17:56:30 (1491242190) [ 1056.194313] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1056.839389] Lustre: DEBUG MARKER: rc=0; [ 1056.839389] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1056.839389] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1056.839389] echo $(hostname -s): $val; [ 1056.839389] rc=$val; [ 1056.839389] fi; [ 1056.839389] exit $rc [ 1057.423894] Lustre: DEBUG MARKER: dmesg [ 1057.819983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 17:56:32 \(1491242192\) [ 1057.981980] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 17:56:32 (1491242192) [ 1058.220223] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1058.827634] Lustre: DEBUG MARKER: rc=0; [ 1058.827634] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1058.827634] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1058.827634] echo $(hostname -s): $val; [ 1058.827634] rc=$val; [ 1058.827634] fi; [ 1058.827634] exit $rc [ 1059.413319] Lustre: DEBUG MARKER: dmesg [ 1059.790767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27a: one stripe file ================================================================== 17:56:34 \(1491242194\) [ 1059.943660] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 17:56:34 (1491242194) [ 1060.151561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == test_27a: write to one stripe file ========================= [ 1060.307488] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [ 1060.526053] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1061.142003] Lustre: DEBUG MARKER: rc=0; [ 1061.142003] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1061.142003] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1061.142003] echo $(hostname -s): $val; [ 1061.142003] rc=$val; [ 1061.142003] fi; [ 1061.142003] exit $rc [ 1061.751443] Lustre: DEBUG MARKER: dmesg [ 1062.139154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27b: create and write to two stripe file ============================================== 17:56:37 \(1491242197\) [ 1062.303507] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 17:56:37 (1491242197) [ 1062.515644] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1063.133414] Lustre: DEBUG MARKER: rc=0; [ 1063.133414] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1063.133414] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1063.133414] echo $(hostname -s): $val; [ 1063.133414] rc=$val; [ 1063.133414] fi; [ 1063.133414] exit $rc [ 1063.736177] Lustre: DEBUG MARKER: dmesg [ 1064.131224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27d: create file with default settings ================================================ 17:56:39 \(1491242199\) [ 1064.287793] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 17:56:39 (1491242199) [ 1064.492590] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1065.099300] Lustre: DEBUG MARKER: rc=0; [ 1065.099300] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1065.099300] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1065.099300] echo $(hostname -s): $val; [ 1065.099300] rc=$val; [ 1065.099300] fi; [ 1065.099300] exit $rc [ 1065.694621] Lustre: DEBUG MARKER: dmesg [ 1066.079247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27e: setstripe existing file \(should return error\) ==================================== 17:56:41 \(1491242201\) [ 1066.239336] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 17:56:41 (1491242201) [ 1066.371216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1066.371216] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1066.371216] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1066.714784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1067.321983] Lustre: DEBUG MARKER: rc=0; [ 1067.321983] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1067.321983] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1067.321983] echo $(hostname -s): $val; [ 1067.321983] rc=$val; [ 1067.321983] fi; [ 1067.321983] exit $rc [ 1067.915308] Lustre: DEBUG MARKER: dmesg [ 1068.294689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27f: setstripe with bad stripe size \(should return error\) ============================= 17:56:43 \(1491242203\) [ 1068.448127] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 17:56:43 (1491242203) [ 1068.655481] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1069.259532] Lustre: DEBUG MARKER: rc=0; [ 1069.259532] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1069.259532] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1069.259532] echo $(hostname -s): $val; [ 1069.259532] rc=$val; [ 1069.259532] fi; [ 1069.259532] exit $rc [ 1069.848237] Lustre: DEBUG MARKER: dmesg [ 1070.235209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27g: \/usr\/bin\/lfs getstripe with no objects =========================================== 17:56:45 \(1491242205\) [ 1070.386865] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects =========================================== 17:56:45 (1491242205) [ 1070.586251] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1071.193882] Lustre: DEBUG MARKER: rc=0; [ 1071.193882] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1071.193882] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1071.193882] echo $(hostname -s): $val; [ 1071.193882] rc=$val; [ 1071.193882] fi; [ 1071.193882] exit $rc [ 1071.779074] Lustre: DEBUG MARKER: dmesg [ 1072.166900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27i: \/usr\/bin\/lfs getstripe with some objects ========================================= 17:56:47 \(1491242207\) [ 1072.320774] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects ========================================= 17:56:47 (1491242207) [ 1072.517998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1073.123936] Lustre: DEBUG MARKER: rc=0; [ 1073.123936] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1073.123936] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1073.123936] echo $(hostname -s): $val; [ 1073.123936] rc=$val; [ 1073.123936] fi; [ 1073.123936] exit $rc [ 1073.714403] Lustre: DEBUG MARKER: dmesg [ 1074.104406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27j: setstripe with bad stripe offset \(should return error\) =========================== 17:56:49 \(1491242209\) [ 1074.259966] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 17:56:49 (1491242209) [ 1074.276760] LustreError: 23794:0:(lod_qos.c:1201:lod_alloc_specific()) Start index 7 not found in pool '' [ 1074.460194] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1075.068000] Lustre: DEBUG MARKER: rc=0; [ 1075.068000] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1075.068000] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1075.068000] echo $(hostname -s): $val; [ 1075.068000] rc=$val; [ 1075.068000] fi; [ 1075.068000] exit $rc [ 1075.660586] Lustre: DEBUG MARKER: dmesg [ 1076.079855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27k: limit i_blksize for broken user apps ============================================= 17:56:51 \(1491242211\) [ 1076.232190] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 17:56:51 (1491242211) [ 1076.441059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1077.053920] Lustre: DEBUG MARKER: rc=0; [ 1077.053920] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1077.053920] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1077.053920] echo $(hostname -s): $val; [ 1077.053920] rc=$val; [ 1077.053920] fi; [ 1077.053920] exit $rc [ 1077.636911] Lustre: DEBUG MARKER: dmesg [ 1078.051019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27l: check setstripe permissions \(should return error\) ================================ 17:56:53 \(1491242213\) [ 1078.211491] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 17:56:53 (1491242213) [ 1078.414362] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1079.025270] Lustre: DEBUG MARKER: rc=0; [ 1079.025270] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1079.025270] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1079.025270] echo $(hostname -s): $val; [ 1079.025270] rc=$val; [ 1079.025270] fi; [ 1079.025270] exit $rc [ 1079.614068] Lustre: DEBUG MARKER: dmesg [ 1080.012445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_27m skipping SLOW test 27m [ 1080.181087] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 1080.380963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27n: create file with some full OSTs ================================================== 17:56:55 \(1491242215\) [ 1080.546590] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 17:56:55 (1491242215) [ 1081.078144] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1091.358804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1091.358804] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1091.358804] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1091.631398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1091.631398] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1091.631398] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1091.898633] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1092.153217] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1092.405327] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1092.662427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1092.662427] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1092.662427] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1092.930446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1092.930446] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1092.930446] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1093.204247] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1093.461357] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1094.034853] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1094.553359] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1105.154349] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1115.477102] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1116.104920] Lustre: DEBUG MARKER: rc=0; [ 1116.104920] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1116.104920] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1116.104920] echo $(hostname -s): $val; [ 1116.104920] rc=$val; [ 1116.104920] fi; [ 1116.104920] exit $rc [ 1116.689635] Lustre: DEBUG MARKER: dmesg [ 1117.081313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27o: create file with all full OSTs \(should error\) ==================================== 17:57:32 \(1491242252\) [ 1117.255808] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 17:57:32 (1491242252) [ 1117.786318] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1128.059075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1128.059075] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1128.059075] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1128.333696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1128.333696] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1128.333696] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1128.602843] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1128.858048] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1129.111264] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1129.366528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1129.366528] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1129.366528] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1129.633070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1129.633070] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1129.633070] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1129.900131] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1130.156398] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1130.719620] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1131.231055] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1141.502958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1141.502958] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1141.502958] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1141.773648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1141.773648] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1141.773648] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1142.043258] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1142.299240] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1142.568063] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1142.838201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1142.838201] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1142.838201] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1143.116357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1143.116357] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1143.116357] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1143.387134] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1143.641077] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1144.041238] LustreError: 24776:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c84b:0x0]: have 0 want 1 [ 1144.168882] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1144.692478] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1154.965334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1154.965334] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1154.965334] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1155.235693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1155.235693] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1155.235693] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1155.504692] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1155.760588] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_last_id [ 1156.013533] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_next_id [ 1156.274728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1156.274728] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1156.274728] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1156.543476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1156.543476] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1156.543476] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1156.812088] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1157.064388] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1157.468155] LustreError: 23796:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c84d:0x0]: have 0 want 1 [ 1157.602224] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1158.129924] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1168.409819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1168.409819] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1168.409819] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1168.682983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1168.682983] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1168.682983] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1168.953333] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1169.210143] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_last_id [ 1169.465951] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_next_id [ 1169.725499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1169.725499] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1169.725499] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1169.994053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1169.994053] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1169.994053] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1170.263695] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1170.516586] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1170.914118] LustreError: 23794:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c84f:0x0]: have 0 want 1 [ 1171.044710] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1171.563928] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1181.836802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1181.836802] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1181.836802] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1182.105717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1182.105717] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1182.105717] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1182.374157] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1182.628416] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_last_id [ 1182.884738] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_next_id [ 1183.144015] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1183.144015] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1183.144015] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1183.416629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1183.416629] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1183.416629] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1183.684046] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1183.937693] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1184.337190] LustreError: 23795:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c851:0x0]: have 0 want 1 [ 1184.465929] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1184.984264] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1195.255694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1195.255694] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1195.255694] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1195.526100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1195.526100] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1195.526100] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1195.795651] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1196.055209] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_last_id [ 1196.312449] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_next_id [ 1196.570582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1196.570582] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1196.570582] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1196.840660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1196.840660] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1196.840660] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1197.108068] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1197.362225] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1197.773406] LustreError: 23795:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c853:0x0]: have 0 want 1 [ 1197.903172] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1198.428439] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1208.698493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1208.698493] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1208.698493] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1208.966697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1208.966697] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1208.966697] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1209.235697] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1209.488651] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_last_id [ 1209.742871] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_next_id [ 1210.009997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1210.009997] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1210.009997] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1210.285146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1210.285146] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1210.285146] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1210.554594] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1210.809412] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1211.228209] LustreError: 24776:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c855:0x0]: have 0 want 1 [ 1211.357468] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1211.874786] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1222.467054] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1232.847723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1233.470957] Lustre: DEBUG MARKER: rc=0; [ 1233.470957] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1233.470957] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1233.470957] echo $(hostname -s): $val; [ 1233.470957] rc=$val; [ 1233.470957] fi; [ 1233.470957] exit $rc [ 1234.055401] Lustre: DEBUG MARKER: dmesg [ 1234.447697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27p: append to a truncated file with some full OSTs =================================== 17:59:29 \(1491242369\) [ 1234.603172] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 17:59:29 (1491242369) [ 1235.174540] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1245.462239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1245.462239] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1245.462239] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1245.734110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1245.734110] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1245.734110] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1246.004527] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1246.265428] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1246.523802] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1246.786557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1246.786557] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1246.786557] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1247.055345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1247.055345] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1247.055345] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1247.322882] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1247.576226] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1248.177426] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1248.696937] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1259.321206] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1269.644852] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1270.262914] Lustre: DEBUG MARKER: rc=0; [ 1270.262914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1270.262914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1270.262914] echo $(hostname -s): $val; [ 1270.262914] rc=$val; [ 1270.262914] fi; [ 1270.262914] exit $rc [ 1270.845785] Lustre: DEBUG MARKER: dmesg [ 1271.249979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27q: append to truncated file with all OSTs full \(should error\) ======================= 18:00:06 \(1491242406\) [ 1271.406936] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 18:00:06 (1491242406) [ 1271.935572] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1282.224026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1282.224026] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1282.224026] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1282.498437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1282.498437] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1282.498437] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1282.763969] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1283.020356] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1283.272805] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1283.531398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1283.531398] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1283.531398] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1283.801871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1283.801871] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1283.801871] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1284.071429] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1284.323118] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1284.898687] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1285.411461] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1295.690979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1295.690979] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1295.690979] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1295.965291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1295.965291] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1295.965291] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1296.232628] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1296.489724] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1296.742859] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1297.001928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1297.001928] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1297.001928] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1297.271032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1297.271032] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1297.271032] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1297.541423] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1297.797093] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1298.207544] LustreError: 23796:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c89e:0x0]: have 0 want 1 [ 1298.336399] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1298.857250] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1309.138020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1309.138020] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1309.138020] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1309.408498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1309.408498] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1309.408498] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1309.677388] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1309.931854] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_last_id [ 1310.185487] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_next_id [ 1310.445720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1310.445720] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1310.445720] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1310.721596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1310.721596] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1310.721596] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1310.988057] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1311.240387] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1311.767128] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1312.276660] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1322.552372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1322.552372] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1322.552372] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1322.823881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1322.823881] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1322.823881] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1323.094939] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1323.351573] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_last_id [ 1323.603554] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_next_id [ 1323.863663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1323.863663] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1323.863663] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1324.134115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1324.134115] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1324.134115] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1324.402612] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1324.659232] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1325.192892] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1325.702832] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1335.990386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1335.990386] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1335.990386] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1336.260940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1336.260940] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1336.260940] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1336.527879] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1336.782125] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_last_id [ 1337.035714] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_next_id [ 1337.292217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1337.292217] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1337.292217] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1337.565832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1337.565832] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1337.565832] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1337.850324] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1338.114682] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1338.513747] LustreError: 24776:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1c8a4:0x0]: have 0 want 1 [ 1338.517301] LustreError: 24776:0:(lod_qos.c:1273:lod_alloc_specific()) Skipped 2 previous similar messages [ 1338.644099] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1339.145805] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1349.421877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1349.421877] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1349.421877] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1349.693152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1349.693152] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1349.693152] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1349.963455] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1350.229588] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_last_id [ 1350.482987] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_next_id [ 1350.747171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1350.747171] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1350.747171] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1351.015726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1351.015726] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1351.015726] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1351.285719] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1351.540210] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1352.059919] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1352.567455] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1362.844200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1362.844200] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1362.844200] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1363.115483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1363.115483] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1363.115483] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1363.383327] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1363.637991] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_last_id [ 1363.891707] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_next_id [ 1364.149170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1364.149170] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1364.149170] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1364.425738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1364.425738] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1364.425738] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1364.700790] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1364.954959] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1365.468146] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1365.971391] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1376.565349] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1386.873117] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1387.480131] Lustre: DEBUG MARKER: rc=0; [ 1387.480131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1387.480131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1387.480131] echo $(hostname -s): $val; [ 1387.480131] rc=$val; [ 1387.480131] fi; [ 1387.480131] exit $rc [ 1388.071396] Lustre: DEBUG MARKER: dmesg [ 1388.466919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27r: stripe file with some full OSTs \(shouldn\'t LBUG\) ================================= 18:02:03 \(1491242523\) [ 1388.617702] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 18:02:03 (1491242523) [ 1389.153404] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1399.428776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1399.428776] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1399.428776] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1399.695710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1399.695710] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1399.695710] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1399.964721] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1400.217094] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1400.472857] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1400.731778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1400.731778] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1400.731778] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1401.004909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1401.004909] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1401.004909] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1401.275454] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1401.527299] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1402.103801] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1402.605986] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1413.198628] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1423.517950] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1424.121202] Lustre: DEBUG MARKER: rc=0; [ 1424.121202] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1424.121202] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1424.121202] echo $(hostname -s): $val; [ 1424.121202] rc=$val; [ 1424.121202] fi; [ 1424.121202] exit $rc [ 1424.702709] Lustre: DEBUG MARKER: dmesg [ 1425.097663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27s: lsm_xfersize overflow \(should error\) \(bug 10725\) ================================= 18:02:40 \(1491242560\) [ 1425.256474] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 18:02:40 (1491242560) [ 1425.456579] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1426.058668] Lustre: DEBUG MARKER: rc=0; [ 1426.058668] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1426.058668] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1426.058668] echo $(hostname -s): $val; [ 1426.058668] rc=$val; [ 1426.058668] fi; [ 1426.058668] exit $rc [ 1426.636225] Lustre: DEBUG MARKER: dmesg [ 1427.027161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27t: check that utils parse path correctly ============================================ 18:02:42 \(1491242562\) [ 1427.189836] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 18:02:42 (1491242562) [ 1427.391032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1428.002723] Lustre: DEBUG MARKER: rc=0; [ 1428.002723] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1428.002723] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1428.002723] echo $(hostname -s): $val; [ 1428.002723] rc=$val; [ 1428.002723] fi; [ 1428.002723] exit $rc [ 1428.593813] Lustre: DEBUG MARKER: dmesg [ 1429.023697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27u: skip object creation on OSC w\/o objects ========================================== 18:02:44 \(1491242564\) [ 1429.187717] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 18:02:44 (1491242564) [ 1429.365973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x139 [ 1429.538987] Lustre: *** cfs_fail_loc=139, val=0*** [ 1430.040410] Lustre: *** cfs_fail_loc=139, val=0*** [ 1430.042171] Lustre: Skipped 361 previous similar messages [ 1431.042242] Lustre: *** cfs_fail_loc=139, val=0*** [ 1431.044038] Lustre: Skipped 727 previous similar messages [ 1431.280817] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 1433.771635] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1434.400091] Lustre: DEBUG MARKER: rc=0; [ 1434.400091] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1434.400091] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1434.400091] echo $(hostname -s): $val; [ 1434.400091] rc=$val; [ 1434.400091] fi; [ 1434.400091] exit $rc [ 1434.988696] Lustre: DEBUG MARKER: dmesg [ 1435.386686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27v: skip object creation on slow OST ================================================= 18:02:50 \(1491242570\) [ 1435.542114] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 18:02:50 (1491242570) [ 1435.837317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1435.837317] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1435.837317] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1436.113871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1436.113871] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1436.113871] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1436.384092] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1436.637900] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1436.892809] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1437.151348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1437.151348] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1437.151348] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1437.420997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1437.420997] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1437.420997] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1437.688138] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1437.942538] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1438.526944] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1439.035552] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1449.307850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1449.307850] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1449.307850] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1449.585962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1449.585962] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1449.585962] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1449.850285] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1450.107854] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1450.362293] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1450.620658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1450.620658] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1450.620658] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1450.898134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1450.898134] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1450.898134] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1451.174324] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1451.429185] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1451.820912] LustreError: 23794:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000401:0x1ccd9:0x0]: have 0 want 1 [ 1451.824546] LustreError: 23794:0:(lod_qos.c:1273:lod_alloc_specific()) Skipped 2 previous similar messages [ 1451.951426] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1452.455599] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1462.727023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1462.727023] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1462.727023] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1462.997986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1462.997986] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1462.997986] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1463.267423] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1463.523743] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_last_id [ 1463.779068] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_next_id [ 1464.046215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1464.046215] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1464.046215] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1464.315173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1464.315173] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1464.315173] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1464.584089] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1464.838985] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1465.352370] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1465.866328] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1476.143535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1476.143535] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1476.143535] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1476.414158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1476.414158] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1476.414158] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1476.681814] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1476.938430] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_last_id [ 1477.193108] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_next_id [ 1477.450200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1477.450200] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1477.450200] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1477.722104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1477.722104] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1477.722104] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1477.988959] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1478.240499] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1478.764634] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1479.268724] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1489.544233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1489.544233] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1489.544233] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1489.816388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1489.816388] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1489.816388] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1490.085738] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1490.341444] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_last_id [ 1490.595554] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_next_id [ 1490.854922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1490.854922] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1490.854922] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1491.122546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1491.122546] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1491.122546] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1491.387564] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1491.640968] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1492.151187] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1492.664622] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1502.944268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1502.944268] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1502.944268] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1503.213282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1503.213282] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1503.213282] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1503.480772] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1503.734753] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_last_id [ 1503.990429] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_next_id [ 1504.252451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1504.252451] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1504.252451] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1504.519710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1504.519710] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1504.519710] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1504.787743] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1505.042271] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1505.557321] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1506.058005] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1516.333712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1516.333712] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1516.333712] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1516.605238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1516.605238] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1516.605238] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1516.869728] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1517.122756] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_last_id [ 1517.374342] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_next_id [ 1517.633351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1517.633351] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1517.633351] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1517.902659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1517.902659] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1517.902659] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1518.169027] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1518.423356] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1518.948350] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1519.457934] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1530.041380] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1552.175909] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1562.493786] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1563.099518] Lustre: DEBUG MARKER: rc=0; [ 1563.099518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1563.099518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1563.099518] echo $(hostname -s): $val; [ 1563.099518] rc=$val; [ 1563.099518] fi; [ 1563.099518] exit $rc [ 1563.679458] Lustre: DEBUG MARKER: dmesg [ 1564.101744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27w: check \/usr\/bin\/lfs setstripe -S option =========================================== 18:04:59 \(1491242699\) [ 1564.254315] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option =========================================== 18:04:59 (1491242699) [ 1564.467900] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1565.075227] Lustre: DEBUG MARKER: rc=0; [ 1565.075227] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1565.075227] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1565.075227] echo $(hostname -s): $val; [ 1565.075227] rc=$val; [ 1565.075227] fi; [ 1565.075227] exit $rc [ 1565.657851] Lustre: DEBUG MARKER: dmesg [ 1566.050752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27wa: check \/usr\/bin\/lfs setstripe -c -i options ====================================== 18:05:01 \(1491242701\) [ 1566.208528] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options ====================================== 18:05:01 (1491242701) [ 1566.523918] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1567.139896] Lustre: DEBUG MARKER: rc=0; [ 1567.139896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1567.139896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1567.139896] echo $(hostname -s): $val; [ 1567.139896] rc=$val; [ 1567.139896] fi; [ 1567.139896] exit $rc [ 1567.724908] Lustre: DEBUG MARKER: dmesg [ 1568.124598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27x: create files while OST0 is degraded ============================================== 18:05:03 \(1491242703\) [ 1568.278199] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 18:05:03 (1491242703) [ 1568.711769] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1579.343911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1579.950736] Lustre: DEBUG MARKER: rc=0; [ 1579.950736] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1579.950736] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1579.950736] echo $(hostname -s): $val; [ 1579.950736] rc=$val; [ 1579.950736] fi; [ 1579.950736] exit $rc [ 1580.545281] Lustre: DEBUG MARKER: dmesg [ 1580.950413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 18:05:16 \(1491242716\) [ 1581.111977] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 18:05:16 (1491242716) [ 1581.286225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1581.286225] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1581.286225] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1581.557475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1581.557475] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1581.557475] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1581.823053] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1582.077091] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1582.331953] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1582.586418] Lustre: DEBUG MARKER: lctl dl [ 1582.856004] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 1582.985491] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1583.121036] Lustre: DEBUG MARKER: lctl --device %lustre-OST0002-osc-MDT0000 deactivate [ 1583.381795] Lustre: DEBUG MARKER: lctl --device %lustre-OST0003-osc-MDT0000 deactivate [ 1583.512184] Lustre: setting import lustre-OST0003_UUID INACTIVE by administrator request [ 1583.514299] Lustre: Skipped 1 previous similar message [ 1583.648995] Lustre: DEBUG MARKER: lctl --device %lustre-OST0004-osc-MDT0000 deactivate [ 1583.907176] Lustre: DEBUG MARKER: lctl --device %lustre-OST0005-osc-MDT0000 deactivate [ 1584.165938] Lustre: DEBUG MARKER: lctl --device %lustre-OST0006-osc-MDT0000 deactivate [ 1584.742452] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1595.278618] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 1595.408133] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1595.418943] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1595.422795] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 1595.425028] Lustre: Skipped 27 previous similar messages [ 1595.559073] Lustre: DEBUG MARKER: lctl --device %lustre-OST0002-osc-MDT0000 activate [ 1595.818619] Lustre: DEBUG MARKER: lctl --device %lustre-OST0003-osc-MDT0000 activate [ 1595.946448] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1595.950707] Lustre: Skipped 1 previous similar message [ 1595.957607] LustreError: 167-0: lustre-OST0003-osc-MDT0000: This client was evicted by lustre-OST0003; in progress operations using this service will fail. [ 1595.961472] LustreError: Skipped 1 previous similar message [ 1596.098748] Lustre: DEBUG MARKER: lctl --device %lustre-OST0004-osc-MDT0000 activate [ 1596.357163] Lustre: DEBUG MARKER: lctl --device %lustre-OST0005-osc-MDT0000 activate [ 1596.620018] Lustre: DEBUG MARKER: lctl --device %lustre-OST0006-osc-MDT0000 activate [ 1596.872038] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1607.197984] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1607.804182] Lustre: DEBUG MARKER: rc=0; [ 1607.804182] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1607.804182] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1607.804182] echo $(hostname -s): $val; [ 1607.804182] rc=$val; [ 1607.804182] fi; [ 1607.804182] exit $rc [ 1608.395160] Lustre: DEBUG MARKER: dmesg [ 1608.822757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27z: check SEQ\/OID on the MDT and OST filesystems ===================================== 18:05:43 \(1491242743\) [ 1608.982588] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 18:05:43 (1491242743) [ 1614.631728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-1 [ 1614.790414] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 1614.988529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000401, oid 0x1cd21 ver 0x0 [ 1615.138663] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1cd21 ver 0x0 [ 1615.333334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000401, oid 0x1cd21, count: 1 [ 1615.482744] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1cd21, count: 1 [ 1615.670362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:0 oid:276\/0x114 seq:0 [ 1615.820258] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:276/0x114 seq:0 [ 1617.983462] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1617.985900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1617.990153] Lustre: Skipped 3 previous similar messages [ 1618.798694] Lustre: lustre-MDT0000: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 1618.801208] Lustre: Skipped 5 previous similar messages [ 1619.681145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1619.835127] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1628.133901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-2 [ 1628.285720] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 1628.491913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000401, oid 0x1cd23 ver 0x0 [ 1628.647753] Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x1cd23 ver 0x0 [ 1628.838616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000401, oid 0x1cd23, count: 7 [ 1628.990989] Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x1cd23, count: 7 [ 1629.185444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:6 oid:250\/0xfa seq:0 [ 1629.342317] Lustre: DEBUG MARKER: want: stripe:0 ost:6 oid:250/0xfa seq:0 [ 1631.808018] LustreError: 11-0: lustre-OST0006-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1631.810548] Lustre: lustre-OST0006-osc-MDT0000: Connection to lustre-OST0006 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1633.130498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1633.292176] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1637.430123] Lustre: lustre-OST0006-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 1637.434336] Lustre: Skipped 2 previous similar messages [ 1637.758941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:1 ost:0 oid:277\/0x115 seq:0 [ 1637.916158] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:277/0x115 seq:0 [ 1641.652853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1641.809774] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1642.847463] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1642.850361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1644.020784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:2 ost:1 oid:252\/0xfc seq:0 [ 1644.189767] Lustre: DEBUG MARKER: want: stripe:2 ost:1 oid:252/0xfc seq:0 [ 1645.503589] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1648.024940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1648.184776] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1653.149877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:3 ost:2 oid:250\/0xfa seq:0 [ 1653.301201] Lustre: DEBUG MARKER: want: stripe:3 ost:2 oid:250/0xfa seq:0 [ 1655.791567] LustreError: 11-0: lustre-OST0002-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1655.794580] Lustre: lustre-OST0002-osc-MDT0000: Connection to lustre-OST0002 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1655.799995] Lustre: Skipped 1 previous similar message [ 1659.105744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1659.260121] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1663.145439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:4 ost:3 oid:250\/0xfa seq:0 [ 1663.305648] Lustre: DEBUG MARKER: want: stripe:4 ost:3 oid:250/0xfa seq:0 [ 1666.079525] LustreError: 11-0: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1667.129217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1667.284611] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1671.446709] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 1671.449838] Lustre: Skipped 7 previous similar messages [ 1671.769443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:5 ost:4 oid:250\/0xfa seq:0 [ 1671.936238] Lustre: DEBUG MARKER: want: stripe:5 ost:4 oid:250/0xfa seq:0 [ 1675.765457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1675.924160] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1676.351561] Lustre: lustre-OST0004-osc-MDT0000: Connection to lustre-OST0004 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1676.357256] Lustre: Skipped 1 previous similar message [ 1680.429277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:6 ost:5 oid:250\/0xfa seq:0 [ 1680.583515] Lustre: DEBUG MARKER: want: stripe:6 ost:5 oid:250/0xfa seq:0 [ 1684.401929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1684.565380] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1686.770384] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1687.391198] Lustre: DEBUG MARKER: rc=0; [ 1687.391198] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1687.391198] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1687.391198] echo $(hostname -s): $val; [ 1687.391198] rc=$val; [ 1687.391198] fi; [ 1687.391198] exit $rc [ 1687.992266] Lustre: DEBUG MARKER: dmesg [ 1688.400640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 18:07:03 \(1491242823\) [ 1688.555993] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 18:07:03 (1491242823) [ 1688.575893] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1688.578594] Lustre: Skipped 4 previous similar messages [ 1688.811358] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1689.418932] Lustre: DEBUG MARKER: rc=0; [ 1689.418932] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1689.418932] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1689.418932] echo $(hostname -s): $val; [ 1689.418932] rc=$val; [ 1689.418932] fi; [ 1689.418932] exit $rc [ 1690.015341] Lustre: DEBUG MARKER: dmesg [ 1690.418469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27B: call setstripe on open unlinked file\/rename victim =============================== 18:07:05 \(1491242825\) [ 1690.588546] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 18:07:05 (1491242825) [ 1690.809794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1691.425758] Lustre: DEBUG MARKER: rc=0; [ 1691.425758] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1691.425758] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1691.425758] echo $(hostname -s): $val; [ 1691.425758] rc=$val; [ 1691.425758] fi; [ 1691.425758] exit $rc [ 1692.022530] Lustre: DEBUG MARKER: dmesg [ 1692.426584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27C: check full striping across all OSTs ============================================== 18:07:07 \(1491242827\) [ 1692.584342] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 18:07:07 (1491242827) [ 1692.942105] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1693.574930] Lustre: DEBUG MARKER: rc=0; [ 1693.574930] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1693.574930] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1693.574930] echo $(hostname -s): $val; [ 1693.574930] rc=$val; [ 1693.574930] fi; [ 1693.574930] exit $rc [ 1694.185765] Lustre: DEBUG MARKER: dmesg [ 1694.581448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27D: validate llapi_layout API ======================================================== 18:07:09 \(1491242829\) [ 1694.737987] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 18:07:09 (1491242829) [ 1694.893296] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1697.114630] LustreError: 7230:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 1699.158920] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1699.427491] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1699.734713] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.testpool lustre-OST[0-6/1] [ 1710.023114] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1710.023114] sort -u | tr '\n' ' ' [ 1710.278206] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1710.278206] sort -u | tr '\n' ' ' [ 1710.563156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1710.563156] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1710.563156] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1711.123894] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1711.395821] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1711.666729] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1717.933224] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 1728.194883] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [ 1736.456103] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [ 1744.716222] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [ 1752.981722] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [ 1759.242203] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [ 1767.504127] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1771.769676] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1772.025688] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1773.387690] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1774.002189] Lustre: DEBUG MARKER: rc=0; [ 1774.002189] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1774.002189] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1774.002189] echo $(hostname -s): $val; [ 1774.002189] rc=$val; [ 1774.002189] fi; [ 1774.002189] exit $rc [ 1774.585459] Lustre: DEBUG MARKER: dmesg [ 1774.991672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27E: check that default extended attribute size properly increases ==================== 18:08:30 \(1491242910\) [ 1775.147961] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 18:08:30 (1491242910) [ 1775.386103] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1776.001515] Lustre: DEBUG MARKER: rc=0; [ 1776.001515] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1776.001515] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1776.001515] echo $(hostname -s): $val; [ 1776.001515] rc=$val; [ 1776.001515] fi; [ 1776.001515] exit $rc [ 1776.587699] Lustre: DEBUG MARKER: dmesg [ 1777.002309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 18:08:32 \(1491242912\) [ 1777.156587] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 18:08:32 (1491242912) [ 1777.290302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 1777.290302] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 1777.290302] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 1778.095554] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1778.097714] LustreError: Skipped 2 previous similar messages [ 1778.099377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1778.103128] Lustre: Skipped 1 previous similar message [ 1783.961512] Lustre: MGS: Connection restored to 54b83b0c-69e3-7863-749f-8e4a979818a6 (at 10.9.5.244@tcp) [ 1783.963676] Lustre: Skipped 4 previous similar messages [ 1784.873682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1785.029395] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1787.734350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1787.892795] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1790.589930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1790.746483] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1793.437407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1793.598330] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1796.310016] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1796.464670] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1799.202254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1799.356735] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1802.049169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1802.208889] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1806.269246] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1806.894326] Lustre: DEBUG MARKER: rc=0; [ 1806.894326] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1806.894326] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1806.894326] echo $(hostname -s): $val; [ 1806.894326] rc=$val; [ 1806.894326] fi; [ 1806.894326] exit $rc [ 1807.505110] Lustre: DEBUG MARKER: dmesg [ 1807.908538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 28: create\/mknod\/mkdir with bad file types ============================================ 18:09:03 \(1491242943\) [ 1808.065698] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 18:09:03 (1491242943) [ 1808.396781] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1809.031302] Lustre: DEBUG MARKER: rc=0; [ 1809.031302] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1809.031302] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1809.031302] echo $(hostname -s): $val; [ 1809.031302] rc=$val; [ 1809.031302] fi; [ 1809.031302] exit $rc [ 1809.630377] Lustre: DEBUG MARKER: dmesg [ 1810.024239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 29: IT_GETATTR regression ============================================================ 18:09:05 \(1491242945\) [ 1810.180004] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 18:09:05 (1491242945) [ 1811.452237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first d29 [ 1811.601367] Lustre: DEBUG MARKER: first d29 [ 1811.803332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark second d29 [ 1811.959457] Lustre: DEBUG MARKER: second d29 [ 1812.149321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 1812.304224] Lustre: DEBUG MARKER: done [ 1812.495584] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1813.109645] Lustre: DEBUG MARKER: rc=0; [ 1813.109645] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1813.109645] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1813.109645] echo $(hostname -s): $val; [ 1813.109645] rc=$val; [ 1813.109645] fi; [ 1813.109645] exit $rc [ 1813.689342] Lustre: DEBUG MARKER: dmesg [ 1814.099391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30a: execute binary from Lustre \(execve\) ============================================== 18:09:09 \(1491242949\) [ 1814.261080] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 18:09:09 (1491242949) [ 1814.465220] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1815.074246] Lustre: DEBUG MARKER: rc=0; [ 1815.074246] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1815.074246] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1815.074246] echo $(hostname -s): $val; [ 1815.074246] rc=$val; [ 1815.074246] fi; [ 1815.074246] exit $rc [ 1815.665066] Lustre: DEBUG MARKER: dmesg [ 1816.073220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30b: execute binary from Lustre as non-root =========================================== 18:09:11 \(1491242951\) [ 1816.237999] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 18:09:11 (1491242951) [ 1816.448878] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1817.072655] Lustre: DEBUG MARKER: rc=0; [ 1817.072655] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1817.072655] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1817.072655] echo $(hostname -s): $val; [ 1817.072655] rc=$val; [ 1817.072655] fi; [ 1817.072655] exit $rc [ 1817.659781] Lustre: DEBUG MARKER: dmesg [ 1818.053275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30c: execute binary from Lustre without read perms ==================================== 18:09:13 \(1491242953\) [ 1818.210312] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 18:09:13 (1491242953) [ 1818.481346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1819.091079] Lustre: DEBUG MARKER: rc=0; [ 1819.091079] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1819.091079] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1819.091079] echo $(hostname -s): $val; [ 1819.091079] rc=$val; [ 1819.091079] fi; [ 1819.091079] exit $rc [ 1819.688184] Lustre: DEBUG MARKER: dmesg [ 1820.087745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31a: open-unlink file ================================================================= 18:09:15 \(1491242955\) [ 1820.243498] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 18:09:15 (1491242955) [ 1820.492287] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1821.096854] Lustre: DEBUG MARKER: rc=0; [ 1821.096854] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1821.096854] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1821.096854] echo $(hostname -s): $val; [ 1821.096854] rc=$val; [ 1821.096854] fi; [ 1821.096854] exit $rc [ 1821.689045] Lustre: DEBUG MARKER: dmesg [ 1822.097935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31b: unlink file with multiple links while open ======================================= 18:09:17 \(1491242957\) [ 1822.251608] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 18:09:17 (1491242957) [ 1822.452870] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1823.058951] Lustre: DEBUG MARKER: rc=0; [ 1823.058951] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1823.058951] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1823.058951] echo $(hostname -s): $val; [ 1823.058951] rc=$val; [ 1823.058951] fi; [ 1823.058951] exit $rc [ 1823.640639] Lustre: DEBUG MARKER: dmesg [ 1824.045187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31c: open-unlink file with multiple links ============================================= 18:09:19 \(1491242959\) [ 1824.198834] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 18:09:19 (1491242959) [ 1824.402785] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1825.010540] Lustre: DEBUG MARKER: rc=0; [ 1825.010540] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1825.010540] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1825.010540] echo $(hostname -s): $val; [ 1825.010540] rc=$val; [ 1825.010540] fi; [ 1825.010540] exit $rc [ 1825.610028] Lustre: DEBUG MARKER: dmesg [ 1826.060505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31d: remove of open directory ========================================================= 18:09:21 \(1491242961\) [ 1826.213743] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 18:09:21 (1491242961) [ 1826.415541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1827.030179] Lustre: DEBUG MARKER: rc=0; [ 1827.030179] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1827.030179] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1827.030179] echo $(hostname -s): $val; [ 1827.030179] rc=$val; [ 1827.030179] fi; [ 1827.030179] exit $rc [ 1827.617141] Lustre: DEBUG MARKER: dmesg [ 1828.029760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31e: remove of open non-empty directory =============================================== 18:09:23 \(1491242963\) [ 1828.187036] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 18:09:23 (1491242963) [ 1828.387409] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1829.010332] Lustre: DEBUG MARKER: rc=0; [ 1829.010332] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1829.010332] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1829.010332] echo $(hostname -s): $val; [ 1829.010332] rc=$val; [ 1829.010332] fi; [ 1829.010332] exit $rc [ 1829.602511] Lustre: DEBUG MARKER: dmesg [ 1830.005411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31f: remove of open directory with open-unlink file =================================== 18:09:25 \(1491242965\) [ 1830.171968] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 18:09:25 (1491242965) [ 1836.448370] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1837.059760] Lustre: DEBUG MARKER: rc=0; [ 1837.059760] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1837.059760] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1837.059760] echo $(hostname -s): $val; [ 1837.059760] rc=$val; [ 1837.059760] fi; [ 1837.059760] exit $rc [ 1837.647226] Lustre: DEBUG MARKER: dmesg [ 1838.089266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31g: cross directory link============================================================== 18:09:33 \(1491242973\) [ 1838.249380] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 18:09:33 (1491242973) [ 1838.466282] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1839.074781] Lustre: DEBUG MARKER: rc=0; [ 1839.074781] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1839.074781] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1839.074781] echo $(hostname -s): $val; [ 1839.074781] rc=$val; [ 1839.074781] fi; [ 1839.074781] exit $rc [ 1839.656813] Lustre: DEBUG MARKER: dmesg [ 1840.065900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31h: cross directory link under child================================================== 18:09:35 \(1491242975\) [ 1840.229680] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 18:09:35 (1491242975) [ 1840.437371] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1841.058344] Lustre: DEBUG MARKER: rc=0; [ 1841.058344] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1841.058344] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1841.058344] echo $(hostname -s): $val; [ 1841.058344] rc=$val; [ 1841.058344] fi; [ 1841.058344] exit $rc [ 1841.647429] Lustre: DEBUG MARKER: dmesg [ 1842.050744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31i: cross directory link under parent================================================= 18:09:37 \(1491242977\) [ 1842.205551] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 18:09:37 (1491242977) [ 1842.417259] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1843.033408] Lustre: DEBUG MARKER: rc=0; [ 1843.033408] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1843.033408] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1843.033408] echo $(hostname -s): $val; [ 1843.033408] rc=$val; [ 1843.033408] fi; [ 1843.033408] exit $rc [ 1843.630241] Lustre: DEBUG MARKER: dmesg [ 1844.045254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31j: link for directory================================================================ 18:09:39 \(1491242979\) [ 1844.200287] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 18:09:39 (1491242979) [ 1844.413530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1845.045459] Lustre: DEBUG MARKER: rc=0; [ 1845.045459] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1845.045459] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1845.045459] echo $(hostname -s): $val; [ 1845.045459] rc=$val; [ 1845.045459] fi; [ 1845.045459] exit $rc [ 1845.636680] Lustre: DEBUG MARKER: dmesg [ 1846.093981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31k: link to file: the same, non-existing, dir========================================= 18:09:41 \(1491242981\) [ 1846.250428] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 18:09:41 (1491242981) [ 1846.469108] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1847.073281] Lustre: DEBUG MARKER: rc=0; [ 1847.073281] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1847.073281] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1847.073281] echo $(hostname -s): $val; [ 1847.073281] rc=$val; [ 1847.073281] fi; [ 1847.073281] exit $rc [ 1847.650617] Lustre: DEBUG MARKER: dmesg [ 1848.059666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31m: link to file: the same, non-existing, dir========================================= 18:09:43 \(1491242983\) [ 1848.219077] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 18:09:43 (1491242983) [ 1848.442700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1849.048237] Lustre: DEBUG MARKER: rc=0; [ 1849.048237] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1849.048237] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1849.048237] echo $(hostname -s): $val; [ 1849.048237] rc=$val; [ 1849.048237] fi; [ 1849.048237] exit $rc [ 1849.629916] Lustre: DEBUG MARKER: dmesg [ 1850.044034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31n: check link count of unlinked file ================================================ 18:09:45 \(1491242985\) [ 1850.200087] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 18:09:45 (1491242985) [ 1850.409082] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1851.008970] Lustre: DEBUG MARKER: rc=0; [ 1851.008970] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1851.008970] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1851.008970] echo $(hostname -s): $val; [ 1851.008970] rc=$val; [ 1851.008970] fi; [ 1851.008970] exit $rc [ 1851.601911] Lustre: DEBUG MARKER: dmesg [ 1852.030200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31o: duplicate hard links with same filename ========================================== 18:09:47 \(1491242987\) [ 1852.181585] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 18:09:47 (1491242987) [ 1858.179049] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1858.804345] Lustre: DEBUG MARKER: rc=0; [ 1858.804345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1858.804345] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1858.804345] echo $(hostname -s): $val; [ 1858.804345] rc=$val; [ 1858.804345] fi; [ 1858.804345] exit $rc [ 1859.400139] Lustre: DEBUG MARKER: dmesg [ 1859.819343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31p: remove of open striped directory ================================================= 18:09:54 \(1491242994\) [ 1859.989675] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 18:09:54 (1491242994) [ 1860.188725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_31p needs \>= 2 MDTs [ 1860.345552] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 1860.540972] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1861.155957] Lustre: DEBUG MARKER: rc=0; [ 1861.155957] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1861.155957] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1861.155957] echo $(hostname -s): $val; [ 1861.155957] rc=$val; [ 1861.155957] fi; [ 1861.155957] exit $rc [ 1861.755422] Lustre: DEBUG MARKER: dmesg [ 1862.192666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32a: stat d32a\/ext2-mountpoint\/.. ===================================================== 18:09:57 \(1491242997\) [ 1862.347587] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 18:09:57 (1491242997) [ 1862.638726] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1863.254790] Lustre: DEBUG MARKER: rc=0; [ 1863.254790] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1863.254790] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1863.254790] echo $(hostname -s): $val; [ 1863.254790] rc=$val; [ 1863.254790] fi; [ 1863.254790] exit $rc [ 1863.841896] Lustre: DEBUG MARKER: dmesg [ 1864.251407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32b: open d32b\/ext2-mountpoint\/.. ===================================================== 18:09:59 \(1491242999\) [ 1864.414807] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 18:09:59 (1491242999) [ 1864.732476] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1865.363924] Lustre: DEBUG MARKER: rc=0; [ 1865.363924] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1865.363924] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1865.363924] echo $(hostname -s): $val; [ 1865.363924] rc=$val; [ 1865.363924] fi; [ 1865.363924] exit $rc [ 1865.963829] Lustre: DEBUG MARKER: dmesg [ 1866.406649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32c: stat d32c\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 18:10:01 \(1491243001\) [ 1866.559896] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 18:10:01 (1491243001) [ 1866.831092] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1867.465904] Lustre: DEBUG MARKER: rc=0; [ 1867.465904] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1867.465904] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1867.465904] echo $(hostname -s): $val; [ 1867.465904] rc=$val; [ 1867.465904] fi; [ 1867.465904] exit $rc [ 1868.048838] Lustre: DEBUG MARKER: dmesg [ 1868.462347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32d: open d32d\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 18:10:03 \(1491243003\) [ 1868.622330] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 18:10:03 (1491243003) [ 1868.870069] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1869.482914] Lustre: DEBUG MARKER: rc=0; [ 1869.482914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1869.482914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1869.482914] echo $(hostname -s): $val; [ 1869.482914] rc=$val; [ 1869.482914] fi; [ 1869.482914] exit $rc [ 1870.081910] Lustre: DEBUG MARKER: dmesg [ 1870.500117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32e: stat d32e\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 18:10:05 \(1491243005\) [ 1870.653547] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 18:10:05 (1491243005) [ 1870.867153] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1871.480884] Lustre: DEBUG MARKER: rc=0; [ 1871.480884] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1871.480884] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1871.480884] echo $(hostname -s): $val; [ 1871.480884] rc=$val; [ 1871.480884] fi; [ 1871.480884] exit $rc [ 1872.066824] Lustre: DEBUG MARKER: dmesg [ 1872.497961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32f: open d32f\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 18:10:07 \(1491243007\) [ 1872.662380] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 18:10:07 (1491243007) [ 1872.877056] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1873.486669] Lustre: DEBUG MARKER: rc=0; [ 1873.486669] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1873.486669] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1873.486669] echo $(hostname -s): $val; [ 1873.486669] rc=$val; [ 1873.486669] fi; [ 1873.486669] exit $rc [ 1874.085808] Lustre: DEBUG MARKER: dmesg [ 1874.492019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32g: stat d32g\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 18:10:09 \(1491243009\) [ 1874.649816] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 18:10:09 (1491243009) [ 1874.872550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1875.486387] Lustre: DEBUG MARKER: rc=0; [ 1875.486387] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1875.486387] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1875.486387] echo $(hostname -s): $val; [ 1875.486387] rc=$val; [ 1875.486387] fi; [ 1875.486387] exit $rc [ 1876.084720] Lustre: DEBUG MARKER: dmesg [ 1876.495505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32h: open d32h\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 18:10:11 \(1491243011\) [ 1876.657286] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 18:10:11 (1491243011) [ 1876.876704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1877.488064] Lustre: DEBUG MARKER: rc=0; [ 1877.488064] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1877.488064] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1877.488064] echo $(hostname -s): $val; [ 1877.488064] rc=$val; [ 1877.488064] fi; [ 1877.488064] exit $rc [ 1878.071385] Lustre: DEBUG MARKER: dmesg [ 1878.490756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32i: stat d32i\/ext2-mountpoint\/..\/test_file =========================================== 18:10:13 \(1491243013\) [ 1878.651167] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 18:10:13 (1491243013) [ 1878.884886] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1879.506972] Lustre: DEBUG MARKER: rc=0; [ 1879.506972] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1879.506972] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1879.506972] echo $(hostname -s): $val; [ 1879.506972] rc=$val; [ 1879.506972] fi; [ 1879.506972] exit $rc [ 1880.096849] Lustre: DEBUG MARKER: dmesg [ 1880.507565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32j: open d32j\/ext2-mountpoint\/..\/test_file =========================================== 18:10:15 \(1491243015\) [ 1880.663559] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 18:10:15 (1491243015) [ 1880.922455] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1881.529951] Lustre: DEBUG MARKER: rc=0; [ 1881.529951] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1881.529951] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1881.529951] echo $(hostname -s): $val; [ 1881.529951] rc=$val; [ 1881.529951] fi; [ 1881.529951] exit $rc [ 1882.118415] Lustre: DEBUG MARKER: dmesg [ 1882.535277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32k: stat d32k\/ext2-mountpoint\/..\/d2\/test_file ======================================== 18:10:17 \(1491243017\) [ 1882.687693] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 18:10:17 (1491243017) [ 1882.947005] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1883.565521] Lustre: DEBUG MARKER: rc=0; [ 1883.565521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1883.565521] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1883.565521] echo $(hostname -s): $val; [ 1883.565521] rc=$val; [ 1883.565521] fi; [ 1883.565521] exit $rc [ 1884.159969] Lustre: DEBUG MARKER: dmesg [ 1884.576519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32l: open d32l\/ext2-mountpoint\/..\/d2\/test_file ======================================== 18:10:19 \(1491243019\) [ 1884.736140] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 18:10:19 (1491243019) [ 1884.993202] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1885.605433] Lustre: DEBUG MARKER: rc=0; [ 1885.605433] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1885.605433] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1885.605433] echo $(hostname -s): $val; [ 1885.605433] rc=$val; [ 1885.605433] fi; [ 1885.605433] exit $rc [ 1886.203200] Lustre: DEBUG MARKER: dmesg [ 1886.632457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32m: stat d32m\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 18:10:21 \(1491243021\) [ 1886.791737] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 18:10:21 (1491243021) [ 1886.997434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1887.623415] Lustre: DEBUG MARKER: rc=0; [ 1887.623415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1887.623415] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1887.623415] echo $(hostname -s): $val; [ 1887.623415] rc=$val; [ 1887.623415] fi; [ 1887.623415] exit $rc [ 1888.227638] Lustre: DEBUG MARKER: dmesg [ 1888.639985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32n: open d32n\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 18:10:23 \(1491243023\) [ 1888.797055] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 18:10:23 (1491243023) [ 1889.006885] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1889.619087] Lustre: DEBUG MARKER: rc=0; [ 1889.619087] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1889.619087] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1889.619087] echo $(hostname -s): $val; [ 1889.619087] rc=$val; [ 1889.619087] fi; [ 1889.619087] exit $rc [ 1890.215180] Lustre: DEBUG MARKER: dmesg [ 1890.624147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32o: stat d32o\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 18:10:25 \(1491243025\) [ 1890.785878] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 18:10:25 (1491243025) [ 1890.999360] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1891.621847] Lustre: DEBUG MARKER: rc=0; [ 1891.621847] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1891.621847] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1891.621847] echo $(hostname -s): $val; [ 1891.621847] rc=$val; [ 1891.621847] fi; [ 1891.621847] exit $rc [ 1892.211710] Lustre: DEBUG MARKER: dmesg [ 1892.630843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32p: open d32p\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 18:10:27 \(1491243027\) [ 1892.800973] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 18:10:27 (1491243027) [ 1892.995792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_1 [ 1893.153087] Lustre: DEBUG MARKER: 32p_1 [ 1893.343335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_2 [ 1893.497866] Lustre: DEBUG MARKER: 32p_2 [ 1893.696551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_3 [ 1893.855095] Lustre: DEBUG MARKER: 32p_3 [ 1894.067998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_4 [ 1894.228899] Lustre: DEBUG MARKER: 32p_4 [ 1894.434280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_5 [ 1894.594241] Lustre: DEBUG MARKER: 32p_5 [ 1894.788434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_6 [ 1894.950181] Lustre: DEBUG MARKER: 32p_6 [ 1895.157079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_7 [ 1895.315638] Lustre: DEBUG MARKER: 32p_7 [ 1895.522466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_8 [ 1895.676433] Lustre: DEBUG MARKER: 32p_8 [ 1895.878919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_9 [ 1896.036152] Lustre: DEBUG MARKER: 32p_9 [ 1896.252393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_10 [ 1896.406291] Lustre: DEBUG MARKER: 32p_10 [ 1896.598581] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1897.214613] Lustre: DEBUG MARKER: rc=0; [ 1897.214613] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1897.214613] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1897.214613] echo $(hostname -s): $val; [ 1897.214613] rc=$val; [ 1897.214613] fi; [ 1897.214613] exit $rc [ 1897.811090] Lustre: DEBUG MARKER: dmesg [ 1898.239917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32q: stat follows mountpoints in Lustre \(should return error\) ========================= 18:10:33 \(1491243033\) [ 1898.394376] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 18:10:33 (1491243033) [ 1898.659960] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1899.282704] Lustre: DEBUG MARKER: rc=0; [ 1899.282704] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1899.282704] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1899.282704] echo $(hostname -s): $val; [ 1899.282704] rc=$val; [ 1899.282704] fi; [ 1899.282704] exit $rc [ 1899.884774] Lustre: DEBUG MARKER: dmesg [ 1900.305910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32r: opendir follows mountpoints in Lustre \(should return error\) ====================== 18:10:35 \(1491243035\) [ 1900.465997] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 18:10:35 (1491243035) [ 1900.716156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1901.336269] Lustre: DEBUG MARKER: rc=0; [ 1901.336269] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1901.336269] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1901.336269] echo $(hostname -s): $val; [ 1901.336269] rc=$val; [ 1901.336269] fi; [ 1901.336269] exit $rc [ 1901.941192] Lustre: DEBUG MARKER: dmesg [ 1902.360611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33aa: write file with mode 444 \(should return error\) ================================== 18:10:37 \(1491243037\) [ 1902.515637] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 18:10:37 (1491243037) [ 1902.739293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_1 [ 1902.903462] Lustre: DEBUG MARKER: 33_1 [ 1903.107580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_2 [ 1903.264180] Lustre: DEBUG MARKER: 33_2 [ 1903.477956] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1904.083445] Lustre: DEBUG MARKER: rc=0; [ 1904.083445] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1904.083445] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1904.083445] echo $(hostname -s): $val; [ 1904.083445] rc=$val; [ 1904.083445] fi; [ 1904.083445] exit $rc [ 1904.679317] Lustre: DEBUG MARKER: dmesg [ 1905.098124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33a: test open file\(mode=0444\) with O_RDWR \(should return error\) ====================== 18:10:40 \(1491243040\) [ 1905.259103] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 18:10:40 (1491243040) [ 1905.470585] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1906.090181] Lustre: DEBUG MARKER: rc=0; [ 1906.090181] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1906.090181] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1906.090181] echo $(hostname -s): $val; [ 1906.090181] rc=$val; [ 1906.090181] fi; [ 1906.090181] exit $rc [ 1906.685345] Lustre: DEBUG MARKER: dmesg [ 1907.106436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33b: test open file with malformed flags \(No panic\) =================================== 18:10:42 \(1491243042\) [ 1907.267081] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 18:10:42 (1491243042) [ 1907.488494] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1908.108405] Lustre: DEBUG MARKER: rc=0; [ 1908.108405] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1908.108405] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1908.108405] echo $(hostname -s): $val; [ 1908.108405] rc=$val; [ 1908.108405] fi; [ 1908.108405] exit $rc [ 1908.718429] Lustre: DEBUG MARKER: dmesg [ 1909.137580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33c: test llobdstat and write_bytes =================================================== 18:10:44 \(1491243044\) [ 1909.298732] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 18:10:44 (1491243044) [ 1910.090988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1910.702733] Lustre: DEBUG MARKER: rc=0; [ 1910.702733] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1910.702733] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1910.702733] echo $(hostname -s): $val; [ 1910.702733] rc=$val; [ 1910.702733] fi; [ 1910.702733] exit $rc [ 1911.283690] Lustre: DEBUG MARKER: dmesg [ 1911.697679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 18:10:46 \(1491243046\) [ 1911.867815] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 18:10:46 (1491243046) [ 1912.064768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33d needs \>= 2 MDTs [ 1912.228519] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 1912.431757] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1913.060400] Lustre: DEBUG MARKER: rc=0; [ 1913.060400] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1913.060400] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1913.060400] echo $(hostname -s): $val; [ 1913.060400] rc=$val; [ 1913.060400] fi; [ 1913.060400] exit $rc [ 1913.662749] Lustre: DEBUG MARKER: dmesg [ 1914.082316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33e: mkdir and striped directory should have same mode ================================ 18:10:49 \(1491243049\) [ 1914.242213] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 18:10:49 (1491243049) [ 1914.429928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33e needs \>= 2 MDTs [ 1914.579160] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 1914.769189] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1915.380036] Lustre: DEBUG MARKER: rc=0; [ 1915.380036] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1915.380036] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1915.380036] echo $(hostname -s): $val; [ 1915.380036] rc=$val; [ 1915.380036] fi; [ 1915.380036] exit $rc [ 1915.960166] Lustre: DEBUG MARKER: dmesg [ 1916.376645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 18:10:51 \(1491243051\) [ 1916.539463] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 18:10:51 (1491243051) [ 1916.737401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33f needs \>= 2 MDTs [ 1916.891181] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 1917.088800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1917.713221] Lustre: DEBUG MARKER: rc=0; [ 1917.713221] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1917.713221] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1917.713221] echo $(hostname -s): $val; [ 1917.713221] rc=$val; [ 1917.713221] fi; [ 1917.713221] exit $rc [ 1918.311989] Lustre: DEBUG MARKER: dmesg [ 1918.818314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33g: nonroot user create already existing root created file =========================== 18:10:53 \(1491243053\) [ 1918.998395] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 18:10:53 (1491243053) [ 1919.202522] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1919.838505] Lustre: DEBUG MARKER: rc=0; [ 1919.838505] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1919.838505] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1919.838505] echo $(hostname -s): $val; [ 1919.838505] rc=$val; [ 1919.838505] fi; [ 1919.838505] exit $rc [ 1920.454928] Lustre: DEBUG MARKER: dmesg [ 1920.912515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34a: truncate file that has not been opened =========================================== 18:10:56 \(1491243056\) [ 1921.076460] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 18:10:56 (1491243056) [ 1921.295801] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1921.930778] Lustre: DEBUG MARKER: rc=0; [ 1921.930778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1921.930778] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1921.930778] echo $(hostname -s): $val; [ 1921.930778] rc=$val; [ 1921.930778] fi; [ 1921.930778] exit $rc [ 1922.535081] Lustre: DEBUG MARKER: dmesg [ 1922.940609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34b: O_RDONLY opening file doesn\'t create objects ===================================== 18:10:58 \(1491243058\) [ 1923.104801] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 18:10:58 (1491243058) [ 1923.310567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1923.927957] Lustre: DEBUG MARKER: rc=0; [ 1923.927957] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1923.927957] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1923.927957] echo $(hostname -s): $val; [ 1923.927957] rc=$val; [ 1923.927957] fi; [ 1923.927957] exit $rc [ 1924.515972] Lustre: DEBUG MARKER: dmesg [ 1924.930528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34c: O_RDWR opening file-with-size works ============================================== 18:11:00 \(1491243060\) [ 1925.083129] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 18:11:00 (1491243060) [ 1925.285139] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1925.898227] Lustre: DEBUG MARKER: rc=0; [ 1925.898227] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1925.898227] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1925.898227] echo $(hostname -s): $val; [ 1925.898227] rc=$val; [ 1925.898227] fi; [ 1925.898227] exit $rc [ 1926.485285] Lustre: DEBUG MARKER: dmesg [ 1926.907647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34d: write to sparse file ============================================================= 18:11:02 \(1491243062\) [ 1927.063875] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 18:11:02 (1491243062) [ 1927.256287] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1927.864901] Lustre: DEBUG MARKER: rc=0; [ 1927.864901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1927.864901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1927.864901] echo $(hostname -s): $val; [ 1927.864901] rc=$val; [ 1927.864901] fi; [ 1927.864901] exit $rc [ 1928.456296] Lustre: DEBUG MARKER: dmesg [ 1928.869732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34e: create objects, some with size and some without ================================== 18:11:03 \(1491243063\) [ 1929.027052] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 18:11:03 (1491243063) [ 1929.237283] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1929.845395] Lustre: DEBUG MARKER: rc=0; [ 1929.845395] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1929.845395] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1929.845395] echo $(hostname -s): $val; [ 1929.845395] rc=$val; [ 1929.845395] fi; [ 1929.845395] exit $rc [ 1930.437243] Lustre: DEBUG MARKER: dmesg [ 1930.863393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34f: read from a file with no objects until EOF ======================================= 18:11:05 \(1491243065\) [ 1931.015938] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 18:11:05 (1491243065) [ 1931.232345] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1931.855729] Lustre: DEBUG MARKER: rc=0; [ 1931.855729] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1931.855729] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1931.855729] echo $(hostname -s): $val; [ 1931.855729] rc=$val; [ 1931.855729] fi; [ 1931.855729] exit $rc [ 1932.441026] Lustre: DEBUG MARKER: dmesg [ 1932.856754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34g: truncate long file =============================================================== 18:11:07 \(1491243067\) [ 1933.014644] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 18:11:07 (1491243067) [ 1933.271708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1933.887506] Lustre: DEBUG MARKER: rc=0; [ 1933.887506] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1933.887506] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1933.887506] echo $(hostname -s): $val; [ 1933.887506] rc=$val; [ 1933.887506] fi; [ 1933.887506] exit $rc [ 1934.472063] Lustre: DEBUG MARKER: dmesg [ 1934.893361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34h: ftruncate file under grouplock should not block ================================== 18:11:10 \(1491243070\) [ 1935.056469] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 18:11:10 (1491243070) [ 1937.665144] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1938.287540] Lustre: DEBUG MARKER: rc=0; [ 1938.287540] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1938.287540] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1938.287540] echo $(hostname -s): $val; [ 1938.287540] rc=$val; [ 1938.287540] fi; [ 1938.287540] exit $rc [ 1938.880052] Lustre: DEBUG MARKER: dmesg [ 1939.301371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 35a: exec file with mode 444 \(should return and not leak\) ============================= 18:11:14 \(1491243074\) [ 1939.454926] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 18:11:14 (1491243074) [ 1939.680109] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1940.295076] Lustre: DEBUG MARKER: rc=0; [ 1940.295076] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1940.295076] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1940.295076] echo $(hostname -s): $val; [ 1940.295076] rc=$val; [ 1940.295076] fi; [ 1940.295076] exit $rc [ 1940.914892] Lustre: DEBUG MARKER: dmesg [ 1941.333246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) =================================================== 18:11:16 \(1491243076\) [ 1941.495293] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 18:11:16 (1491243076) [ 1941.699318] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1942.313900] Lustre: DEBUG MARKER: rc=0; [ 1942.313900] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1942.313900] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1942.313900] echo $(hostname -s): $val; [ 1942.313900] rc=$val; [ 1942.313900] fi; [ 1942.313900] exit $rc [ 1942.912508] Lustre: DEBUG MARKER: dmesg [ 1943.324987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ==================================================== 18:11:18 \(1491243078\) [ 1943.479933] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 18:11:18 (1491243078) [ 1943.676145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1944.291110] Lustre: DEBUG MARKER: rc=0; [ 1944.291110] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1944.291110] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1944.291110] echo $(hostname -s): $val; [ 1944.291110] rc=$val; [ 1944.291110] fi; [ 1944.291110] exit $rc [ 1944.881855] Lustre: DEBUG MARKER: dmesg [ 1945.303362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================== 18:11:20 \(1491243080\) [ 1945.463569] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 18:11:20 (1491243080) [ 1945.686847] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1946.293220] Lustre: DEBUG MARKER: rc=0; [ 1946.293220] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1946.293220] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1946.293220] echo $(hostname -s): $val; [ 1946.293220] rc=$val; [ 1946.293220] fi; [ 1946.293220] exit $rc [ 1946.887124] Lustre: DEBUG MARKER: dmesg [ 1947.299799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) =========================================== 18:11:22 \(1491243082\) [ 1947.462422] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 18:11:22 (1491243082) [ 1947.667993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1948.304886] Lustre: DEBUG MARKER: rc=0; [ 1948.304886] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1948.304886] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1948.304886] echo $(hostname -s): $val; [ 1948.304886] rc=$val; [ 1948.304886] fi; [ 1948.304886] exit $rc [ 1948.897749] Lustre: DEBUG MARKER: dmesg [ 1949.310405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ==================================== 18:11:24 \(1491243084\) [ 1949.468650] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 18:11:24 (1491243084) [ 1949.674496] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1950.291367] Lustre: DEBUG MARKER: rc=0; [ 1950.291367] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1950.291367] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1950.291367] echo $(hostname -s): $val; [ 1950.291367] rc=$val; [ 1950.291367] fi; [ 1950.291367] exit $rc [ 1950.885311] Lustre: DEBUG MARKER: dmesg [ 1951.303581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ========================================== 18:11:26 \(1491243086\) [ 1951.463839] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 18:11:26 (1491243086) [ 1952.698422] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1953.317396] Lustre: DEBUG MARKER: rc=0; [ 1953.317396] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1953.317396] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1953.317396] echo $(hostname -s): $val; [ 1953.317396] rc=$val; [ 1953.317396] fi; [ 1953.317396] exit $rc [ 1953.913751] Lustre: DEBUG MARKER: dmesg [ 1954.338889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36g: filter mod data cache expiry ===================================================== 18:11:29 \(1491243089\) [ 1954.494919] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 18:11:29 (1491243089) [ 1997.485779] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1998.089310] Lustre: DEBUG MARKER: rc=0; [ 1998.089310] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1998.089310] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1998.089310] echo $(hostname -s): $val; [ 1998.089310] rc=$val; [ 1998.089310] fi; [ 1998.089310] exit $rc [ 1998.684140] Lustre: DEBUG MARKER: dmesg [ 1999.093588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ========================================== 18:12:14 \(1491243134\) [ 1999.248431] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 18:12:14 (1491243134) [ 2000.475114] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2001.081664] Lustre: DEBUG MARKER: rc=0; [ 2001.081664] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2001.081664] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2001.081664] echo $(hostname -s): $val; [ 2001.081664] rc=$val; [ 2001.081664] fi; [ 2001.081664] exit $rc [ 2001.693319] Lustre: DEBUG MARKER: dmesg [ 2002.149904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36i: change mtime on striped directory ================================================ 18:12:17 \(1491243137\) [ 2002.328478] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 18:12:17 (1491243137) [ 2002.557724] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_36i needs \>= 2 MDTs [ 2002.722419] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 2002.916353] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2003.533695] Lustre: DEBUG MARKER: rc=0; [ 2003.533695] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2003.533695] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2003.533695] echo $(hostname -s): $val; [ 2003.533695] rc=$val; [ 2003.533695] fi; [ 2003.533695] exit $rc [ 2004.117494] Lustre: DEBUG MARKER: dmesg [ 2004.541527] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 18:12:19 \(1491243139\) [ 2004.702439] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 18:12:19 (1491243139) [ 2004.898288] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2005.512716] Lustre: DEBUG MARKER: rc=0; [ 2005.512716] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2005.512716] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2005.512716] echo $(hostname -s): $val; [ 2005.512716] rc=$val; [ 2005.512716] fi; [ 2005.512716] exit $rc [ 2006.096307] Lustre: DEBUG MARKER: dmesg [ 2006.523675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39a: mtime changed on create ========================================================== 18:12:21 \(1491243141\) [ 2006.683886] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ========================================================== 18:12:21 (1491243141) [ 2008.897321] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2009.525715] Lustre: DEBUG MARKER: rc=0; [ 2009.525715] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2009.525715] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2009.525715] echo $(hostname -s): $val; [ 2009.525715] rc=$val; [ 2009.525715] fi; [ 2009.525715] exit $rc [ 2010.129655] Lustre: DEBUG MARKER: dmesg [ 2010.552796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 18:12:25 \(1491243145\) [ 2010.716758] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 18:12:25 (1491243145) [ 2012.048050] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2012.655307] Lustre: DEBUG MARKER: rc=0; [ 2012.655307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2012.655307] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2012.655307] echo $(hostname -s): $val; [ 2012.655307] rc=$val; [ 2012.655307] fi; [ 2012.655307] exit $rc [ 2013.248504] Lustre: DEBUG MARKER: dmesg [ 2013.681786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39c: mtime change on rename =========================================================== 18:12:28 \(1491243148\) [ 2013.851095] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 18:12:28 (1491243148) [ 2016.114122] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2016.735593] Lustre: DEBUG MARKER: rc=0; [ 2016.735593] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2016.735593] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2016.735593] echo $(hostname -s): $val; [ 2016.735593] rc=$val; [ 2016.735593] fi; [ 2016.735593] exit $rc [ 2017.333444] Lustre: DEBUG MARKER: dmesg [ 2017.742662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39d: create, utime, stat ============================================================== 18:12:32 \(1491243152\) [ 2017.899532] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 18:12:32 (1491243152) [ 2018.115595] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2018.742701] Lustre: DEBUG MARKER: rc=0; [ 2018.742701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2018.742701] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2018.742701] echo $(hostname -s): $val; [ 2018.742701] rc=$val; [ 2018.742701] fi; [ 2018.742701] exit $rc [ 2019.351677] Lustre: DEBUG MARKER: dmesg [ 2019.761655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ======================================================== 18:12:34 \(1491243154\) [ 2019.919820] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 18:12:34 (1491243154) [ 2020.141706] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2020.754447] Lustre: DEBUG MARKER: rc=0; [ 2020.754447] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2020.754447] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2020.754447] echo $(hostname -s): $val; [ 2020.754447] rc=$val; [ 2020.754447] fi; [ 2020.754447] exit $rc [ 2021.354364] Lustre: DEBUG MARKER: dmesg [ 2021.785020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat ================================================= 18:12:36 \(1491243156\) [ 2021.948277] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 18:12:36 (1491243156) [ 2024.161065] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2024.798218] Lustre: DEBUG MARKER: rc=0; [ 2024.798218] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2024.798218] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2024.798218] echo $(hostname -s): $val; [ 2024.798218] rc=$val; [ 2024.798218] fi; [ 2024.798218] exit $rc [ 2025.416840] Lustre: DEBUG MARKER: dmesg [ 2025.858182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39g: write, chmod, stat =============================================================== 18:12:40 \(1491243160\) [ 2026.031755] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 18:12:40 (1491243160) [ 2028.244495] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2028.858159] Lustre: DEBUG MARKER: rc=0; [ 2028.858159] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2028.858159] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2028.858159] echo $(hostname -s): $val; [ 2028.858159] rc=$val; [ 2028.858159] fi; [ 2028.858159] exit $rc [ 2029.464136] Lustre: DEBUG MARKER: dmesg [ 2029.923996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ============================================= 18:12:45 \(1491243165\) [ 2030.089930] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 18:12:45 (1491243165) [ 2031.324178] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2031.937701] Lustre: DEBUG MARKER: rc=0; [ 2031.937701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2031.937701] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2031.937701] echo $(hostname -s): $val; [ 2031.937701] rc=$val; [ 2031.937701] fi; [ 2031.937701] exit $rc [ 2032.542534] Lustre: DEBUG MARKER: dmesg [ 2032.959738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39i: write, rename, stat ============================================================== 18:12:48 \(1491243168\) [ 2033.111799] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 18:12:48 (1491243168) [ 2034.351780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2034.966402] Lustre: DEBUG MARKER: rc=0; [ 2034.966402] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2034.966402] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2034.966402] echo $(hostname -s): $val; [ 2034.966402] rc=$val; [ 2034.966402] fi; [ 2034.966402] exit $rc [ 2035.557159] Lustre: DEBUG MARKER: dmesg [ 2035.988406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ======================================================= 18:12:51 \(1491243171\) [ 2036.149160] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 18:12:51 (1491243171) [ 2036.288467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2036.611536] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 2036.959482] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 2039.452847] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 2040.050202] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2040.389351] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2041.005590] Lustre: DEBUG MARKER: rc=0; [ 2041.005590] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2041.005590] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2041.005590] echo $(hostname -s): $val; [ 2041.005590] rc=$val; [ 2041.005590] fi; [ 2041.005590] exit $rc [ 2041.593332] Lustre: DEBUG MARKER: dmesg [ 2042.026543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ======================================================== 18:12:57 \(1491243177\) [ 2042.182945] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 18:12:57 (1491243177) [ 2043.538130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2044.153951] Lustre: DEBUG MARKER: rc=0; [ 2044.153951] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2044.153951] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2044.153951] echo $(hostname -s): $val; [ 2044.153951] rc=$val; [ 2044.153951] fi; [ 2044.153951] exit $rc [ 2044.758856] Lustre: DEBUG MARKER: dmesg [ 2045.189870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39l: directory atime update =========================================================== 18:13:00 \(1491243180\) [ 2045.355754] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 18:13:00 (1491243180) [ 2045.506131] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 2045.783345] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=2 [ 2049.060211] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 2052.334485] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 2052.652881] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2053.282191] Lustre: DEBUG MARKER: rc=0; [ 2053.282191] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2053.282191] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2053.282191] echo $(hostname -s): $val; [ 2053.282191] rc=$val; [ 2053.282191] fi; [ 2053.282191] exit $rc [ 2053.878148] Lustre: DEBUG MARKER: dmesg [ 2054.312891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ================================================= 18:13:09 \(1491243189\) [ 2054.467789] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 18:13:09 (1491243189) [ 2056.708372] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2057.334865] Lustre: DEBUG MARKER: rc=0; [ 2057.334865] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2057.334865] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2057.334865] echo $(hostname -s): $val; [ 2057.334865] rc=$val; [ 2057.334865] fi; [ 2057.334865] exit $rc [ 2057.926816] Lustre: DEBUG MARKER: dmesg [ 2058.349456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39n: check that O_NOATIME is honored ================================================== 18:13:13 \(1491243193\) [ 2058.509171] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 18:13:13 (1491243193) [ 2058.661166] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 2058.915868] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=1 [ 2069.230479] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 2069.544083] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2070.164418] Lustre: DEBUG MARKER: rc=0; [ 2070.164418] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2070.164418] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2070.164418] echo $(hostname -s): $val; [ 2070.164418] rc=$val; [ 2070.164418] fi; [ 2070.164418] exit $rc [ 2070.757106] Lustre: DEBUG MARKER: dmesg [ 2071.171931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39o: directory cached attributes updated after create ================================= 18:13:26 \(1491243206\) [ 2071.332477] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 18:13:26 (1491243206) [ 2071.549365] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2072.171943] Lustre: DEBUG MARKER: rc=0; [ 2072.171943] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2072.171943] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2072.171943] echo $(hostname -s): $val; [ 2072.171943] rc=$val; [ 2072.171943] fi; [ 2072.171943] exit $rc [ 2072.777756] Lustre: DEBUG MARKER: dmesg [ 2073.206981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39p: remote directory cached attributes updated after create ========================== 18:13:28 \(1491243208\) [ 2073.362677] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 18:13:28 (1491243208) [ 2073.557048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_39p needs \>= 2 MDTs [ 2073.716188] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 2073.906677] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2074.535005] Lustre: DEBUG MARKER: rc=0; [ 2074.535005] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2074.535005] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2074.535005] echo $(hostname -s): $val; [ 2074.535005] rc=$val; [ 2074.535005] fi; [ 2074.535005] exit $rc [ 2075.123543] Lustre: DEBUG MARKER: dmesg [ 2075.548943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39q: close won\'t zero out atime ======================================================= 18:13:30 \(1491243210\) [ 2075.712312] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 18:13:30 (1491243210) [ 2075.927808] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2076.552185] Lustre: DEBUG MARKER: rc=0; [ 2076.552185] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2076.552185] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2076.552185] echo $(hostname -s): $val; [ 2076.552185] rc=$val; [ 2076.552185] fi; [ 2076.552185] exit $rc [ 2077.142217] Lustre: DEBUG MARKER: dmesg [ 2077.571593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ============================================= 18:13:32 \(1491243212\) [ 2077.732321] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 18:13:32 (1491243212) [ 2077.936124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2078.555558] Lustre: DEBUG MARKER: rc=0; [ 2078.555558] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2078.555558] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2078.555558] echo $(hostname -s): $val; [ 2078.555558] rc=$val; [ 2078.555558] fi; [ 2078.555558] exit $rc [ 2079.156256] Lustre: DEBUG MARKER: dmesg [ 2079.599101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 41: test small file write + fstat ===================================================== 18:13:34 \(1491243214\) [ 2079.758598] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 18:13:34 (1491243214) [ 2080.975547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2081.596390] Lustre: DEBUG MARKER: rc=0; [ 2081.596390] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2081.596390] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2081.596390] echo $(hostname -s): $val; [ 2081.596390] rc=$val; [ 2081.596390] fi; [ 2081.596390] exit $rc [ 2082.191874] Lustre: DEBUG MARKER: dmesg [ 2082.595858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 2082.751766] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 2082.957827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 2083.117758] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 2083.308525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 2083.470615] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 2083.666752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 2083.823644] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 2084.037269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 18:13:39 \(1491243219\) [ 2084.190613] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 18:13:39 (1491243219) [ 2121.900961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2122.527329] Lustre: DEBUG MARKER: rc=0; [ 2122.527329] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2122.527329] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2122.527329] echo $(hostname -s): $val; [ 2122.527329] rc=$val; [ 2122.527329] fi; [ 2122.527329] exit $rc [ 2123.117243] Lustre: DEBUG MARKER: dmesg [ 2123.558926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 18:14:18 \(1491243258\) [ 2123.720988] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 18:14:18 (1491243258) [ 2125.116474] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2125.912620] Lustre: DEBUG MARKER: rc=0; [ 2125.912620] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2125.912620] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2125.912620] echo $(hostname -s): $val; [ 2125.912620] rc=$val; [ 2125.912620] fi; [ 2125.912620] exit $rc [ 2126.511573] Lustre: DEBUG MARKER: dmesg [ 2126.930799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================= 18:14:22 \(1491243262\) [ 2127.084822] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 18:14:22 (1491243262) [ 2127.303663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2127.916851] Lustre: DEBUG MARKER: rc=0; [ 2127.916851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2127.916851] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2127.916851] echo $(hostname -s): $val; [ 2127.916851] rc=$val; [ 2127.916851] fi; [ 2127.916851] exit $rc [ 2128.513974] Lustre: DEBUG MARKER: dmesg [ 2128.956046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 18:14:24 \(1491243264\) [ 2129.107718] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 18:14:24 (1491243264) [ 2129.320059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2129.932256] Lustre: DEBUG MARKER: rc=0; [ 2129.932256] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2129.932256] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2129.932256] echo $(hostname -s): $val; [ 2129.932256] rc=$val; [ 2129.932256] fi; [ 2129.932256] exit $rc [ 2130.536110] Lustre: DEBUG MARKER: dmesg [ 2130.972190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre ======================================================= 18:14:26 \(1491243266\) [ 2131.146880] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre ======================================================= 18:14:26 (1491243266) [ 2131.380763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2132.013110] Lustre: DEBUG MARKER: rc=0; [ 2132.013110] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2132.013110] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2132.013110] echo $(hostname -s): $val; [ 2132.013110] rc=$val; [ 2132.013110] fi; [ 2132.013110] exit $rc [ 2132.605843] Lustre: DEBUG MARKER: dmesg [ 2133.039420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44A: zero length read from a sparse stripe ============================================ 18:14:28 \(1491243268\) [ 2133.203366] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ============================================ 18:14:28 (1491243268) [ 2133.406521] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2134.030700] Lustre: DEBUG MARKER: rc=0; [ 2134.030700] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2134.030700] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2134.030700] echo $(hostname -s): $val; [ 2134.030700] rc=$val; [ 2134.030700] fi; [ 2134.030700] exit $rc [ 2134.632641] Lustre: DEBUG MARKER: dmesg [ 2135.047199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44a: test sparse pwrite =============================================================== 18:14:30 \(1491243270\) [ 2135.204589] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 18:14:30 (1491243270) [ 2135.687625] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2136.300504] Lustre: DEBUG MARKER: rc=0; [ 2136.300504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2136.300504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2136.300504] echo $(hostname -s): $val; [ 2136.300504] rc=$val; [ 2136.300504] fi; [ 2136.300504] exit $rc [ 2136.924789] Lustre: DEBUG MARKER: dmesg [ 2137.322924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 2137.488396] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 2137.700774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ================================================ 18:14:32 \(1491243272\) [ 2137.859731] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 18:14:32 (1491243272) [ 2138.251377] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2138.857016] Lustre: DEBUG MARKER: rc=0; [ 2138.857016] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2138.857016] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2138.857016] echo $(hostname -s): $val; [ 2138.857016] rc=$val; [ 2138.857016] fi; [ 2138.857016] exit $rc [ 2139.453500] Lustre: DEBUG MARKER: dmesg [ 2139.870085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48a: Access renamed working dir \(should return errors\)================================= 18:14:34 \(1491243274\) [ 2140.035488] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 18:14:34 (1491243274) [ 2140.273993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2140.887113] Lustre: DEBUG MARKER: rc=0; [ 2140.887113] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2140.887113] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2140.887113] echo $(hostname -s): $val; [ 2140.887113] rc=$val; [ 2140.887113] fi; [ 2140.887113] exit $rc [ 2141.496841] Lustre: DEBUG MARKER: dmesg [ 2141.964935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48b: Access removed working dir \(should return errors\)================================= 18:14:37 \(1491243277\) [ 2142.146894] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 18:14:37 (1491243277) [ 2142.364449] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2143.003217] Lustre: DEBUG MARKER: rc=0; [ 2143.003217] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2143.003217] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2143.003217] echo $(hostname -s): $val; [ 2143.003217] rc=$val; [ 2143.003217] fi; [ 2143.003217] exit $rc [ 2143.626081] Lustre: DEBUG MARKER: dmesg [ 2144.112801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48c: Access removed working subdir \(should return errors\) ============================= 18:14:39 \(1491243279\) [ 2144.277412] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 18:14:39 (1491243279) [ 2144.510718] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2145.136231] Lustre: DEBUG MARKER: rc=0; [ 2145.136231] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2145.136231] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2145.136231] echo $(hostname -s): $val; [ 2145.136231] rc=$val; [ 2145.136231] fi; [ 2145.136231] exit $rc [ 2145.729962] Lustre: DEBUG MARKER: dmesg [ 2146.193181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48d: Access removed parent subdir \(should return errors\) ============================== 18:14:41 \(1491243281\) [ 2146.348410] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 18:14:41 (1491243281) [ 2146.576590] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2147.191711] Lustre: DEBUG MARKER: rc=0; [ 2147.191711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2147.191711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2147.191711] echo $(hostname -s): $val; [ 2147.191711] rc=$val; [ 2147.191711] fi; [ 2147.191711] exit $rc [ 2147.791192] Lustre: DEBUG MARKER: dmesg [ 2148.211421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48e: Access to recreated parent subdir \(should return errors\) ========================= 18:14:43 \(1491243283\) [ 2148.366927] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 18:14:43 (1491243283) [ 2148.585312] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2149.206341] Lustre: DEBUG MARKER: rc=0; [ 2149.206341] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2149.206341] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2149.206341] echo $(hostname -s): $val; [ 2149.206341] rc=$val; [ 2149.206341] fi; [ 2149.206341] exit $rc [ 2149.801678] Lustre: DEBUG MARKER: dmesg [ 2150.230558] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 49: Change max_pages_per_rpc won\'t break osc extent =================================== 18:14:45 \(1491243285\) [ 2150.397590] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 18:14:45 (1491243285) [ 2150.897867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2151.516329] Lustre: DEBUG MARKER: rc=0; [ 2151.516329] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2151.516329] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2151.516329] echo $(hostname -s): $val; [ 2151.516329] rc=$val; [ 2151.516329] fi; [ 2151.516329] exit $rc [ 2152.113088] Lustre: DEBUG MARKER: dmesg [ 2152.539989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 50: special situations: \/proc symlinks =============================================== 18:14:47 \(1491243287\) [ 2152.693741] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 18:14:47 (1491243287) [ 2152.887800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2153.503441] Lustre: DEBUG MARKER: rc=0; [ 2153.503441] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2153.503441] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2153.503441] echo $(hostname -s): $val; [ 2153.503441] rc=$val; [ 2153.503441] fi; [ 2153.503441] exit $rc [ 2154.109718] Lustre: DEBUG MARKER: dmesg [ 2154.532086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51a: special situations: split htree with empty entry ================================= 18:14:49 \(1491243289\) [ 2154.690880] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 18:14:49 (1491243289) [ 2155.160568] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2155.767198] Lustre: DEBUG MARKER: rc=0; [ 2155.767198] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2155.767198] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2155.767198] echo $(hostname -s): $val; [ 2155.767198] rc=$val; [ 2155.767198] fi; [ 2155.767198] exit $rc [ 2156.362548] Lustre: DEBUG MARKER: dmesg [ 2156.808796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ===================== 18:14:51 \(1491243291\) [ 2156.971530] Lustre: DEBUG MARKER: == sanity test 51b: exceed 64k subdirectory nlink limit on create, verify unlink ===================== 18:14:51 (1491243291) [ 2328.919864] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2329.547426] Lustre: DEBUG MARKER: rc=0; [ 2329.547426] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2329.547426] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2329.547426] echo $(hostname -s): $val; [ 2329.547426] rc=$val; [ 2329.547426] fi; [ 2329.547426] exit $rc [ 2330.151381] Lustre: DEBUG MARKER: dmesg [ 2330.596489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51d: check object distribution ======================================================== 18:17:45 \(1491243465\) [ 2330.758751] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 18:17:45 (1491243465) [ 2333.151525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0 has 143 objects, 143 are index 0 [ 2333.313521] Lustre: DEBUG MARKER: OST0 has 143 objects, 143 are index 0 [ 2333.525266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST1 has 143 objects, 143 are index 0 [ 2333.682250] Lustre: DEBUG MARKER: OST1 has 143 objects, 143 are index 0 [ 2333.883743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST2 has 143 objects, 143 are index 0 [ 2334.040708] Lustre: DEBUG MARKER: OST2 has 143 objects, 143 are index 0 [ 2334.260675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST3 has 142 objects, 142 are index 0 [ 2334.415200] Lustre: DEBUG MARKER: OST3 has 142 objects, 142 are index 0 [ 2334.622490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST4 has 143 objects, 143 are index 0 [ 2334.776992] Lustre: DEBUG MARKER: OST4 has 143 objects, 143 are index 0 [ 2334.984062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST5 has 143 objects, 143 are index 0 [ 2335.156545] Lustre: DEBUG MARKER: OST5 has 143 objects, 143 are index 0 [ 2335.362693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST6 has 143 objects, 143 are index 0 [ 2335.516442] Lustre: DEBUG MARKER: OST6 has 143 objects, 143 are index 0 [ 2337.384875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2338.016640] Lustre: DEBUG MARKER: rc=0; [ 2338.016640] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2338.016640] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2338.016640] echo $(hostname -s): $val; [ 2338.016640] rc=$val; [ 2338.016640] fi; [ 2338.016640] exit $rc [ 2338.617742] Lustre: DEBUG MARKER: dmesg [ 2339.064629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51e: check file nlink limit =========================================================== 18:17:54 \(1491243474\) [ 2339.224475] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 18:17:54 (1491243474) [ 2545.229455] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2545.840120] Lustre: DEBUG MARKER: rc=0; [ 2545.840120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2545.840120] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2545.840120] echo $(hostname -s): $val; [ 2545.840120] rc=$val; [ 2545.840120] fi; [ 2545.840120] exit $rc [ 2546.451704] Lustre: DEBUG MARKER: dmesg [ 2546.896955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51f: check many open files limit ====================================================== 18:21:22 \(1491243682\) [ 2547.054927] Lustre: DEBUG MARKER: == sanity test 51f: check many open files limit ====================================================== 18:21:22 (1491243682) [ 2878.763933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2879.392179] Lustre: DEBUG MARKER: rc=0; [ 2879.392179] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2879.392179] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2879.392179] echo $(hostname -s): $val; [ 2879.392179] rc=$val; [ 2879.392179] fi; [ 2879.392179] exit $rc [ 2879.999556] Lustre: DEBUG MARKER: dmesg [ 2880.529730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52a: append-only flag test \(should return errors\) ===================================== 18:26:55 \(1491244015\) [ 2880.683812] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 18:26:55 (1491244015) [ 2880.990768] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2881.612742] Lustre: DEBUG MARKER: rc=0; [ 2881.612742] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2881.612742] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2881.612742] echo $(hostname -s): $val; [ 2881.612742] rc=$val; [ 2881.612742] fi; [ 2881.612742] exit $rc [ 2882.212083] Lustre: DEBUG MARKER: dmesg [ 2882.640567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52b: immutable flag test \(should return errors\) ======================================= 18:26:57 \(1491244017\) [ 2882.804628] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 18:26:57 (1491244017) [ 2883.231864] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2883.852056] Lustre: DEBUG MARKER: rc=0; [ 2883.852056] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2883.852056] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2883.852056] echo $(hostname -s): $val; [ 2883.852056] rc=$val; [ 2883.852056] fi; [ 2883.852056] exit $rc [ 2884.452293] Lustre: DEBUG MARKER: dmesg [ 2884.882813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 18:26:59 \(1491244019\) [ 2885.038220] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 18:26:59 (1491244019) [ 2885.229449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2885.229449] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2885.229449] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2885.517066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2885.517066] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2885.517066] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2885.793141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2885.793141] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2885.793141] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2886.064214] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2886.324387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.*OST*-osc-MDT0000.prealloc_last_id [ 2886.594810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_seq [ 2886.853569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2887.389536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_seq [ 2887.647149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 2888.182501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_last_seq [ 2888.440403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0002-osc-MDT0000.prealloc_last_id [ 2888.966510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_last_seq [ 2889.225572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0003-osc-MDT0000.prealloc_last_id [ 2889.751118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_last_seq [ 2890.011351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0004-osc-MDT0000.prealloc_last_id [ 2890.540571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_last_seq [ 2890.799256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0005-osc-MDT0000.prealloc_last_id [ 2891.329183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_last_seq [ 2891.586736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0006-osc-MDT0000.prealloc_last_id [ 2892.168530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2892.788458] Lustre: DEBUG MARKER: rc=0; [ 2892.788458] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2892.788458] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2892.788458] echo $(hostname -s): $val; [ 2892.788458] rc=$val; [ 2892.788458] fi; [ 2892.788458] exit $rc [ 2893.387099] Lustre: DEBUG MARKER: dmesg [ 2893.816857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54a: unix domain socket test ========================================================== 18:27:08 \(1491244028\) [ 2893.976334] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 18:27:08 (1491244028) [ 2894.249378] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2894.868553] Lustre: DEBUG MARKER: rc=0; [ 2894.868553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2894.868553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2894.868553] echo $(hostname -s): $val; [ 2894.868553] rc=$val; [ 2894.868553] fi; [ 2894.868553] exit $rc [ 2895.472582] Lustre: DEBUG MARKER: dmesg [ 2895.932496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54b: char device works in lustre ====================================================== 18:27:11 \(1491244031\) [ 2896.101349] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 18:27:11 (1491244031) [ 2896.301576] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2896.915967] Lustre: DEBUG MARKER: rc=0; [ 2896.915967] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2896.915967] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2896.915967] echo $(hostname -s): $val; [ 2896.915967] rc=$val; [ 2896.915967] fi; [ 2896.915967] exit $rc [ 2897.508609] Lustre: DEBUG MARKER: dmesg [ 2897.950463] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54c: block device works in lustre ===================================================== 18:27:13 \(1491244033\) [ 2898.105701] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 18:27:13 (1491244033) [ 2898.763169] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2899.384576] Lustre: DEBUG MARKER: rc=0; [ 2899.384576] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2899.384576] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2899.384576] echo $(hostname -s): $val; [ 2899.384576] rc=$val; [ 2899.384576] fi; [ 2899.384576] exit $rc [ 2899.984776] Lustre: DEBUG MARKER: dmesg [ 2900.409487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54d: fifo device works in lustre ====================================================== 18:27:15 \(1491244035\) [ 2900.571976] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 18:27:15 (1491244035) [ 2900.772368] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2901.383293] Lustre: DEBUG MARKER: rc=0; [ 2901.383293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2901.383293] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2901.383293] echo $(hostname -s): $val; [ 2901.383293] rc=$val; [ 2901.383293] fi; [ 2901.383293] exit $rc [ 2901.981601] Lustre: DEBUG MARKER: dmesg [ 2902.406847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54e: console\/tty device works in lustre =============================================== 18:27:17 \(1491244037\) [ 2902.560913] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 18:27:17 (1491244037) [ 2902.758478] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2903.366246] Lustre: DEBUG MARKER: rc=0; [ 2903.366246] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2903.366246] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2903.366246] echo $(hostname -s): $val; [ 2903.366246] rc=$val; [ 2903.366246] fi; [ 2903.366246] exit $rc [ 2903.957815] Lustre: DEBUG MARKER: dmesg [ 2904.382033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56a: check \/usr\/bin\/lfs getstripe ===================================================== 18:27:19 \(1491244039\) [ 2904.536494] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe ===================================================== 18:27:19 (1491244039) [ 2904.550220] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2904.552420] Lustre: Skipped 11 previous similar messages [ 2904.911830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2905.526979] Lustre: DEBUG MARKER: rc=0; [ 2905.526979] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2905.526979] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2905.526979] echo $(hostname -s): $val; [ 2905.526979] rc=$val; [ 2905.526979] fi; [ 2905.526979] exit $rc [ 2906.128466] Lustre: DEBUG MARKER: dmesg [ 2906.571845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56b: check \/usr\/bin\/lfs getdirstripe ================================================== 18:27:21 \(1491244041\) [ 2906.735472] Lustre: DEBUG MARKER: == sanity test 56b: check /usr/bin/lfs getdirstripe ================================================== 18:27:21 (1491244041) [ 2906.961134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2907.570623] Lustre: DEBUG MARKER: rc=0; [ 2907.570623] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2907.570623] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2907.570623] echo $(hostname -s): $val; [ 2907.570623] rc=$val; [ 2907.570623] fi; [ 2907.570623] exit $rc [ 2908.173434] Lustre: DEBUG MARKER: dmesg [ 2908.613376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56g: check lfs find -name ============================================================= 18:27:23 \(1491244043\) [ 2908.771664] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 18:27:23 (1491244043) [ 2908.783871] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2908.786074] Lustre: Skipped 5 previous similar messages [ 2909.077231] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2909.691129] Lustre: DEBUG MARKER: rc=0; [ 2909.691129] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2909.691129] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2909.691129] echo $(hostname -s): $val; [ 2909.691129] rc=$val; [ 2909.691129] fi; [ 2909.691129] exit $rc [ 2909.996223] LustreError: 16313:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2910.290441] Lustre: DEBUG MARKER: dmesg [ 2910.735056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56h: check lfs find ! -name =========================================================== 18:27:25 \(1491244045\) [ 2910.890935] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 18:27:25 (1491244045) [ 2910.904035] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2910.906272] Lustre: Skipped 5 previous similar messages [ 2911.197685] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2911.812202] Lustre: DEBUG MARKER: rc=0; [ 2911.812202] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2911.812202] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2911.812202] echo $(hostname -s): $val; [ 2911.812202] rc=$val; [ 2911.812202] fi; [ 2911.812202] exit $rc [ 2912.414790] Lustre: DEBUG MARKER: dmesg [ 2912.840396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56i: check \'lfs find -ost UUID\' skips directories ===================================== 18:27:27 \(1491244047\) [ 2912.997168] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 18:27:27 (1491244047) [ 2913.208123] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2913.819923] Lustre: DEBUG MARKER: rc=0; [ 2913.819923] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2913.819923] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2913.819923] echo $(hostname -s): $val; [ 2913.819923] rc=$val; [ 2913.819923] fi; [ 2913.819923] exit $rc [ 2914.431456] Lustre: DEBUG MARKER: dmesg [ 2914.849906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56j: check lfs find -type d =========================================================== 18:27:29 \(1491244049\) [ 2915.024521] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 18:27:29 (1491244049) [ 2915.338354] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2915.964415] Lustre: DEBUG MARKER: rc=0; [ 2915.964415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2915.964415] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2915.964415] echo $(hostname -s): $val; [ 2915.964415] rc=$val; [ 2915.964415] fi; [ 2915.964415] exit $rc [ 2916.569212] Lustre: DEBUG MARKER: dmesg [ 2917.016841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56k: check lfs find -type f =========================================================== 18:27:32 \(1491244052\) [ 2917.171520] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 18:27:32 (1491244052) [ 2917.488946] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2918.124564] Lustre: DEBUG MARKER: rc=0; [ 2918.124564] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2918.124564] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2918.124564] echo $(hostname -s): $val; [ 2918.124564] rc=$val; [ 2918.124564] fi; [ 2918.124564] exit $rc [ 2918.734478] Lustre: DEBUG MARKER: dmesg [ 2919.211862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56l: check lfs find -type b =========================================================== 18:27:34 \(1491244054\) [ 2919.375205] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 18:27:34 (1491244054) [ 2919.706740] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2919.989532] LustreError: 17721:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2920.353248] Lustre: DEBUG MARKER: rc=0; [ 2920.353248] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2920.353248] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2920.353248] echo $(hostname -s): $val; [ 2920.353248] rc=$val; [ 2920.353248] fi; [ 2920.353248] exit $rc [ 2920.999608] Lustre: DEBUG MARKER: dmesg [ 2921.460163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56m: check lfs find -type c =========================================================== 18:27:36 \(1491244056\) [ 2921.620652] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 18:27:36 (1491244056) [ 2921.940713] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2922.569522] Lustre: DEBUG MARKER: rc=0; [ 2922.569522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2922.569522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2922.569522] echo $(hostname -s): $val; [ 2922.569522] rc=$val; [ 2922.569522] fi; [ 2922.569522] exit $rc [ 2923.176113] Lustre: DEBUG MARKER: dmesg [ 2923.620555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56n: check lfs find -type l =========================================================== 18:27:38 \(1491244058\) [ 2923.792506] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 18:27:38 (1491244058) [ 2924.111923] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2924.725620] Lustre: DEBUG MARKER: rc=0; [ 2924.725620] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2924.725620] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2924.725620] echo $(hostname -s): $val; [ 2924.725620] rc=$val; [ 2924.725620] fi; [ 2924.725620] exit $rc [ 2925.327363] Lustre: DEBUG MARKER: dmesg [ 2925.776266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56o: check lfs find -mtime for old files ============================================== 18:27:40 \(1491244060\) [ 2925.950822] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 18:27:40 (1491244060) [ 2926.498921] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2927.128511] Lustre: DEBUG MARKER: rc=0; [ 2927.128511] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2927.128511] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2927.128511] echo $(hostname -s): $val; [ 2927.128511] rc=$val; [ 2927.128511] fi; [ 2927.128511] exit $rc [ 2927.733149] Lustre: DEBUG MARKER: dmesg [ 2928.170139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56p: check lfs find -uid and ! -uid =================================================== 18:27:43 \(1491244063\) [ 2928.346517] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 18:27:43 (1491244063) [ 2928.683744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2929.313709] Lustre: DEBUG MARKER: rc=0; [ 2929.313709] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2929.313709] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2929.313709] echo $(hostname -s): $val; [ 2929.313709] rc=$val; [ 2929.313709] fi; [ 2929.313709] exit $rc [ 2929.917327] Lustre: DEBUG MARKER: dmesg [ 2930.369356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56q: check lfs find -gid and ! -gid =================================================== 18:27:45 \(1491244065\) [ 2930.526422] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 18:27:45 (1491244065) [ 2930.843753] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2931.460803] Lustre: DEBUG MARKER: rc=0; [ 2931.460803] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2931.460803] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2931.460803] echo $(hostname -s): $val; [ 2931.460803] rc=$val; [ 2931.460803] fi; [ 2931.460803] exit $rc [ 2932.069295] Lustre: DEBUG MARKER: dmesg [ 2932.518138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56r: check lfs find -size works ======================================================= 18:27:47 \(1491244067\) [ 2932.680397] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 18:27:47 (1491244067) [ 2933.221765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2933.846215] Lustre: DEBUG MARKER: rc=0; [ 2933.846215] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2933.846215] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2933.846215] echo $(hostname -s): $val; [ 2933.846215] rc=$val; [ 2933.846215] fi; [ 2933.846215] exit $rc [ 2934.455137] Lustre: DEBUG MARKER: dmesg [ 2934.890742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56s: check lfs find -stripe-count works =============================================== 18:27:50 \(1491244070\) [ 2935.054945] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 18:27:50 (1491244070) [ 2935.457326] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2936.100533] Lustre: DEBUG MARKER: rc=0; [ 2936.100533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2936.100533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2936.100533] echo $(hostname -s): $val; [ 2936.100533] rc=$val; [ 2936.100533] fi; [ 2936.100533] exit $rc [ 2936.699019] Lustre: DEBUG MARKER: dmesg [ 2937.140402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56t: check lfs find -stripe-size works ================================================ 18:27:52 \(1491244072\) [ 2937.323981] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 18:27:52 (1491244072) [ 2937.768178] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2938.390091] Lustre: DEBUG MARKER: rc=0; [ 2938.390091] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2938.390091] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2938.390091] echo $(hostname -s): $val; [ 2938.390091] rc=$val; [ 2938.390091] fi; [ 2938.390091] exit $rc [ 2938.993662] Lustre: DEBUG MARKER: dmesg [ 2939.447303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56u: check lfs find -stripe-index works =============================================== 18:27:54 \(1491244074\) [ 2939.603939] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 18:27:54 (1491244074) [ 2940.028421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2940.662655] Lustre: DEBUG MARKER: rc=0; [ 2940.662655] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2940.662655] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2940.662655] echo $(hostname -s): $val; [ 2940.662655] rc=$val; [ 2940.662655] fi; [ 2940.662655] exit $rc [ 2941.276769] Lustre: DEBUG MARKER: dmesg [ 2941.703722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56v: check \'lfs find -mdt match with lfs getstripe -M\' ================================ 18:27:56 \(1491244076\) [ 2941.859539] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 18:27:56 (1491244076) [ 2942.274641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2942.890780] Lustre: DEBUG MARKER: rc=0; [ 2942.890780] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2942.890780] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2942.890780] echo $(hostname -s): $val; [ 2942.890780] rc=$val; [ 2942.890780] fi; [ 2942.890780] exit $rc [ 2943.511096] Lustre: DEBUG MARKER: dmesg [ 2943.970672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 18:27:59 \(1491244079\) [ 2944.139454] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 18:27:59 (1491244079) [ 2949.110655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2949.110655] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2949.110655] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2957.915749] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2958.538503] Lustre: DEBUG MARKER: rc=0; [ 2958.538503] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2958.538503] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2958.538503] echo $(hostname -s): $val; [ 2958.538503] rc=$val; [ 2958.538503] fi; [ 2958.538503] exit $rc [ 2959.134759] Lustre: DEBUG MARKER: dmesg [ 2959.556292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56x: lfs migration support ============================================================ 18:28:14 \(1491244094\) [ 2959.719346] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 18:28:14 (1491244094) [ 2960.092107] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2960.704348] Lustre: DEBUG MARKER: rc=0; [ 2960.704348] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2960.704348] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2960.704348] echo $(hostname -s): $val; [ 2960.704348] rc=$val; [ 2960.704348] fi; [ 2960.704348] exit $rc [ 2961.302124] Lustre: DEBUG MARKER: dmesg [ 2961.733253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56xa: lfs migration --block support =================================================== 18:28:16 \(1491244096\) [ 2961.898597] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 18:28:16 (1491244096) [ 2962.283246] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2962.899397] Lustre: DEBUG MARKER: rc=0; [ 2962.899397] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2962.899397] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2962.899397] echo $(hostname -s): $val; [ 2962.899397] rc=$val; [ 2962.899397] fi; [ 2962.899397] exit $rc [ 2963.499730] Lustre: DEBUG MARKER: dmesg [ 2963.937515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56y: lfs find -L raid0\|released ======================================================= 18:28:19 \(1491244099\) [ 2964.093850] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 18:28:19 (1491244099) [ 2964.228876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2964.228876] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2964.228876] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2964.612256] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2965.227610] Lustre: DEBUG MARKER: rc=0; [ 2965.227610] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2965.227610] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2965.227610] echo $(hostname -s): $val; [ 2965.227610] rc=$val; [ 2965.227610] fi; [ 2965.227610] exit $rc [ 2965.830286] Lustre: DEBUG MARKER: dmesg [ 2966.273307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56z: lfs find should continue after an error ========================================== 18:28:21 \(1491244101\) [ 2966.436913] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 18:28:21 (1491244101) [ 2966.788615] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2967.404309] Lustre: DEBUG MARKER: rc=0; [ 2967.404309] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2967.404309] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2967.404309] echo $(hostname -s): $val; [ 2967.404309] rc=$val; [ 2967.404309] fi; [ 2967.404309] exit $rc [ 2968.008247] Lustre: DEBUG MARKER: dmesg [ 2968.462226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56aa: lfs find --size under striped dir =============================================== 18:28:23 \(1491244103\) [ 2968.617865] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 18:28:23 (1491244103) [ 2968.832381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_56aa needs \>= 2 MDTs [ 2969.006877] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 2969.222520] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2969.869042] Lustre: DEBUG MARKER: rc=0; [ 2969.869042] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2969.869042] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2969.869042] echo $(hostname -s): $val; [ 2969.869042] rc=$val; [ 2969.869042] fi; [ 2969.869042] exit $rc [ 2970.485102] Lustre: DEBUG MARKER: dmesg [ 2970.950135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57a: verify MDS filesystem created with large inodes ================================== 18:28:26 \(1491244106\) [ 2971.125447] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 18:28:26 (1491244106) [ 2971.282216] Lustre: DEBUG MARKER: lctl get_param -n osd*.*MDT*.mntdev [ 2971.554272] Lustre: DEBUG MARKER: lctl get_param -n osd*.*MDT*.mntdev [ 2971.824619] Lustre: DEBUG MARKER: dumpe2fs -h /dev/mapper/lvm--Role_MDS-P1 [ 2972.148754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2972.779510] Lustre: DEBUG MARKER: rc=0; [ 2972.779510] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2972.779510] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2972.779510] echo $(hostname -s): $val; [ 2972.779510] rc=$val; [ 2972.779510] fi; [ 2972.779510] exit $rc [ 2973.386931] Lustre: DEBUG MARKER: dmesg [ 2973.907229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 18:28:29 \(1491244109\) [ 2974.082062] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 18:28:29 (1491244109) [ 2975.413104] Lustre: DEBUG MARKER: lctl get_param -n osd*.*MDT0000.kbytesfree [ 2976.967426] Lustre: DEBUG MARKER: lctl get_param -n osd*.*MDT0000.kbytesfree [ 2977.418321] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2978.038362] Lustre: DEBUG MARKER: rc=0; [ 2978.038362] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2978.038362] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2978.038362] echo $(hostname -s): $val; [ 2978.038362] rc=$val; [ 2978.038362] fi; [ 2978.038362] exit $rc [ 2978.654150] Lustre: DEBUG MARKER: dmesg [ 2979.091274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 58: verify cross-platform wire constants ============================================== 18:28:34 \(1491244114\) [ 2979.253586] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 18:28:34 (1491244114) [ 2979.447221] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2980.069207] Lustre: DEBUG MARKER: rc=0; [ 2980.069207] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2980.069207] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2980.069207] echo $(hostname -s): $val; [ 2980.069207] rc=$val; [ 2980.069207] fi; [ 2980.069207] exit $rc [ 2980.681344] Lustre: DEBUG MARKER: dmesg [ 2981.191254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 59: verify cancellation of llog records async ========================================= 18:28:36 \(1491244116\) [ 2981.370386] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 18:28:36 (1491244116) [ 2982.056254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2982.320953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 2982.619140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2983.904499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2985.196554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2986.481488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2987.767380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2989.049539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2990.332522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2991.684026] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2992.310082] Lustre: DEBUG MARKER: rc=0; [ 2992.310082] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2992.310082] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2992.310082] echo $(hostname -s): $val; [ 2992.310082] rc=$val; [ 2992.310082] fi; [ 2992.310082] exit $rc [ 2992.993334] Lustre: DEBUG MARKER: dmesg [ 2993.436097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 18:28:48 \(1491244128\) [ 2993.597669] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 18:28:48 (1491244128) [ 2993.746175] Lustre: DEBUG MARKER: ! which run-llog.sh &> /dev/null [ 2994.083387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_60 run 2614 - from kernel mode [ 2994.245031] Lustre: DEBUG MARKER: test_60 run 2614 - from kernel mode [ 2994.383398] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=warning; /usr/sbin/lctl dk > /dev/null [ 2994.772780] Lustre: DEBUG MARKER: sh run-llog.sh [ 2995.097964] Lustre: 25509:0:(llog_test.c:2082:llog_test_setup()) Setup llog-test device over MGS device [ 2995.100181] Lustre: 25509:0:(llog_test.c:111:llog_test_1()) 1a: create a log with name: 5cc3fc58 [ 2995.102645] Lustre: 25509:0:(llog_test.c:128:llog_test_1()) 1b: close newly-created log [ 2995.104592] Lustre: 25509:0:(llog_test.c:159:llog_test_2()) 2a: re-open a log with name: 5cc3fc58 [ 2995.106838] Lustre: 25509:0:(llog_test.c:179:llog_test_2()) 2b: create a log without specified NAME & LOGID [ 2995.108988] Lustre: 25509:0:(llog_test.c:197:llog_test_2()) 2b: write 1 llog records, check llh_count [ 2995.111263] Lustre: 25509:0:(llog_test.c:210:llog_test_2()) 2c: re-open the log by LOGID and verify llh_count [ 2995.113986] Lustre: 25509:0:(llog_test.c:257:llog_test_2()) 2d: destroy this log [ 2995.115853] Lustre: 25509:0:(llog_test.c:411:llog_test_3()) 3a: write 1023 fixed-size llog records [ 2995.121968] Lustre: 25509:0:(llog_test.c:377:llog_test3_process()) test3: processing records from index 501 to the end [ 2995.125404] Lustre: 25509:0:(llog_test.c:386:llog_test3_process()) test3: total 525 records processed with 0 paddings [ 2995.127494] Lustre: 25509:0:(llog_test.c:467:llog_test_3()) 3b: write 566 variable size llog records [ 2995.133500] Lustre: 25509:0:(llog_test.c:536:llog_test_3()) 3c: write records with variable size until BITMAP_SIZE, return -ENOSPC [ 2995.403890] Lustre: 25509:0:(llog_test.c:559:llog_test_3()) 3c: wrote 63962 more records before end of llog is reached [ 2995.406453] Lustre: 25509:0:(llog_test.c:588:llog_test_4()) 4a: create a catalog log with name: 5cc3fc59 [ 2995.408702] Lustre: 25509:0:(llog_test.c:603:llog_test_4()) 4b: write 1 record into the catalog [ 2995.411287] Lustre: 25509:0:(llog_test.c:618:llog_test_4()) 4c: cancel 1 log record [ 2995.413338] Lustre: 25509:0:(llog_test.c:630:llog_test_4()) 4d: write 65536 more log records [ 2995.748106] Lustre: 25509:0:(llog_test.c:646:llog_test_4()) 4e: add 5 large records, one record per block [ 2995.750469] Lustre: 25509:0:(llog_test.c:666:llog_test_4()) 4f: put newly-created catalog [ 2995.752393] Lustre: 25509:0:(llog_test.c:764:llog_test_5()) 5a: re-open catalog by id [ 2995.754263] Lustre: 25509:0:(llog_test.c:777:llog_test_5()) 5b: print the catalog entries.. we expect 2 [ 2995.756252] Lustre: 25515:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0x26:0x0] in log [0xa:0x10:0x0] [ 2995.758451] Lustre: 25509:0:(llog_test.c:789:llog_test_5()) 5c: Cancel 65536 records, see one log zapped [ 2996.019913] Lustre: 25509:0:(llog_test.c:797:llog_test_5()) 5c: print the catalog entries.. we expect 1 [ 2996.022306] Lustre: 25509:0:(llog_test.c:809:llog_test_5()) 5d: add 1 record to the log with many canceled empty pages [ 2996.024630] Lustre: 25509:0:(llog_test.c:817:llog_test_5()) 5e: print plain log entries.. expect 6 [ 2996.026969] Lustre: 25509:0:(llog_test.c:829:llog_test_5()) 5f: print plain log entries reversely.. expect 6 [ 2996.029701] Lustre: 25509:0:(llog_test.c:843:llog_test_5()) 5g: close re-opened catalog [ 2996.031655] Lustre: 25509:0:(llog_test.c:873:llog_test_6()) 6a: re-open log 5cc3fc58 using client API [ 2996.033898] Lustre: MGS: non-config logname received: 5cc3fc58 [ 2996.035826] Lustre: 25509:0:(llog_test.c:905:llog_test_6()) 6b: process log 5cc3fc58 using client API [ 2996.086245] Lustre: 25509:0:(llog_test.c:909:llog_test_6()) 6b: processed 63962 records [ 2996.088415] Lustre: 25509:0:(llog_test.c:916:llog_test_6()) 6c: process log 5cc3fc58 reversely using client API [ 2996.399444] Lustre: 25509:0:(llog_test.c:920:llog_test_6()) 6c: processed 63962 records [ 2996.401705] Lustre: 25509:0:(llog_test.c:1067:llog_test_7()) 7a: test llog_logid_rec [ 2997.069517] Lustre: 25509:0:(llog_test.c:1078:llog_test_7()) 7b: test llog_unlink64_rec [ 2998.188365] Lustre: 25509:0:(llog_test.c:1089:llog_test_7()) 7c: test llog_setattr64_rec [ 3000.264898] Lustre: 25509:0:(llog_test.c:1100:llog_test_7()) 7d: test llog_size_change_rec [ 3000.879051] Lustre: 25509:0:(llog_test.c:1111:llog_test_7()) 7e: test llog_changelog_rec [ 3001.162735] Lustre: 25509:0:(llog_test.c:1017:llog_test_7_sub()) 7_sub: records are not aligned, written 64071 from 64767 [ 3001.556515] Lustre: 25509:0:(llog_test.c:1123:llog_test_7()) 7f: test llog_changelog_user_rec [ 3001.829422] Lustre: 25509:0:(llog_test.c:1017:llog_test_7_sub()) 7_sub: records are not aligned, written 64452 from 64767 [ 3002.048796] Lustre: 25509:0:(llog_test.c:1134:llog_test_7()) 7g: test llog_gen_rec [ 3002.615054] Lustre: 25509:0:(llog_test.c:1229:llog_test_8()) 8a: fill the first plain llog [ 3002.618724] Lustre: 25509:0:(llog_test.c:1262:llog_test_8()) 8a: pin llog [0x1:0x31:0x0] [ 3002.620583] Lustre: 25509:0:(llog_test.c:1272:llog_test_8()) 8b: fill the second plain llog [ 3002.623339] Lustre: 25509:0:(llog_test.c:1293:llog_test_8()) 8b: second llog [0x1:0x33:0x0] [ 3002.625149] Lustre: 25509:0:(llog_test.c:1303:llog_test_8()) 8c: drop two records from the first plain llog [ 3002.679906] Lustre: 25509:0:(llog_test.c:1306:llog_test_8()) 8d: count survived records [ 3002.681994] LustreError: 25509:0:(llog.c:616:llog_process_thread()) Local llog found corrupted [ 3002.684123] Lustre: 25509:0:(llog_test.c:1333:llog_test_8()) 8d: close re-opened catalog [ 3002.686019] Lustre: 25509:0:(llog_test.c:1395:llog_test_9()) 9a: test llog_logid_rec [ 3002.688056] Lustre: 25509:0:(llog_test.c:1379:llog_test_9_sub()) 9_sub: record type 1064553b in log 0x1:0x35:0x0 [ 3002.690036] Lustre: 25509:0:(llog_test.c:1406:llog_test_9()) 9b: test llog_obd_cfg_rec [ 3002.692018] Lustre: 25509:0:(llog_test.c:1417:llog_test_9()) 9c: test llog_changelog_rec [ 3002.693956] Lustre: 25509:0:(llog_test.c:1429:llog_test_9()) 9d: test llog_changelog_user_rec [ 3002.695851] Lustre: 25509:0:(llog_test.c:1466:llog_test_10()) 10a: create a catalog log with name: 5cc3fc5a [ 3003.427831] Lustre: 25509:0:(llog_test.c:1494:llog_test_10()) 10b: write 65536 log records [ 3003.991251] Lustre: 25509:0:(llog_test.c:1518:llog_test_10()) 10c: write 131072 more log records [ 3004.819140] Lustre: 25509:0:(llog_test.c:1546:llog_test_10()) 10c: write 65536 more log records [ 3005.134999] Lustre: 25509:0:(llog_cat.c:89:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 3005.162913] Lustre: 25509:0:(llog_test.c:1571:llog_test_10()) 10c: wrote 61704 records then 3832 failed with ENOSPC [ 3005.165160] Lustre: 25509:0:(llog_test.c:1588:llog_test_10()) 10d: Cancel 65536 records, see one log zapped [ 3005.367922] Lustre: 25509:0:(llog_test.c:1600:llog_test_10()) 10d: print the catalog entries.. we expect 3 [ 3005.370223] Lustre: 25525:0:(llog_test.c:695:cat_print_cb()) seeing record at index 2 - [0x1:0x3a:0x0] in log [0xa:0x11:0x0] [ 3005.372327] Lustre: 25525:0:(llog_test.c:695:cat_print_cb()) Skipped 2 previous similar messages [ 3005.541602] Lustre: 25509:0:(llog_test.c:1628:llog_test_10()) 10e: write 65536 more log records [ 3005.865805] Lustre: 25509:0:(llog_cat.c:89:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 3005.867999] Lustre: 25509:0:(llog_cat.c:89:llog_cat_new_log()) Skipped 3831 previous similar messages [ 3005.876483] Lustre: 25509:0:(llog_test.c:1653:llog_test_10()) 10e: wrote 64578 records then 958 failed with ENOSPC [ 3005.878453] Lustre: 25509:0:(llog_test.c:1655:llog_test_10()) 10e: print the catalog entries.. we expect 4 [ 3005.880742] Lustre: 25509:0:(llog_test.c:1691:llog_test_10()) 10e: catalog successfully wrap around, last_idx 1, first 1 [ 3005.996257] Lustre: 25509:0:(llog_test.c:1704:llog_test_10()) 10f: Cancel 65536 records, see one log zapped [ 3005.998407] Lustre: 25509:0:(llog_cat.c:869:llog_cat_process_or_fork()) catlog 0x11:10 crosses index zero [ 3006.202317] Lustre: 25509:0:(llog_test.c:1716:llog_test_10()) 10f: print the catalog entries.. we expect 3 [ 3006.321852] Lustre: 25509:0:(llog_test.c:1744:llog_test_10()) 10f: write 65536 more log records [ 3006.657489] Lustre: 25509:0:(llog_test.c:1769:llog_test_10()) 10f: wrote 64578 records then 958 failed with ENOSPC [ 3006.786853] Lustre: 25509:0:(llog_test.c:1812:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 3006.789013] Lustre: 25509:0:(llog_cat.c:869:llog_cat_process_or_fork()) catlog 0x11:10 crosses index zero [ 3006.992866] Lustre: 25509:0:(llog_test.c:1824:llog_test_10()) 10g: print the catalog entries.. we expect 3 [ 3006.995150] Lustre: 25528:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf35:0x0] in log [0xa:0x11:0x0] [ 3006.997325] Lustre: 25528:0:(llog_test.c:695:cat_print_cb()) Skipped 9 previous similar messages [ 3007.101737] Lustre: 25509:0:(llog_test.c:1851:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 3007.516795] Lustre: 25509:0:(llog_test.c:1863:llog_test_10()) 10g: print the catalog entries.. we expect 2 [ 3007.577569] Lustre: 25509:0:(llog_test.c:1898:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 3011.440634] Lustre: 25509:0:(llog_test.c:1910:llog_test_10()) 10g: print the catalog entries.. we expect 1 [ 3011.442948] Lustre: 25530:0:(llog_test.c:695:cat_print_cb()) seeing record at index 2 - [0x1:0x12f4:0x0] in log [0xa:0x11:0x0] [ 3011.446364] Lustre: 25530:0:(llog_test.c:695:cat_print_cb()) Skipped 4 previous similar messages [ 3011.448676] Lustre: 25509:0:(llog_test.c:1935:llog_test_10()) 10g: llh_cat_idx has also successfully wrapped! [ 3011.450747] Lustre: 25509:0:(llog_test.c:1941:llog_test_10()) 10: put newly-created catalog [ 3011.596218] Lustre: DEBUG MARKER: /usr/sbin/lctl dk [ 3011.936640] Lustre: DEBUG MARKER: which llog_reader 2> /dev/null [ 3012.196676] Lustre: DEBUG MARKER: ls -d /sbin/llog_reader [ 3012.549675] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3012.812785] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3012.943840] Lustre: Failing over lustre-MDT0000 [ 3016.106646] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.110845] LustreError: Skipped 4 previous similar messages [ 3018.994056] Lustre: 25898:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491244148/real 1491244148] req@ffff880039236a00 x1563680144402128/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491244154 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3019.285214] Lustre: server umount lustre-MDT0000 complete [ 3019.415533] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3019.735043] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3019.982989] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3020.112527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3020.250788] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d21/53 [ 3020.655765] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d21/53 [ 3020.910139] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d21/53 [ 3021.171602] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d22/54 [ 3021.424189] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d22/54 [ 3021.682170] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d23/55 [ 3021.933698] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d23/55 [ 3022.193908] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d24/56 [ 3022.447433] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/O/1/d24/56 [ 3022.736081] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3022.987922] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3023.257521] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3023.568702] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3023.851231] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3024.102329] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3024.349318] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3024.506100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.524650] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 3024.528064] Lustre: Skipped 14 previous similar messages [ 3024.576371] LustreError: 27341:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3024.586228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.735746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3024.997298] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3025.489616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.500467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.657175] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.657176] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.804853] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3026.062707] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3026.326430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3026.653401] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3027.277833] Lustre: DEBUG MARKER: rc=0; [ 3027.277833] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3027.277833] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3027.277833] echo $(hostname -s): $val; [ 3027.277833] rc=$val; [ 3027.277833] fi; [ 3027.277833] exit $rc [ 3027.894707] Lustre: DEBUG MARKER: dmesg [ 3028.338675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60aa: llog_print works with FIDs and simple names ===================================== 18:29:23 \(1491244163\) [ 3028.513146] Lustre: DEBUG MARKER: == sanity test 60aa: llog_print works with FIDs and simple names ===================================== 18:29:23 (1491244163) [ 3028.649943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3028.649943] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3028.649943] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3028.929122] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 3029.188172] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %MGS llog_print \$lustre-client [ 3029.317728] ------------[ cut here ]------------ [ 3029.319912] WARNING: at lib/vsprintf.c:1742 vsnprintf+0x691/0x6a0() [ 3029.322153] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iosf_mbi crc32_pclmul ghash_clmulni_intel ib_core aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon parport_pc parport i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common sysfillrect 8139too sysimgblt fb_sys_fops ttm crc32c_intel drm virtio_pci 8139cp mii virtio_ring virtio serio_raw i2c_core ata_piix libata floppy [last unloaded: llog_test] [ 3029.346488] CPU: 1 PID: 28602 Comm: llog_process_th Tainted: G OE ------------ 3.10.0-514.10.2.el7_lustre.x86_64 #1 [ 3029.351050] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 3029.353397] 0000000000000000 0000000087d9bcc4 ffff88007c0d3bd0 ffffffff8168663b [ 3029.355908] ffff88007c0d3c08 ffffffff81085940 ffff880077e8e0e0 ffff88007ac4ffad [ 3029.358372] ffff88007ac50000 ffff88007ac50004 ffff880077e8e0f0 ffff88007c0d3c18 [ 3029.360822] Call Trace: [ 3029.362773] [] dump_stack+0x19/0x1b [ 3029.364968] [] warn_slowpath_common+0x70/0xb0 [ 3029.367126] [] warn_slowpath_null+0x1a/0x20 [ 3029.369295] [] vsnprintf+0x691/0x6a0 [ 3029.371323] [] ? __brelse+0x3d/0x50 [ 3029.373362] [] snprintf+0x49/0x70 [ 3029.375324] [] ? osd_read+0x95/0xc0 [osd_ldiskfs] [ 3029.377410] [] class_config_yaml_output+0x3b7/0x7e0 [obdclass] [ 3029.379474] [] llog_print_cb+0x425/0x520 [obdclass] [ 3029.381506] [] llog_process_thread+0x5a5/0x1240 [obdclass] [ 3029.383543] [] llog_process_thread_daemonize+0x9d/0xe0 [obdclass] [ 3029.385683] [] ? llog_backup+0x500/0x500 [obdclass] [ 3029.387661] [] kthread+0xcf/0xe0 [ 3029.389545] [] ? kthread_create_on_node+0x140/0x140 [ 3029.391499] [] ret_from_fork+0x58/0x90 [ 3029.393432] [] ? kthread_create_on_node+0x140/0x140 [ 3029.395394] ---[ end trace bdc484d97e107437 ]--- [ 3029.397283] LustreError: 28602:0:(llog_ioctl.c:253:llog_print_cb()) not enough space for print log records [ 3029.531038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3029.531038] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3029.531038] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3029.809546] Lustre: DEBUG MARKER: /usr/sbin/lctl --device MGS llog_print lustre-client [ 3029.938180] LustreError: 28750:0:(llog_ioctl.c:253:llog_print_cb()) not enough space for print log records [ 3030.136869] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3030.774440] Lustre: DEBUG MARKER: rc=0; [ 3030.774440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3030.774440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3030.774440] echo $(hostname -s): $val; [ 3030.774440] rc=$val; [ 3030.774440] fi; [ 3030.774440] exit $rc [ 3031.377235] Lustre: DEBUG MARKER: dmesg [ 3031.826380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60b: limit repeated messages from CERROR\/CWARN ======================================== 18:29:26 \(1491244166\) [ 3031.984422] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 18:29:26 (1491244166) [ 3038.107154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3038.463165] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3038.714385] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3039.334881] Lustre: DEBUG MARKER: rc=0; [ 3039.334881] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3039.334881] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3039.334881] echo $(hostname -s): $val; [ 3039.334881] rc=$val; [ 3039.334881] fi; [ 3039.334881] exit $rc [ 3039.926907] Lustre: DEBUG MARKER: dmesg [ 3040.372541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60c: unlink file when mds full ======================================================== 18:29:35 \(1491244175\) [ 3040.546995] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 18:29:35 (1491244175) [ 3057.219892] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3057.842889] Lustre: DEBUG MARKER: rc=0; [ 3057.842889] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3057.842889] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3057.842889] echo $(hostname -s): $val; [ 3057.842889] rc=$val; [ 3057.842889] fi; [ 3057.842889] exit $rc [ 3058.461755] Lustre: DEBUG MARKER: dmesg [ 3058.987116] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60d: test printk console message masking ============================================== 18:29:54 \(1491244194\) [ 3059.409662] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 18:29:54 (1491244194) [ 3059.653557] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3060.270518] Lustre: DEBUG MARKER: rc=0; [ 3060.270518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3060.270518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3060.270518] echo $(hostname -s): $val; [ 3060.270518] rc=$val; [ 3060.270518] fi; [ 3060.270518] exit $rc [ 3060.882237] Lustre: DEBUG MARKER: dmesg [ 3061.319795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60e: no space while new llog is being created ========================================= 18:29:56 \(1491244196\) [ 3061.486566] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 18:29:56 (1491244196) [ 3061.678717] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x15b [ 3061.822195] Lustre: *** cfs_fail_loc=15b, val=0*** [ 3061.824203] Lustre: *** cfs_fail_loc=15b, val=0*** [ 3062.011160] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3062.636845] Lustre: DEBUG MARKER: rc=0; [ 3062.636845] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3062.636845] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3062.636845] echo $(hostname -s): $val; [ 3062.636845] rc=$val; [ 3062.636845] fi; [ 3062.636845] exit $rc [ 3063.243029] Lustre: DEBUG MARKER: dmesg [ 3063.694295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 61: mmap\(\) writes don\'t make sync hang ================================================ 18:29:58 \(1491244198\) [ 3063.854405] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 18:29:58 (1491244198) [ 3064.667143] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3065.278080] Lustre: DEBUG MARKER: rc=0; [ 3065.278080] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3065.278080] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3065.278080] echo $(hostname -s): $val; [ 3065.278080] rc=$val; [ 3065.278080] fi; [ 3065.278080] exit $rc [ 3065.891553] Lustre: DEBUG MARKER: dmesg [ 3066.337309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 18:30:01 \(1491244201\) [ 3066.502058] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 18:30:01 (1491244201) [ 3126.747309] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3127.367876] Lustre: DEBUG MARKER: rc=0; [ 3127.367876] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3127.367876] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3127.367876] echo $(hostname -s): $val; [ 3127.367876] rc=$val; [ 3127.367876] fi; [ 3127.367876] exit $rc [ 3127.983022] Lustre: DEBUG MARKER: dmesg [ 3128.422388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63b: async write errors should be returned to fsync =================================== 18:31:03 \(1491244263\) [ 3128.580574] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 18:31:03 (1491244263) [ 3128.715470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3132.160610] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3132.502953] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3133.132437] Lustre: DEBUG MARKER: rc=0; [ 3133.132437] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3133.132437] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3133.132437] echo $(hostname -s): $val; [ 3133.132437] rc=$val; [ 3133.132437] fi; [ 3133.132437] exit $rc [ 3133.734655] Lustre: DEBUG MARKER: dmesg [ 3134.170130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64a: verify filter grant calculations \(in kernel\) ===================================== 18:31:09 \(1491244269\) [ 3134.331827] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 18:31:09 (1491244269) [ 3134.539214] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3135.155677] Lustre: DEBUG MARKER: rc=0; [ 3135.155677] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3135.155677] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3135.155677] echo $(hostname -s): $val; [ 3135.155677] rc=$val; [ 3135.155677] fi; [ 3135.155677] exit $rc [ 3135.752469] Lustre: DEBUG MARKER: dmesg [ 3136.179354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_64b skipping SLOW test 64b [ 3136.352930] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 3136.579119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64c: verify grant shrink ========================------ =============================== 18:31:11 \(1491244271\) [ 3136.743681] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ =============================== 18:31:11 (1491244271) [ 3136.939393] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3137.553891] Lustre: DEBUG MARKER: rc=0; [ 3137.553891] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3137.553891] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3137.553891] echo $(hostname -s): $val; [ 3137.553891] rc=$val; [ 3137.553891] fi; [ 3137.553891] exit $rc [ 3138.155571] Lustre: DEBUG MARKER: dmesg [ 3138.633190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65a: directory with no stripe info ==================================================== 18:31:13 \(1491244273\) [ 3138.805397] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 18:31:13 (1491244273) [ 3139.094949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3139.711585] Lustre: DEBUG MARKER: rc=0; [ 3139.711585] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3139.711585] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3139.711585] echo $(hostname -s): $val; [ 3139.711585] rc=$val; [ 3139.711585] fi; [ 3139.711585] exit $rc [ 3140.309889] Lustre: DEBUG MARKER: dmesg [ 3140.750436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 18:31:15 \(1491244275\) [ 3140.905657] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 18:31:15 (1491244275) [ 3141.128189] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3141.741788] Lustre: DEBUG MARKER: rc=0; [ 3141.741788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3141.741788] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3141.741788] echo $(hostname -s): $val; [ 3141.741788] rc=$val; [ 3141.741788] fi; [ 3141.741788] exit $rc [ 3142.335276] Lustre: DEBUG MARKER: dmesg [ 3142.775357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 6 =================================== 18:31:17 \(1491244277\) [ 3142.936733] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 6 =================================== 18:31:17 (1491244277) [ 3143.154580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3143.765823] Lustre: DEBUG MARKER: rc=0; [ 3143.765823] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3143.765823] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3143.765823] echo $(hostname -s): $val; [ 3143.765823] rc=$val; [ 3143.765823] fi; [ 3143.765823] exit $rc [ 3144.375646] Lustre: DEBUG MARKER: dmesg [ 3144.818340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 18:31:19 \(1491244279\) [ 3144.974768] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 18:31:19 (1491244279) [ 3145.204860] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3145.823213] Lustre: DEBUG MARKER: rc=0; [ 3145.823213] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3145.823213] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3145.823213] echo $(hostname -s): $val; [ 3145.823213] rc=$val; [ 3145.823213] fi; [ 3145.823213] exit $rc [ 3146.417889] Lustre: DEBUG MARKER: dmesg [ 3146.854799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65e: directory setstripe defaults ===================================================== 18:31:21 \(1491244281\) [ 3147.017607] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 18:31:21 (1491244281) [ 3147.245186] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3147.863621] Lustre: DEBUG MARKER: rc=0; [ 3147.863621] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3147.863621] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3147.863621] echo $(hostname -s): $val; [ 3147.863621] rc=$val; [ 3147.863621] fi; [ 3147.863621] exit $rc [ 3148.466664] Lustre: DEBUG MARKER: dmesg [ 3148.907517] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65f: dir setstripe permission \(should return error\) =================================== 18:31:24 \(1491244284\) [ 3149.066659] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 18:31:24 (1491244284) [ 3149.350343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3149.970540] Lustre: DEBUG MARKER: rc=0; [ 3149.970540] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3149.970540] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3149.970540] echo $(hostname -s): $val; [ 3149.970540] rc=$val; [ 3149.970540] fi; [ 3149.970540] exit $rc [ 3150.586026] Lustre: DEBUG MARKER: dmesg [ 3151.081037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65g: directory setstripe -d =========================================================== 18:31:26 \(1491244286\) [ 3151.240296] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 18:31:26 (1491244286) [ 3151.471842] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3152.127523] Lustre: DEBUG MARKER: rc=0; [ 3152.127523] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3152.127523] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3152.127523] echo $(hostname -s): $val; [ 3152.127523] rc=$val; [ 3152.127523] fi; [ 3152.127523] exit $rc [ 3152.748301] Lustre: DEBUG MARKER: dmesg [ 3153.190604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65h: directory stripe info inherit ==================================================== 18:31:28 \(1491244288\) [ 3153.351709] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 18:31:28 (1491244288) [ 3153.570589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3154.198655] Lustre: DEBUG MARKER: rc=0; [ 3154.198655] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3154.198655] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3154.198655] echo $(hostname -s): $val; [ 3154.198655] rc=$val; [ 3154.198655] fi; [ 3154.198655] exit $rc [ 3154.811594] Lustre: DEBUG MARKER: dmesg [ 3155.246841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65i: set non-default striping on root directory \(bug 6367\)============================= 18:31:30 \(1491244290\) [ 3155.462500] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 18:31:30 (1491244290) [ 3155.477934] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 3155.480172] Lustre: Skipped 5 previous similar messages [ 3155.675447] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3156.299245] Lustre: DEBUG MARKER: rc=0; [ 3156.299245] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3156.299245] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3156.299245] echo $(hostname -s): $val; [ 3156.299245] rc=$val; [ 3156.299245] fi; [ 3156.299245] exit $rc [ 3156.912062] Lustre: DEBUG MARKER: dmesg [ 3157.352472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ia: getstripe on -1 default directory striping ====================================== 18:31:32 \(1491244292\) [ 3157.509430] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 18:31:32 (1491244292) [ 3158.175602] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3158.794731] Lustre: DEBUG MARKER: rc=0; [ 3158.794731] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3158.794731] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3158.794731] echo $(hostname -s): $val; [ 3158.794731] rc=$val; [ 3158.794731] fi; [ 3158.794731] exit $rc [ 3159.411765] Lustre: DEBUG MARKER: dmesg [ 3159.863287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 18:31:34 \(1491244294\) [ 3160.034507] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 18:31:34 (1491244294) [ 3160.638181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3161.156587] LustreError: 2670:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripecount': rc = -34 [ 3161.251832] Lustre: DEBUG MARKER: rc=0; [ 3161.251832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3161.251832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3161.251832] echo $(hostname -s): $val; [ 3161.251832] rc=$val; [ 3161.251832] fi; [ 3161.251832] exit $rc [ 3161.864467] Lustre: DEBUG MARKER: dmesg [ 3162.314693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ic: new find on -1 default directory striping ======================================= 18:31:37 \(1491244297\) [ 3162.478969] Lustre: DEBUG MARKER: == sanity test 65ic: new find on -1 default directory striping ======================================= 18:31:37 (1491244297) [ 3165.050861] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3165.666491] Lustre: DEBUG MARKER: rc=0; [ 3165.666491] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3165.666491] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3165.666491] echo $(hostname -s): $val; [ 3165.666491] rc=$val; [ 3165.666491] fi; [ 3165.666491] exit $rc [ 3166.279417] Lustre: DEBUG MARKER: dmesg [ 3166.744655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65j: set default striping on root directory \(bug 6367\)================================= 18:31:41 \(1491244301\) [ 3166.909942] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 18:31:41 (1491244301) [ 3167.963442] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 3167.965714] Lustre: Skipped 5 previous similar messages [ 3168.171482] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3168.786405] Lustre: DEBUG MARKER: rc=0; [ 3168.786405] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3168.786405] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3168.786405] echo $(hostname -s): $val; [ 3168.786405] rc=$val; [ 3168.786405] fi; [ 3168.786405] exit $rc [ 3169.395887] Lustre: DEBUG MARKER: dmesg [ 3169.885255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 18:31:44 \(1491244304\) [ 3170.042435] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 18:31:44 (1491244304) [ 3170.196647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3170.196647] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3170.196647] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3170.472722] Lustre: DEBUG MARKER: lctl dl [ 3170.729345] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 3170.857335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3170.861426] Lustre: Skipped 6 previous similar messages [ 3170.872377] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 3170.874540] Lustre: Skipped 13 previous similar messages [ 3171.004410] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 3171.162847] LustreError: 3786:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3171.166786] LustreError: 3786:0:(obd_config.c:1392:class_process_proc_param()) Skipped 1 previous similar message [ 3171.260314] Lustre: DEBUG MARKER: lctl --device %lustre-OST0002-osc-MDT0000 activate [ 3171.519819] Lustre: DEBUG MARKER: lctl --device %lustre-OST0003-osc-MDT0000 activate [ 3171.779230] Lustre: DEBUG MARKER: lctl --device %lustre-OST0004-osc-MDT0000 activate [ 3172.035791] Lustre: DEBUG MARKER: lctl --device %lustre-OST0005-osc-MDT0000 activate [ 3172.293352] Lustre: DEBUG MARKER: lctl --device %lustre-OST0006-osc-MDT0000 activate [ 3172.566869] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3174.509280] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 deactivate [ 3174.640453] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3174.642593] Lustre: Skipped 3 previous similar messages [ 3174.774965] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000*MDT0000.create_count [ 3175.033891] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000*MDT0000.max_create_count [ 3175.300933] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.max_create_count=0 [ 3177.270775] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.max_create_count=20000 [ 3177.528092] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.create_count=32 [ 3177.785041] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 3177.919233] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3177.923305] LustreError: Skipped 3 previous similar messages [ 3178.069211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3178.069211] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3178.069211] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3178.359492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3178.359492] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3178.359492] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3178.648640] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3178.983815] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3179.244654] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3179.736875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.*OST*-osc-MDT0000.ost_server_uuid 40 [ 3179.739663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.*OST*-osc-MDT0000.ost_server_uuid 40 [ 3179.895978] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.*OST*-osc-MDT0000.ost_server_uuid 40 [ 3179.901490] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.*OST*-osc-MDT0000.ost_server_uuid 40 [ 3180.086764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3180.090686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3180.247300] Lustre: DEBUG MARKER: osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3180.247302] Lustre: DEBUG MARKER: osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3180.394637] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3182.314055] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 3182.442825] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3182.576805] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0001*MDT0000.create_count [ 3182.833437] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0001*MDT0000.max_create_count [ 3183.094295] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.max_create_count=0 [ 3185.074317] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.max_create_count=20000 [ 3185.332466] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.create_count=32 [ 3185.590751] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 3185.721445] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3185.725984] Lustre: Skipped 7 previous similar messages [ 3185.731503] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3185.893074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3185.893074] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3185.893074] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3186.173145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3186.173145] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3186.173145] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3186.449711] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3186.707733] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3186.970307] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3187.463803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3187.463805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3187.625794] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3187.634395] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3187.822332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3187.825455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3187.989312] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3187.990821] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3188.136170] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3190.101770] Lustre: DEBUG MARKER: lctl --device %lustre-OST0002-osc-MDT0000 deactivate [ 3190.232118] Lustre: setting import lustre-OST0002_UUID INACTIVE by administrator request [ 3190.364391] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0002*MDT0000.create_count [ 3190.621801] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0002*MDT0000.max_create_count [ 3190.878587] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0002*MDT0000.max_create_count=0 [ 3192.887366] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0002*MDT0000.max_create_count=20000 [ 3193.148421] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0002*MDT0000.create_count=32 [ 3193.407229] Lustre: DEBUG MARKER: lctl --device %lustre-OST0002-osc-MDT0000 activate [ 3193.539438] LustreError: 167-0: lustre-OST0002-osc-MDT0000: This client was evicted by lustre-OST0002; in progress operations using this service will fail. [ 3193.683841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3193.683841] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3193.683841] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3193.965499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3193.965499] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3193.965499] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3194.240430] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3194.508413] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3194.769505] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3195.260054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3195.262314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3195.424131] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3195.424133] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3195.625452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3195.625454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3195.790927] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3195.793826] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3195.936007] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3197.806948] Lustre: DEBUG MARKER: lctl --device %lustre-OST0003-osc-MDT0000 deactivate [ 3197.935897] Lustre: setting import lustre-OST0003_UUID INACTIVE by administrator request [ 3198.069920] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0003*MDT0000.create_count [ 3198.328744] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0003*MDT0000.max_create_count [ 3198.595861] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0003*MDT0000.max_create_count=0 [ 3200.572205] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0003*MDT0000.max_create_count=20000 [ 3200.844268] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0003*MDT0000.create_count=32 [ 3201.104917] Lustre: DEBUG MARKER: lctl --device %lustre-OST0003-osc-MDT0000 activate [ 3201.242021] LustreError: 167-0: lustre-OST0003-osc-MDT0000: This client was evicted by lustre-OST0003; in progress operations using this service will fail. [ 3201.388066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3201.388066] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3201.388066] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3201.668400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3201.668400] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3201.668400] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3201.961517] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3202.221764] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3202.485770] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3202.977663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [ 3202.982227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [ 3203.138996] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [ 3203.141431] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [ 3203.334302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3203.334486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3203.502530] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3203.506240] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3203.652610] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3205.604345] Lustre: DEBUG MARKER: lctl --device %lustre-OST0004-osc-MDT0000 deactivate [ 3205.731762] Lustre: setting import lustre-OST0004_UUID INACTIVE by administrator request [ 3205.865392] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0004*MDT0000.create_count [ 3206.125164] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0004*MDT0000.max_create_count [ 3206.387086] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0004*MDT0000.max_create_count=0 [ 3208.398543] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0004*MDT0000.max_create_count=20000 [ 3208.666303] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0004*MDT0000.create_count=32 [ 3208.924425] Lustre: DEBUG MARKER: lctl --device %lustre-OST0004-osc-MDT0000 activate [ 3209.054659] Lustre: lustre-OST0004-osc-MDT0000: Connection to lustre-OST0004 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3209.059489] Lustre: Skipped 2 previous similar messages [ 3209.067875] LustreError: 167-0: lustre-OST0004-osc-MDT0000: This client was evicted by lustre-OST0004; in progress operations using this service will fail. [ 3209.072626] Lustre: lustre-OST0004-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 3209.075448] Lustre: Skipped 10 previous similar messages [ 3209.213192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3209.213192] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3209.213192] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3209.490857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3209.490857] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3209.490857] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3209.768106] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3210.027502] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3210.288524] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3210.781595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [ 3210.783270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [ 3210.946642] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [ 3210.947116] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [ 3211.136926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3211.137086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3211.300314] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3211.310999] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3211.455417] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3213.413483] Lustre: DEBUG MARKER: lctl --device %lustre-OST0005-osc-MDT0000 deactivate [ 3213.686769] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0005*MDT0000.create_count [ 3213.948368] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0005*MDT0000.max_create_count [ 3214.211154] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0005*MDT0000.max_create_count=0 [ 3216.166468] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0005*MDT0000.max_create_count=20000 [ 3216.424422] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0005*MDT0000.create_count=32 [ 3216.682351] Lustre: DEBUG MARKER: lctl --device %lustre-OST0005-osc-MDT0000 activate [ 3216.952949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3216.952949] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3216.952949] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3217.233600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3217.233600] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3217.233600] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3217.510164] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3217.771195] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3218.063101] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3218.561752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [ 3218.565154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [ 3218.730398] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [ 3218.730399] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [ 3218.922427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3218.928047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.088635] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.088637] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.236414] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 3221.233929] Lustre: DEBUG MARKER: lctl --device %lustre-OST0006-osc-MDT0000 deactivate [ 3221.373652] Lustre: setting import lustre-OST0006_UUID INACTIVE by administrator request [ 3221.376322] Lustre: Skipped 1 previous similar message [ 3221.514123] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0006*MDT0000.create_count [ 3221.787108] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0006*MDT0000.max_create_count [ 3222.060964] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0006*MDT0000.max_create_count=0 [ 3224.049246] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0006*MDT0000.max_create_count=20000 [ 3224.304576] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0006*MDT0000.create_count=32 [ 3224.565876] Lustre: DEBUG MARKER: lctl --device %lustre-OST0006-osc-MDT0000 activate [ 3224.699299] LustreError: 167-0: lustre-OST0006-osc-MDT0000: This client was evicted by lustre-OST0006; in progress operations using this service will fail. [ 3224.703787] LustreError: Skipped 1 previous similar message [ 3224.856328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3224.856328] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3224.856328] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3225.136666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3225.136666] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3225.136666] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3225.414727] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3225.681695] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 3225.946228] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3226.438090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [ 3226.444561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [ 3226.603888] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [ 3226.617694] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [ 3226.797733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3226.802348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3226.965337] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3226.965338] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3227.166092] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3227.832290] Lustre: DEBUG MARKER: rc=0; [ 3227.832290] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3227.832290] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3227.832290] echo $(hostname -s): $val; [ 3227.832290] rc=$val; [ 3227.832290] fi; [ 3227.832290] exit $rc [ 3228.458106] Lustre: DEBUG MARKER: dmesg [ 3228.948159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65l: lfs find on -1 stripe dir ======================================================== 18:32:44 \(1491244364\) [ 3229.117846] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 18:32:44 (1491244364) [ 3229.339401] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3229.976464] Lustre: DEBUG MARKER: rc=0; [ 3229.976464] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3229.976464] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3229.976464] echo $(hostname -s): $val; [ 3229.976464] rc=$val; [ 3229.976464] fi; [ 3229.976464] exit $rc [ 3230.595135] Lustre: DEBUG MARKER: dmesg [ 3231.049934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65m: normal user can\'t set filesystem default stripe ================================== 18:32:46 \(1491244366\) [ 3231.206328] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 18:32:46 (1491244366) [ 3231.405846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3232.022357] Lustre: DEBUG MARKER: rc=0; [ 3232.022357] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3232.022357] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3232.022357] echo $(hostname -s): $val; [ 3232.022357] rc=$val; [ 3232.022357] fi; [ 3232.022357] exit $rc [ 3232.625179] Lustre: DEBUG MARKER: dmesg [ 3233.065796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 66: update inode blocks count on client =============================================== 18:32:48 \(1491244368\) [ 3233.224465] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 18:32:48 (1491244368) [ 3233.450551] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 3234.139506] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 3234.811933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3235.428209] Lustre: DEBUG MARKER: rc=0; [ 3235.428209] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3235.428209] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3235.428209] echo $(hostname -s): $val; [ 3235.428209] rc=$val; [ 3235.428209] fi; [ 3235.428209] exit $rc [ 3236.034294] Lustre: DEBUG MARKER: dmesg [ 3236.478137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 69: verify oa2dentry return -ENOENT doesn\'t LBUG ====================================== 18:32:51 \(1491244371\) [ 3236.645171] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 18:32:51 (1491244371) [ 3238.646359] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3239.270799] Lustre: DEBUG MARKER: rc=0; [ 3239.270799] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3239.270799] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3239.270799] echo $(hostname -s): $val; [ 3239.270799] rc=$val; [ 3239.270799] fi; [ 3239.270799] exit $rc [ 3239.865990] Lustre: DEBUG MARKER: dmesg [ 3240.313309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_71 skipping SLOW test 71 [ 3240.475301] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 3240.688266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72a: Test that remove suid works properly \(bug5695\) =================================== 18:32:55 \(1491244375\) [ 3240.858237] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 18:32:55 (1491244375) [ 3241.145432] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3241.775868] Lustre: DEBUG MARKER: rc=0; [ 3241.775868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3241.775868] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3241.775868] echo $(hostname -s): $val; [ 3241.775868] rc=$val; [ 3241.775868] fi; [ 3241.775868] exit $rc [ 3242.377631] Lustre: DEBUG MARKER: dmesg [ 3242.817555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72b: Test that we keep mode setting if without file data changed \(bug 24226\) ========== 18:32:57 \(1491244377\) [ 3242.982853] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 18:32:57 (1491244377) [ 3243.304379] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3243.932830] Lustre: DEBUG MARKER: rc=0; [ 3243.932830] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3243.932830] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3243.932830] echo $(hostname -s): $val; [ 3243.932830] rc=$val; [ 3243.932830] fi; [ 3243.932830] exit $rc [ 3244.537592] Lustre: DEBUG MARKER: dmesg [ 3244.976876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 73: multiple MDC requests \(should not deadlock\) ======================================= 18:33:00 \(1491244380\) [ 3245.134836] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 18:33:00 (1491244380) [ 3271.372344] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3271.994522] Lustre: DEBUG MARKER: rc=0; [ 3271.994522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3271.994522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3271.994522] echo $(hostname -s): $val; [ 3271.994522] rc=$val; [ 3271.994522] fi; [ 3271.994522] exit $rc [ 3272.595345] Lustre: DEBUG MARKER: dmesg [ 3273.048024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74a: ldlm_enqueue freed-export error path, ls \(shouldn\'t LBUG\) ======================== 18:33:28 \(1491244408\) [ 3273.213627] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 18:33:28 (1491244408) [ 3273.428207] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3274.052377] Lustre: DEBUG MARKER: rc=0; [ 3274.052377] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3274.052377] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3274.052377] echo $(hostname -s): $val; [ 3274.052377] rc=$val; [ 3274.052377] fi; [ 3274.052377] exit $rc [ 3274.654818] Lustre: DEBUG MARKER: dmesg [ 3275.103417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74b: ldlm_enqueue freed-export error path, touch \(shouldn\'t LBUG\) ===================== 18:33:30 \(1491244410\) [ 3275.259727] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 18:33:30 (1491244410) [ 3275.470488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3276.087865] Lustre: DEBUG MARKER: rc=0; [ 3276.087865] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3276.087865] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3276.087865] echo $(hostname -s): $val; [ 3276.087865] rc=$val; [ 3276.087865] fi; [ 3276.087865] exit $rc [ 3276.696126] Lustre: DEBUG MARKER: dmesg [ 3277.154308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74c: ldlm_lock_create error path, \(shouldn\'t LBUG\) ==================================== 18:33:32 \(1491244412\) [ 3277.314450] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 18:33:32 (1491244412) [ 3277.513094] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3278.139093] Lustre: DEBUG MARKER: rc=0; [ 3278.139093] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3278.139093] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3278.139093] echo $(hostname -s): $val; [ 3278.139093] rc=$val; [ 3278.139093] fi; [ 3278.139093] exit $rc [ 3278.751016] Lustre: DEBUG MARKER: dmesg [ 3279.197216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 3279.361224] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 3279.576927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77a: normal checksum read\/write operation ============================================= 18:33:34 \(1491244414\) [ 3279.733950] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 18:33:34 (1491244414) [ 3280.767829] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3281.389767] Lustre: DEBUG MARKER: rc=0; [ 3281.389767] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3281.389767] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3281.389767] echo $(hostname -s): $val; [ 3281.389767] rc=$val; [ 3281.389767] fi; [ 3281.389767] exit $rc [ 3281.993147] Lustre: DEBUG MARKER: dmesg [ 3282.444688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77b: checksum error on client write, read ============================================= 18:33:37 \(1491244417\) [ 3282.608258] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 18:33:37 (1491244417) [ 3284.451584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 3284.606357] Lustre: DEBUG MARKER: set checksum type to crc32 [ 3286.202878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 3286.355107] Lustre: DEBUG MARKER: set checksum type to adler [ 3287.967642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 3288.131075] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3289.637104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 3289.795288] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3290.007913] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3290.642502] Lustre: DEBUG MARKER: rc=0; [ 3290.642502] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3290.642502] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3290.642502] echo $(hostname -s): $val; [ 3290.642502] rc=$val; [ 3290.642502] fi; [ 3290.642502] exit $rc [ 3291.355134] Lustre: DEBUG MARKER: dmesg [ 3291.906509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77d: checksum error on OST direct write, read ========================================= 18:33:47 \(1491244427\) [ 3292.071318] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 18:33:47 (1491244427) [ 3294.644452] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3295.260884] Lustre: DEBUG MARKER: rc=0; [ 3295.260884] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3295.260884] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3295.260884] echo $(hostname -s): $val; [ 3295.260884] rc=$val; [ 3295.260884] fi; [ 3295.260884] exit $rc [ 3295.859925] Lustre: DEBUG MARKER: dmesg [ 3296.309317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77f: repeat checksum error on write \(expect error\) ==================================== 18:33:51 \(1491244431\) [ 3296.464877] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 18:33:51 (1491244431) [ 3296.670648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 3296.833702] Lustre: DEBUG MARKER: set checksum type to crc32 [ 3352.813432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 3352.973224] Lustre: DEBUG MARKER: set checksum type to adler [ 3408.598665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 3408.766777] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3457.626123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 3457.791737] Lustre: DEBUG MARKER: set checksum type to crc32c [ 3457.984554] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3458.616277] Lustre: DEBUG MARKER: rc=0; [ 3458.616277] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3458.616277] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3458.616277] echo $(hostname -s): $val; [ 3458.616277] rc=$val; [ 3458.616277] fi; [ 3458.616277] exit $rc [ 3459.239163] Lustre: DEBUG MARKER: dmesg [ 3459.711473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77g: checksum error on OST write, read ================================================ 18:36:34 \(1491244594\) [ 3459.870410] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 18:36:34 (1491244594) [ 3463.548772] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3464.167368] Lustre: DEBUG MARKER: rc=0; [ 3464.167368] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3464.167368] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3464.167368] echo $(hostname -s): $val; [ 3464.167368] rc=$val; [ 3464.167368] fi; [ 3464.167368] exit $rc [ 3464.766526] Lustre: DEBUG MARKER: dmesg [ 3465.209206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77j: client only supporting ADLER32 =================================================== 18:36:40 \(1491244600\) [ 3465.372339] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 18:36:40 (1491244600) [ 3479.083245] Lustre: MGS: Connection restored to f12920f1-3d5b-1e78-d777-e27fb9937371 (at 10.9.5.237@tcp) [ 3479.085417] Lustre: Skipped 2 previous similar messages [ 3479.505930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3479.663017] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3479.855153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3480.010861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3480.389513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3480.549797] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3480.744085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3480.903359] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3481.275698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3481.439141] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3481.628445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3481.787486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3482.174311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3482.343097] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3482.534377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3482.692510] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3483.074557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3483.237512] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3483.424883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3483.578183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3483.953778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3484.119690] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3484.303784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3484.461400] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3484.852420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uui [ 3485.011915] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0001-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0002-osc-ffff880041e28800.ost_server_uuid osc.lustre-OST0003 [ 3485.197290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3485.351479] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880041e28800.ost_server_uuid in FULL state after 0 sec [ 3485.798744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3486.459281] Lustre: DEBUG MARKER: rc=0; [ 3486.459281] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3486.459281] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3486.459281] echo $(hostname -s): $val; [ 3486.459281] rc=$val; [ 3486.459281] fi; [ 3486.459281] exit $rc [ 3487.078005] Lustre: DEBUG MARKER: dmesg [ 3487.532110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 78: handle large O_DIRECT writes correctly ============================================ 18:37:02 \(1491244622\) [ 3487.696757] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 18:37:02 (1491244622) [ 3490.271618] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3490.898036] Lustre: DEBUG MARKER: rc=0; [ 3490.898036] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3490.898036] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3490.898036] echo $(hostname -s): $val; [ 3490.898036] rc=$val; [ 3490.898036] fi; [ 3490.898036] exit $rc [ 3491.515780] Lustre: DEBUG MARKER: dmesg [ 3491.962490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 79: df report consistency check ======================================================= 18:37:07 \(1491244627\) [ 3492.121806] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 18:37:07 (1491244627) [ 3492.270566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3492.534657] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3492.830021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3494.111982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3495.392408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3496.684642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3497.978208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3499.272500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3500.559438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3501.843562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3503.128224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3504.514039] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3505.135545] Lustre: DEBUG MARKER: rc=0; [ 3505.135545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3505.135545] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3505.135545] echo $(hostname -s): $val; [ 3505.135545] rc=$val; [ 3505.135545] fi; [ 3505.135545] exit $rc [ 3505.748343] Lustre: DEBUG MARKER: dmesg [ 3506.190844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 80: Page eviction is equally fast at high offsets too ================================ 18:37:21 \(1491244641\) [ 3506.351104] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 18:37:21 (1491244641) [ 3507.910301] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3508.535738] Lustre: DEBUG MARKER: rc=0; [ 3508.535738] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3508.535738] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3508.535738] echo $(hostname -s): $val; [ 3508.535738] rc=$val; [ 3508.535738] fi; [ 3508.535738] exit $rc [ 3509.138762] Lustre: DEBUG MARKER: dmesg [ 3509.594610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 18:37:24 \(1491244644\) [ 3509.759221] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 18:37:24 (1491244644) [ 3510.311574] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3510.928479] Lustre: DEBUG MARKER: rc=0; [ 3510.928479] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3510.928479] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3510.928479] echo $(hostname -s): $val; [ 3510.928479] rc=$val; [ 3510.928479] fi; [ 3510.928479] exit $rc [ 3511.535101] Lustre: DEBUG MARKER: dmesg [ 3512.000108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 18:37:27 \(1491244647\) [ 3512.169233] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 18:37:27 (1491244647) [ 3512.697924] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3513.313961] Lustre: DEBUG MARKER: rc=0; [ 3513.313961] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3513.313961] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3513.313961] echo $(hostname -s): $val; [ 3513.313961] rc=$val; [ 3513.313961] fi; [ 3513.313961] exit $rc [ 3513.916675] Lustre: DEBUG MARKER: dmesg [ 3514.382605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 82: Basic grouplock test ============================================================== 18:37:29 \(1491244649\) [ 3514.541609] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 18:37:29 (1491244649) [ 3516.974609] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3517.602833] Lustre: DEBUG MARKER: rc=0; [ 3517.602833] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3517.602833] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3517.602833] echo $(hostname -s): $val; [ 3517.602833] rc=$val; [ 3517.602833] fi; [ 3517.602833] exit $rc [ 3518.202016] Lustre: DEBUG MARKER: dmesg [ 3518.664175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99a: cvs init ========================================================================= 18:37:33 \(1491244653\) [ 3518.830088] Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 18:37:33 (1491244653) [ 3519.685434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3520.297735] Lustre: DEBUG MARKER: rc=0; [ 3520.297735] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3520.297735] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3520.297735] echo $(hostname -s): $val; [ 3520.297735] rc=$val; [ 3520.297735] fi; [ 3520.297735] exit $rc [ 3520.903663] Lustre: DEBUG MARKER: dmesg [ 3521.376794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99b: cvs import ======================================================================= 18:37:36 \(1491244656\) [ 3521.542233] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 18:37:36 (1491244656) [ 3521.806080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3522.428237] Lustre: DEBUG MARKER: rc=0; [ 3522.428237] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3522.428237] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3522.428237] echo $(hostname -s): $val; [ 3522.428237] rc=$val; [ 3522.428237] fi; [ 3522.428237] exit $rc [ 3523.044339] Lustre: DEBUG MARKER: dmesg [ 3523.515100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99c: cvs checkout ===================================================================== 18:37:38 \(1491244658\) [ 3523.683311] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 18:37:38 (1491244658) [ 3524.897646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3525.513742] Lustre: DEBUG MARKER: rc=0; [ 3525.513742] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3525.513742] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3525.513742] echo $(hostname -s): $val; [ 3525.513742] rc=$val; [ 3525.513742] fi; [ 3525.513742] exit $rc [ 3526.130258] Lustre: DEBUG MARKER: dmesg [ 3526.594418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99d: cvs add ========================================================================== 18:37:41 \(1491244661\) [ 3526.761663] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 18:37:41 (1491244661) [ 3527.010085] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3527.655054] Lustre: DEBUG MARKER: rc=0; [ 3527.655054] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3527.655054] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3527.655054] echo $(hostname -s): $val; [ 3527.655054] rc=$val; [ 3527.655054] fi; [ 3527.655054] exit $rc [ 3528.252971] Lustre: DEBUG MARKER: dmesg [ 3528.710998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99e: cvs update ======================================================================= 18:37:43 \(1491244663\) [ 3528.885653] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 18:37:43 (1491244663) [ 3529.142816] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3529.761814] Lustre: DEBUG MARKER: rc=0; [ 3529.761814] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3529.761814] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3529.761814] echo $(hostname -s): $val; [ 3529.761814] rc=$val; [ 3529.761814] fi; [ 3529.761814] exit $rc [ 3530.357347] Lustre: DEBUG MARKER: dmesg [ 3530.839796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99f: cvs commit ======================================================================= 18:37:45 \(1491244665\) [ 3531.001226] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 18:37:45 (1491244665) [ 3531.944309] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3532.557521] Lustre: DEBUG MARKER: rc=0; [ 3532.557521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3532.557521] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3532.557521] echo $(hostname -s): $val; [ 3532.557521] rc=$val; [ 3532.557521] fi; [ 3532.557521] exit $rc [ 3533.151902] Lustre: DEBUG MARKER: dmesg [ 3533.606560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 100: check local port using privileged port =========================================== 18:37:48 \(1491244668\) [ 3533.767788] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 18:37:48 (1491244668) [ 3534.535794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3535.154147] Lustre: DEBUG MARKER: rc=0; [ 3535.154147] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3535.154147] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3535.154147] echo $(hostname -s): $val; [ 3535.154147] rc=$val; [ 3535.154147] fi; [ 3535.154147] exit $rc [ 3535.762976] Lustre: DEBUG MARKER: dmesg [ 3536.239672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101a: check read-ahead for random reads =============================================== 18:37:51 \(1491244671\) [ 3536.397209] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 18:37:51 (1491244671) [ 3563.008918] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3563.702259] Lustre: DEBUG MARKER: rc=0; [ 3563.702259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3563.702259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3563.702259] echo $(hostname -s): $val; [ 3563.702259] rc=$val; [ 3563.702259] fi; [ 3563.702259] exit $rc [ 3564.315965] Lustre: DEBUG MARKER: dmesg [ 3564.795677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101b: check stride-io mode read-ahead ================================================= 18:38:19 \(1491244699\) [ 3564.957615] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 18:38:19 (1491244699) [ 3572.169943] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3572.798570] Lustre: DEBUG MARKER: rc=0; [ 3572.798570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3572.798570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3572.798570] echo $(hostname -s): $val; [ 3572.798570] rc=$val; [ 3572.798570] fi; [ 3572.798570] exit $rc [ 3573.422457] Lustre: DEBUG MARKER: dmesg [ 3573.880384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101c: check stripe_size aligned read-ahead ============================================ 18:38:28 \(1491244708\) [ 3574.077305] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 18:38:28 (1491244708) [ 3584.702930] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3585.330162] Lustre: DEBUG MARKER: rc=0; [ 3585.330162] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3585.330162] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3585.330162] echo $(hostname -s): $val; [ 3585.330162] rc=$val; [ 3585.330162] fi; [ 3585.330162] exit $rc [ 3585.954075] Lustre: DEBUG MARKER: dmesg [ 3586.409478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101d: file read with and without read-ahead enabled =================================== 18:38:41 \(1491244721\) [ 3586.576898] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 18:38:41 (1491244721) [ 3729.710762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3729.980725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3730.360512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3731.647322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3732.924757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3734.204822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3735.488981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3736.775722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3738.078149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3739.365763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3740.654114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3742.008619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3742.643518] Lustre: DEBUG MARKER: rc=0; [ 3742.643518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3742.643518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3742.643518] echo $(hostname -s): $val; [ 3742.643518] rc=$val; [ 3742.643518] fi; [ 3742.643518] exit $rc [ 3743.259791] Lustre: DEBUG MARKER: dmesg [ 3743.714195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101e: check read-ahead for small read\(1k\) for small files\(500k\) ======================= 18:41:18 \(1491244878\) [ 3743.871038] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 18:41:18 (1491244878) [ 3752.024271] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3752.649789] Lustre: DEBUG MARKER: rc=0; [ 3752.649789] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3752.649789] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3752.649789] echo $(hostname -s): $val; [ 3752.649789] rc=$val; [ 3752.649789] fi; [ 3752.649789] exit $rc [ 3753.254490] Lustre: DEBUG MARKER: dmesg [ 3753.756663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101f: check mmap read performance ===================================================== 18:41:28 \(1491244888\) [ 3753.926539] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 18:41:28 (1491244888) [ 3759.580037] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3760.195113] Lustre: DEBUG MARKER: rc=0; [ 3760.195113] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3760.195113] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3760.195113] echo $(hostname -s): $val; [ 3760.195113] rc=$val; [ 3760.195113] fi; [ 3760.195113] exit $rc [ 3760.798198] Lustre: DEBUG MARKER: dmesg [ 3761.257172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101g: Big bulk\(4\/16 MiB\) readahead ==================================================== 18:41:36 \(1491244896\) [ 3761.416050] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ==================================================== 18:41:36 (1491244896) [ 3764.572676] Lustre: MGS: Connection restored to f12920f1-3d5b-1e78-d777-e27fb9937371 (at 10.9.5.237@tcp) [ 3764.574688] Lustre: Skipped 3 previous similar messages [ 3799.024029] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3799.669334] Lustre: DEBUG MARKER: rc=0; [ 3799.669334] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3799.669334] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3799.669334] echo $(hostname -s): $val; [ 3799.669334] rc=$val; [ 3799.669334] fi; [ 3799.669334] exit $rc [ 3800.269886] Lustre: DEBUG MARKER: dmesg [ 3800.721498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102a: user xattr test ================================================================= 18:42:15 \(1491244935\) [ 3800.874271] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 18:42:15 (1491244935) [ 3801.173872] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3801.798275] Lustre: DEBUG MARKER: rc=0; [ 3801.798275] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3801.798275] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3801.798275] echo $(hostname -s): $val; [ 3801.798275] rc=$val; [ 3801.798275] fi; [ 3801.798275] exit $rc [ 3802.389226] Lustre: DEBUG MARKER: dmesg [ 3802.840783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102b: getfattr\/setfattr for trusted.lov EAs =========================================== 18:42:17 \(1491244937\) [ 3802.998481] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 18:42:17 (1491244937) [ 3803.232673] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3803.850506] Lustre: DEBUG MARKER: rc=0; [ 3803.850506] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3803.850506] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3803.850506] echo $(hostname -s): $val; [ 3803.850506] rc=$val; [ 3803.850506] fi; [ 3803.850506] exit $rc [ 3804.449691] Lustre: DEBUG MARKER: dmesg [ 3804.915773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102c: non-root getfattr\/setfattr for lustre.lov EAs =================================== 18:42:20 \(1491244940\) [ 3805.071247] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 18:42:20 (1491244940) [ 3805.317460] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3805.931517] Lustre: DEBUG MARKER: rc=0; [ 3805.931517] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3805.931517] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3805.931517] echo $(hostname -s): $val; [ 3805.931517] rc=$val; [ 3805.931517] fi; [ 3805.931517] exit $rc [ 3806.533040] Lustre: DEBUG MARKER: dmesg [ 3806.980969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 18:42:22 \(1491244942\) [ 3807.135141] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 18:42:22 (1491244942) [ 3807.340585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102d lustre-aware tar is not installed [ 3807.497457] Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed [ 3807.691250] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3808.314523] Lustre: DEBUG MARKER: rc=0; [ 3808.314523] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3808.314523] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3808.314523] echo $(hostname -s): $val; [ 3808.314523] rc=$val; [ 3808.314523] fi; [ 3808.314523] exit $rc [ 3808.933702] Lustre: DEBUG MARKER: dmesg [ 3809.387447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102f: tar copy files, not keep osts =================================================== 18:42:24 \(1491244944\) [ 3809.543502] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 18:42:24 (1491244944) [ 3809.755604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102f lustre-aware tar is not installed [ 3809.910232] Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed [ 3810.108145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3810.727483] Lustre: DEBUG MARKER: rc=0; [ 3810.727483] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3810.727483] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3810.727483] echo $(hostname -s): $val; [ 3810.727483] rc=$val; [ 3810.727483] fi; [ 3810.727483] exit $rc [ 3811.333203] Lustre: DEBUG MARKER: dmesg [ 3811.785409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102h: grow xattr from inside inode to external block ================================== 18:42:26 \(1491244946\) [ 3811.947589] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 18:42:26 (1491244946) [ 3812.167110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102h.sanity [ 3812.328562] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 3812.528231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3812.681965] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 3812.890798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3813.047652] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 3813.252757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trusted.big still valid after growing trusted.sml [ 3813.406546] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3813.602825] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3814.219766] Lustre: DEBUG MARKER: rc=0; [ 3814.219766] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3814.219766] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3814.219766] echo $(hostname -s): $val; [ 3814.219766] rc=$val; [ 3814.219766] fi; [ 3814.219766] exit $rc [ 3814.842776] Lustre: DEBUG MARKER: dmesg [ 3815.527253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102ha: grow xattr from inside inode to external inode ================================= 18:42:30 \(1491244950\) [ 3815.987156] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 18:42:30 (1491244950) [ 3816.317113] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [ 3816.317113] grep -E -q '(ea_inode|large_xattr)' [ 3816.637532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102ha large_xattr disabled [ 3816.798707] Lustre: DEBUG MARKER: SKIP: sanity test_102ha large_xattr disabled [ 3817.000550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3817.620583] Lustre: DEBUG MARKER: rc=0; [ 3817.620583] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3817.620583] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3817.620583] echo $(hostname -s): $val; [ 3817.620583] rc=$val; [ 3817.620583] fi; [ 3817.620583] exit $rc [ 3818.219909] Lustre: DEBUG MARKER: dmesg [ 3818.670508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102i: lgetxattr test on symbolic link ================================================= 18:42:33 \(1491244953\) [ 3818.835414] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 18:42:33 (1491244953) [ 3819.041123] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3819.665530] Lustre: DEBUG MARKER: rc=0; [ 3819.665530] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3819.665530] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3819.665530] echo $(hostname -s): $val; [ 3819.665530] rc=$val; [ 3819.665530] fi; [ 3819.665530] exit $rc [ 3820.272486] Lustre: DEBUG MARKER: dmesg [ 3820.729536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 18:42:35 \(1491244955\) [ 3820.886523] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 18:42:35 (1491244955) [ 3821.089369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102j lustre-aware tar is not installed [ 3821.246958] Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed [ 3821.441624] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3822.062305] Lustre: DEBUG MARKER: rc=0; [ 3822.062305] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3822.062305] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3822.062305] echo $(hostname -s): $val; [ 3822.062305] rc=$val; [ 3822.062305] fi; [ 3822.062305] exit $rc [ 3822.658103] Lustre: DEBUG MARKER: dmesg [ 3823.124071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102k: setfattr without parameter of value shouldn\'t cause a crash ===================== 18:42:38 \(1491244958\) [ 3823.283594] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 18:42:38 (1491244958) [ 3823.517606] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3824.142504] Lustre: DEBUG MARKER: rc=0; [ 3824.142504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3824.142504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3824.142504] echo $(hostname -s): $val; [ 3824.142504] rc=$val; [ 3824.142504] fi; [ 3824.142504] exit $rc [ 3824.756450] Lustre: DEBUG MARKER: dmesg [ 3825.205609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102l: listxattr size test ============================================================= 18:42:40 \(1491244960\) [ 3825.362395] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 18:42:40 (1491244960) [ 3825.569669] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3826.186205] Lustre: DEBUG MARKER: rc=0; [ 3826.186205] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3826.186205] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3826.186205] echo $(hostname -s): $val; [ 3826.186205] rc=$val; [ 3826.186205] fi; [ 3826.186205] exit $rc [ 3826.789467] Lustre: DEBUG MARKER: dmesg [ 3827.246914] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 18:42:42 \(1491244962\) [ 3827.399629] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 18:42:42 (1491244962) [ 3827.592401] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3828.217430] Lustre: DEBUG MARKER: rc=0; [ 3828.217430] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3828.217430] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3828.217430] echo $(hostname -s): $val; [ 3828.217430] rc=$val; [ 3828.217430] fi; [ 3828.217430] exit $rc [ 3828.813869] Lustre: DEBUG MARKER: dmesg [ 3829.276849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 18:42:44 \(1491244964\) [ 3829.435641] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 18:42:44 (1491244964) [ 3829.575626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3829.575626] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3829.575626] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3829.861951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3829.861951] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3829.861951] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3830.186908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3830.186908] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3830.186908] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3830.620195] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3831.247103] Lustre: DEBUG MARKER: rc=0; [ 3831.247103] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3831.247103] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3831.247103] echo $(hostname -s): $val; [ 3831.247103] rc=$val; [ 3831.247103] fi; [ 3831.247103] exit $rc [ 3831.843229] Lustre: DEBUG MARKER: dmesg [ 3832.312069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102p: check setxattr\(2\) correctly fails without permission ============================ 18:42:47 \(1491244967\) [ 3832.470179] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 18:42:47 (1491244967) [ 3832.609638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3832.609638] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3832.609638] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3833.000139] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3833.623231] Lustre: DEBUG MARKER: rc=0; [ 3833.623231] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3833.623231] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3833.623231] echo $(hostname -s): $val; [ 3833.623231] rc=$val; [ 3833.623231] fi; [ 3833.623231] exit $rc [ 3834.221646] Lustre: DEBUG MARKER: dmesg [ 3834.686689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 18:42:49 \(1491244969\) [ 3834.840630] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 18:42:49 (1491244969) [ 3834.977104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3834.977104] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3834.977104] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3835.318407] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3835.938297] Lustre: DEBUG MARKER: rc=0; [ 3835.938297] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3835.938297] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3835.938297] echo $(hostname -s): $val; [ 3835.938297] rc=$val; [ 3835.938297] fi; [ 3835.938297] exit $rc [ 3836.554580] Lustre: DEBUG MARKER: dmesg [ 3837.019456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102r: set EAs with empty values ======================================================= 18:42:52 \(1491244972\) [ 3837.183734] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 18:42:52 (1491244972) [ 3837.316166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3837.316166] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3837.316166] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3837.699707] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3838.329104] Lustre: DEBUG MARKER: rc=0; [ 3838.329104] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3838.329104] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3838.329104] echo $(hostname -s): $val; [ 3838.329104] rc=$val; [ 3838.329104] fi; [ 3838.329104] exit $rc [ 3838.933330] Lustre: DEBUG MARKER: dmesg [ 3839.397157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103a: acl test ======================================================================== 18:42:54 \(1491244974\) [ 3839.564551] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 18:42:54 (1491244974) [ 3839.750263] Lustre: DEBUG MARKER: gpasswd -a daemon bin [ 3840.062252] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [ 3840.315898] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [ 3840.568133] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall /usr/sbin/l_getidentity [ 3840.819236] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [ 3842.159933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3842.159933] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3842.159933] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3842.436264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3842.436264] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3842.436264] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3842.707478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3842.707478] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3842.707478] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3847.754992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3848.409414] Lustre: DEBUG MARKER: rc=0; [ 3848.409414] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3848.409414] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3848.409414] echo $(hostname -s): $val; [ 3848.409414] rc=$val; [ 3848.409414] fi; [ 3848.409414] exit $rc [ 3849.021893] Lustre: DEBUG MARKER: dmesg [ 3849.481957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103b: MDS mount option \'noacl\' ======================================================== 18:43:04 \(1491244984\) [ 3849.646412] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' ======================================================== 18:43:04 (1491244984) [ 3849.814980] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3850.072244] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3850.202819] Lustre: Failing over lustre-MDT0000 [ 3853.229171] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.229172] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.229175] LustreError: Skipped 8 previous similar messages [ 3853.239168] LustreError: Skipped 5 previous similar messages [ 3853.817413] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.821560] LustreError: Skipped 1 previous similar message [ 3856.261052] Lustre: 3950:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491244985/real 1491244985] req@ffff88005cf98c00 x1563680144650432/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491244991 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3856.378859] Lustre: server umount lustre-MDT0000 complete [ 3856.506200] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3856.764199] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3857.028705] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3857.279969] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3857.527253] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o noacl /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3857.670723] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3857.670723] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3857.670723] [ 3857.747530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: noacl,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.895438] LustreError: 4366:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3857.912498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.074205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3858.354653] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3858.858252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3858.861855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3859.017733] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3859.017735] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3859.165365] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3859.426129] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3859.684537] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3859.951286] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3860.207310] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3863.450762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3863.817761] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3863.991678] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3864.248583] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3864.376194] Lustre: Failing over lustre-MDT0000 [ 3868.227850] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.233399] LustreError: Skipped 6 previous similar messages [ 3870.416053] Lustre: 5424:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491245000/real 1491245000] req@ffff880062eea400 x1563680144668016/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491245006 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3870.500548] Lustre: server umount lustre-MDT0000 complete [ 3870.628174] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3870.888654] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3871.156255] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3871.407347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3871.656653] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3871.816339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.881285] LustreError: 5842:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3871.892494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.041425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3872.300897] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3872.620106] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3872.794910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3872.798568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3872.959429] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3872.963781] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3873.106165] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3873.372298] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3873.475405] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3873.636156] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3873.899718] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3874.156768] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3874.482311] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3875.114229] Lustre: DEBUG MARKER: rc=0; [ 3875.114229] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3875.114229] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3875.114229] echo $(hostname -s): $val; [ 3875.114229] rc=$val; [ 3875.114229] fi; [ 3875.114229] exit $rc [ 3875.731117] Lustre: DEBUG MARKER: dmesg [ 3876.197876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103c: \'cp -rp\' won\'t set empty acl ==================================================== 18:43:31 \(1491245011\) [ 3876.357651] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 18:43:31 (1491245011) [ 3876.604256] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3877.226570] Lustre: DEBUG MARKER: rc=0; [ 3877.226570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3877.226570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3877.226570] echo $(hostname -s): $val; [ 3877.226570] rc=$val; [ 3877.226570] fi; [ 3877.226570] exit $rc [ 3877.859787] Lustre: DEBUG MARKER: dmesg [ 3878.339881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104a: lfs df [-ih] [path] test ======================================================== 18:43:33 \(1491245013\) [ 3878.495370] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 18:43:33 (1491245013) [ 3878.924147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88005ac6e800.ost_server_uuid osc.lustre-OST0001-osc-ffff88005ac6e800.ost_server_uuid osc.lustre-OST0002-osc-ffff88005ac6e800.ost_server_uui [ 3879.078881] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff88005ac6e800.ost_server_uuid osc.lustre-OST0001-osc-ffff88005ac6e800.ost_server_uuid osc.lustre-OST0002-osc-ffff88005ac6e800.ost_server_uuid osc.lustre-OST0003 [ 3879.262124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff88005ac6e800.ost_server_uuid in FULL state after 0 sec [ 3879.411886] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88005ac6e800.ost_server_uuid in FULL state after 0 sec [ 3879.613046] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3880.249169] Lustre: DEBUG MARKER: rc=0; [ 3880.249169] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3880.249169] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3880.249169] echo $(hostname -s): $val; [ 3880.249169] rc=$val; [ 3880.249169] fi; [ 3880.249169] exit $rc [ 3880.857410] Lustre: DEBUG MARKER: dmesg [ 3881.323509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 18:43:36 \(1491245016\) [ 3881.490334] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 18:43:36 (1491245016) [ 3881.697612] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3882.334057] Lustre: DEBUG MARKER: rc=0; [ 3882.334057] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3882.334057] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3882.334057] echo $(hostname -s): $val; [ 3882.334057] rc=$val; [ 3882.334057] fi; [ 3882.334057] exit $rc [ 3882.952513] Lustre: DEBUG MARKER: dmesg [ 3883.434383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105a: flock when mounted without -o flock test ======================================== 18:43:38 \(1491245018\) [ 3883.599225] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 18:43:38 (1491245018) [ 3883.855919] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3884.478705] Lustre: DEBUG MARKER: rc=0; [ 3884.478705] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3884.478705] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3884.478705] echo $(hostname -s): $val; [ 3884.478705] rc=$val; [ 3884.478705] fi; [ 3884.478705] exit $rc [ 3885.091454] Lustre: DEBUG MARKER: dmesg [ 3885.572080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105b: fcntl when mounted without -o flock test ======================================== 18:43:40 \(1491245020\) [ 3885.734367] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 18:43:40 (1491245020) [ 3885.949630] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3886.571287] Lustre: DEBUG MARKER: rc=0; [ 3886.571287] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3886.571287] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3886.571287] echo $(hostname -s): $val; [ 3886.571287] rc=$val; [ 3886.571287] fi; [ 3886.571287] exit $rc [ 3887.178730] Lustre: DEBUG MARKER: dmesg [ 3887.656506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105c: lockf when mounted without -o flock test ======================================== 18:43:42 \(1491245022\) [ 3887.816811] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 18:43:42 (1491245022) [ 3888.034744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3888.658832] Lustre: DEBUG MARKER: rc=0; [ 3888.658832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3888.658832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3888.658832] echo $(hostname -s): $val; [ 3888.658832] rc=$val; [ 3888.658832] fi; [ 3888.658832] exit $rc [ 3889.262828] Lustre: DEBUG MARKER: dmesg [ 3889.728187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105d: flock race \(should not freeze\) ================================================== 18:43:44 \(1491245024\) [ 3889.911760] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 18:43:44 (1491245024) [ 3900.133996] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3900.750073] Lustre: DEBUG MARKER: rc=0; [ 3900.750073] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3900.750073] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3900.750073] echo $(hostname -s): $val; [ 3900.750073] rc=$val; [ 3900.750073] fi; [ 3900.750073] exit $rc [ 3901.348522] Lustre: DEBUG MARKER: dmesg [ 3901.817699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105e: Two conflicting flocks from same process ======================================== 18:43:56 \(1491245036\) [ 3901.974719] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 18:43:56 (1491245036) [ 3902.184187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3902.832861] Lustre: DEBUG MARKER: rc=0; [ 3902.832861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3902.832861] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3902.832861] echo $(hostname -s): $val; [ 3902.832861] rc=$val; [ 3902.832861] fi; [ 3902.832861] exit $rc [ 3903.450144] Lustre: DEBUG MARKER: dmesg [ 3903.983879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 18:43:59 \(1491245039\) [ 3904.139631] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 18:43:59 (1491245039) [ 3904.343103] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3904.961840] Lustre: DEBUG MARKER: rc=0; [ 3904.961840] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3904.961840] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3904.961840] echo $(hostname -s): $val; [ 3904.961840] rc=$val; [ 3904.961840] fi; [ 3904.961840] exit $rc [ 3905.558131] Lustre: DEBUG MARKER: dmesg [ 3906.016574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 107: Coredump on SIG ================================================================== 18:44:01 \(1491245041\) [ 3906.177142] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 18:44:01 (1491245041) [ 3907.407266] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3908.031205] Lustre: DEBUG MARKER: rc=0; [ 3908.031205] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3908.031205] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3908.031205] echo $(hostname -s): $val; [ 3908.031205] rc=$val; [ 3908.031205] fi; [ 3908.031205] exit $rc [ 3908.627354] Lustre: DEBUG MARKER: dmesg [ 3909.090234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 110: filename length checking ========================================================= 18:44:04 \(1491245044\) [ 3909.249133] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 18:44:04 (1491245044) [ 3909.486235] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3910.111676] Lustre: DEBUG MARKER: rc=0; [ 3910.111676] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3910.111676] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3910.111676] echo $(hostname -s): $val; [ 3910.111676] rc=$val; [ 3910.111676] fi; [ 3910.111676] exit $rc [ 3910.724897] Lustre: DEBUG MARKER: dmesg [ 3911.182774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_115 skipping SLOW test 115 [ 3911.369530] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 3911.598683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116a: stripe QOS: free space balance ================================================== 18:44:06 \(1491245046\) [ 3911.763523] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 18:44:06 (1491245046) [ 3911.925820] Lustre: DEBUG MARKER: lctl get_param -n lo*.*-mdtlov.qos_prio_free [ 3912.208400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3912.480836] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3912.767504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3914.056742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3915.342102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3916.625684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3917.906257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3919.189959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3920.477574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3921.773129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3923.057609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3924.364352] Lustre: DEBUG MARKER: lctl get_param -n *.*MDT0000-mdtlov.qos_threshold_rr [ 3937.346323] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3947.627573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3956.986981] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3967.262395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3969.654293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3969.930241] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3970.304130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3971.627954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3972.923264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3974.218072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3975.505070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3976.785044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3978.076876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3979.445821] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3980.094627] Lustre: DEBUG MARKER: rc=0; [ 3980.094627] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3980.094627] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3980.094627] echo $(hostname -s): $val; [ 3980.094627] rc=$val; [ 3980.094627] fi; [ 3980.094627] exit $rc [ 3980.712993] Lustre: DEBUG MARKER: dmesg [ 3981.175723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116b: QoS shouldn\'t LBUG if not enough OSTs found on the 2nd pass ===================== 18:45:16 \(1491245116\) [ 3981.334677] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 18:45:16 (1491245116) [ 3981.495458] Lustre: DEBUG MARKER: lctl get_param -n lo*.lustre-MDT0000-mdtlov.qos_threshold_rr [ 3981.759106] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=0 [ 3982.019664] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x147 [ 3982.170756] Lustre: *** cfs_fail_loc=147, val=0*** [ 3982.335307] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3982.616827] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=17% [ 3982.941081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3983.575983] Lustre: DEBUG MARKER: rc=0; [ 3983.575983] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3983.575983] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3983.575983] echo $(hostname -s): $val; [ 3983.575983] rc=$val; [ 3983.575983] fi; [ 3983.575983] exit $rc [ 3984.184726] Lustre: DEBUG MARKER: dmesg [ 3984.647784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 117: verify osd extend ================================================================ 18:45:19 \(1491245119\) [ 3984.805087] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 18:45:19 (1491245119) [ 3985.225450] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3985.842479] Lustre: DEBUG MARKER: rc=0; [ 3985.842479] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3985.842479] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3985.842479] echo $(hostname -s): $val; [ 3985.842479] rc=$val; [ 3985.842479] fi; [ 3985.842479] exit $rc [ 3986.442673] Lustre: DEBUG MARKER: dmesg [ 3986.916226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118a: verify O_SYNC works ============================================================= 18:45:22 \(1491245122\) [ 3987.084784] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 18:45:22 (1491245122) [ 3987.578791] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3988.197631] Lustre: DEBUG MARKER: rc=0; [ 3988.197631] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3988.197631] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3988.197631] echo $(hostname -s): $val; [ 3988.197631] rc=$val; [ 3988.197631] fi; [ 3988.197631] exit $rc [ 3988.802341] Lustre: DEBUG MARKER: dmesg [ 3989.267031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 18:45:24 \(1491245124\) [ 3989.424308] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 18:45:24 (1491245124) [ 3990.735841] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3991.356178] Lustre: DEBUG MARKER: rc=0; [ 3991.356178] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3991.356178] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3991.356178] echo $(hostname -s): $val; [ 3991.356178] rc=$val; [ 3991.356178] fi; [ 3991.356178] exit $rc [ 3991.970583] Lustre: DEBUG MARKER: dmesg [ 3992.429289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 18:45:27 \(1491245127\) [ 3992.588477] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 18:45:27 (1491245127) [ 3995.434999] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3996.070060] Lustre: DEBUG MARKER: rc=0; [ 3996.070060] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3996.070060] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3996.070060] echo $(hostname -s): $val; [ 3996.070060] rc=$val; [ 3996.070060] fi; [ 3996.070060] exit $rc [ 3996.683978] Lustre: DEBUG MARKER: dmesg [ 3997.154616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 18:45:32 \(1491245132\) [ 3997.315719] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 18:45:32 (1491245132) [ 4003.408250] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4004.026465] Lustre: DEBUG MARKER: rc=0; [ 4004.026465] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4004.026465] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4004.026465] echo $(hostname -s): $val; [ 4004.026465] rc=$val; [ 4004.026465] fi; [ 4004.026465] exit $rc [ 4004.627065] Lustre: DEBUG MARKER: dmesg [ 4005.092507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 18:45:40 \(1491245140\) [ 4005.246979] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 18:45:40 (1491245140) [ 4005.759942] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4006.380162] Lustre: DEBUG MARKER: rc=0; [ 4006.380162] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4006.380162] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4006.380162] echo $(hostname -s): $val; [ 4006.380162] rc=$val; [ 4006.380162] fi; [ 4006.380162] exit $rc [ 4006.983904] Lustre: DEBUG MARKER: dmesg [ 4007.448636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118g: Don\'t stay in wait if we got local -ENOMEM ===================================== 18:45:42 \(1491245142\) [ 4007.615616] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 18:45:42 (1491245142) [ 4008.108064] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4008.732724] Lustre: DEBUG MARKER: rc=0; [ 4008.732724] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4008.732724] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4008.732724] echo $(hostname -s): $val; [ 4008.732724] rc=$val; [ 4008.732724] fi; [ 4008.732724] exit $rc [ 4009.333401] Lustre: DEBUG MARKER: dmesg [ 4009.799924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118h: Verify timeout in handling recoverables errors ================================= 18:45:44 \(1491245144\) [ 4009.956304] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 18:45:44 (1491245144) [ 4021.324653] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4021.949647] Lustre: DEBUG MARKER: rc=0; [ 4021.949647] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4021.949647] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4021.949647] echo $(hostname -s): $val; [ 4021.949647] rc=$val; [ 4021.949647] fi; [ 4021.949647] exit $rc [ 4022.553176] Lustre: DEBUG MARKER: dmesg [ 4023.025157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118i: Fix error before timeout in recoverable error ================================== 18:45:58 \(1491245158\) [ 4023.182673] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 18:45:58 (1491245158) [ 4030.019988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4030.652877] Lustre: DEBUG MARKER: rc=0; [ 4030.652877] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4030.652877] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4030.652877] echo $(hostname -s): $val; [ 4030.652877] rc=$val; [ 4030.652877] fi; [ 4030.652877] exit $rc [ 4031.259411] Lustre: DEBUG MARKER: dmesg [ 4031.742915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118j: Simulate unrecoverable OST side error =========================================== 18:46:06 \(1491245166\) [ 4031.895189] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 18:46:06 (1491245166) [ 4033.033974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4033.658420] Lustre: DEBUG MARKER: rc=0; [ 4033.658420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4033.658420] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4033.658420] echo $(hostname -s): $val; [ 4033.658420] rc=$val; [ 4033.658420] fi; [ 4033.658420] exit $rc [ 4034.262238] Lustre: DEBUG MARKER: dmesg [ 4034.741575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 18:46:09 \(1491245169\) [ 4034.897092] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 18:46:09 (1491245169) [ 4040.801790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4041.427913] Lustre: DEBUG MARKER: rc=0; [ 4041.427913] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4041.427913] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4041.427913] echo $(hostname -s): $val; [ 4041.427913] rc=$val; [ 4041.427913] fi; [ 4041.427913] exit $rc [ 4042.755888] Lustre: DEBUG MARKER: dmesg [ 4044.836744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118l: fsync dir ======================================================================= 18:46:19 \(1491245179\) [ 4045.000826] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 18:46:19 (1491245179) [ 4045.213192] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4045.843300] Lustre: DEBUG MARKER: rc=0; [ 4045.843300] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4045.843300] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4045.843300] echo $(hostname -s): $val; [ 4045.843300] rc=$val; [ 4045.843300] fi; [ 4045.843300] exit $rc [ 4046.448017] Lustre: DEBUG MARKER: dmesg [ 4046.920043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118m: fdatasync dir =================================================================== 18:46:22 \(1491245182\) [ 4047.071023] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 18:46:22 (1491245182) [ 4047.325663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4047.946267] Lustre: DEBUG MARKER: rc=0; [ 4047.946267] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4047.946267] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4047.946267] echo $(hostname -s): $val; [ 4047.946267] rc=$val; [ 4047.946267] fi; [ 4047.946267] exit $rc [ 4048.552723] Lustre: DEBUG MARKER: dmesg [ 4049.046324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ============================== 18:46:24 \(1491245184\) [ 4049.203217] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 18:46:24 (1491245184) [ 4049.453135] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4050.087634] Lustre: DEBUG MARKER: rc=0; [ 4050.087634] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4050.087634] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4050.087634] echo $(hostname -s): $val; [ 4050.087634] rc=$val; [ 4050.087634] fi; [ 4050.087634] exit $rc [ 4050.699819] Lustre: DEBUG MARKER: dmesg [ 4051.162922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119b: Sparse directIO read must return actual read amount ============================= 18:46:26 \(1491245186\) [ 4051.321700] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 18:46:26 (1491245186) [ 4051.692285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4052.317130] Lustre: DEBUG MARKER: rc=0; [ 4052.317130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4052.317130] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4052.317130] echo $(hostname -s): $val; [ 4052.317130] rc=$val; [ 4052.317130] fi; [ 4052.317130] exit $rc [ 4052.926744] Lustre: DEBUG MARKER: dmesg [ 4053.393305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119c: Testing for direct read hitting hole ============================================ 18:46:28 \(1491245188\) [ 4053.552867] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 18:46:28 (1491245188) [ 4053.855749] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4054.484142] Lustre: DEBUG MARKER: rc=0; [ 4054.484142] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4054.484142] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4054.484142] echo $(hostname -s): $val; [ 4054.484142] rc=$val; [ 4054.484142] fi; [ 4054.484142] exit $rc [ 4055.094349] Lustre: DEBUG MARKER: dmesg [ 4055.568814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 18:46:30 \(1491245190\) [ 4055.727222] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 18:46:30 (1491245190) [ 4057.070233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark the DIO writes have completed, now wait for the reads \(should not block very long\) [ 4057.234146] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 4059.435159] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4060.061416] Lustre: DEBUG MARKER: rc=0; [ 4060.061416] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4060.061416] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4060.061416] echo $(hostname -s): $val; [ 4060.061416] rc=$val; [ 4060.061416] fi; [ 4060.061416] exit $rc [ 4060.670659] Lustre: DEBUG MARKER: dmesg [ 4061.143909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 18:46:36 \(1491245196\) [ 4061.307521] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 18:46:36 (1491245196) [ 4061.505385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4061.768915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4062.090149] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4062.714363] Lustre: DEBUG MARKER: rc=0; [ 4062.714363] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4062.714363] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4062.714363] echo $(hostname -s): $val; [ 4062.714363] rc=$val; [ 4062.714363] fi; [ 4062.714363] exit $rc [ 4063.322227] Lustre: DEBUG MARKER: dmesg [ 4063.792928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120b: Early Lock Cancel: create test ================================================== 18:46:38 \(1491245198\) [ 4063.961211] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 18:46:38 (1491245198) [ 4064.142757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4064.428372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4064.748550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4065.373042] Lustre: DEBUG MARKER: rc=0; [ 4065.373042] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4065.373042] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4065.373042] echo $(hostname -s): $val; [ 4065.373042] rc=$val; [ 4065.373042] fi; [ 4065.373042] exit $rc [ 4065.979820] Lustre: DEBUG MARKER: dmesg [ 4066.452939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120c: Early Lock Cancel: link test ==================================================== 18:46:41 \(1491245201\) [ 4066.614065] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 18:46:41 (1491245201) [ 4066.808724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4067.073686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4067.404801] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4068.030897] Lustre: DEBUG MARKER: rc=0; [ 4068.030897] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4068.030897] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4068.030897] echo $(hostname -s): $val; [ 4068.030897] rc=$val; [ 4068.030897] fi; [ 4068.030897] exit $rc [ 4068.636861] Lustre: DEBUG MARKER: dmesg [ 4069.117192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120d: Early Lock Cancel: setattr test ================================================= 18:46:44 \(1491245204\) [ 4069.294222] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 18:46:44 (1491245204) [ 4069.481315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4069.745025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4070.079530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4070.705288] Lustre: DEBUG MARKER: rc=0; [ 4070.705288] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4070.705288] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4070.705288] echo $(hostname -s): $val; [ 4070.705288] rc=$val; [ 4070.705288] fi; [ 4070.705288] exit $rc [ 4071.312409] Lustre: DEBUG MARKER: dmesg [ 4071.787831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120e: Early Lock Cancel: unlink test ================================================== 18:46:46 \(1491245206\) [ 4071.953863] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 18:46:46 (1491245206) [ 4074.163356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4079.469769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4079.802345] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4080.442202] Lustre: DEBUG MARKER: rc=0; [ 4080.442202] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4080.442202] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4080.442202] echo $(hostname -s): $val; [ 4080.442202] rc=$val; [ 4080.442202] fi; [ 4080.442202] exit $rc [ 4081.059281] Lustre: DEBUG MARKER: dmesg [ 4081.605300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120f: Early Lock Cancel: rename test ================================================== 18:46:56 \(1491245216\) [ 4081.769083] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 18:46:56 (1491245216) [ 4083.999325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4089.270175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4089.601671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4090.224187] Lustre: DEBUG MARKER: rc=0; [ 4090.224187] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4090.224187] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4090.224187] echo $(hostname -s): $val; [ 4090.224187] rc=$val; [ 4090.224187] fi; [ 4090.224187] exit $rc [ 4090.824102] Lustre: DEBUG MARKER: dmesg [ 4091.307484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120g: Early Lock Cancel: performance test ============================================= 18:47:06 \(1491245226\) [ 4091.497932] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 18:47:06 (1491245226) [ 4091.685089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4103.389925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4111.462697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 4113.795577] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4114.427955] Lustre: DEBUG MARKER: rc=0; [ 4114.427955] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4114.427955] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4114.427955] echo $(hostname -s): $val; [ 4114.427955] rc=$val; [ 4114.427955] fi; [ 4114.427955] exit $rc [ 4115.050279] Lustre: DEBUG MARKER: dmesg [ 4115.512224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 121: read cancel race ================================================================= 18:47:30 \(1491245250\) [ 4115.677033] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 18:47:30 (1491245250) [ 4115.911767] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4116.537908] Lustre: DEBUG MARKER: rc=0; [ 4116.537908] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4116.537908] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4116.537908] echo $(hostname -s): $val; [ 4116.537908] rc=$val; [ 4116.537908] fi; [ 4116.537908] exit $rc [ 4117.152825] Lustre: DEBUG MARKER: dmesg [ 4117.644274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123a: verify statahead work =========================================================== 18:47:32 \(1491245252\) [ 4117.816558] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 18:47:32 (1491245252) [ 4118.263141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files without statahead: 0 sec [ 4118.421860] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [ 4118.707758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files with statahead: 0 sec [ 4118.872815] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [ 4121.224512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files without statahead: 1 sec [ 4121.383899] Lustre: DEBUG MARKER: ls 1000 files without statahead: 1 sec [ 4122.025853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files with statahead: 1 sec [ 4122.188155] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [ 4143.400543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files without statahead: 11 sec [ 4143.554465] Lustre: DEBUG MARKER: ls 10000 files without statahead: 11 sec [ 4147.035169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files with statahead: 3 sec [ 4147.191149] Lustre: DEBUG MARKER: ls 10000 files with statahead: 3 sec [ 4147.398120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 4147.553577] Lustre: DEBUG MARKER: ls done [ 4153.473046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm -r \/mnt\/lustre\/d123a.sanity\/: 6 seconds [ 4153.724363] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 6 seconds [ 4154.036238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm done [ 4154.274233] Lustre: DEBUG MARKER: rm done [ 4154.589933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4155.525729] Lustre: DEBUG MARKER: rc=0; [ 4155.525729] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4155.525729] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4155.525729] echo $(hostname -s): $val; [ 4155.525729] rc=$val; [ 4155.525729] fi; [ 4155.525729] exit $rc [ 4156.136761] Lustre: DEBUG MARKER: dmesg [ 4156.605631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123b: not panic with network error in statahead enqueue \(bug 15027\) =================== 18:48:11 \(1491245291\) [ 4156.770178] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 18:48:11 (1491245291) [ 4159.395989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 4159.562831] Lustre: DEBUG MARKER: ls done [ 4160.365808] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4160.981701] Lustre: DEBUG MARKER: rc=0; [ 4160.981701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4160.981701] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4160.981701] echo $(hostname -s): $val; [ 4160.981701] rc=$val; [ 4160.981701] fi; [ 4160.981701] exit $rc [ 4161.577213] Lustre: DEBUG MARKER: dmesg [ 4162.037203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124a: lru resize ====================================================================== 18:48:17 \(1491245297\) [ 4162.191284] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 18:48:17 (1491245297) [ 4162.404156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create 2000 files at \/mnt\/lustre\/d124a.sanity [ 4162.562162] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 4165.756515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88005ac6e800 [ 4165.922972] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff88005ac6e800 [ 4166.141220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88005ac6e800 [ 4166.309279] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff88005ac6e800 [ 4166.512311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LRU=2003 [ 4166.678315] Lustre: DEBUG MARKER: LRU=2003 [ 4166.877094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LIMIT=83999 [ 4167.031483] Lustre: DEBUG MARKER: LIMIT=83999 [ 4167.228694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LVF=50323 [ 4167.380254] Lustre: DEBUG MARKER: LVF=50323 [ 4167.579631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OLD_LVF=1 [ 4167.739296] Lustre: DEBUG MARKER: OLD_LVF=1 [ 4167.936079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Sleep 50 sec [ 4168.098252] Lustre: DEBUG MARKER: Sleep 50 sec [ 4218.334756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Dropped 606 locks in 50s [ 4218.506592] Lustre: DEBUG MARKER: Dropped 606 locks in 50s [ 4218.740834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark unlink 2000 files at \/mnt\/lustre\/d124a.sanity [ 4218.905065] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 4220.579212] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4221.228562] Lustre: DEBUG MARKER: rc=0; [ 4221.228562] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4221.228562] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4221.228562] echo $(hostname -s): $val; [ 4221.228562] rc=$val; [ 4221.228562] fi; [ 4221.228562] exit $rc [ 4221.840292] Lustre: DEBUG MARKER: dmesg [ 4222.314268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124b: lru resize \(performance test\) =================================================== 18:49:17 \(1491245357\) [ 4222.482737] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 18:49:17 (1491245357) [ 4227.466895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/disable_lru_resize 3 times [ 4227.633477] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 4232.657493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 5 seconds [ 4232.809427] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 4233.007045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 200 [ 4233.163205] Lustre: DEBUG MARKER: lru_size = 200 [ 4242.648637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/enable_lru_resize 3 times [ 4242.813849] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 4247.530784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 4 seconds [ 4247.685365] Lustre: DEBUG MARKER: ls -la time: 4 seconds [ 4247.890134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 4006 [ 4248.048804] Lustre: DEBUG MARKER: lru_size = 4006 [ 4248.250476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la is 20% faster with lru resize enabled [ 4248.423193] Lustre: DEBUG MARKER: ls -la is 20% faster with lru resize enabled [ 4251.008360] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4251.669295] Lustre: DEBUG MARKER: rc=0; [ 4251.669295] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4251.669295] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4251.669295] echo $(hostname -s): $val; [ 4251.669295] rc=$val; [ 4251.669295] fi; [ 4251.669295] exit $rc [ 4252.269221] Lustre: DEBUG MARKER: dmesg [ 4252.779584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124c: LRUR cancel very aged locks ===================================================== 18:49:47 \(1491245387\) [ 4252.977923] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 18:49:47 (1491245387) [ 4273.674520] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4274.300285] Lustre: DEBUG MARKER: rc=0; [ 4274.300285] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4274.300285] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4274.300285] echo $(hostname -s): $val; [ 4274.300285] rc=$val; [ 4274.300285] fi; [ 4274.300285] exit $rc [ 4274.935932] Lustre: DEBUG MARKER: dmesg [ 4275.415597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 125: don\'t return EPROTO when a dir has a non-default striping and ACLs =============== 18:50:10 \(1491245410\) [ 4275.570974] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 18:50:10 (1491245410) [ 4275.812679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4276.448952] Lustre: DEBUG MARKER: rc=0; [ 4276.448952] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4276.448952] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4276.448952] echo $(hostname -s): $val; [ 4276.448952] rc=$val; [ 4276.448952] fi; [ 4276.448952] exit $rc [ 4277.058871] Lustre: DEBUG MARKER: dmesg [ 4277.525762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 126: check that the fsgid provided by the client is taken into account ================ 18:50:12 \(1491245412\) [ 4277.687221] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 18:50:12 (1491245412) [ 4277.892238] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4278.506973] Lustre: DEBUG MARKER: rc=0; [ 4278.506973] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4278.506973] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4278.506973] echo $(hostname -s): $val; [ 4278.506973] rc=$val; [ 4278.506973] fi; [ 4278.506973] exit $rc [ 4279.125246] Lustre: DEBUG MARKER: dmesg [ 4279.613788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127a: verify the client stats are sane ================================================ 18:50:14 \(1491245414\) [ 4279.785868] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 18:50:14 (1491245414) [ 4280.170713] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4280.800658] Lustre: DEBUG MARKER: rc=0; [ 4280.800658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4280.800658] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4280.800658] echo $(hostname -s): $val; [ 4280.800658] rc=$val; [ 4280.800658] fi; [ 4280.800658] exit $rc [ 4281.415110] Lustre: DEBUG MARKER: dmesg [ 4281.881616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127b: verify the llite client stats are sane ========================================== 18:50:16 \(1491245416\) [ 4282.035056] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 18:50:16 (1491245416) [ 4282.273924] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4282.895389] Lustre: DEBUG MARKER: rc=0; [ 4282.895389] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4282.895389] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4282.895389] echo $(hostname -s): $val; [ 4282.895389] rc=$val; [ 4282.895389] fi; [ 4282.895389] exit $rc [ 4283.498941] Lustre: DEBUG MARKER: dmesg [ 4283.962378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 128: interactive lfs for 2 consecutive find\'s ========================================= 18:50:19 \(1491245419\) [ 4284.118550] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 18:50:19 (1491245419) [ 4284.381748] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4285.002954] Lustre: DEBUG MARKER: rc=0; [ 4285.002954] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4285.002954] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4285.002954] echo $(hostname -s): $val; [ 4285.002954] rc=$val; [ 4285.002954] fi; [ 4285.002954] exit $rc [ 4285.625300] Lustre: DEBUG MARKER: dmesg [ 4286.110898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 129: test directory size limit ======================================================== 18:50:21 \(1491245421\) [ 4286.269739] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 18:50:21 (1491245421) [ 4286.408187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4286.408187] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4286.408187] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4286.738889] Lustre: DEBUG MARKER: dv=$(lctl get_param -n *.lustre-MDT0000.mntdev); [ 4286.738889] if foo=$(lvdisplay -c $dv 2>/dev/null); then [ 4286.738889] echo dm-${foo##*:}; [ 4286.738889] else [ 4286.738889] echo $(basename $dv); [ 4286.738889] fi; [ 4287.026047] Lustre: DEBUG MARKER: test -e /proc/fs/ldiskfs/dm-0/max_dir_size [ 4287.276966] Lustre: DEBUG MARKER: echo 20480 >/sys/fs/ldiskfs/dm-0/max_dir_size [ 4287.528085] Lustre: DEBUG MARKER: test -e /sys/fs/ldiskfs/dm-0/warning_dir_size [ 4287.779904] Lustre: DEBUG MARKER: echo 20480 >/sys/fs/ldiskfs/dm-0/warning_dir_size [ 4288.048140] Lustre: DEBUG MARKER: dmesg | tail -3 | grep -q is approaching [ 4289.277226] Lustre: 5825:0:(osd_handler.c:467:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 2444, FID: [0x200001b73:0x81a0:0x0]) is approaching maximum entry limit [ 4290.346988] Lustre: 5825:0:(osd_handler.c:467:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 2444, FID: [0x200001b73:0x81a0:0x0]) has reached maximum entry limit [ 4290.498769] Lustre: DEBUG MARKER: dv=$(lctl get_param -n *.lustre-MDT0000.mntdev); [ 4290.498769] if foo=$(lvdisplay -c $dv 2>/dev/null); then [ 4290.498769] echo dm-${foo##*:}; [ 4290.498769] else [ 4290.498769] echo $(basename $dv); [ 4290.498769] fi; [ 4290.771226] Lustre: DEBUG MARKER: test -e /proc/fs/ldiskfs/dm-0/max_dir_size [ 4291.025697] Lustre: DEBUG MARKER: echo 0 >/sys/fs/ldiskfs/dm-0/max_dir_size [ 4291.280271] Lustre: DEBUG MARKER: test -e /sys/fs/ldiskfs/dm-0/warning_dir_size [ 4291.531844] Lustre: DEBUG MARKER: echo 0 >/sys/fs/ldiskfs/dm-0/warning_dir_size [ 4292.864846] Lustre: DEBUG MARKER: dmesg | tail -3 | grep -q has reached [ 4293.158120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4293.158120] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4293.158120] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4293.500731] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4294.140091] Lustre: DEBUG MARKER: rc=0; [ 4294.140091] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4294.140091] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4294.140091] echo $(hostname -s): $val; [ 4294.140091] rc=$val; [ 4294.140091] fi; [ 4294.140091] exit $rc [ 4294.777623] Lustre: DEBUG MARKER: dmesg [ 4295.322580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130a: FIEMAP \(1-stripe file\) ========================================================== 18:50:30 \(1491245430\) [ 4295.488135] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 18:50:30 (1491245430) [ 4295.934548] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4296.563047] Lustre: DEBUG MARKER: rc=0; [ 4296.563047] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4296.563047] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4296.563047] echo $(hostname -s): $val; [ 4296.563047] rc=$val; [ 4296.563047] fi; [ 4296.563047] exit $rc [ 4297.168718] Lustre: DEBUG MARKER: dmesg [ 4297.645159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130b: FIEMAP \(7-stripe file\) ========================================================== 18:50:32 \(1491245432\) [ 4297.802371] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (7-stripe file) ========================================================== 18:50:32 (1491245432) [ 4298.266457] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4298.886523] Lustre: DEBUG MARKER: rc=0; [ 4298.886523] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4298.886523] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4298.886523] echo $(hostname -s): $val; [ 4298.886523] rc=$val; [ 4298.886523] fi; [ 4298.886523] exit $rc [ 4299.498072] Lustre: DEBUG MARKER: dmesg [ 4299.986965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130c: FIEMAP \(2-stripe file with hole\) ================================================ 18:50:35 \(1491245435\) [ 4300.147773] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 18:50:35 (1491245435) [ 4300.416078] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4301.051440] Lustre: DEBUG MARKER: rc=0; [ 4301.051440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4301.051440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4301.051440] echo $(hostname -s): $val; [ 4301.051440] rc=$val; [ 4301.051440] fi; [ 4301.051440] exit $rc [ 4301.662450] Lustre: DEBUG MARKER: dmesg [ 4302.133774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130d: FIEMAP \(N-stripe file\) ========================================================== 18:50:37 \(1491245437\) [ 4302.303575] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 18:50:37 (1491245437) [ 4302.749194] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4303.374367] Lustre: DEBUG MARKER: rc=0; [ 4303.374367] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4303.374367] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4303.374367] echo $(hostname -s): $val; [ 4303.374367] rc=$val; [ 4303.374367] fi; [ 4303.374367] exit $rc [ 4303.987734] Lustre: DEBUG MARKER: dmesg [ 4304.479889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130e: FIEMAP \(test continuation FIEMAP calls\) ========================================= 18:50:39 \(1491245439\) [ 4304.653769] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 18:50:39 (1491245439) [ 4311.824208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4312.461350] Lustre: DEBUG MARKER: rc=0; [ 4312.461350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4312.461350] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4312.461350] echo $(hostname -s): $val; [ 4312.461350] rc=$val; [ 4312.461350] fi; [ 4312.461350] exit $rc [ 4313.076258] Lustre: DEBUG MARKER: dmesg [ 4313.544937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ============================ 18:50:48 \(1491245448\) [ 4313.706167] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 18:50:48 (1491245448) [ 4314.023774] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4314.650672] Lustre: DEBUG MARKER: rc=0; [ 4314.650672] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4314.650672] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4314.650672] echo $(hostname -s): $val; [ 4314.650672] rc=$val; [ 4314.650672] fi; [ 4314.650672] exit $rc [ 4315.261408] Lustre: DEBUG MARKER: dmesg [ 4315.729935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131b: test append writev ============================================================== 18:50:50 \(1491245450\) [ 4315.888961] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 18:50:50 (1491245450) [ 4316.154231] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4316.783241] Lustre: DEBUG MARKER: rc=0; [ 4316.783241] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4316.783241] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4316.783241] echo $(hostname -s): $val; [ 4316.783241] rc=$val; [ 4316.783241] fi; [ 4316.783241] exit $rc [ 4317.392860] Lustre: DEBUG MARKER: dmesg [ 4317.870749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ============================================= 18:50:52 \(1491245452\) [ 4318.053345] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 18:50:52 (1491245452) [ 4318.260032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4318.897528] Lustre: DEBUG MARKER: rc=0; [ 4318.897528] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4318.897528] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4318.897528] echo $(hostname -s): $val; [ 4318.897528] rc=$val; [ 4318.897528] fi; [ 4318.897528] exit $rc [ 4319.516867] Lustre: DEBUG MARKER: dmesg [ 4319.994603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131d: test short read ================================================================= 18:50:55 \(1491245455\) [ 4320.162383] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 18:50:55 (1491245455) [ 4320.397234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4321.042733] Lustre: DEBUG MARKER: rc=0; [ 4321.042733] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4321.042733] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4321.042733] echo $(hostname -s): $val; [ 4321.042733] rc=$val; [ 4321.042733] fi; [ 4321.042733] exit $rc [ 4321.658615] Lustre: DEBUG MARKER: dmesg [ 4322.124347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131e: test read hitting hole ========================================================== 18:50:57 \(1491245457\) [ 4322.282486] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 18:50:57 (1491245457) [ 4322.516056] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4323.137065] Lustre: DEBUG MARKER: rc=0; [ 4323.137065] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4323.137065] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4323.137065] echo $(hostname -s): $val; [ 4323.137065] rc=$val; [ 4323.137065] fi; [ 4323.137065] exit $rc [ 4323.734534] Lustre: DEBUG MARKER: dmesg [ 4324.219203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133a: Verifying MDT stats ============================================================= 18:50:59 \(1491245459\) [ 4324.374236] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 18:50:59 (1491245459) [ 4324.565196] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 4324.822028] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4325.346820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4325.611851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4325.874645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4326.136268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4326.136268] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4326.136268] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4326.412637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4326.675392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4326.942842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4327.210493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4327.479712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4327.742418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4328.077758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4328.711497] Lustre: DEBUG MARKER: rc=0; [ 4328.711497] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4328.711497] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4328.711497] echo $(hostname -s): $val; [ 4328.711497] rc=$val; [ 4328.711497] fi; [ 4328.711497] exit $rc [ 4329.327670] Lustre: DEBUG MARKER: dmesg [ 4329.802110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133b: Verifying extra MDT stats ======================================================= 18:51:04 \(1491245464\) [ 4329.956048] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 18:51:04 (1491245464) [ 4330.179471] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4330.700073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4330.959093] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4331.221453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4331.221453] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4331.221453] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4331.505163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4331.770096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 4332.112144] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4332.742029] Lustre: DEBUG MARKER: rc=0; [ 4332.742029] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4332.742029] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4332.742029] echo $(hostname -s): $val; [ 4332.742029] rc=$val; [ 4332.742029] fi; [ 4332.742029] exit $rc [ 4333.356595] Lustre: DEBUG MARKER: dmesg [ 4333.847948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133c: Verifying OST stats ============================================================= 18:51:08 \(1491245468\) [ 4334.011101] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 18:51:08 (1491245468) [ 4334.320083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4334.586788] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4334.872130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4336.158345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4337.439693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4338.717180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4340.006599] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 4341.433644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4341.694104] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4341.999141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4343.289912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4344.577842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4345.862515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4347.146948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4348.432978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4350.067184] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4350.698317] Lustre: DEBUG MARKER: rc=0; [ 4350.698317] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4350.698317] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4350.698317] echo $(hostname -s): $val; [ 4350.698317] rc=$val; [ 4350.698317] fi; [ 4350.698317] exit $rc [ 4351.318156] Lustre: DEBUG MARKER: dmesg [ 4351.796347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133d: Verifying rename_stats ========================================================== 18:51:26 \(1491245486\) [ 4351.959322] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 18:51:26 (1491245486) [ 4352.146027] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 4352.398409] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 4353.279465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4353.538702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4353.799512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4354.053116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 4354.326402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4354.584202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4354.841497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4355.102188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4355.809136] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4356.443659] Lustre: DEBUG MARKER: rc=0; [ 4356.443659] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4356.443659] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4356.443659] echo $(hostname -s): $val; [ 4356.443659] rc=$val; [ 4356.443659] fi; [ 4356.443659] exit $rc [ 4357.068950] Lustre: DEBUG MARKER: dmesg [ 4357.547056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133e: Verifying OST {read,write}_bytes nid stats ====================================== 18:51:32 \(1491245492\) [ 4357.708868] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 18:51:32 (1491245492) [ 4359.205878] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4359.842525] Lustre: DEBUG MARKER: rc=0; [ 4359.842525] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4359.842525] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4359.842525] echo $(hostname -s): $val; [ 4359.842525] rc=$val; [ 4359.842525] fi; [ 4359.842525] exit $rc [ 4360.460649] Lustre: DEBUG MARKER: dmesg [ 4360.942653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133f: Check for LBUGs\/Oopses\/unreadable files in \/proc ================================ 18:51:36 \(1491245496\) [ 4361.109037] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc ================================ 18:51:36 (1491245496) [ 4361.970127] Lustre: DEBUG MARKER: \ls -d /{proc,sys}/{fs,sys,kernel/debug}/{lustre,lnet}/ [ 4362.222813] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ ! -name req_history -exec cat {} \; [ 4363.283040] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ ! -name req_history -type f -exec cat {} \; [ 4367.073678] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4367.708454] Lustre: DEBUG MARKER: rc=0; [ 4367.708454] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4367.708454] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4367.708454] echo $(hostname -s): $val; [ 4367.708454] rc=$val; [ 4367.708454] fi; [ 4367.708454] exit $rc [ 4368.321366] Lustre: DEBUG MARKER: dmesg [ 4368.810074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133g: Check for Oopses on bad io area writes\/reads in \/proc =========================== 18:51:43 \(1491245503\) [ 4368.965179] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc =========================== 18:51:43 (1491245503) [ 4369.699736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4369.699736] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4369.699736] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4369.973147] Lustre: DEBUG MARKER: \ls -d /{proc,sys}/{fs,sys,kernel/debug}/{lustre,lnet}/ [ 4370.229667] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ -type f -not -name force_lbug -not -name changelog_mask -exec badarea_io {} \; [ 4370.461264] Lustre: 6268:0:(mdt_coordinator.c:1804:mdt_hsm_policy_seq_write()) lustre-MDT0000: '' is unknown, supported policies are: [ 4370.509557] LustreError: 6316:0:(mdt_coordinator.c:1956:mdt_hsm_cdt_control_seq_write()) lustre-MDT0000: Valid coordinator control commands are: enabled shutdown disabled purge help [ 4370.515990] Lustre: 6318:0:(lprocfs_status.c:2376:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash due to bad address, rc = -14 [ 4370.520445] Lustre: 6318:0:(lprocfs_status.c:2372:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash to "", needs uid:gid format, rc = -22 [ 4370.525863] LustreError: 6319:0:(genops.c:1529:obd_export_evict_by_uuid()) lustre-MDT0000: can't disconnect : no exports found [ 4370.531917] LustreError: 6321:0:(mdt_lproc.c:366:lprocfs_identity_info_seq_write()) lustre-MDT0000: invalid data count = 5, size = 1048 [ 4370.536419] LustreError: 6321:0:(mdt_lproc.c:383:lprocfs_identity_info_seq_write()) lustre-MDT0000: MDS identity downcall bad params [ 4370.541849] Lustre: 6322:0:(lprocfs_status.c:2444:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids to "", bad address rc = -14 [ 4370.546738] Lustre: 6322:0:(lprocfs_status.c:2448:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids due to string too long rc = -22 [ 4370.555088] LustreError: 6326:0:(mdt_lproc.c:298:mdt_identity_upcall_seq_write()) lustre-MDT0000: identity upcall too long [ 4370.585630] LustreError: 6359:0:(mgs_nids.c:796:lprocfs_wr_ir_state()) Unable to process command: (-22) [ 4370.843951] LustreError: 6665:0:(lproc_fid.c:177:lprocfs_server_fid_width_seq_write()) ctl-lustre-MDT0000: invalid FID sequence width: rc = -14 [ 4370.923577] LustreError: 6752:0:(ldlm_resource.c:105:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [ 4370.946472] LustreError: 6764:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [ 4370.999112] Lustre: 6808:0:(libcfs_string.c:127:cfs_str2mask()) unknown mask ''. [ 4370.999112] mask usage: [+|-] ... [ 4371.005777] Lustre: 0 MB is too small for debug buffer size, setting it to 2 MB. [ 4371.032514] LustreError: 6841:0:(ldlm_resource.c:354:lru_size_store()) lru_size: invalid value written [ 4373.140760] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4373.396348] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4373.525379] Lustre: Failing over lustre-MDT0000 [ 4377.637941] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.568048] Lustre: 7226:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491245509/real 1491245509] req@ffff880058c20300 x1563680145224256/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491245515 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4379.658335] Lustre: server umount lustre-MDT0000 complete [ 4379.785749] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4390.127501] Lustre: DEBUG MARKER: hostname [ 4390.471252] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4390.739098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4390.990346] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4391.146385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.170189] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 4391.173121] Lustre: Skipped 31 previous similar messages [ 4391.210651] LustreError: 7643:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4391.221529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.370187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4391.634517] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4392.139311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4392.139619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4392.302724] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4392.303529] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4392.447674] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4392.705921] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4392.959715] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4393.223087] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4394.003884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.008206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.180610] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.264565] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4399.636835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4400.488449] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4401.430764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4401.553383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4401.608580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4401.759285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4404.136655] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4404.406428] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 4405.261305] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 4405.263946] Lustre: Skipped 1 previous similar message [ 4405.535058] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880058c21800 x1563680145225616/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 4410.257850] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 4410.260761] Lustre: Skipped 6 previous similar messages [ 4415.257990] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.257995] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.258004] LustreError: Skipped 8 previous similar messages [ 4415.271355] LustreError: Skipped 5 previous similar messages [ 4416.576041] Lustre: 8897:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491245546/real 1491245546] req@ffff88003465f900 x1563680145225808/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491245552 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4416.701612] Lustre: server umount lustre-MDT0000 complete [ 4416.831961] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4433.986190] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 4433.986190] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 4433.986190] if [ $running -ne $mpts ]; then [ 4433.986190] echo $(hostname) env are INSANE!; [ 4433.986190] exit 1; [ 4433.986190] fi [ 4434.390337] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 4434.390337] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 4434.390337] if [ $running -ne $mpts ]; then [ 4434.390337] echo $(hostname) env are INSANE!; [ 4434.390337] exit 1; [ 4434.390337] fi [ 4435.391421] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4435.661821] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4435.922487] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4436.171804] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4436.324914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.384302] LustreError: 9470:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4436.393948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.544526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4436.806188] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4437.295695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4437.301236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4437.454647] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4437.469477] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4437.614682] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4437.884246] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4438.144361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4438.405899] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4438.661489] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4440.724787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4440.877040] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4441.390090] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491245572/real 1491245572] req@ffff8800667f4300 x1563680145226080/t0(0) o8->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491245577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4443.612618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4443.761420] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4446.555686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4446.705152] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4449.438983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4449.600806] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4452.389266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4452.547528] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4455.275269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4455.445604] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4458.175860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4458.334931] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4460.492695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4460.645083] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4462.077133] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 4462.403515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 4462.564486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4462.697684] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 4468.055820] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4468.691203] Lustre: DEBUG MARKER: rc=0; [ 4468.691203] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4468.691203] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4468.691203] echo $(hostname -s): $val; [ 4468.691203] rc=$val; [ 4468.691203] fi; [ 4468.691203] exit $rc [ 4469.293336] Lustre: DEBUG MARKER: dmesg [ 4469.785340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 18:53:24 \(1491245604\) [ 4469.949694] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 18:53:24 (1491245604) [ 4470.107330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4470.107330] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4470.107330] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4471.499664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x327 [ 4471.754247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4471.884763] Lustre: *** cfs_fail_loc=327, val=0*** [ 4482.071713] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4482.328142] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4483.723724] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4484.340785] Lustre: DEBUG MARKER: rc=0; [ 4484.340785] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4484.340785] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4484.340785] echo $(hostname -s): $val; [ 4484.340785] rc=$val; [ 4484.340785] fi; [ 4484.340785] exit $rc [ 4484.943421] Lustre: DEBUG MARKER: dmesg [ 4485.435986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 18:53:40 \(1491245620\) [ 4485.593622] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 18:53:40 (1491245620) [ 4485.749534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4485.749534] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4485.749534] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4486.032688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.lock_reclaim_threshold_mb [ 4486.288146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=0 [ 4486.541310] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x328 [ 4486.797603] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4486.933548] Lustre: *** cfs_fail_loc=328, val=0*** [ 4486.935468] Lustre: Skipped 512 previous similar messages [ 4493.664490] Lustre: *** cfs_fail_loc=328, val=0*** [ 4493.666139] Lustre: Skipped 501 previous similar messages [ 4499.665469] Lustre: *** cfs_fail_loc=328, val=0*** [ 4505.666290] Lustre: *** cfs_fail_loc=328, val=0*** [ 4507.067617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4507.328117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4507.582987] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=336m [ 4512.832960] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4513.455329] Lustre: DEBUG MARKER: rc=0; [ 4513.455329] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4513.455329] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4513.455329] echo $(hostname -s): $val; [ 4513.455329] rc=$val; [ 4513.455329] fi; [ 4513.455329] exit $rc [ 4514.062735] Lustre: DEBUG MARKER: dmesg [ 4514.547763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 140: Check reasonable stack depth \(shouldn\'t LBUG\) ==================================== 18:54:09 \(1491245649\) [ 4514.713211] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 18:54:09 (1491245649) [ 4516.095520] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4516.722681] Lustre: DEBUG MARKER: rc=0; [ 4516.722681] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4516.722681] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4516.722681] echo $(hostname -s): $val; [ 4516.722681] rc=$val; [ 4516.722681] fi; [ 4516.722681] exit $rc [ 4517.362622] Lustre: DEBUG MARKER: dmesg [ 4517.873263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150: truncate\/append tests ============================================================ 18:54:12 \(1491245652\) [ 4518.043467] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 18:54:12 (1491245652) [ 4518.748152] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4519.371793] Lustre: DEBUG MARKER: rc=0; [ 4519.371793] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4519.371793] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4519.371793] echo $(hostname -s): $val; [ 4519.371793] rc=$val; [ 4519.371793] fi; [ 4519.371793] exit $rc [ 4519.982444] Lustre: DEBUG MARKER: dmesg [ 4520.464886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 151: test cache on oss and controls =================================================== 18:54:15 \(1491245655\) [ 4520.629379] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 18:54:15 (1491245655) [ 4528.575761] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4529.197182] Lustre: DEBUG MARKER: rc=0; [ 4529.197182] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4529.197182] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4529.197182] echo $(hostname -s): $val; [ 4529.197182] rc=$val; [ 4529.197182] fi; [ 4529.197182] exit $rc [ 4529.793195] Lustre: DEBUG MARKER: dmesg [ 4530.276538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 152: test read\/write with enomem ====================================================== 18:54:25 \(1491245665\) [ 4530.434635] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 18:54:25 (1491245665) [ 4530.699066] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4531.313871] Lustre: DEBUG MARKER: rc=0; [ 4531.313871] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4531.313871] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4531.313871] echo $(hostname -s): $val; [ 4531.313871] rc=$val; [ 4531.313871] fi; [ 4531.313871] exit $rc [ 4531.908560] Lustre: DEBUG MARKER: dmesg [ 4532.393899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 153: test if fdatasync does not crash ================================================= 18:54:27 \(1491245667\) [ 4532.552941] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 18:54:27 (1491245667) [ 4532.799407] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4533.433488] Lustre: DEBUG MARKER: rc=0; [ 4533.433488] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4533.433488] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4533.433488] echo $(hostname -s): $val; [ 4533.433488] rc=$val; [ 4533.433488] fi; [ 4533.433488] exit $rc [ 4534.038912] Lustre: DEBUG MARKER: dmesg [ 4534.525234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 18:54:29 \(1491245669\) [ 4534.678683] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 18:54:29 (1491245669) [ 4534.814191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4534.814191] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4534.814191] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4535.153123] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4535.781600] Lustre: DEBUG MARKER: rc=0; [ 4535.781600] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4535.781600] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4535.781600] echo $(hostname -s): $val; [ 4535.781600] rc=$val; [ 4535.781600] fi; [ 4535.781600] exit $rc [ 4536.387280] Lustre: DEBUG MARKER: dmesg [ 4536.862727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154B: verify the ll_decode_linkea tool ================================================ 18:54:31 \(1491245671\) [ 4537.024873] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ================================================ 18:54:31 (1491245671) [ 4537.159966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4537.159966] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4537.159966] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4537.517196] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4538.148788] Lustre: DEBUG MARKER: rc=0; [ 4538.148788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4538.148788] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4538.148788] echo $(hostname -s): $val; [ 4538.148788] rc=$val; [ 4538.148788] fi; [ 4538.148788] exit $rc [ 4538.742985] Lustre: DEBUG MARKER: dmesg [ 4539.228445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154a: Open-by-FID ===================================================================== 18:54:34 \(1491245674\) [ 4539.387440] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 18:54:34 (1491245674) [ 4539.524883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4539.524883] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4539.524883] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4539.752333] LustreError: 9455:0:(fld_handler.c:256:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 4539.783179] Lustre: 9455:0:(mdd_device.c:582:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [f154a.sanity] in .lustre/fid, FID format should be [0x200000400:0x1:0x0] [ 4539.806192] Lustre: 9455:0:(mdd_device.c:592:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [0x1:0x2:0x0] in .lustre/fid, sequence should be >= 0x200000400 or within [0xc,0xffffffff]. [ 4539.941254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4539.941254] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4539.941254] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4540.257209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4540.257209] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4540.257209] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4540.638652] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4541.275879] Lustre: DEBUG MARKER: rc=0; [ 4541.275879] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4541.275879] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4541.275879] echo $(hostname -s): $val; [ 4541.275879] rc=$val; [ 4541.275879] fi; [ 4541.275879] exit $rc [ 4541.875251] Lustre: DEBUG MARKER: dmesg [ 4542.356137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154b: Open-by-FID for remote directory ================================================ 18:54:37 \(1491245677\) [ 4542.513411] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 18:54:37 (1491245677) [ 4542.656193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4542.656193] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4542.656193] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4542.999609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_154b needs \>= 2 MDTs [ 4543.150516] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 4543.345545] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4543.975506] Lustre: DEBUG MARKER: rc=0; [ 4543.975506] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4543.975506] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4543.975506] echo $(hostname -s): $val; [ 4543.975506] rc=$val; [ 4543.975506] fi; [ 4543.975506] exit $rc [ 4544.575363] Lustre: DEBUG MARKER: dmesg [ 4545.066655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 18:54:40 \(1491245680\) [ 4545.237560] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 18:54:40 (1491245680) [ 4545.372465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4545.372465] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4545.372465] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4545.738723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4546.370690] Lustre: DEBUG MARKER: rc=0; [ 4546.370690] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4546.370690] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4546.370690] echo $(hostname -s): $val; [ 4546.370690] rc=$val; [ 4546.370690] fi; [ 4546.370690] exit $rc [ 4546.965275] Lustre: DEBUG MARKER: dmesg [ 4547.450224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154d: Verify open file fid ============================================================ 18:54:42 \(1491245682\) [ 4547.621346] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 18:54:42 (1491245682) [ 4547.774752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4547.774752] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4547.774752] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4548.081694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.exports.'10\.9\.5\.237@tcp'.open_files [ 4548.431248] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4549.063319] Lustre: DEBUG MARKER: rc=0; [ 4549.063319] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4549.063319] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4549.063319] echo $(hostname -s): $val; [ 4549.063319] rc=$val; [ 4549.063319] fi; [ 4549.063319] exit $rc [ 4549.663726] Lustre: DEBUG MARKER: dmesg [ 4550.129608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154e: .lustre is not returned by readdir ============================================== 18:54:45 \(1491245685\) [ 4550.300373] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 18:54:45 (1491245685) [ 4550.438105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4550.438105] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4550.438105] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4550.784364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4551.401360] Lustre: DEBUG MARKER: rc=0; [ 4551.401360] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4551.401360] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4551.401360] echo $(hostname -s): $val; [ 4551.401360] rc=$val; [ 4551.401360] fi; [ 4551.401360] exit $rc [ 4551.993852] Lustre: DEBUG MARKER: dmesg [ 4552.466745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154f: get parent fids by reading link ea ============================================== 18:54:47 \(1491245687\) [ 4552.625945] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 18:54:47 (1491245687) [ 4552.940910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4553.561229] Lustre: DEBUG MARKER: rc=0; [ 4553.561229] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4553.561229] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4553.561229] echo $(hostname -s): $val; [ 4553.561229] rc=$val; [ 4553.561229] fi; [ 4553.561229] exit $rc [ 4554.151558] Lustre: DEBUG MARKER: dmesg [ 4554.641353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154g: various llapi FID tests ========================================================= 18:54:49 \(1491245689\) [ 4554.798780] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 18:54:49 (1491245689) [ 4554.933388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4554.933388] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4554.933388] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4589.363361] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4589.982628] Lustre: DEBUG MARKER: rc=0; [ 4589.982628] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4589.982628] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4589.982628] echo $(hostname -s): $val; [ 4589.982628] rc=$val; [ 4589.982628] fi; [ 4589.982628] exit $rc [ 4590.584157] Lustre: DEBUG MARKER: dmesg [ 4591.109818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 18:55:26 \(1491245726\) [ 4591.283459] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 18:55:26 (1491245726) [ 4619.809879] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4620.451901] Lustre: DEBUG MARKER: rc=0; [ 4620.451901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4620.451901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4620.451901] echo $(hostname -s): $val; [ 4620.451901] rc=$val; [ 4620.451901] fi; [ 4620.451901] exit $rc [ 4621.043425] Lustre: DEBUG MARKER: dmesg [ 4621.532153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 18:55:56 \(1491245756\) [ 4621.685160] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 18:55:56 (1491245756) [ 4650.267687] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4650.902900] Lustre: DEBUG MARKER: rc=0; [ 4650.902900] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4650.902900] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4650.902900] echo $(hostname -s): $val; [ 4650.902900] rc=$val; [ 4650.902900] fi; [ 4650.902900] exit $rc [ 4651.489562] Lustre: DEBUG MARKER: dmesg [ 4651.965105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 18:56:27 \(1491245787\) [ 4652.124922] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 18:56:27 (1491245787) [ 4680.600507] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4681.237087] Lustre: DEBUG MARKER: rc=0; [ 4681.237087] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4681.237087] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4681.237087] echo $(hostname -s): $val; [ 4681.237087] rc=$val; [ 4681.237087] fi; [ 4681.237087] exit $rc [ 4681.835825] Lustre: DEBUG MARKER: dmesg [ 4682.320253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 18:56:57 \(1491245817\) [ 4682.474307] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 18:56:57 (1491245817) [ 4710.996395] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4711.637402] Lustre: DEBUG MARKER: rc=0; [ 4711.637402] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4711.637402] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4711.637402] echo $(hostname -s): $val; [ 4711.637402] rc=$val; [ 4711.637402] fi; [ 4711.637402] exit $rc [ 4712.236157] Lustre: DEBUG MARKER: dmesg [ 4712.794923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 18:57:27 \(1491245847\) [ 4712.959575] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 18:57:27 (1491245847) [ 4717.251575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4745.082572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4745.720905] Lustre: DEBUG MARKER: rc=0; [ 4745.720905] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4745.720905] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4745.720905] echo $(hostname -s): $val; [ 4745.720905] rc=$val; [ 4745.720905] fi; [ 4745.720905] exit $rc [ 4746.340818] Lustre: DEBUG MARKER: dmesg [ 4746.842871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 18:58:01 \(1491245881\) [ 4746.998738] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 18:58:01 (1491245881) [ 4751.299384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4779.085985] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4779.738404] Lustre: DEBUG MARKER: rc=0; [ 4779.738404] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4779.738404] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4779.738404] echo $(hostname -s): $val; [ 4779.738404] rc=$val; [ 4779.738404] fi; [ 4779.738404] exit $rc [ 4780.335126] Lustre: DEBUG MARKER: dmesg [ 4780.839635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 18:58:35 \(1491245915\) [ 4781.000762] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 18:58:35 (1491245915) [ 4785.329279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4813.186085] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4813.831015] Lustre: DEBUG MARKER: rc=0; [ 4813.831015] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4813.831015] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4813.831015] echo $(hostname -s): $val; [ 4813.831015] rc=$val; [ 4813.831015] fi; [ 4813.831015] exit $rc [ 4814.426040] Lustre: DEBUG MARKER: dmesg [ 4814.923540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 18:59:10 \(1491245950\) [ 4815.085851] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 18:59:10 (1491245950) [ 4819.397630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4847.057721] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4847.707168] Lustre: DEBUG MARKER: rc=0; [ 4847.707168] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4847.707168] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4847.707168] echo $(hostname -s): $val; [ 4847.707168] rc=$val; [ 4847.707168] fi; [ 4847.707168] exit $rc [ 4848.316743] Lustre: DEBUG MARKER: dmesg [ 4848.818672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 156: Verification of tunables ========================================================= 18:59:43 \(1491245983\) [ 4848.976155] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 18:59:43 (1491245983) [ 4857.104099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Turn on read and write cache [ 4857.269487] Lustre: DEBUG MARKER: Turn on read and write cache [ 4858.112719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write data and read it back. [ 4858.270656] Lustre: DEBUG MARKER: Write data and read it back. [ 4858.472705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read should be satisfied from the cache. [ 4858.628884] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 4859.581278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8276, after: 8279 [ 4859.745020] Lustre: DEBUG MARKER: cache hits:: before: 8276, after: 8279 [ 4859.945932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read again\; it should be satisfied from the cache. [ 4860.101898] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 4860.649636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8279, after: 8282 [ 4860.812194] Lustre: DEBUG MARKER: cache hits:: before: 8279, after: 8282 [ 4861.021291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Turn off the read cache and turn on the write cache [ 4861.180306] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 4862.035492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read again\; it should be satisfied from the cache. [ 4862.196458] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 4863.074334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8282, after: 8285 [ 4863.229889] Lustre: DEBUG MARKER: cache hits:: before: 8282, after: 8285 [ 4863.432785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read again\; it should not be satisfied from the cache. [ 4863.602179] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 4864.134842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8285, after: 8285 [ 4864.291504] Lustre: DEBUG MARKER: cache hits:: before: 8285, after: 8285 [ 4864.498461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write data and read it back. [ 4864.661163] Lustre: DEBUG MARKER: Write data and read it back. [ 4864.861189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read should be satisfied from the cache. [ 4865.025676] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 4865.914570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8285, after: 8288 [ 4866.072248] Lustre: DEBUG MARKER: cache hits:: before: 8285, after: 8288 [ 4866.314590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read again\; it should not be satisfied from the cache. [ 4866.481639] Lustre: DEBUG MARKER: Read again; it should not be satisfied from the cache. [ 4867.019905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8288, after: 8288 [ 4867.178113] Lustre: DEBUG MARKER: cache hits:: before: 8288, after: 8288 [ 4867.378282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Turn off read and write cache [ 4867.538205] Lustre: DEBUG MARKER: Turn off read and write cache [ 4868.373502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write data and read it back [ 4868.529777] Lustre: DEBUG MARKER: Write data and read it back [ 4868.728414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark It should not be satisfied from the cache. [ 4868.905658] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 4869.766828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8288, after: 8288 [ 4869.930671] Lustre: DEBUG MARKER: cache hits:: before: 8288, after: 8288 [ 4870.129384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Turn on the read cache and turn off the write cache [ 4870.286213] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 4871.147476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write data and read it back [ 4871.305711] Lustre: DEBUG MARKER: Write data and read it back [ 4871.510485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark It should not be satisfied from the cache. [ 4871.673516] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 4872.561289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8288, after: 8288 [ 4872.718739] Lustre: DEBUG MARKER: cache hits:: before: 8288, after: 8288 [ 4872.925151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Read again\; it should be satisfied from the cache. [ 4873.081535] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 4873.942249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cache hits:: before: 8288, after: 8291 [ 4874.103707] Lustre: DEBUG MARKER: cache hits:: before: 8288, after: 8291 [ 4898.350254] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4899.023714] Lustre: DEBUG MARKER: rc=0; [ 4899.023714] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4899.023714] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4899.023714] echo $(hostname -s): $val; [ 4899.023714] rc=$val; [ 4899.023714] fi; [ 4899.023714] exit $rc [ 4899.615371] Lustre: DEBUG MARKER: dmesg [ 4900.103459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160a: changelog sanity ================================================================ 19:00:35 \(1491246035\) [ 4900.266150] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 19:00:35 (1491246035) [ 4900.417609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4900.417609] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4900.417609] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4900.687433] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4900.819472] Lustre: lustre-MDD0000: changelog on [ 4900.947763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4901.246750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4901.499104] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-MKDIR [ 4901.754866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4902.007800] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-CLOSE [ 4902.272397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4902.527725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+MKDIR [ 4902.784434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4903.038757] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+CLOSE [ 4903.343745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4903.606048] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4903.874057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4904.153391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4904.426349] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4904.681670] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4904.809648] Lustre: Failing over lustre-MDT0000 [ 4907.318518] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.322251] LustreError: Skipped 3 previous similar messages [ 4910.860064] Lustre: 24137:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491246040/real 1491246040] req@ffff88005cfb3900 x1563680145277824/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491246046 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4910.865818] Lustre: 24137:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4910.897519] Lustre: server umount lustre-MDT0000 complete [ 4911.025225] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4911.280376] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4911.551712] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4911.804761] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4912.054480] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4912.209865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.229510] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 4912.231684] Lustre: Skipped 29 previous similar messages [ 4912.271896] LustreError: 24554:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4912.281436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4912.288873] Lustre: lustre-MDD0000: changelog on [ 4912.434360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4912.691984] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4913.173767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4913.178194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4913.332071] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4913.338246] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4913.443603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4913.479344] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4913.735370] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4913.988241] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4914.251865] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4914.506198] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4914.765363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4915.020698] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl1 [ 4915.148383] Lustre: lustre-MDD0000: changelog off [ 4915.280795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4915.540760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4915.869712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4916.501656] Lustre: DEBUG MARKER: rc=0; [ 4916.501656] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4916.501656] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4916.501656] echo $(hostname -s): $val; [ 4916.501656] rc=$val; [ 4916.501656] fi; [ 4916.501656] exit $rc [ 4917.099506] Lustre: DEBUG MARKER: dmesg [ 4917.645893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160b: Verify that very long rename doesn\'t crash in changelog ========================= 19:00:52 \(1491246052\) [ 4917.806376] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 19:00:52 (1491246052) [ 4917.965021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4917.965021] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4917.965021] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4918.243435] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4918.374244] Lustre: lustre-MDD0000: changelog on [ 4918.509525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4919.547895] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4919.721650] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl2 [ 4919.850380] Lustre: lustre-MDD0000: changelog off [ 4920.048915] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4920.675834] Lustre: DEBUG MARKER: rc=0; [ 4920.675834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4920.675834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4920.675834] echo $(hostname -s): $val; [ 4920.675834] rc=$val; [ 4920.675834] fi; [ 4920.675834] exit $rc [ 4921.281095] Lustre: DEBUG MARKER: dmesg [ 4921.806346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160c: verify that changelog log catch the truncate event ============================== 19:00:56 \(1491246056\) [ 4921.973099] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 19:00:56 (1491246056) [ 4922.127401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4922.127401] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4922.127401] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4922.401361] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4922.528713] Lustre: lustre-MDD0000: changelog on [ 4922.666552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4922.928628] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-TRUNC [ 4923.202284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4923.474279] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+TRUNC [ 4923.760651] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl3 [ 4923.888951] Lustre: lustre-MDD0000: changelog off [ 4924.096230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4924.732223] Lustre: DEBUG MARKER: rc=0; [ 4924.732223] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4924.732223] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4924.732223] echo $(hostname -s): $val; [ 4924.732223] rc=$val; [ 4924.732223] fi; [ 4924.732223] exit $rc [ 4925.323158] Lustre: DEBUG MARKER: dmesg [ 4925.833873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160d: verify that changelog log catch the migrate event =============================== 19:01:00 \(1491246060\) [ 4925.992379] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 19:01:00 (1491246060) [ 4926.206771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_160d needs \>= 2 MDTs [ 4926.366774] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 4926.577957] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4927.203434] Lustre: DEBUG MARKER: rc=0; [ 4927.203434] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4927.203434] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4927.203434] echo $(hostname -s): $val; [ 4927.203434] rc=$val; [ 4927.203434] fi; [ 4927.203434] exit $rc [ 4927.801801] Lustre: DEBUG MARKER: dmesg [ 4928.376726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160e: changelog negative testing ====================================================== 19:01:03 \(1491246063\) [ 4928.543369] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing ====================================================== 19:01:03 (1491246063) [ 4928.688424] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4928.815386] Lustre: lustre-MDD0000: changelog on [ 4928.950557] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl77 [ 4929.084691] Lustre: 27959:0:(mdd_device.c:1448:mdd_changelog_user_purge()) lustre-MDD0000: No changelog for user 77; rc=0 [ 4929.287061] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4929.920719] Lustre: DEBUG MARKER: rc=0; [ 4929.920719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4929.920719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4929.920719] echo $(hostname -s): $val; [ 4929.920719] rc=$val; [ 4929.920719] fi; [ 4929.920719] exit $rc [ 4930.524355] Lustre: DEBUG MARKER: dmesg [ 4930.918321] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl4 [ 4931.047176] Lustre: lustre-MDD0000: changelog off [ 4931.265825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161a: link ea sanity ================================================================== 19:01:06 \(1491246066\) [ 4931.423723] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 19:01:06 (1491246066) [ 4933.911722] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4934.547185] Lustre: DEBUG MARKER: rc=0; [ 4934.547185] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4934.547185] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4934.547185] echo $(hostname -s): $val; [ 4934.547185] rc=$val; [ 4934.547185] fi; [ 4934.547185] exit $rc [ 4935.155437] Lustre: DEBUG MARKER: dmesg [ 4935.763805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161b: link ea sanity under remote directory =========================================== 19:01:10 \(1491246070\) [ 4935.926659] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 19:01:10 (1491246070) [ 4936.126399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_161b skipping remote directory test [ 4936.279066] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 4936.478947] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4937.105788] Lustre: DEBUG MARKER: rc=0; [ 4937.105788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4937.105788] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4937.105788] echo $(hostname -s): $val; [ 4937.105788] rc=$val; [ 4937.105788] fi; [ 4937.105788] exit $rc [ 4937.709311] Lustre: DEBUG MARKER: dmesg [ 4938.233236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 19:01:13 \(1491246073\) [ 4938.402676] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 19:01:13 (1491246073) [ 4938.568453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4938.568453] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4938.568453] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4938.848677] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4938.983591] Lustre: lustre-MDD0000: changelog on [ 4939.230638] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl5 [ 4939.360785] Lustre: lustre-MDD0000: changelog off [ 4939.561253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4940.188456] Lustre: DEBUG MARKER: rc=0; [ 4940.188456] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4940.188456] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4940.188456] echo $(hostname -s): $val; [ 4940.188456] rc=$val; [ 4940.188456] fi; [ 4940.188456] exit $rc [ 4940.778264] Lustre: DEBUG MARKER: dmesg [ 4941.263038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161d: create with concurrent .lustre\/fid access ======================================= 19:01:16 \(1491246076\) [ 4941.426227] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ======================================= 19:01:16 (1491246076) [ 4941.563456] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4946.834348] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl6 [ 4947.153558] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4947.801379] Lustre: DEBUG MARKER: rc=0; [ 4947.801379] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4947.801379] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4947.801379] echo $(hostname -s): $val; [ 4947.801379] rc=$val; [ 4947.801379] fi; [ 4947.801379] exit $rc [ 4948.393897] Lustre: DEBUG MARKER: dmesg [ 4948.903234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162a: path lookup sanity ============================================================== 19:01:24 \(1491246084\) [ 4949.063794] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 19:01:24 (1491246084) [ 4949.359206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4949.989329] Lustre: DEBUG MARKER: rc=0; [ 4949.989329] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4949.989329] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4949.989329] echo $(hostname -s): $val; [ 4949.989329] rc=$val; [ 4949.989329] fi; [ 4949.989329] exit $rc [ 4950.579936] Lustre: DEBUG MARKER: dmesg [ 4951.060656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162b: striped directory path lookup sanity ============================================ 19:01:26 \(1491246086\) [ 4951.219871] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 19:01:26 (1491246086) [ 4951.448428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_162b needs \>= 2 MDTs [ 4951.611308] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 4951.827625] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4952.477253] Lustre: DEBUG MARKER: rc=0; [ 4952.477253] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4952.477253] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4952.477253] echo $(hostname -s): $val; [ 4952.477253] rc=$val; [ 4952.477253] fi; [ 4952.477253] exit $rc [ 4953.094883] Lustre: DEBUG MARKER: dmesg [ 4953.589166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 19:01:28 \(1491246088\) [ 4953.761375] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 19:01:28 (1491246088) [ 4953.899714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4953.899714] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4953.899714] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4956.526253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4957.164317] Lustre: DEBUG MARKER: rc=0; [ 4957.164317] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4957.164317] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4957.164317] echo $(hostname -s): $val; [ 4957.164317] rc=$val; [ 4957.164317] fi; [ 4957.164317] exit $rc [ 4957.773917] Lustre: DEBUG MARKER: dmesg [ 4958.272480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 169: parallel read and truncate should not deadlock =================================== 19:01:33 \(1491246093\) [ 4958.449617] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 19:01:33 (1491246093) [ 4958.669160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating a 10 Mb file [ 4958.855952] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4959.256149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark starting reads [ 4959.410387] Lustre: DEBUG MARKER: starting reads [ 4959.656363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating the file [ 4959.943259] Lustre: DEBUG MARKER: truncating the file [ 4960.154315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark killing dd [ 4960.311141] Lustre: DEBUG MARKER: killing dd [ 4960.537075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing the temporary file [ 4960.690885] Lustre: DEBUG MARKER: removing the temporary file [ 4960.896714] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4961.530137] Lustre: DEBUG MARKER: rc=0; [ 4961.530137] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4961.530137] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4961.530137] echo $(hostname -s): $val; [ 4961.530137] rc=$val; [ 4961.530137] fi; [ 4961.530137] exit $rc [ 4962.136287] Lustre: DEBUG MARKER: dmesg [ 4962.634711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 170: test lctl df to handle corrupted log ============================================= 19:01:37 \(1491246097\) [ 4962.799399] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 19:01:37 (1491246097) [ 4965.062846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4965.727925] Lustre: DEBUG MARKER: rc=0; [ 4965.727925] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4965.727925] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4965.727925] echo $(hostname -s): $val; [ 4965.727925] rc=$val; [ 4965.727925] fi; [ 4965.727925] exit $rc [ 4966.326362] Lustre: DEBUG MARKER: dmesg [ 4966.843466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit\(\) ============================== 19:01:41 \(1491246101\) [ 4967.003745] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 19:01:41 (1491246101) [ 4970.254307] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4970.900063] Lustre: DEBUG MARKER: rc=0; [ 4970.900063] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4970.900063] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4970.900063] echo $(hostname -s): $val; [ 4970.900063] rc=$val; [ 4970.900063] fi; [ 4970.900063] exit $rc [ 4971.499116] Lustre: DEBUG MARKER: dmesg [ 4971.988038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 180a: test obdecho on osc ============================================================= 19:01:47 \(1491246107\) [ 4972.147832] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ============================================================= 19:01:47 (1491246107) [ 4972.685725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4973.344656] Lustre: DEBUG MARKER: rc=0; [ 4973.344656] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4973.344656] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4973.344656] echo $(hostname -s): $val; [ 4973.344656] rc=$val; [ 4973.344656] fi; [ 4973.344656] exit $rc [ 4973.946420] Lustre: DEBUG MARKER: dmesg [ 4974.470667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 180b: test obdecho directly on obdfilter ============================================== 19:01:49 \(1491246109\) [ 4974.632438] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter ============================================== 19:01:49 (1491246109) [ 4978.291096] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4978.942678] Lustre: DEBUG MARKER: rc=0; [ 4978.942678] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4978.942678] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4978.942678] echo $(hostname -s): $val; [ 4978.942678] rc=$val; [ 4978.942678] fi; [ 4978.942678] exit $rc [ 4979.563519] Lustre: DEBUG MARKER: dmesg [ 4980.076621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 180c: test huge bulk I\/O size on obdfilter, don\'t LASSERT ============================= 19:01:55 \(1491246115\) [ 4980.230332] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ============================= 19:01:55 (1491246115) [ 4980.403676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 4980.403676] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 4980.403676] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 4981.179894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_module obdecho\/obdecho [ 4981.351951] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_module obdecho/obdecho [ 5009.576989] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5010.217847] Lustre: DEBUG MARKER: rc=0; [ 5010.217847] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5010.217847] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5010.217847] echo $(hostname -s): $val; [ 5010.217847] rc=$val; [ 5010.217847] fi; [ 5010.217847] exit $rc [ 5010.828752] Lustre: DEBUG MARKER: dmesg [ 5011.327191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 181: Test open-unlinked dir =========================================================== 19:02:26 \(1491246146\) [ 5011.507415] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 19:02:26 (1491246146) [ 5020.554775] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5021.194435] Lustre: DEBUG MARKER: rc=0; [ 5021.194435] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5021.194435] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5021.194435] echo $(hostname -s): $val; [ 5021.194435] rc=$val; [ 5021.194435] fi; [ 5021.194435] exit $rc [ 5021.813501] Lustre: DEBUG MARKER: dmesg [ 5022.413868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 182: Test parallel modify metadata operations ========================================= 19:02:37 \(1491246157\) [ 5022.571108] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 19:02:37 (1491246157) [ 5030.023835] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5030.653606] Lustre: DEBUG MARKER: rc=0; [ 5030.653606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5030.653606] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5030.653606] echo $(hostname -s): $val; [ 5030.653606] rc=$val; [ 5030.653606] fi; [ 5030.653606] exit $rc [ 5031.262759] Lustre: DEBUG MARKER: dmesg [ 5031.755896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 183: No crash or request leak in case of strange dispositions ========================= 19:02:46 \(1491246166\) [ 5031.936968] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========================= 19:02:46 (1491246166) [ 5032.087599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5032.087599] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5032.087599] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5032.482426] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x148 [ 5032.649618] Lustre: *** cfs_fail_loc=148, val=0*** [ 5032.803863] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 5033.205089] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5033.902440] Lustre: DEBUG MARKER: rc=0; [ 5033.902440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5033.902440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5033.902440] echo $(hostname -s): $val; [ 5033.902440] rc=$val; [ 5033.902440] fi; [ 5033.902440] exit $rc [ 5034.511047] Lustre: DEBUG MARKER: dmesg [ 5035.000542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184a: Basic layout swap =============================================================== 19:02:50 \(1491246170\) [ 5035.166276] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =============================================================== 19:02:50 (1491246170) [ 5035.435484] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5036.081080] Lustre: DEBUG MARKER: rc=0; [ 5036.081080] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5036.081080] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5036.081080] echo $(hostname -s): $val; [ 5036.081080] rc=$val; [ 5036.081080] fi; [ 5036.081080] exit $rc [ 5036.683477] Lustre: DEBUG MARKER: dmesg [ 5037.165808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184b: Forbidden layout swap \(will generate errors\) ==================================== 19:02:52 \(1491246172\) [ 5037.320993] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ==================================== 19:02:52 (1491246172) [ 5037.612910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5038.246828] Lustre: DEBUG MARKER: rc=0; [ 5038.246828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5038.246828] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5038.246828] echo $(hostname -s): $val; [ 5038.246828] rc=$val; [ 5038.246828] fi; [ 5038.246828] exit $rc [ 5038.842318] Lustre: DEBUG MARKER: dmesg [ 5039.354532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184c: Concurrent write and layout swap ================================================ 19:02:54 \(1491246174\) [ 5039.519604] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ================================================ 19:02:54 (1491246174) [ 5050.226936] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5050.844383] Lustre: DEBUG MARKER: rc=0; [ 5050.844383] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5050.844383] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5050.844383] echo $(hostname -s): $val; [ 5050.844383] rc=$val; [ 5050.844383] fi; [ 5050.844383] exit $rc [ 5051.433971] Lustre: DEBUG MARKER: dmesg [ 5051.946706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184d: allow stripeless layouts swap =================================================== 19:03:07 \(1491246187\) [ 5052.111181] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap =================================================== 19:03:07 (1491246187) [ 5052.360886] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5053.004553] Lustre: DEBUG MARKER: rc=0; [ 5053.004553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5053.004553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5053.004553] echo $(hostname -s): $val; [ 5053.004553] rc=$val; [ 5053.004553] fi; [ 5053.004553] exit $rc [ 5053.606545] Lustre: DEBUG MARKER: dmesg [ 5054.099883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 19:03:09 \(1491246189\) [ 5054.256399] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 19:03:09 (1491246189) [ 5054.397938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5054.397938] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5054.397938] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5054.794853] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5055.456864] Lustre: DEBUG MARKER: rc=0; [ 5055.456864] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5055.456864] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5055.456864] echo $(hostname -s): $val; [ 5055.456864] rc=$val; [ 5055.456864] fi; [ 5055.456864] exit $rc [ 5056.082426] Lustre: DEBUG MARKER: dmesg [ 5056.576784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 185: Volatile file support ============================================================ 19:03:11 \(1491246191\) [ 5056.733990] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ============================================================ 19:03:11 (1491246191) [ 5056.883716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5056.883716] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5056.883716] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5058.259644] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5058.894034] Lustre: DEBUG MARKER: rc=0; [ 5058.894034] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5058.894034] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5058.894034] echo $(hostname -s): $val; [ 5058.894034] rc=$val; [ 5058.894034] fi; [ 5058.894034] exit $rc [ 5059.500656] Lustre: DEBUG MARKER: dmesg [ 5060.011688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 187a: Test data version change ======================================================== 19:03:15 \(1491246195\) [ 5060.176346] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ======================================================== 19:03:15 (1491246195) [ 5060.328042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5060.328042] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5060.328042] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5061.692949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5062.330371] Lustre: DEBUG MARKER: rc=0; [ 5062.330371] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5062.330371] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5062.330371] echo $(hostname -s): $val; [ 5062.330371] rc=$val; [ 5062.330371] fi; [ 5062.330371] exit $rc [ 5062.928285] Lustre: DEBUG MARKER: dmesg [ 5063.426384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 187b: Test data version change on volatile file ======================================= 19:03:18 \(1491246198\) [ 5063.581612] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ======================================= 19:03:18 (1491246198) [ 5063.740595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5063.740595] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5063.740595] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5064.160758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5064.792359] Lustre: DEBUG MARKER: rc=0; [ 5064.792359] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5064.792359] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5064.792359] echo $(hostname -s): $val; [ 5064.792359] rc=$val; [ 5064.792359] fi; [ 5064.792359] exit $rc [ 5065.408183] Lustre: DEBUG MARKER: dmesg [ 5065.885997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 200: OST pools ======================================================================== 19:03:20 \(1491246200\) [ 5066.052476] Lustre: DEBUG MARKER: == sanity test 200: OST pools ======================================================================== 19:03:20 (1491246200) [ 5066.191908] Lustre: DEBUG MARKER: lctl pool_new lustre.cea1 [ 5074.453027] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 2>/dev/null || echo foo [ 5074.711788] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 2>/dev/null || echo foo [ 5075.020245] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.cea1 lustre-OST[0-6/2] [ 5081.292427] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 5081.292427] sort -u | tr '\n' ' ' [ 5081.551357] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 5081.551357] sort -u | tr '\n' ' ' [ 5085.377360] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.cea1 lustre-OST0000_UUID [ 5093.640700] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 5093.640700] grep lustre-OST0000_UUID [ 5093.900344] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 5093.900344] grep lustre-OST0000_UUID [ 5094.206683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.cea1 lustre-OST0002_UUID [ 5100.465311] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.cea1 lustre-OST0004_UUID [ 5110.724787] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.cea1 lustre-OST0006_UUID [ 5120.986896] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 [ 5121.239095] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 [ 5121.535666] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.cea1 [ 5131.905029] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5132.549139] Lustre: DEBUG MARKER: rc=0; [ 5132.549139] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5132.549139] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5132.549139] echo $(hostname -s): $val; [ 5132.549139] rc=$val; [ 5132.549139] fi; [ 5132.549139] exit $rc [ 5133.156152] Lustre: DEBUG MARKER: dmesg [ 5133.643629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204a: Print default stripe attributes ================================================= 19:04:28 \(1491246268\) [ 5133.798863] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ================================================= 19:04:28 (1491246268) [ 5134.027299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5134.652709] Lustre: DEBUG MARKER: rc=0; [ 5134.652709] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5134.652709] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5134.652709] echo $(hostname -s): $val; [ 5134.652709] rc=$val; [ 5134.652709] fi; [ 5134.652709] exit $rc [ 5135.266575] Lustre: DEBUG MARKER: dmesg [ 5135.748920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204b: Print default stripe size and offset =========================================== 19:04:30 \(1491246270\) [ 5135.928440] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset =========================================== 19:04:30 (1491246270) [ 5136.141561] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5136.767950] Lustre: DEBUG MARKER: rc=0; [ 5136.767950] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5136.767950] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5136.767950] echo $(hostname -s): $val; [ 5136.767950] rc=$val; [ 5136.767950] fi; [ 5136.767950] exit $rc [ 5137.376123] Lustre: DEBUG MARKER: dmesg [ 5137.872228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204c: Print default stripe count and offset =========================================== 19:04:32 \(1491246272\) [ 5138.032518] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset =========================================== 19:04:32 (1491246272) [ 5138.245340] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5138.863317] Lustre: DEBUG MARKER: rc=0; [ 5138.863317] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5138.863317] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5138.863317] echo $(hostname -s): $val; [ 5138.863317] rc=$val; [ 5138.863317] fi; [ 5138.863317] exit $rc [ 5139.469591] Lustre: DEBUG MARKER: dmesg [ 5140.072535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204d: Print default stripe count and size ============================================= 19:04:35 \(1491246275\) [ 5140.232802] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ============================================= 19:04:35 (1491246275) [ 5140.455394] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5141.081954] Lustre: DEBUG MARKER: rc=0; [ 5141.081954] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5141.081954] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5141.081954] echo $(hostname -s): $val; [ 5141.081954] rc=$val; [ 5141.081954] fi; [ 5141.081954] exit $rc [ 5141.680170] Lustre: DEBUG MARKER: dmesg [ 5142.175908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204e: Print raw stripe attributes ===================================================== 19:04:37 \(1491246277\) [ 5142.332959] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ===================================================== 19:04:37 (1491246277) [ 5142.543456] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5143.175130] Lustre: DEBUG MARKER: rc=0; [ 5143.175130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5143.175130] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5143.175130] echo $(hostname -s): $val; [ 5143.175130] rc=$val; [ 5143.175130] fi; [ 5143.175130] exit $rc [ 5143.787173] Lustre: DEBUG MARKER: dmesg [ 5144.296506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204f: Print raw stripe size and offset =============================================== 19:04:39 \(1491246279\) [ 5144.454807] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset =============================================== 19:04:39 (1491246279) [ 5144.664126] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5145.284614] Lustre: DEBUG MARKER: rc=0; [ 5145.284614] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5145.284614] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5145.284614] echo $(hostname -s): $val; [ 5145.284614] rc=$val; [ 5145.284614] fi; [ 5145.284614] exit $rc [ 5145.888935] Lustre: DEBUG MARKER: dmesg [ 5146.402188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204g: Print raw stripe count and offset =============================================== 19:04:41 \(1491246281\) [ 5146.561686] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset =============================================== 19:04:41 (1491246281) [ 5146.768674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5147.413427] Lustre: DEBUG MARKER: rc=0; [ 5147.413427] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5147.413427] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5147.413427] echo $(hostname -s): $val; [ 5147.413427] rc=$val; [ 5147.413427] fi; [ 5147.413427] exit $rc [ 5148.020649] Lustre: DEBUG MARKER: dmesg [ 5148.494422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204h: Print raw stripe count and size ================================================= 19:04:43 \(1491246283\) [ 5148.650843] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ================================================= 19:04:43 (1491246283) [ 5148.855367] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5149.478083] Lustre: DEBUG MARKER: rc=0; [ 5149.478083] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5149.478083] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5149.478083] echo $(hostname -s): $val; [ 5149.478083] rc=$val; [ 5149.478083] fi; [ 5149.478083] exit $rc [ 5150.077837] Lustre: DEBUG MARKER: dmesg [ 5150.583105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 205: Verify job stats ================================================================= 19:04:45 \(1491246285\) [ 5150.737581] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats ================================================================= 19:04:45 (1491246285) [ 5150.874151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5150.874151] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5150.874151] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5151.205812] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=nodelocal [ 5157.574658] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_register -n [ 5157.701288] Lustre: lustre-MDD0000: changelog on [ 5157.703119] Lustre: Skipped 1 previous similar message [ 5157.833291] Lustre: DEBUG MARKER: lctl get_param -n mdt.*.job_cleanup_interval [ 5158.088769] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.job_cleanup_interval=5 [ 5158.418312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mkdir \/mnt\/lustre\/d205.sanity [ 5158.569493] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [ 5158.777117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mkdir.28154 [ 5158.930061] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.28154 [ 5159.067633] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5159.397316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: rmdir \/mnt\/lustre\/d205.sanity [ 5159.574495] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [ 5159.772930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.rmdir.20651 [ 5159.930687] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.20651 [ 5160.065659] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5160.400746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mknod \/mnt\/lustre\/f205.sanity c 1 3 [ 5160.556347] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [ 5160.755351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mknod.8693 [ 5160.912824] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.8693 [ 5161.052140] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5161.380888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: rm -f \/mnt\/lustre\/f205.sanity [ 5161.533442] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [ 5161.731100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.rm.27208 [ 5161.889842] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.27208 [ 5162.026615] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5162.356338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: \/usr\/bin\/lfs setstripe -i 0 -c 1 \/mnt\/lustre\/f205.sanity [ 5162.517331] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [ 5162.716953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.lfs.24026 [ 5162.870296] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.24026 [ 5163.074277] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5163.414375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: touch \/mnt\/lustre\/f205.sanity [ 5163.575686] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [ 5163.786484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.touch.21236 [ 5163.944232] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.21236 [ 5164.083924] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5164.673587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: dd if=\/dev\/zero of=\/mnt\/lustre\/f205.sanity bs=1M count=1 oflag=sync [ 5164.835160] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [ 5165.036802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.dd.1424 [ 5165.191565] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.1424 [ 5165.722159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: dd if=\/mnt\/lustre\/f205.sanity of=\/dev\/null bs=1M count=1 iflag=direct [ 5165.880802] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [ 5166.094302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.dd.28236 [ 5166.253106] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.28236 [ 5166.742265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: \/usr\/lib64\/lustre\/tests\/truncate \/mnt\/lustre\/f205.sanity 0 [ 5166.897799] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [ 5167.095267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.truncate.27467 [ 5167.256628] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.27467 [ 5167.392671] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5167.978073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mv -f \/mnt\/lustre\/f205.sanity \/mnt\/lustre\/d205.sanity.rename [ 5168.135985] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [ 5168.340541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mv.23520 [ 5168.493397] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.23520 [ 5168.634190] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5168.974793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mkdir \/mnt\/lustre\/d205.sanity.expire [ 5169.152635] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [ 5169.357344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mkdir.18223 [ 5169.549191] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.18223 [ 5169.690665] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 5169.951385] Lustre: DEBUG MARKER: lctl get_param *.*.job_stats [ 5170.211703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5170.211703] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5170.211703] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5170.500458] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=disable [ 5176.889004] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.job_cleanup_interval=600 [ 5177.143153] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=procname_uid [ 5184.534269] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl7 [ 5184.664463] Lustre: lustre-MDD0000: changelog off [ 5184.666636] Lustre: Skipped 1 previous similar message [ 5184.859405] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5185.504069] Lustre: DEBUG MARKER: rc=0; [ 5185.504069] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5185.504069] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5185.504069] echo $(hostname -s): $val; [ 5185.504069] rc=$val; [ 5185.504069] fi; [ 5185.504069] exit $rc [ 5186.114351] Lustre: DEBUG MARKER: dmesg [ 5186.626719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 206: fail lov_init_raid0\(\) doesn\'t lbug =============================================== 19:05:21 \(1491246321\) [ 5186.780646] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug =============================================== 19:05:21 (1491246321) [ 5187.002756] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5187.629174] Lustre: DEBUG MARKER: rc=0; [ 5187.629174] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5187.629174] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5187.629174] echo $(hostname -s): $val; [ 5187.629174] rc=$val; [ 5187.629174] fi; [ 5187.629174] exit $rc [ 5188.237960] Lustre: DEBUG MARKER: dmesg [ 5188.751827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207a: can refresh layout at glimpse =================================================== 19:05:23 \(1491246323\) [ 5188.913668] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse =================================================== 19:05:23 (1491246323) [ 5189.127770] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5189.754039] Lustre: DEBUG MARKER: rc=0; [ 5189.754039] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5189.754039] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5189.754039] echo $(hostname -s): $val; [ 5189.754039] rc=$val; [ 5189.754039] fi; [ 5189.754039] exit $rc [ 5190.354543] Lustre: DEBUG MARKER: dmesg [ 5190.839471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207b: can refresh layout at open ====================================================== 19:05:25 \(1491246325\) [ 5191.004658] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ====================================================== 19:05:25 (1491246325) [ 5191.237694] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5191.874766] Lustre: DEBUG MARKER: rc=0; [ 5191.874766] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5191.874766] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5191.874766] echo $(hostname -s): $val; [ 5191.874766] rc=$val; [ 5191.874766] fi; [ 5191.874766] exit $rc [ 5192.491135] Lustre: DEBUG MARKER: dmesg [ 5192.984713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 208: Exclusive open =================================================================== 19:05:28 \(1491246328\) [ 5193.145080] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open =================================================================== 19:05:28 (1491246328) [ 5193.298503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5193.298503] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5193.298503] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5196.602743] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5196.857937] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5196.986589] Lustre: Failing over lustre-MDT0000 [ 5197.324779] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.324783] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.324785] LustreError: Skipped 1 previous similar message [ 5197.335142] LustreError: Skipped 5 previous similar messages [ 5203.026043] Lustre: 12962:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491246332/real 1491246332] req@ffff8800649cc900 x1563680145523408/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491246338 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5203.143886] Lustre: server umount lustre-MDT0000 complete [ 5203.271676] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5213.594128] Lustre: DEBUG MARKER: hostname [ 5213.920220] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5214.174635] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5214.423771] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5214.579123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.642675] LustreError: 13382:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 5214.653183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.802197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5215.064755] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5215.557428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5215.557501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5215.718489] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5215.720111] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5215.864167] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5216.122302] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5216.375205] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5216.636847] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5217.412653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.452680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.633305] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.672079] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5221.560977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5224.324061] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5224.891427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5224.984536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5225.039736] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5225.191516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5227.516061] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5227.776379] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5227.906130] Lustre: Failing over lustre-MDT0000 [ 5233.992635] Lustre: server umount lustre-MDT0000 complete [ 5234.123812] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5244.450991] Lustre: DEBUG MARKER: hostname [ 5244.787088] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5245.038656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5245.288488] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5245.448253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.526328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.675523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5245.936835] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5246.432644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5246.439567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5246.598329] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5246.600415] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5246.745694] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5247.004642] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5247.260024] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5247.524614] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5248.342799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5248.342801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5248.641823] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5248.655703] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.313882] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5253.587779] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5253.911534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5253.928173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5254.132071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5254.153338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5255.579389] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5256.231341] Lustre: DEBUG MARKER: rc=0; [ 5256.231341] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5256.231341] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5256.231341] echo $(hostname -s): $val; [ 5256.231341] rc=$val; [ 5256.231341] fi; [ 5256.231341] exit $rc [ 5256.854219] Lustre: DEBUG MARKER: dmesg [ 5257.360892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 209: read-only open\/close requests should be freed promptly =========================== 19:06:32 \(1491246392\) [ 5257.537810] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly =========================== 19:06:32 (1491246392) [ 5263.983191] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5264.644788] Lustre: DEBUG MARKER: rc=0; [ 5264.644788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5264.644788] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5264.644788] echo $(hostname -s): $val; [ 5264.644788] rc=$val; [ 5264.644788] fi; [ 5264.644788] exit $rc [ 5265.252060] Lustre: DEBUG MARKER: dmesg [ 5265.752366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 212: Sendfile test ==================================================================== 19:06:40 \(1491246400\) [ 5265.910354] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 19:06:40 (1491246400) [ 5267.419205] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5268.044119] Lustre: DEBUG MARKER: rc=0; [ 5268.044119] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5268.044119] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5268.044119] echo $(hostname -s): $val; [ 5268.044119] rc=$val; [ 5268.044119] fi; [ 5268.044119] exit $rc [ 5268.649389] Lustre: DEBUG MARKER: dmesg [ 5269.156237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 213: OSC lock completion and cancel race don\'t crash - bug 18829 ====================== 19:06:44 \(1491246404\) [ 5269.308816] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 19:06:44 (1491246404) [ 5269.549326] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5270.178357] Lustre: DEBUG MARKER: rc=0; [ 5270.178357] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5270.178357] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5270.178357] echo $(hostname -s): $val; [ 5270.178357] rc=$val; [ 5270.178357] fi; [ 5270.178357] exit $rc [ 5270.782120] Lustre: DEBUG MARKER: dmesg [ 5271.278015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 19:06:46 \(1491246406\) [ 5271.444402] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 19:06:46 (1491246406) [ 5272.928451] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5273.564314] Lustre: DEBUG MARKER: rc=0; [ 5273.564314] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5273.564314] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5273.564314] echo $(hostname -s): $val; [ 5273.564314] rc=$val; [ 5273.564314] fi; [ 5273.564314] exit $rc [ 5274.181914] Lustre: DEBUG MARKER: dmesg [ 5274.693272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 19:06:49 \(1491246409\) [ 5274.888540] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 19:06:49 (1491246409) [ 5275.202158] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5275.829648] Lustre: DEBUG MARKER: rc=0; [ 5275.829648] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5275.829648] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5275.829648] echo $(hostname -s): $val; [ 5275.829648] rc=$val; [ 5275.829648] fi; [ 5275.829648] exit $rc [ 5276.446197] Lustre: DEBUG MARKER: dmesg [ 5276.960055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 19:06:52 \(1491246412\) [ 5277.119206] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 19:06:52 (1491246412) [ 5320.324386] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5321.017639] Lustre: DEBUG MARKER: rc=0; [ 5321.017639] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5321.017639] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5321.017639] echo $(hostname -s): $val; [ 5321.017639] rc=$val; [ 5321.017639] fi; [ 5321.017639] exit $rc [ 5321.629430] Lustre: DEBUG MARKER: dmesg [ 5322.147058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 217: check lctl ping for hostnames with hiphen \(\'-\'\) ================================== 19:07:37 \(1491246457\) [ 5322.304144] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 19:07:37 (1491246457) [ 5322.810197] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [ 5323.439959] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5324.073522] Lustre: DEBUG MARKER: rc=0; [ 5324.073522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5324.073522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5324.073522] echo $(hostname -s): $val; [ 5324.073522] rc=$val; [ 5324.073522] fi; [ 5324.073522] exit $rc [ 5324.665093] Lustre: DEBUG MARKER: dmesg [ 5325.193329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 218: parallel read and truncate should not deadlock =================================== 19:07:40 \(1491246460\) [ 5325.361153] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 19:07:40 (1491246460) [ 5325.568423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating a 10 Mb file [ 5325.737866] Lustre: DEBUG MARKER: creating a 10 Mb file [ 5326.144361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark starting reads [ 5326.306787] Lustre: DEBUG MARKER: starting reads [ 5326.602526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating the file [ 5326.758769] Lustre: DEBUG MARKER: truncating the file [ 5326.974060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark killing dd [ 5327.137827] Lustre: DEBUG MARKER: killing dd [ 5327.343838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing the temporary file [ 5327.494852] Lustre: DEBUG MARKER: removing the temporary file [ 5327.693281] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5328.336878] Lustre: DEBUG MARKER: rc=0; [ 5328.336878] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5328.336878] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5328.336878] echo $(hostname -s): $val; [ 5328.336878] rc=$val; [ 5328.336878] fi; [ 5328.336878] exit $rc [ 5328.947052] Lustre: DEBUG MARKER: dmesg [ 5329.460699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 219: LU-394: Write partial won\'t cause uncontiguous pages vec at LND ================== 19:07:44 \(1491246464\) [ 5329.613088] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ================== 19:07:44 (1491246464) [ 5329.874236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5330.500769] Lustre: DEBUG MARKER: rc=0; [ 5330.500769] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5330.500769] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5330.500769] echo $(hostname -s): $val; [ 5330.500769] rc=$val; [ 5330.500769] fi; [ 5330.500769] exit $rc [ 5331.097418] Lustre: DEBUG MARKER: dmesg [ 5331.592411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 220: preallocated MDS objects still used if ENOSPC from OST =========================== 19:07:46 \(1491246466\) [ 5331.751102] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST =========================== 19:07:46 (1491246466) [ 5331.957664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5331.957664] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5331.957664] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5332.232228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5332.232228] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5332.232228] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5332.504988] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 5332.760280] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 5333.015606] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 5333.777780] Lustre: DEBUG MARKER: lctl pool_new lustre.test_220 [ 5342.042077] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_220 2>/dev/null || echo foo [ 5342.296027] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_220 2>/dev/null || echo foo [ 5342.598374] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_220 lustre-OST0000 [ 5350.915215] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 5351.171603] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 5351.931713] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_220 lustre-OST0000 [ 5360.201091] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_220 [ 5368.580993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5369.214795] Lustre: DEBUG MARKER: rc=0; [ 5369.214795] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5369.214795] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5369.214795] echo $(hostname -s): $val; [ 5369.214795] rc=$val; [ 5369.214795] fi; [ 5369.214795] exit $rc [ 5369.812360] Lustre: DEBUG MARKER: dmesg [ 5370.229611] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 5370.579147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 19:08:25 \(1491246505\) [ 5370.738284] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 19:08:25 (1491246505) [ 5371.416895] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5372.053208] Lustre: DEBUG MARKER: rc=0; [ 5372.053208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5372.053208] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5372.053208] echo $(hostname -s): $val; [ 5372.053208] rc=$val; [ 5372.053208] fi; [ 5372.053208] exit $rc [ 5372.657568] Lustre: DEBUG MARKER: dmesg [ 5373.238424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 19:08:28 \(1491246508\) [ 5373.401694] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 19:08:28 (1491246508) [ 5373.649467] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5374.283113] Lustre: DEBUG MARKER: rc=0; [ 5374.283113] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5374.283113] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5374.283113] echo $(hostname -s): $val; [ 5374.283113] rc=$val; [ 5374.283113] fi; [ 5374.283113] exit $rc [ 5374.890286] Lustre: DEBUG MARKER: dmesg [ 5375.407416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 19:08:30 \(1491246510\) [ 5375.564923] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 19:08:30 (1491246510) [ 5375.815726] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5376.455130] Lustre: DEBUG MARKER: rc=0; [ 5376.455130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5376.455130] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5376.455130] echo $(hostname -s): $val; [ 5376.455130] rc=$val; [ 5376.455130] fi; [ 5376.455130] exit $rc [ 5377.065275] Lustre: DEBUG MARKER: dmesg [ 5377.558515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 19:08:32 \(1491246512\) [ 5377.730849] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 19:08:32 (1491246512) [ 5377.983897] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5378.621087] Lustre: DEBUG MARKER: rc=0; [ 5378.621087] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5378.621087] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5378.621087] echo $(hostname -s): $val; [ 5378.621087] rc=$val; [ 5378.621087] fi; [ 5378.621087] exit $rc [ 5379.233753] Lustre: DEBUG MARKER: dmesg [ 5379.734764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224a: Don\'t panic on bulk IO failure ================================================== 19:08:34 \(1491246514\) [ 5379.899203] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ================================================== 19:08:34 (1491246514) [ 5380.185998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5380.821594] Lustre: DEBUG MARKER: rc=0; [ 5380.821594] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5380.821594] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5380.821594] echo $(hostname -s): $val; [ 5380.821594] rc=$val; [ 5380.821594] fi; [ 5380.821594] exit $rc [ 5381.457649] Lustre: DEBUG MARKER: dmesg [ 5381.968176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224b: Don\'t panic on bulk IO failure ================================================== 19:08:37 \(1491246517\) [ 5382.120157] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ================================================== 19:08:37 (1491246517) [ 5383.350786] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5383.984224] Lustre: DEBUG MARKER: rc=0; [ 5383.984224] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5383.984224] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5383.984224] echo $(hostname -s): $val; [ 5383.984224] rc=$val; [ 5383.984224] fi; [ 5383.984224] exit $rc [ 5384.582824] Lustre: DEBUG MARKER: dmesg [ 5385.081882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224c: Don\'t hang if one of md lost during large bulk RPC ============================== 19:08:40 \(1491246520\) [ 5385.243191] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 19:08:40 (1491246520) [ 5389.298751] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='0' [ 5396.697760] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.timeout='5' [ 5406.856278] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='600' [ 5417.286195] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.timeout='20' [ 5451.315352] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5452.160204] Lustre: DEBUG MARKER: rc=0; [ 5452.160204] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5452.160204] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5452.160204] echo $(hostname -s): $val; [ 5452.160204] rc=$val; [ 5452.160204] fi; [ 5452.160204] exit $rc [ 5452.750800] Lustre: DEBUG MARKER: dmesg [ 5453.259434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 19:09:48 \(1491246588\) [ 5453.418820] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 19:09:48 (1491246588) [ 5453.573384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5453.573384] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5453.573384] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5453.850936] Lustre: DEBUG MARKER: lctl dl [ 5454.399241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5455.354078] LustreError: 23748:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [ 5455.356166] LustreError: 23748:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [ 5455.668443] LustreError: 23814:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [ 5458.911410] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5459.532545] Lustre: DEBUG MARKER: rc=0; [ 5459.532545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5459.532545] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5459.532545] echo $(hostname -s): $val; [ 5459.532545] rc=$val; [ 5459.532545] fi; [ 5459.532545] exit $rc [ 5460.153773] Lustre: DEBUG MARKER: dmesg [ 5460.683776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 19:09:55 \(1491246595\) [ 5460.847241] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 19:09:55 (1491246595) [ 5461.005516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5461.005516] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5461.005516] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5461.296955] Lustre: DEBUG MARKER: lctl dl [ 5461.957885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5462.915490] LustreError: 25226:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [ 5462.917647] LustreError: 25226:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [ 5463.227353] LustreError: 25292:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [ 5463.231413] LustreError: 25292:0:(lod_object.c:1370:lod_verify_md_striping()) Skipped 1 previous similar message [ 5466.503949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5467.158245] Lustre: DEBUG MARKER: rc=0; [ 5467.158245] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5467.158245] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5467.158245] echo $(hostname -s): $val; [ 5467.158245] rc=$val; [ 5467.158245] fi; [ 5467.158245] exit $rc [ 5467.805294] Lustre: DEBUG MARKER: dmesg [ 5468.334239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 226a: call path2fid and fid2path on files of all type ================================= 19:10:03 \(1491246603\) [ 5468.493833] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ================================= 19:10:03 (1491246603) [ 5468.834925] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5469.463358] Lustre: DEBUG MARKER: rc=0; [ 5469.463358] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5469.463358] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5469.463358] echo $(hostname -s): $val; [ 5469.463358] rc=$val; [ 5469.463358] fi; [ 5469.463358] exit $rc [ 5470.067224] Lustre: DEBUG MARKER: dmesg [ 5470.593447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 19:10:05 \(1491246605\) [ 5470.751596] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 19:10:05 (1491246605) [ 5470.946835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_226b needs \>= 2 MDTs [ 5471.100684] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 5471.299157] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5471.921460] Lustre: DEBUG MARKER: rc=0; [ 5471.921460] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5471.921460] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5471.921460] echo $(hostname -s): $val; [ 5471.921460] rc=$val; [ 5471.921460] fi; [ 5471.921460] exit $rc [ 5472.524231] Lustre: DEBUG MARKER: dmesg [ 5473.035793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 227: running truncated executable does not cause OOM ================================== 19:10:08 \(1491246608\) [ 5473.197796] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ================================== 19:10:08 (1491246608) [ 5473.409446] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5474.039038] Lustre: DEBUG MARKER: rc=0; [ 5474.039038] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5474.039038] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5474.039038] echo $(hostname -s): $val; [ 5474.039038] rc=$val; [ 5474.039038] fi; [ 5474.039038] exit $rc [ 5474.652648] Lustre: DEBUG MARKER: dmesg [ 5475.173331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228a: try to reuse idle OI blocks ===================================================== 19:10:10 \(1491246610\) [ 5475.335482] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ===================================================== 19:10:10 (1491246610) [ 5486.257464] Lustre: DEBUG MARKER: sync [ 5492.648988] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.45" /dev/lvm-Role_MDS/P1 [ 5506.408237] Lustre: DEBUG MARKER: sync [ 5512.801969] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.45" /dev/lvm-Role_MDS/P1 [ 5513.117352] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5513.754984] Lustre: DEBUG MARKER: rc=0; [ 5513.754984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5513.754984] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5513.754984] echo $(hostname -s): $val; [ 5513.754984] rc=$val; [ 5513.754984] fi; [ 5513.754984] exit $rc [ 5514.367468] Lustre: DEBUG MARKER: dmesg [ 5514.902369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228b: idle OI blocks can be reused after MDT restart ================================== 19:10:49 \(1491246649\) [ 5515.055824] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ================================== 19:10:49 (1491246649) [ 5525.875939] Lustre: DEBUG MARKER: sync [ 5532.245055] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.46" /dev/lvm-Role_MDS/P1 [ 5544.051182] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5544.305732] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5544.432985] Lustre: Failing over lustre-MDT0000 [ 5544.583697] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5544.587653] LustreError: Skipped 10 previous similar messages [ 5550.518034] Lustre: 28059:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491246680/real 1491246680] req@ffff880054188900 x1563680145871328/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491246686 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5550.524311] Lustre: 28059:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5550.655994] Lustre: server umount lustre-MDT0000 complete [ 5550.783454] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5551.042035] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 5551.311896] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5551.565906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5551.813223] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5551.958497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.976454] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 5551.979908] Lustre: Skipped 39 previous similar messages [ 5552.032520] LustreError: 28475:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 5552.037095] LustreError: 28475:0:(obd_config.c:1392:class_process_proc_param()) Skipped 1 previous similar message [ 5552.049401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.172610] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5552.291711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5552.578698] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5553.071270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5553.074870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5553.231837] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5553.233476] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5553.378392] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5553.636338] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5553.890646] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5554.151952] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5554.409342] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5554.588770] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5556.807907] Lustre: DEBUG MARKER: sync [ 5563.160397] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.46" /dev/lvm-Role_MDS/P1 [ 5563.492416] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5564.138636] Lustre: DEBUG MARKER: rc=0; [ 5564.138636] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5564.138636] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5564.138636] echo $(hostname -s): $val; [ 5564.138636] rc=$val; [ 5564.138636] fi; [ 5564.138636] exit $rc [ 5564.742848] Lustre: DEBUG MARKER: dmesg [ 5565.246068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 19:11:40 \(1491246700\) [ 5565.406400] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 19:11:40 (1491246700) [ 5587.448728] Lustre: DEBUG MARKER: sync [ 5593.935728] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.49" /dev/lvm-Role_MDS/P1 [ 5619.648222] Lustre: DEBUG MARKER: sync [ 5626.194476] Lustre: DEBUG MARKER: debugfs -c -R "stat oi.16.49" /dev/lvm-Role_MDS/P1 [ 5626.526780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5627.161779] Lustre: DEBUG MARKER: rc=0; [ 5627.161779] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5627.161779] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5627.161779] echo $(hostname -s): $val; [ 5627.161779] rc=$val; [ 5627.161779] fi; [ 5627.161779] exit $rc [ 5627.777564] Lustre: DEBUG MARKER: dmesg [ 5628.377519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 229: getstripe\/stat\/rm\/attr changes work on released files ============================ 19:12:43 \(1491246763\) [ 5628.537815] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 19:12:43 (1491246763) [ 5628.674368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5628.674368] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5628.674368] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5629.042144] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5629.665501] Lustre: DEBUG MARKER: rc=0; [ 5629.665501] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5629.665501] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5629.665501] echo $(hostname -s): $val; [ 5629.665501] rc=$val; [ 5629.665501] fi; [ 5629.665501] exit $rc [ 5630.274273] Lustre: DEBUG MARKER: dmesg [ 5630.788243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230a: Create remote directory and files under the remote directory ==================== 19:12:45 \(1491246765\) [ 5630.948095] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ==================== 19:12:45 (1491246765) [ 5631.151071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230a needs \>= 2 MDTs [ 5631.301874] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 5631.497431] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5632.138554] Lustre: DEBUG MARKER: rc=0; [ 5632.138554] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5632.138554] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5632.138554] echo $(hostname -s): $val; [ 5632.138554] rc=$val; [ 5632.138554] fi; [ 5632.138554] exit $rc [ 5632.737172] Lustre: DEBUG MARKER: dmesg [ 5633.242377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230b: migrate directory =============================================================== 19:12:48 \(1491246768\) [ 5633.394068] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =============================================================== 19:12:48 (1491246768) [ 5633.584650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230b needs \>= 2 MDTs [ 5633.742126] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 5633.938784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5634.562959] Lustre: DEBUG MARKER: rc=0; [ 5634.562959] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5634.562959] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5634.562959] echo $(hostname -s): $val; [ 5634.562959] rc=$val; [ 5634.562959] fi; [ 5634.562959] exit $rc [ 5635.168350] Lustre: DEBUG MARKER: dmesg [ 5635.678084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230c: check directory accessiblity if migration is failed ============================= 19:12:50 \(1491246770\) [ 5635.830611] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed ============================= 19:12:50 (1491246770) [ 5636.046277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230c needs \>= 2 MDTs [ 5636.206646] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 5636.409376] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5637.051816] Lustre: DEBUG MARKER: rc=0; [ 5637.051816] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5637.051816] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5637.051816] echo $(hostname -s): $val; [ 5637.051816] rc=$val; [ 5637.051816] fi; [ 5637.051816] exit $rc [ 5637.662579] Lustre: DEBUG MARKER: dmesg [ 5638.178534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230d: check migrate big directory ===================================================== 19:12:53 \(1491246773\) [ 5638.335744] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory ===================================================== 19:12:53 (1491246773) [ 5638.539801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230d needs \>= 2 MDTs [ 5638.690100] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [ 5638.884765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5639.505345] Lustre: DEBUG MARKER: rc=0; [ 5639.505345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5639.505345] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5639.505345] echo $(hostname -s): $val; [ 5639.505345] rc=$val; [ 5639.505345] fi; [ 5639.505345] exit $rc [ 5640.106878] Lustre: DEBUG MARKER: dmesg [ 5640.607636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230e: migrate mulitple local link files =============================================== 19:12:55 \(1491246775\) [ 5640.762208] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files =============================================== 19:12:55 (1491246775) [ 5640.954743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230e needs \>= 2 MDTs [ 5641.104539] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 5641.306087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5641.928978] Lustre: DEBUG MARKER: rc=0; [ 5641.928978] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5641.928978] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5641.928978] echo $(hostname -s): $val; [ 5641.928978] rc=$val; [ 5641.928978] fi; [ 5641.928978] exit $rc [ 5642.539259] Lustre: DEBUG MARKER: dmesg [ 5643.051210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230f: migrate mulitple remote link files ============================================== 19:12:58 \(1491246778\) [ 5643.209143] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files ============================================== 19:12:58 (1491246778) [ 5643.403637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230f needs \>= 2 MDTs [ 5643.556416] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 5643.747998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5644.384947] Lustre: DEBUG MARKER: rc=0; [ 5644.384947] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5644.384947] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5644.384947] echo $(hostname -s): $val; [ 5644.384947] rc=$val; [ 5644.384947] fi; [ 5644.384947] exit $rc [ 5645.001488] Lustre: DEBUG MARKER: dmesg [ 5645.523224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230g: migrate dir to non-exist MDT ==================================================== 19:13:00 \(1491246780\) [ 5645.677098] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ==================================================== 19:13:00 (1491246780) [ 5645.873512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230g needs \>= 2 MDTs [ 5646.030269] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 5646.222879] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5646.849223] Lustre: DEBUG MARKER: rc=0; [ 5646.849223] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5646.849223] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5646.849223] echo $(hostname -s): $val; [ 5646.849223] rc=$val; [ 5646.849223] fi; [ 5646.849223] exit $rc [ 5647.444152] Lustre: DEBUG MARKER: dmesg [ 5647.964730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230h: migrate .. and root ============================================================= 19:13:03 \(1491246783\) [ 5648.116810] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ============================================================= 19:13:03 (1491246783) [ 5648.314827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230h needs \>= 2 MDTs [ 5648.467086] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 5648.666274] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5649.295834] Lustre: DEBUG MARKER: rc=0; [ 5649.295834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5649.295834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5649.295834] echo $(hostname -s): $val; [ 5649.295834] rc=$val; [ 5649.295834] fi; [ 5649.295834] exit $rc [ 5649.898725] Lustre: DEBUG MARKER: dmesg [ 5650.405256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 19:13:05 \(1491246785\) [ 5650.563527] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 19:13:05 (1491246785) [ 5650.761195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230i needs \>= 2 MDTs [ 5650.912534] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 5651.115401] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5651.738440] Lustre: DEBUG MARKER: rc=0; [ 5651.738440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5651.738440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5651.738440] echo $(hostname -s): $val; [ 5651.738440] rc=$val; [ 5651.738440] fi; [ 5651.738440] exit $rc [ 5652.347939] Lustre: DEBUG MARKER: dmesg [ 5652.849562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 231a: checking that reading\/writing of BRW RPC size results in one RPC ================ 19:13:07 \(1491246787\) [ 5653.014585] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ================ 19:13:07 (1491246787) [ 5654.403777] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5655.035246] Lustre: DEBUG MARKER: rc=0; [ 5655.035246] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5655.035246] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5655.035246] echo $(hostname -s): $val; [ 5655.035246] rc=$val; [ 5655.035246] fi; [ 5655.035246] exit $rc [ 5655.641941] Lustre: DEBUG MARKER: dmesg [ 5656.150226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 19:13:11 \(1491246791\) [ 5656.310609] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 19:13:11 (1491246791) [ 5660.071666] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5660.705062] Lustre: DEBUG MARKER: rc=0; [ 5660.705062] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5660.705062] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5660.705062] echo $(hostname -s): $val; [ 5660.705062] rc=$val; [ 5660.705062] fi; [ 5660.705062] exit $rc [ 5661.301742] Lustre: DEBUG MARKER: dmesg [ 5661.836102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 232: failed lock should not block umount ============================================== 19:13:16 \(1491246796\) [ 5662.002262] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount ============================================== 19:13:16 (1491246796) [ 5662.883910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5663.510644] Lustre: DEBUG MARKER: rc=0; [ 5663.510644] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5663.510644] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5663.510644] echo $(hostname -s): $val; [ 5663.510644] rc=$val; [ 5663.510644] fi; [ 5663.510644] exit $rc [ 5664.112654] Lustre: DEBUG MARKER: dmesg [ 5664.630263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 19:13:19 \(1491246799\) [ 5664.785609] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 19:13:19 (1491246799) [ 5664.924798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5664.924798] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5664.924798] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5665.261655] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5665.902607] Lustre: DEBUG MARKER: rc=0; [ 5665.902607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5665.902607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5665.902607] echo $(hostname -s): $val; [ 5665.902607] rc=$val; [ 5665.902607] fi; [ 5665.902607] exit $rc [ 5666.513827] Lustre: DEBUG MARKER: dmesg [ 5667.061622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 19:13:22 \(1491246802\) [ 5667.219979] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 19:13:22 (1491246802) [ 5667.359933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5667.359933] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5667.359933] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5667.715737] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5668.365122] Lustre: DEBUG MARKER: rc=0; [ 5668.365122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5668.365122] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5668.365122] echo $(hostname -s): $val; [ 5668.365122] rc=$val; [ 5668.365122] fi; [ 5668.365122] exit $rc [ 5668.980493] Lustre: DEBUG MARKER: dmesg [ 5669.513663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 19:13:24 \(1491246804\) [ 5669.697170] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 19:13:24 (1491246804) [ 5670.033305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.46 too old [ 5670.202816] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.46 too old [ 5670.455764] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5671.082396] Lustre: DEBUG MARKER: rc=0; [ 5671.082396] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5671.082396] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5671.082396] echo $(hostname -s): $val; [ 5671.082396] rc=$val; [ 5671.082396] fi; [ 5671.082396] exit $rc [ 5671.694527] Lustre: DEBUG MARKER: dmesg [ 5672.225280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 19:13:27 \(1491246807\) [ 5672.380874] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 19:13:27 (1491246807) [ 5672.519038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5672.519038] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5672.519038] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5674.919320] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5675.552173] Lustre: DEBUG MARKER: rc=0; [ 5675.552173] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5675.552173] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5675.552173] echo $(hostname -s): $val; [ 5675.552173] rc=$val; [ 5675.552173] fi; [ 5675.552173] exit $rc [ 5676.181235] Lustre: DEBUG MARKER: dmesg [ 5676.697077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 236: Layout swap on open unlinked file ================================================ 19:13:31 \(1491246811\) [ 5676.853651] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ================================================ 19:13:31 (1491246811) [ 5677.120230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5677.740431] Lustre: DEBUG MARKER: rc=0; [ 5677.740431] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5677.740431] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5677.740431] echo $(hostname -s): $val; [ 5677.740431] rc=$val; [ 5677.740431] fi; [ 5677.740431] exit $rc [ 5678.338327] Lustre: DEBUG MARKER: dmesg [ 5678.861565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 237: Verify name_to_handle_at\/open_by_handle_at syscalls ============================== 19:13:33 \(1491246813\) [ 5679.023357] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls ============================== 19:13:33 (1491246813) [ 5679.226963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5679.856465] Lustre: DEBUG MARKER: rc=0; [ 5679.856465] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5679.856465] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5679.856465] echo $(hostname -s): $val; [ 5679.856465] rc=$val; [ 5679.856465] fi; [ 5679.856465] exit $rc [ 5680.469881] Lustre: DEBUG MARKER: dmesg [ 5680.995944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 238: Verify linkea consistency ======================================================== 19:13:36 \(1491246816\) [ 5681.151897] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ======================================================== 19:13:36 (1491246816) [ 5681.286305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5681.286305] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5681.286305] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5681.650798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5682.281931] Lustre: DEBUG MARKER: rc=0; [ 5682.281931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5682.281931] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5682.281931] echo $(hostname -s): $val; [ 5682.281931] rc=$val; [ 5682.281931] fi; [ 5682.281931] exit $rc [ 5682.894808] Lustre: DEBUG MARKER: dmesg [ 5683.404206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239: osp_sync test ==================================================================== 19:13:38 \(1491246818\) [ 5683.559960] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test ==================================================================== 19:13:38 (1491246818) [ 5683.697356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5683.697356] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5683.697356] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5694.916491] Lustre: DEBUG MARKER: lctl set_param -n osp*.*.sync_changes 1 [ 5699.936614] Lustre: DEBUG MARKER: lctl get_param -n osc.*MDT*.sync_changes osc.*MDT*.sync_in_flight [ 5700.288738] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5700.916507] Lustre: DEBUG MARKER: rc=0; [ 5700.916507] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5700.916507] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5700.916507] echo $(hostname -s): $val; [ 5700.916507] rc=$val; [ 5700.916507] fi; [ 5700.916507] exit $rc [ 5701.517280] Lustre: DEBUG MARKER: dmesg [ 5702.020944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239a: process invalid osp sync record correctly ======================================= 19:13:57 \(1491246837\) [ 5702.181745] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ======================================= 19:13:57 (1491246837) [ 5702.342881] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x2100 [ 5702.628642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5702.888891] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 5703.048071] Lustre: *** cfs_fail_loc=2100, val=0*** [ 5703.180081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5704.456171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5705.734721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5707.022747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5708.324377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5709.680146] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5710.304438] Lustre: DEBUG MARKER: rc=0; [ 5710.304438] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5710.304438] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5710.304438] echo $(hostname -s): $val; [ 5710.304438] rc=$val; [ 5710.304438] fi; [ 5710.304438] exit $rc [ 5710.906263] Lustre: DEBUG MARKER: dmesg [ 5711.424866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 19:14:06 \(1491246846\) [ 5711.583240] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 19:14:06 (1491246846) [ 5711.736316] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x2101 [ 5712.022713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5713.287713] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 5713.575209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5714.903029] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5715.533791] Lustre: DEBUG MARKER: rc=0; [ 5715.533791] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5715.533791] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5715.533791] echo $(hostname -s): $val; [ 5715.533791] rc=$val; [ 5715.533791] fi; [ 5715.533791] exit $rc [ 5716.135201] Lustre: DEBUG MARKER: dmesg [ 5716.646437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 240: race between ldlm enqueue and the connection RPC \(no ASSERT\) ===================== 19:14:11 \(1491246851\) [ 5716.812238] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ===================== 19:14:11 (1491246851) [ 5717.016936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_240 needs \>= 2 MDTs [ 5717.182831] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 5717.387067] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5718.016480] Lustre: DEBUG MARKER: rc=0; [ 5718.016480] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5718.016480] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5718.016480] echo $(hostname -s): $val; [ 5718.016480] rc=$val; [ 5718.016480] fi; [ 5718.016480] exit $rc [ 5718.623238] Lustre: DEBUG MARKER: dmesg [ 5719.220872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241a: bio vs dio ====================================================================== 19:14:14 \(1491246854\) [ 5719.384333] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ====================================================================== 19:14:14 (1491246854) [ 5728.324038] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5728.983503] Lustre: DEBUG MARKER: rc=0; [ 5728.983503] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5728.983503] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5728.983503] echo $(hostname -s): $val; [ 5728.983503] rc=$val; [ 5728.983503] fi; [ 5728.983503] exit $rc [ 5729.579254] Lustre: DEBUG MARKER: dmesg [ 5730.088302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241b: dio vs dio ====================================================================== 19:14:25 \(1491246865\) [ 5730.246720] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ====================================================================== 19:14:25 (1491246865) [ 5736.254535] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5736.898625] Lustre: DEBUG MARKER: rc=0; [ 5736.898625] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5736.898625] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5736.898625] echo $(hostname -s): $val; [ 5736.898625] rc=$val; [ 5736.898625] fi; [ 5736.898625] exit $rc [ 5737.499434] Lustre: DEBUG MARKER: dmesg [ 5738.037553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 19:14:33 \(1491246873\) [ 5738.194135] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 19:14:33 (1491246873) [ 5738.355569] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x105 [ 5738.487869] Lustre: *** cfs_fail_loc=105, val=0*** [ 5738.621674] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 5738.943122] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5739.568863] Lustre: DEBUG MARKER: rc=0; [ 5739.568863] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5739.568863] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5739.568863] echo $(hostname -s): $val; [ 5739.568863] rc=$val; [ 5739.568863] fi; [ 5739.568863] exit $rc [ 5740.163646] Lustre: DEBUG MARKER: dmesg [ 5740.667148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 243: various group lock tests ========================================================= 19:14:35 \(1491246875\) [ 5740.821437] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ========================================================= 19:14:35 (1491246875) [ 5754.877782] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5755.512608] Lustre: DEBUG MARKER: rc=0; [ 5755.512608] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5755.512608] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5755.512608] echo $(hostname -s): $val; [ 5755.512608] rc=$val; [ 5755.512608] fi; [ 5755.512608] exit $rc [ 5756.120380] Lustre: DEBUG MARKER: dmesg [ 5756.654606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 244: sendfile with group lock tests =================================================== 19:14:51 \(1491246891\) [ 5756.814969] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests =================================================== 19:14:51 (1491246891) [ 5770.109181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5770.754239] Lustre: DEBUG MARKER: rc=0; [ 5770.754239] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5770.754239] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5770.754239] echo $(hostname -s): $val; [ 5770.754239] rc=$val; [ 5770.754239] fi; [ 5770.754239] exit $rc [ 5771.376407] Lustre: DEBUG MARKER: dmesg [ 5771.950204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 245: check mdc connection flag\/data: multiple modify RPCs ============================= 19:15:07 \(1491246907\) [ 5772.105451] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ============================= 19:15:07 (1491246907) [ 5772.313693] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5772.958903] Lustre: DEBUG MARKER: rc=0; [ 5772.958903] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5772.958903] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5772.958903] echo $(hostname -s): $val; [ 5772.958903] rc=$val; [ 5772.958903] fi; [ 5772.958903] exit $rc [ 5773.566985] Lustre: DEBUG MARKER: dmesg [ 5774.078480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 246: Read file of size 4095 should return right length ================================ 19:15:09 \(1491246909\) [ 5774.260681] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length ================================ 19:15:09 (1491246909) [ 5775.328019] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5775.978147] Lustre: DEBUG MARKER: rc=0; [ 5775.978147] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5775.978147] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5775.978147] echo $(hostname -s): $val; [ 5775.978147] rc=$val; [ 5775.978147] fi; [ 5775.978147] exit $rc [ 5776.598200] Lustre: DEBUG MARKER: dmesg [ 5777.111810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247a: mount subdir as fileset ========================================================= 19:15:12 \(1491246912\) [ 5777.281893] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ========================================================= 19:15:12 (1491246912) [ 5777.799141] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5778.430332] Lustre: DEBUG MARKER: rc=0; [ 5778.430332] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5778.430332] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5778.430332] echo $(hostname -s): $val; [ 5778.430332] rc=$val; [ 5778.430332] fi; [ 5778.430332] exit $rc [ 5779.027986] Lustre: DEBUG MARKER: dmesg [ 5779.562676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247b: mount subdir that dose not exist ================================================ 19:15:14 \(1491246914\) [ 5779.761817] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ================================================ 19:15:14 (1491246914) [ 5780.029939] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5780.653294] Lustre: DEBUG MARKER: rc=0; [ 5780.653294] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5780.653294] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5780.653294] echo $(hostname -s): $val; [ 5780.653294] rc=$val; [ 5780.653294] fi; [ 5780.653294] exit $rc [ 5781.264296] Lustre: DEBUG MARKER: dmesg [ 5781.819177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247c: running fid2path outside root =================================================== 19:15:16 \(1491246916\) [ 5781.987729] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside root =================================================== 19:15:16 (1491246916) [ 5782.488328] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5783.120287] Lustre: DEBUG MARKER: rc=0; [ 5783.120287] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5783.120287] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5783.120287] echo $(hostname -s): $val; [ 5783.120287] rc=$val; [ 5783.120287] fi; [ 5783.120287] exit $rc [ 5783.729026] Lustre: DEBUG MARKER: dmesg [ 5784.228406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247d: running fid2path inside root ==================================================== 19:15:19 \(1491246919\) [ 5784.414449] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside root ==================================================== 19:15:19 (1491246919) [ 5784.942116] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5785.573528] Lustre: DEBUG MARKER: rc=0; [ 5785.573528] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5785.573528] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5785.573528] echo $(hostname -s): $val; [ 5785.573528] rc=$val; [ 5785.573528] fi; [ 5785.573528] exit $rc [ 5786.176890] Lustre: DEBUG MARKER: dmesg [ 5786.709249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247e: mount .. as fileset ============================================================= 19:15:21 \(1491246921\) [ 5786.870316] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ============================================================= 19:15:21 (1491246921) [ 5787.249935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5787.876994] Lustre: DEBUG MARKER: rc=0; [ 5787.876994] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5787.876994] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5787.876994] echo $(hostname -s): $val; [ 5787.876994] rc=$val; [ 5787.876994] fi; [ 5787.876994] exit $rc [ 5788.477632] Lustre: DEBUG MARKER: dmesg [ 5789.003858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 248: fast read verification =========================================================== 19:15:24 \(1491246924\) [ 5789.162450] Lustre: DEBUG MARKER: == sanity test 248: fast read verification =========================================================== 19:15:24 (1491246924) [ 5798.969774] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5799.610234] Lustre: DEBUG MARKER: rc=0; [ 5799.610234] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5799.610234] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5799.610234] echo $(hostname -s): $val; [ 5799.610234] rc=$val; [ 5799.610234] fi; [ 5799.610234] exit $rc [ 5800.216186] Lustre: DEBUG MARKER: dmesg [ 5800.732178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 249: Write above 2T file size ========================================================= 19:15:35 \(1491246935\) [ 5800.894760] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ========================================================= 19:15:35 (1491246935) [ 5801.038832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5801.038832] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5801.038832] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5801.380398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5802.007373] Lustre: DEBUG MARKER: rc=0; [ 5802.007373] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5802.007373] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5802.007373] echo $(hostname -s): $val; [ 5802.007373] rc=$val; [ 5802.007373] fi; [ 5802.007373] exit $rc [ 5802.622744] Lustre: DEBUG MARKER: dmesg [ 5803.147334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 250: Write above 16T limit ============================================================ 19:15:38 \(1491246938\) [ 5803.316220] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ============================================================ 19:15:38 (1491246938) [ 5803.538521] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5804.175661] Lustre: DEBUG MARKER: rc=0; [ 5804.175661] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5804.175661] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5804.175661] echo $(hostname -s): $val; [ 5804.175661] rc=$val; [ 5804.175661] fi; [ 5804.175661] exit $rc [ 5804.787433] Lustre: DEBUG MARKER: dmesg [ 5805.310077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 251: Handling short read and write correctly ========================================== 19:15:40 \(1491246940\) [ 5805.467197] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================== 19:15:40 (1491246940) [ 5805.756589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5806.399749] Lustre: DEBUG MARKER: rc=0; [ 5806.399749] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5806.399749] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5806.399749] echo $(hostname -s): $val; [ 5806.399749] rc=$val; [ 5806.399749] fi; [ 5806.399749] exit $rc [ 5807.016036] Lustre: DEBUG MARKER: dmesg [ 5807.547399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 252: check lr_reader tool ============================================================= 19:15:42 \(1491246942\) [ 5807.723147] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ============================================================= 19:15:42 (1491246942) [ 5808.210306] Lustre: DEBUG MARKER: /usr/sbin/lr_reader -h [ 5808.464502] Lustre: DEBUG MARKER: /usr/sbin/lr_reader -c /dev/lvm-Role_MDS/P1 [ 5808.724201] Lustre: DEBUG MARKER: /usr/sbin/lr_reader -cr /dev/lvm-Role_MDS/P1 [ 5809.062821] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5809.705505] Lustre: DEBUG MARKER: rc=0; [ 5809.705505] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5809.705505] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5809.705505] echo $(hostname -s): $val; [ 5809.705505] rc=$val; [ 5809.705505] fi; [ 5809.705505] exit $rc [ 5810.314027] Lustre: DEBUG MARKER: dmesg [ 5810.857128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 253: Check object allocation limit ==================================================== 19:15:45 \(1491246945\) [ 5811.020112] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ==================================================== 19:15:45 (1491246945) [ 5811.185386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5811.185386] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5811.185386] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5811.464225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5811.464225] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5811.464225] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5811.735482] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 5811.990189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_high [ 5812.268697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 5812.524596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 5814.824718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5815.083766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 5815.405688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5816.684993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5817.965804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5819.255224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_high [ 5819.510895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_low [ 5819.772892] Lustre: DEBUG MARKER: lctl pool_new lustre.test_253 [ 5828.035656] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_253 2>/dev/null || echo foo [ 5828.290188] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_253 2>/dev/null || echo foo [ 5828.584050] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_253 lustre-OST0000 [ 5837.088452] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=1716 [ 5837.343226] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=1711 [ 5844.360371] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5854.632267] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5864.906256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status [ 5865.038992] LustreError: 28460:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x2000059f2:0x13ba:0x0]: have 0 want 1 [ 5865.042622] LustreError: 28460:0:(lod_qos.c:1273:lod_alloc_specific()) Skipped 5 previous similar messages [ 5865.354259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5865.617460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 5865.906177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5867.186946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5868.473621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5869.752611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5871.035345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5872.339462] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5882.671610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status [ 5882.932922] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=3 [ 5883.189073] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=1 [ 5883.451291] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_253 lustre-OST0000 [ 5893.710560] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_253 [ 5900.038491] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5900.682399] Lustre: DEBUG MARKER: rc=0; [ 5900.682399] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5900.682399] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5900.682399] echo $(hostname -s): $val; [ 5900.682399] rc=$val; [ 5900.682399] fi; [ 5900.682399] exit $rc [ 5901.300517] Lustre: DEBUG MARKER: dmesg [ 5901.795885] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 5902.167437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 254: Check changelog size ============================================================= 19:17:17 \(1491247037\) [ 5902.328586] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ============================================================= 19:17:17 (1491247037) [ 5902.489303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5902.751197] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5902.876692] Lustre: lustre-MDD0000: changelog on [ 5903.011605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5903.313970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5903.580576] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl1 [ 5903.706583] Lustre: lustre-MDD0000: changelog off [ 5903.907972] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5904.551319] Lustre: DEBUG MARKER: rc=0; [ 5904.551319] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5904.551319] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5904.551319] echo $(hostname -s): $val; [ 5904.551319] rc=$val; [ 5904.551319] fi; [ 5904.551319] exit $rc [ 5905.163695] Lustre: DEBUG MARKER: dmesg [ 5905.718303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 255a: check \'lfs ladvise -a willread\' ================================================= 19:17:20 \(1491247040\) [ 5905.874460] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 19:17:20 (1491247040) [ 6384.146867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6384.806691] Lustre: DEBUG MARKER: rc=0; [ 6384.806691] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6384.806691] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6384.806691] echo $(hostname -s): $val; [ 6384.806691] rc=$val; [ 6384.806691] fi; [ 6384.806691] exit $rc [ 6385.426702] Lustre: DEBUG MARKER: dmesg [ 6385.937042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 255b: check \'lfs ladvise -a dontneed\' ================================================= 19:25:21 \(1491247521\) [ 6386.095542] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ================================================= 19:25:21 (1491247521) [ 6391.875567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6392.520980] Lustre: DEBUG MARKER: rc=0; [ 6392.520980] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6392.520980] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6392.520980] echo $(hostname -s): $val; [ 6392.520980] rc=$val; [ 6392.520980] fi; [ 6392.520980] exit $rc [ 6393.157324] Lustre: DEBUG MARKER: dmesg [ 6393.697637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 256: Check llog delete for empty and not full state =================================== 19:25:28 \(1491247528\) [ 6393.867885] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state =================================== 19:25:28 (1491247528) [ 6394.041761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users | grep cl [ 6394.298612] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 6394.424930] Lustre: lustre-MDD0000: changelog on [ 6394.592197] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6394.845072] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6394.972374] Lustre: Failing over lustre-MDT0000 [ 6396.368703] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.372954] LustreError: Skipped 9 previous similar messages [ 6401.023035] Lustre: 17383:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491247530/real 1491247530] req@ffff8800652fa100 x1563680146397056/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491247536 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6401.150716] Lustre: server umount lustre-MDT0000 complete [ 6401.277407] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6401.537618] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6401.799547] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6402.053347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6402.301475] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6402.447911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6402.467628] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 6402.469897] Lustre: Skipped 24 previous similar messages [ 6402.508254] LustreError: 17799:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6402.518739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.525881] Lustre: lustre-MDD0000: changelog on [ 6402.667791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6402.926282] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6403.416057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6403.417992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6403.582907] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6403.586420] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6403.731388] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6403.988074] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6404.241260] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6404.503643] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6404.758632] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6404.903170] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6409.592322] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6409.783930] Lustre: DEBUG MARKER: debugfs -R "dump changelog_catalog cat.dmp" /dev/lvm-Role_MDS/P1; llog_reader cat.dmp | grep "type=1064553b" | wc -l [ 6410.050398] Lustre: DEBUG MARKER: debugfs -R "dump changelog_catalog cat.dmp" /dev/lvm-Role_MDS/P1; llog_reader cat.dmp | grep "type=1064553b" | wc -l [ 6410.310922] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl2 [ 6410.439661] Lustre: lustre-MDD0000: changelog off [ 6410.637360] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6411.286072] Lustre: DEBUG MARKER: rc=0; [ 6411.286072] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6411.286072] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6411.286072] echo $(hostname -s): $val; [ 6411.286072] rc=$val; [ 6411.286072] fi; [ 6411.286072] exit $rc [ 6411.909879] Lustre: DEBUG MARKER: dmesg [ 6412.431533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 257: xattr locks are not lost ========================================================= 19:25:47 \(1491247547\) [ 6412.596179] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ========================================================= 19:25:47 (1491247547) [ 6412.755224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6412.755224] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6412.755224] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6413.054719] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x80000161 [ 6413.206821] Lustre: *** cfs_fail_loc=161, val=0*** [ 6413.339683] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6413.591101] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6413.719879] Lustre: Failing over lustre-MDT0000 [ 6414.588848] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.593713] LustreError: Skipped 5 previous similar messages [ 6421.760035] Lustre: 19505:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491247551/real 1491247551] req@ffff8800663c7000 x1563680146398560/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491247557 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6421.844072] Lustre: server umount lustre-MDT0000 complete [ 6421.973432] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6422.237469] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6422.506355] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6422.760670] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6423.008399] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6423.215230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.305802] LustreError: 19924:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6423.316513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.465204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6423.725965] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6424.219365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6424.219367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6424.380338] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6424.386348] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6424.528412] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6424.575895] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6424.783779] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6425.037131] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6425.300630] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6425.556476] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6425.876549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6426.531909] Lustre: DEBUG MARKER: rc=0; [ 6426.531909] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6426.531909] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6426.531909] echo $(hostname -s): $val; [ 6426.531909] rc=$val; [ 6426.531909] fi; [ 6426.531909] exit $rc [ 6427.153215] Lustre: DEBUG MARKER: dmesg [ 6427.678394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 260: Check mdc_close fail ============================================================= 19:26:02 \(1491247562\) [ 6427.834923] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ============================================================= 19:26:02 (1491247562) [ 6429.067270] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6429.307325] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6429.943703] Lustre: DEBUG MARKER: rc=0; [ 6429.943703] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6429.943703] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6429.943703] echo $(hostname -s): $val; [ 6429.943703] rc=$val; [ 6429.943703] fi; [ 6429.943703] exit $rc [ 6430.564489] Lustre: DEBUG MARKER: dmesg [ 6431.105966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300a: basic striped dir sanity test =================================================== 19:26:06 \(1491247566\) [ 6431.274633] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test =================================================== 19:26:06 (1491247566) [ 6431.413071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6431.413071] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6431.413071] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6431.763449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300a needs \>= 2 MDTs [ 6431.922747] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 6432.132130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6432.780115] Lustre: DEBUG MARKER: rc=0; [ 6432.780115] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6432.780115] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6432.780115] echo $(hostname -s): $val; [ 6432.780115] rc=$val; [ 6432.780115] fi; [ 6432.780115] exit $rc [ 6433.398916] Lustre: DEBUG MARKER: dmesg [ 6433.915779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300b: check ctime\/mtime for striped dir =============================================== 19:26:09 \(1491247569\) [ 6434.072659] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir =============================================== 19:26:09 (1491247569) [ 6434.209908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6434.209908] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6434.209908] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6434.541637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300b needs \>= 2 MDTs [ 6434.707665] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 6434.905258] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6435.555957] Lustre: DEBUG MARKER: rc=0; [ 6435.555957] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6435.555957] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6435.555957] echo $(hostname -s): $val; [ 6435.555957] rc=$val; [ 6435.555957] fi; [ 6435.555957] exit $rc [ 6436.172894] Lustre: DEBUG MARKER: dmesg [ 6436.700250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300c: chown && check ls under striped directory ======================================= 19:26:11 \(1491247571\) [ 6436.869782] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 6437.002904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6437.002904] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6437.002904] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6437.338549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300c needs \>= 2 MDTs [ 6437.493542] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 6437.694765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6438.325790] Lustre: DEBUG MARKER: rc=0; [ 6438.325790] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6438.325790] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6438.325790] echo $(hostname -s): $val; [ 6438.325790] rc=$val; [ 6438.325790] fi; [ 6438.325790] exit $rc [ 6438.945546] Lustre: DEBUG MARKER: dmesg [ 6439.479065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300d: check default stripe under striped directory ==================================== 19:26:14 \(1491247574\) [ 6439.645327] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ==================================== 19:26:14 (1491247574) [ 6439.784532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6439.784532] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6439.784532] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6440.122313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300d needs \>= 2 MDTs [ 6440.286421] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 6440.486995] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6441.134461] Lustre: DEBUG MARKER: rc=0; [ 6441.134461] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6441.134461] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6441.134461] echo $(hostname -s): $val; [ 6441.134461] rc=$val; [ 6441.134461] fi; [ 6441.134461] exit $rc [ 6441.761197] Lustre: DEBUG MARKER: dmesg [ 6442.283466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300e: check rename under striped directory ============================================ 19:26:17 \(1491247577\) [ 6442.438098] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ============================================ 19:26:17 (1491247577) [ 6442.574407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6442.574407] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6442.574407] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6442.918227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300e needs \>= 2 MDTs [ 6443.078865] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 6443.279358] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6443.913896] Lustre: DEBUG MARKER: rc=0; [ 6443.913896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6443.913896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6443.913896] echo $(hostname -s): $val; [ 6443.913896] rc=$val; [ 6443.913896] fi; [ 6443.913896] exit $rc [ 6444.547356] Lustre: DEBUG MARKER: dmesg [ 6445.074484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300f: check rename cross striped directory ============================================ 19:26:20 \(1491247580\) [ 6445.237508] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ============================================ 19:26:20 (1491247580) [ 6445.370167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6445.370167] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6445.370167] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6445.706949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300f needs \>= 2 MDTs [ 6445.866080] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 6446.066730] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6446.698677] Lustre: DEBUG MARKER: rc=0; [ 6446.698677] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6446.698677] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6446.698677] echo $(hostname -s): $val; [ 6446.698677] rc=$val; [ 6446.698677] fi; [ 6446.698677] exit $rc [ 6447.323084] Lustre: DEBUG MARKER: dmesg [ 6447.851738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300g: check default striped directory for normal directory ============================ 19:26:22 \(1491247582\) [ 6448.013651] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ============================ 19:26:22 (1491247582) [ 6448.153866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6448.153866] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6448.153866] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6448.504124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300g needs \>= 2 MDTs [ 6448.668578] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 6448.860598] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6449.513859] Lustre: DEBUG MARKER: rc=0; [ 6449.513859] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6449.513859] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6449.513859] echo $(hostname -s): $val; [ 6449.513859] rc=$val; [ 6449.513859] fi; [ 6449.513859] exit $rc [ 6450.122932] Lustre: DEBUG MARKER: dmesg [ 6450.651936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300h: check default striped directory for striped directory =========================== 19:26:25 \(1491247585\) [ 6450.823552] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory =========================== 19:26:25 (1491247585) [ 6450.960270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6450.960270] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6450.960270] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6451.312031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300h needs \>= 2 MDTs [ 6451.470006] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 6451.684166] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6452.317570] Lustre: DEBUG MARKER: rc=0; [ 6452.317570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6452.317570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6452.317570] echo $(hostname -s): $val; [ 6452.317570] rc=$val; [ 6452.317570] fi; [ 6452.317570] exit $rc [ 6452.942043] Lustre: DEBUG MARKER: dmesg [ 6453.482393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300i: client handle unknown hash type striped directory =============================== 19:26:28 \(1491247588\) [ 6453.642629] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory =============================== 19:26:28 (1491247588) [ 6453.779857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6453.779857] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6453.779857] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6454.132388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300i needs \>= 2 MDTs [ 6454.292887] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 6454.489820] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6455.128652] Lustre: DEBUG MARKER: rc=0; [ 6455.128652] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6455.128652] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6455.128652] echo $(hostname -s): $val; [ 6455.128652] rc=$val; [ 6455.128652] fi; [ 6455.128652] exit $rc [ 6455.751516] Lustre: DEBUG MARKER: dmesg [ 6456.282230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300j: test large update record ======================================================== 19:26:31 \(1491247591\) [ 6456.441441] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ======================================================== 19:26:31 (1491247591) [ 6456.580092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6456.580092] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6456.580092] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6456.921501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300j needs \>= 2 MDTs [ 6457.077273] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 6457.272929] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6457.951544] Lustre: DEBUG MARKER: rc=0; [ 6457.951544] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6457.951544] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6457.951544] echo $(hostname -s): $val; [ 6457.951544] rc=$val; [ 6457.951544] fi; [ 6457.951544] exit $rc [ 6458.581593] Lustre: DEBUG MARKER: dmesg [ 6459.122398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300k: test large striped directory ==================================================== 19:26:34 \(1491247594\) [ 6459.280713] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ==================================================== 19:26:34 (1491247594) [ 6459.422220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6459.422220] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6459.422220] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6459.761179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300k needs \>= 2 MDTs [ 6459.921806] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 6460.119365] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6460.753931] Lustre: DEBUG MARKER: rc=0; [ 6460.753931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6460.753931] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6460.753931] echo $(hostname -s): $val; [ 6460.753931] rc=$val; [ 6460.753931] fi; [ 6460.753931] exit $rc [ 6461.364771] Lustre: DEBUG MARKER: dmesg [ 6461.884673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 19:26:36 \(1491247596\) [ 6462.049275] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 19:26:36 (1491247596) [ 6462.186480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6462.186480] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6462.186480] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6462.534649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300l needs \>= 2 MDTs [ 6462.688570] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 6462.888046] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6463.525747] Lustre: DEBUG MARKER: rc=0; [ 6463.525747] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6463.525747] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6463.525747] echo $(hostname -s): $val; [ 6463.525747] rc=$val; [ 6463.525747] fi; [ 6463.525747] exit $rc [ 6464.158360] Lustre: DEBUG MARKER: dmesg [ 6464.698229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300m: setstriped directory on single MDT FS =========================================== 19:26:39 \(1491247599\) [ 6464.863443] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS =========================================== 19:26:39 (1491247599) [ 6464.998758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6464.998758] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6464.998758] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6465.380473] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6466.017417] Lustre: DEBUG MARKER: rc=0; [ 6466.017417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6466.017417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6466.017417] echo $(hostname -s): $val; [ 6466.017417] rc=$val; [ 6466.017417] fi; [ 6466.017417] exit $rc [ 6466.644987] Lustre: DEBUG MARKER: dmesg [ 6467.156355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 19:26:42 \(1491247602\) [ 6467.320233] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 19:26:42 (1491247602) [ 6467.455435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6467.455435] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6467.455435] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6467.793741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300n needs \>= 2 MDTs [ 6467.952775] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 6468.179478] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6468.814654] Lustre: DEBUG MARKER: rc=0; [ 6468.814654] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6468.814654] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6468.814654] echo $(hostname -s): $val; [ 6468.814654] rc=$val; [ 6468.814654] fi; [ 6468.814654] exit $rc [ 6469.443301] Lustre: DEBUG MARKER: dmesg [ 6469.982498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300o skipping SLOW test 300o [ 6470.170705] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 6470.395956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300p: create striped directory without space ========================================== 19:26:45 \(1491247605\) [ 6470.549973] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================== 19:26:45 (1491247605) [ 6470.760638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300p needs \>= 2 MDTs [ 6470.916316] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 6471.117898] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6471.753781] Lustre: DEBUG MARKER: rc=0; [ 6471.753781] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6471.753781] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6471.753781] echo $(hostname -s): $val; [ 6471.753781] rc=$val; [ 6471.753781] fi; [ 6471.753781] exit $rc [ 6472.390385] Lustre: DEBUG MARKER: dmesg [ 6472.935574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300q: create remote directory under orphan directory ================================== 19:26:48 \(1491247608\) [ 6473.096193] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ================================== 19:26:48 (1491247608) [ 6473.302048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300q needs \>= 2 MDTs [ 6473.476774] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 6473.697815] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6474.323026] Lustre: DEBUG MARKER: rc=0; [ 6474.323026] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6474.323026] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6474.323026] echo $(hostname -s): $val; [ 6474.323026] rc=$val; [ 6474.323026] fi; [ 6474.323026] exit $rc [ 6474.934645] Lustre: DEBUG MARKER: dmesg [ 6475.453399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310a: open unlink remote file ========================================================= 19:26:50 \(1491247610\) [ 6475.612428] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ========================================================= 19:26:50 (1491247610) [ 6475.819576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310a needs \>= 4 MDTs [ 6475.971683] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 6476.168983] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6476.806932] Lustre: DEBUG MARKER: rc=0; [ 6476.806932] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6476.806932] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6476.806932] echo $(hostname -s): $val; [ 6476.806932] rc=$val; [ 6476.806932] fi; [ 6476.806932] exit $rc [ 6477.430080] Lustre: DEBUG MARKER: dmesg [ 6477.954115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310b: unlink remote file with multiple links while open =============================== 19:26:53 \(1491247613\) [ 6478.110631] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open =============================== 19:26:53 (1491247613) [ 6478.315314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310b needs \>= 4 MDTs [ 6478.477345] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 6478.693654] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6479.344309] Lustre: DEBUG MARKER: rc=0; [ 6479.344309] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6479.344309] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6479.344309] echo $(hostname -s): $val; [ 6479.344309] rc=$val; [ 6479.344309] fi; [ 6479.344309] exit $rc [ 6479.984403] Lustre: DEBUG MARKER: dmesg [ 6480.516775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310c: open-unlink remote file with multiple links ===================================== 19:26:55 \(1491247615\) [ 6480.676364] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ===================================== 19:26:55 (1491247615) [ 6480.881768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310c needs \>= 4 MDTs [ 6481.043198] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 6481.258835] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6481.885198] Lustre: DEBUG MARKER: rc=0; [ 6481.885198] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6481.885198] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6481.885198] echo $(hostname -s): $val; [ 6481.885198] rc=$val; [ 6481.885198] fi; [ 6481.885198] exit $rc [ 6482.512545] Lustre: DEBUG MARKER: dmesg [ 6483.032813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 19:26:58 \(1491247618\) [ 6483.185229] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 19:26:58 (1491247618) [ 6483.324176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6483.324176] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6483.324176] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6484.716699] Lustre: DEBUG MARKER: lctl get_param -n osp.*OST0000*MDT0000.create_count [ 6484.982033] Lustre: DEBUG MARKER: lctl get_param -n osp.*OST0000*MDT0000.max_create_count [ 6485.239504] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.max_create_count=0 [ 6486.605054] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.max_create_count=20000 [ 6486.858849] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.create_count=32 [ 6489.211268] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6489.852062] Lustre: DEBUG MARKER: rc=0; [ 6489.852062] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6489.852062] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6489.852062] echo $(hostname -s): $val; [ 6489.852062] rc=$val; [ 6489.852062] fi; [ 6489.852062] exit $rc [ 6490.473217] Lustre: DEBUG MARKER: dmesg [ 6491.005531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 19:27:06 \(1491247626\) [ 6491.164582] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 19:27:06 (1491247626) [ 6491.379064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_312 the test only applies to zfs [ 6491.544797] Lustre: DEBUG MARKER: SKIP: sanity test_312 the test only applies to zfs [ 6491.750868] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6492.398055] Lustre: DEBUG MARKER: rc=0; [ 6492.398055] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6492.398055] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6492.398055] echo $(hostname -s): $val; [ 6492.398055] rc=$val; [ 6492.398055] fi; [ 6492.398055] exit $rc [ 6493.025603] Lustre: DEBUG MARKER: dmesg [ 6493.548301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 313: io should fail after last_rcvd update fail ======================================= 19:27:08 \(1491247628\) [ 6493.709617] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ======================================= 19:27:08 (1491247628) [ 6494.493288] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6495.122990] Lustre: DEBUG MARKER: rc=0; [ 6495.122990] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6495.122990] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6495.122990] echo $(hostname -s): $val; [ 6495.122990] rc=$val; [ 6495.122990] fi; [ 6495.122990] exit $rc [ 6495.738947] Lustre: DEBUG MARKER: dmesg [ 6496.278182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 399a: fake write should not be slower than normal write =============================== 19:27:11 \(1491247631\) [ 6496.439454] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write =============================== 19:27:11 (1491247631) [ 6570.689500] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6571.328101] Lustre: DEBUG MARKER: rc=0; [ 6571.328101] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6571.328101] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6571.328101] echo $(hostname -s): $val; [ 6571.328101] rc=$val; [ 6571.328101] fi; [ 6571.328101] exit $rc [ 6571.953664] Lustre: DEBUG MARKER: dmesg [ 6572.572496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 399b: fake read should not be slower than normal read ================================= 19:28:27 \(1491247707\) [ 6572.729268] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ================================= 19:28:27 (1491247707) [ 6573.512558] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6574.144081] Lustre: DEBUG MARKER: rc=0; [ 6574.144081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6574.144081] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6574.144081] echo $(hostname -s): $val; [ 6574.144081] rc=$val; [ 6574.144081] fi; [ 6574.144081] exit $rc [ 6574.765116] Lustre: DEBUG MARKER: dmesg [ 6575.302513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 400a: Lustre client api program can compile and link ================================== 19:28:30 \(1491247710\) [ 6575.459667] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link ================================== 19:28:30 (1491247710) [ 6578.152180] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6578.778260] Lustre: DEBUG MARKER: rc=0; [ 6578.778260] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6578.778260] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6578.778260] echo $(hostname -s): $val; [ 6578.778260] rc=$val; [ 6578.778260] fi; [ 6578.778260] exit $rc [ 6579.392808] Lustre: DEBUG MARKER: dmesg [ 6579.936476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 400b: packaged headers can be compiled ================================================ 19:28:35 \(1491247715\) [ 6580.094463] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ================================================ 19:28:35 (1491247715) [ 6580.463288] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6581.099295] Lustre: DEBUG MARKER: rc=0; [ 6581.099295] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6581.099295] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6581.099295] echo $(hostname -s): $val; [ 6581.099295] rc=$val; [ 6581.099295] fi; [ 6581.099295] exit $rc [ 6581.733710] Lustre: DEBUG MARKER: dmesg [ 6582.289053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401a: Verify if \'lctl list_param -R\' can list parameters recursively ================== 19:28:37 \(1491247717\) [ 6582.456775] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ================== 19:28:37 (1491247717) [ 6582.733405] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6583.380668] Lustre: DEBUG MARKER: rc=0; [ 6583.380668] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6583.380668] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6583.380668] echo $(hostname -s): $val; [ 6583.380668] rc=$val; [ 6583.380668] fi; [ 6583.380668] exit $rc [ 6584.026123] Lustre: DEBUG MARKER: dmesg [ 6584.563311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401b: Verify \'lctl {get,set}_param\' continue after error ============================== 19:28:39 \(1491247719\) [ 6584.724824] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ============================== 19:28:39 (1491247719) [ 6584.934365] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6585.573968] Lustre: DEBUG MARKER: rc=0; [ 6585.573968] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6585.573968] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6585.573968] echo $(hostname -s): $val; [ 6585.573968] rc=$val; [ 6585.573968] fi; [ 6585.573968] exit $rc [ 6586.179535] Lustre: DEBUG MARKER: dmesg [ 6586.712662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401c: Verify \'lctl set_param\' without value fails in either format. =================== 19:28:41 \(1491247721\) [ 6586.876624] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. =================== 19:28:41 (1491247721) [ 6587.085074] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6587.724409] Lustre: DEBUG MARKER: rc=0; [ 6587.724409] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6587.724409] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6587.724409] echo $(hostname -s): $val; [ 6587.724409] rc=$val; [ 6587.724409] fi; [ 6587.724409] exit $rc [ 6588.345975] Lustre: DEBUG MARKER: dmesg [ 6588.893779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401d: Verify \'lctl set_param\' accepts values containing \'=\' =========================== 19:28:43 \(1491247723\) [ 6589.059756] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' =========================== 19:28:43 (1491247723) [ 6589.280934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6589.915715] Lustre: DEBUG MARKER: rc=0; [ 6589.915715] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6589.915715] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6589.915715] echo $(hostname -s): $val; [ 6589.915715] rc=$val; [ 6589.915715] fi; [ 6589.915715] exit $rc [ 6590.551386] Lustre: DEBUG MARKER: dmesg [ 6591.089908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 19:28:46 \(1491247726\) [ 6591.262143] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 19:28:46 (1491247726) [ 6591.396661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6591.396661] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6591.396661] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6591.693181] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000015c [ 6591.825916] Lustre: *** cfs_fail_loc=15c, val=0*** [ 6591.827934] LustreError: 19909:0:(lod_lov.c:715:lod_generate_and_set_lovea()) lustre-MDT0000-mdtlov: Can not locate [0x100050000:0x7f22:0x0]: rc = -2 [ 6592.030947] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6592.661597] Lustre: DEBUG MARKER: rc=0; [ 6592.661597] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6592.661597] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6592.661597] echo $(hostname -s): $val; [ 6592.661597] rc=$val; [ 6592.661597] fi; [ 6592.661597] exit $rc [ 6593.288567] Lustre: DEBUG MARKER: dmesg [ 6593.804831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 19:28:48 \(1491247728\) [ 6593.964700] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 19:28:48 (1491247728) [ 6594.255979] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6594.899206] Lustre: DEBUG MARKER: rc=0; [ 6594.899206] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6594.899206] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6594.899206] echo $(hostname -s): $val; [ 6594.899206] rc=$val; [ 6594.899206] fi; [ 6594.899206] exit $rc [ 6595.511030] Lustre: DEBUG MARKER: dmesg [ 6596.038203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 19:28:51 \(1491247731\) [ 6596.195739] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 19:28:51 (1491247731) [ 6596.359635] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6596.622628] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0000-osc-MDT0000 deactivate [ 6596.757202] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6596.889113] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6597.140671] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0000-osc-MDT0000 activate [ 6597.267599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6597.271771] Lustre: Skipped 2 previous similar messages [ 6597.276542] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.409510] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6597.660848] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 6597.916606] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6598.169365] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0001-osc-MDT0000 activate [ 6598.427321] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6598.682162] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0002-osc-MDT0000 deactivate [ 6598.809192] Lustre: setting import lustre-OST0002_UUID INACTIVE by administrator request [ 6598.811344] Lustre: Skipped 1 previous similar message [ 6598.946676] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6599.204807] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0002-osc-MDT0000 activate [ 6599.334336] LustreError: 167-0: lustre-OST0002-osc-MDT0000: This client was evicted by lustre-OST0002; in progress operations using this service will fail. [ 6599.338375] LustreError: Skipped 1 previous similar message [ 6599.468908] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6599.722674] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0003-osc-MDT0000 deactivate [ 6599.980811] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6600.237444] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0003-osc-MDT0000 activate [ 6600.492481] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6600.746143] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0004-osc-MDT0000 deactivate [ 6601.000969] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6601.256230] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0004-osc-MDT0000 activate [ 6601.383455] Lustre: lustre-OST0004-osc-MDT0000: Connection to lustre-OST0004 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6601.387846] Lustre: Skipped 3 previous similar messages [ 6601.522936] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6601.777099] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0005-osc-MDT0000 deactivate [ 6602.031178] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6602.284999] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0005-osc-MDT0000 activate [ 6602.538609] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6602.795649] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0006-osc-MDT0000 deactivate [ 6602.931398] Lustre: setting import lustre-OST0006_UUID INACTIVE by administrator request [ 6602.933592] Lustre: Skipped 3 previous similar messages [ 6603.068866] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6603.336460] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0006-osc-MDT0000 activate [ 6603.462876] LustreError: 167-0: lustre-OST0006-osc-MDT0000: This client was evicted by lustre-OST0006; in progress operations using this service will fail. [ 6603.466992] LustreError: Skipped 3 previous similar messages [ 6603.596290] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6603.932435] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6604.573199] Lustre: DEBUG MARKER: rc=0; [ 6604.573199] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6604.573199] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6604.573199] echo $(hostname -s): $val; [ 6604.573199] rc=$val; [ 6604.573199] fi; [ 6604.573199] exit $rc [ 6605.201808] Lustre: DEBUG MARKER: dmesg [ 6605.733684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 405: Various layout swap lock tests =================================================== 19:29:00 \(1491247740\) [ 6605.897718] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests =================================================== 19:29:00 (1491247740) [ 6606.040739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6606.040739] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6606.040739] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6619.844694] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6620.481188] Lustre: DEBUG MARKER: rc=0; [ 6620.481188] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6620.481188] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6620.481188] echo $(hostname -s): $val; [ 6620.481188] rc=$val; [ 6620.481188] fi; [ 6620.481188] exit $rc [ 6621.109871] Lustre: DEBUG MARKER: dmesg [ 6621.651642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 406: DNE support fs default striping ================================================== 19:29:16 \(1491247756\) [ 6621.806827] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ================================================== 19:29:16 (1491247756) [ 6622.007511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_406 needs \>= 2 MDTs [ 6622.174188] Lustre: DEBUG MARKER: SKIP: sanity test_406 needs >= 2 MDTs [ 6622.367963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6623.001358] Lustre: DEBUG MARKER: rc=0; [ 6623.001358] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6623.001358] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6623.001358] echo $(hostname -s): $val; [ 6623.001358] rc=$val; [ 6623.001358] fi; [ 6623.001358] exit $rc [ 6623.619961] Lustre: DEBUG MARKER: dmesg [ 6624.218359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 6624.404886] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 6624.639961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 408: drop_caches should not hang due to page leaks ==================================== 19:29:19 \(1491247759\) [ 6624.806784] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ==================================== 19:29:19 (1491247759) [ 6625.421488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6626.071888] Lustre: DEBUG MARKER: rc=0; [ 6626.071888] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6626.071888] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6626.071888] echo $(hostname -s): $val; [ 6626.071888] rc=$val; [ 6626.071888] fi; [ 6626.071888] exit $rc [ 6626.705999] Lustre: DEBUG MARKER: dmesg [ 6627.251930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 409: Large amount of cross-MDTs hard links on the same file =========================== 19:29:22 \(1491247762\) [ 6627.418280] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file =========================== 19:29:22 (1491247762) [ 6627.620818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_409 We need at least 2 MDTs for this test [ 6627.781993] Lustre: DEBUG MARKER: SKIP: sanity test_409 We need at least 2 MDTs for this test [ 6627.978835] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6628.622645] Lustre: DEBUG MARKER: rc=0; [ 6628.622645] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6628.622645] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6628.622645] echo $(hostname -s): $val; [ 6628.622645] rc=$val; [ 6628.622645] fi; [ 6628.622645] exit $rc [ 6629.243081] Lustre: DEBUG MARKER: dmesg [ 6629.781770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 801a: write barrier user interfaces and stat machine ================================== 19:29:24 \(1491247764\) [ 6629.942314] Lustre: DEBUG MARKER: == sanity test 801a: write barrier user interfaces and stat machine ================================== 19:29:24 (1491247764) [ 6630.088645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 6630.418197] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 6630.779890] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 6631.075689] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_rescan lustre [ 6631.330383] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x2202 [ 6631.598196] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_freeze lustre 10 [ 6631.724679] LustreError: 5910:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 2202 sleeping for 3000ms [ 6632.598948] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6632.846792] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 6634.726025] LustreError: 5910:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 2202 awake [ 6634.869329] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6635.122946] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6645.380428] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6645.632643] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_freeze lustre 10 [ 6645.887938] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6646.140574] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x2202 [ 6646.395807] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_thaw lustre [ 6646.523356] LustreError: 6561:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 2202 sleeping for 3000ms [ 6647.396164] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6647.650267] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 6649.525033] LustreError: 6561:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 2202 awake [ 6649.656343] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6649.908896] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x2203 [ 6650.160521] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_freeze lustre [ 6650.287740] Lustre: *** cfs_fail_loc=2203, val=0*** [ 6650.418482] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6650.671491] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6650.921480] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_thaw lustre [ 6651.247679] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 6651.853800] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 6652.205900] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6652.839966] Lustre: DEBUG MARKER: rc=0; [ 6652.839966] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6652.839966] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6652.839966] echo $(hostname -s): $val; [ 6652.839966] rc=$val; [ 6652.839966] fi; [ 6652.839966] exit $rc [ 6653.456928] Lustre: DEBUG MARKER: dmesg [ 6653.979172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 801b: modification will be blocked by write barrier =================================== 19:29:49 \(1491247789\) [ 6654.160327] Lustre: DEBUG MARKER: == sanity test 801b: modification will be blocked by write barrier =================================== 19:29:49 (1491247789) [ 6654.297914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 6654.613731] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 6654.968114] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 6655.279567] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_rescan lustre [ 6655.568826] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_freeze lustre 180 [ 6655.835932] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6656.109870] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6661.399635] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6661.652602] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_thaw lustre [ 6661.908853] Lustre: DEBUG MARKER: /usr/sbin/lctl barrier_stat lustre [ 6662.228027] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 6662.831113] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 6663.195681] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6663.831489] Lustre: DEBUG MARKER: rc=0; [ 6663.831489] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6663.831489] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6663.831489] echo $(hostname -s): $val; [ 6663.831489] rc=$val; [ 6663.831489] fi; [ 6663.831489] exit $rc [ 6664.444093] Lustre: DEBUG MARKER: dmesg [ 6665.010303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 801c: rescan barrier bitmap =========================================================== 19:30:00 \(1491247800\) [ 6665.173255] Lustre: DEBUG MARKER: == sanity test 801c: rescan barrier bitmap =========================================================== 19:30:00 (1491247800) [ 6665.370799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_801c needs \>= 2 MDTs [ 6665.528243] Lustre: DEBUG MARKER: SKIP: sanity test_801c needs >= 2 MDTs [ 6665.728531] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6666.360282] Lustre: DEBUG MARKER: rc=0; [ 6666.360282] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6666.360282] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6666.360282] echo $(hostname -s): $val; [ 6666.360282] rc=$val; [ 6666.360282] fi; [ 6666.360282] exit $rc [ 6666.985421] Lustre: DEBUG MARKER: dmesg [ 6667.527712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 802: simulate readonly device ========================================================= 19:30:02 \(1491247802\) [ 6667.686690] Lustre: DEBUG MARKER: == sanity test 802: simulate readonly device ========================================================= 19:30:02 (1491247802) [ 6667.917487] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 6671.586136] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 6672.933926] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6673.187887] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6673.395685] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 6673.395686] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 6673.399608] Lustre: Skipped 5 previous similar messages [ 6673.423085] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880061ba3900 x1563680146426480/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6673.427100] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 6674.463058] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800379bb000 x1563680146426512/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6674.467330] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 6675.487059] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800379bb000 x1563680146426528/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6677.535054] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800379bb000 x1563680146426576/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6677.539725] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 6683.388924] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.393957] LustreError: Skipped 14 previous similar messages [ 6685.350035] Lustre: 9409:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491247814/real 1491247814] req@ffff8800379bb000 x1563680146426672/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491247820 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6685.476526] Lustre: server umount lustre-MDT0000 complete [ 6685.603946] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6693.784576] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6693.784576] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6693.784576] if [ $running -ne $mpts ]; then [ 6693.784576] echo $(hostname) env are INSANE!; [ 6693.784576] exit 1; [ 6693.784576] fi [ 6694.106662] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6694.106662] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6694.106662] if [ $running -ne $mpts ]; then [ 6694.106662] echo $(hostname) env are INSANE!; [ 6694.106662] exit 1; [ 6694.106662] fi [ 6695.104093] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6695.374855] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6695.627698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6695.874155] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o rdonly_dev /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6696.029752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.033909] Turning device dm-0 (0xfc00000) read-only [ 6696.035884] Lustre: lustre-MDT0000-osd: set dev_rdonly on this device [ 6696.088824] LustreError: 9983:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6696.096690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.239285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6696.491406] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6696.991406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6696.995411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6697.154382] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6697.160285] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6697.301304] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6697.555697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6697.806321] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6698.066083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6698.326493] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6700.454268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6700.608512] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6703.378359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6703.535867] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6706.321251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6706.486972] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6709.286960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6709.446156] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6712.267298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6712.425388] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6715.207250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6715.371984] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6718.192760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6718.351441] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6720.155959] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6720.410056] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6726.575049] Lustre: 11515:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491247856/real 1491247856] req@ffff8800379bb000 x1563680146428128/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491247862 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6726.584119] Lustre: 11515:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6726.588254] Removing read-only on unknown block (0xfc00000) [ 6726.607399] Lustre: server umount lustre-MDT0000 complete [ 6726.738034] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6734.680875] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6734.680875] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6734.680875] if [ $running -ne $mpts ]; then [ 6734.680875] echo $(hostname) env are INSANE!; [ 6734.680875] exit 1; [ 6734.680875] fi [ 6735.011960] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6735.011960] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6735.011960] if [ $running -ne $mpts ]; then [ 6735.011960] echo $(hostname) env are INSANE!; [ 6735.011960] exit 1; [ 6735.011960] fi [ 6736.044824] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6736.316053] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6736.570214] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6736.822216] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6736.977821] LDISKFS-fs (dm-0): recovery complete [ 6736.980056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.040947] LustreError: 12088:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6737.051866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.204452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6737.476689] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6737.989625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6737.990710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6738.149120] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6738.152300] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6738.298156] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6738.555943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6738.812904] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6739.082622] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6739.343964] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6741.507799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6741.670782] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6744.471694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6744.630478] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6747.454934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6747.609962] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6750.436708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6750.596261] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6753.454890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6753.612396] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6756.467630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6756.633905] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6759.552346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6759.709741] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6761.901587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6762.062088] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6763.495663] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 6763.834455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 6763.999087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6764.136468] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 6769.507956] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6770.168763] Lustre: DEBUG MARKER: rc=0; [ 6770.168763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6770.168763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6770.168763] echo $(hostname -s): $val; [ 6770.168763] rc=$val; [ 6770.168763] fi; [ 6770.168763] exit $rc [ 6770.790192] Lustre: DEBUG MARKER: dmesg [ 6771.340740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 900: umount should not race with any mgc requeue thread =============================== 19:31:46 \(1491247906\) [ 6771.502767] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 19:31:46 (1491247906) [ 6772.367144] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6772.622157] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6773.603814] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 6773.606657] Lustre: Skipped 6 previous similar messages [ 6775.327082] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007ac31500 x1563680146430400/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6783.595999] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.601495] LustreError: Skipped 13 previous similar messages [ 6784.935725] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6807.792289] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6807.792289] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6807.792289] if [ $running -ne $mpts ]; then [ 6807.792289] echo $(hostname) env are INSANE!; [ 6807.792289] exit 1; [ 6807.792289] fi [ 6808.129805] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6808.129805] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6808.129805] if [ $running -ne $mpts ]; then [ 6808.129805] echo $(hostname) env are INSANE!; [ 6808.129805] exit 1; [ 6808.129805] fi [ 6809.159158] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6809.436938] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6809.690711] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6809.937790] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6810.094476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.161387] LustreError: 14806:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6810.171825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.318124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6810.575081] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6811.069354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6811.071237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6811.228426] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6811.231293] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6811.373378] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6811.631160] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6811.882468] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6812.143452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6812.399976] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6814.557799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6814.721660] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6816.167047] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491247945/real 1491247945] req@ffff8800379b9800 x1563680146430864/t0(0) o8->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491247950 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6816.174498] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6817.502140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6817.662888] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6820.504764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6820.662486] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6823.479901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6823.631359] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6826.465562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6826.627809] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6829.504312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6829.657716] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.529553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.682809] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6834.893042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6835.054134] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6836.493622] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 6836.829753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 6836.987321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6837.130241] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 6842.481937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6843.140870] Lustre: DEBUG MARKER: rc=0; [ 6843.140870] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6843.140870] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6843.140870] echo $(hostname -s): $val; [ 6843.140870] rc=$val; [ 6843.140870] fi; [ 6843.140870] exit $rc [ 6843.770568] Lustre: DEBUG MARKER: dmesg [ 6844.337577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration 6300 sec =========================================================== 19:32:59 \(1491247979\) [ 6844.529389] Lustre: DEBUG MARKER: == sanity test complete, duration 6300 sec =========================================================== 19:32:59 (1491247979) [ 6856.260360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mmp ============----- Mon Apr 3 19:33:11 UTC 2017 [ 6856.409233] Lustre: DEBUG MARKER: -----============= acceptance-small: mmp ============----- Mon Apr 3 19:33:11 UTC 2017 [ 6856.712109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration -o sec ================================================================ 19:33:11 \(1491247991\) [ 6856.860196] Lustre: DEBUG MARKER: == mmp test complete, duration -o sec ================================================================ 19:33:11 (1491247991) [ 6857.933524] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6858.185168] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6858.447067] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062ad4f00 x1563680146556144/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6858.452788] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 6868.591532] Lustre: server umount lustre-MDT0000 complete [ 6868.594092] Lustre: Skipped 1 previous similar message [ 6868.725789] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6880.830985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 6880.991618] Lustre: DEBUG MARKER: excepting tests: [ 6881.151383] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 6881.442584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on MDS, enabling it manually... [ 6881.662708] Lustre: DEBUG MARKER: MMP is not enabled on MDS, enabling it manually... [ 6881.788725] Lustre: DEBUG MARKER: tune2fs -O mmp /dev/lvm-Role_MDS/P1 [ 6882.390476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on OSS, enabling it manually... [ 6882.542213] Lustre: DEBUG MARKER: MMP is not enabled on OSS, enabling it manually... [ 6882.981772] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 6883.542167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 1: two mounts at the same time =========================================================== 19:33:38 \(1491248018\) [ 6883.695825] Lustre: DEBUG MARKER: == mmp test 1: two mounts at the same time =========================================================== 19:33:38 (1491248018) [ 6883.874100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_1 failover pair is needed [ 6884.025835] Lustre: DEBUG MARKER: SKIP: mmp test_1 failover pair is needed [ 6884.195840] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6884.789195] Lustre: DEBUG MARKER: rc=0; [ 6884.789195] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6884.789195] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6884.789195] echo $(hostname -s): $val; [ 6884.789195] rc=$val; [ 6884.789195] fi; [ 6884.789195] exit $rc [ 6885.353571] Lustre: DEBUG MARKER: dmesg [ 6885.840956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 2: one mount delayed by mmp update interval ============================================== 19:33:40 \(1491248020\) [ 6885.990439] Lustre: DEBUG MARKER: == mmp test 2: one mount delayed by mmp update interval ============================================== 19:33:40 (1491248020) [ 6886.173288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_2 failover pair is needed [ 6886.325690] Lustre: DEBUG MARKER: SKIP: mmp test_2 failover pair is needed [ 6886.497265] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6887.098711] Lustre: DEBUG MARKER: rc=0; [ 6887.098711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6887.098711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6887.098711] echo $(hostname -s): $val; [ 6887.098711] rc=$val; [ 6887.098711] fi; [ 6887.098711] exit $rc [ 6887.675474] Lustre: DEBUG MARKER: dmesg [ 6888.169132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 3: one mount delayed by 2x mmp check interval ============================================ 19:33:43 \(1491248023\) [ 6888.318339] Lustre: DEBUG MARKER: == mmp test 3: one mount delayed by 2x mmp check interval ============================================ 19:33:43 (1491248023) [ 6888.495021] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_3 failover pair is needed [ 6888.639879] Lustre: DEBUG MARKER: SKIP: mmp test_3 failover pair is needed [ 6888.808892] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6889.399469] Lustre: DEBUG MARKER: rc=0; [ 6889.399469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6889.399469] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6889.399469] echo $(hostname -s): $val; [ 6889.399469] rc=$val; [ 6889.399469] fi; [ 6889.399469] exit $rc [ 6889.953447] Lustre: DEBUG MARKER: dmesg [ 6890.453928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 4: one mount delayed by \> 2x mmp check interval ========================================== 19:33:45 \(1491248025\) [ 6890.602046] Lustre: DEBUG MARKER: == mmp test 4: one mount delayed by > 2x mmp check interval ========================================== 19:33:45 (1491248025) [ 6890.779822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_4 failover pair is needed [ 6890.923376] Lustre: DEBUG MARKER: SKIP: mmp test_4 failover pair is needed [ 6891.096285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6891.696323] Lustre: DEBUG MARKER: rc=0; [ 6891.696323] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6891.696323] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6891.696323] echo $(hostname -s): $val; [ 6891.696323] rc=$val; [ 6891.696323] fi; [ 6891.696323] exit $rc [ 6892.262192] Lustre: DEBUG MARKER: dmesg [ 6892.759509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 5: mount during unmount of the first filesystem ========================================== 19:33:47 \(1491248027\) [ 6892.901560] Lustre: DEBUG MARKER: == mmp test 5: mount during unmount of the first filesystem ========================================== 19:33:47 (1491248027) [ 6893.075500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_5 failover pair is needed [ 6893.229490] Lustre: DEBUG MARKER: SKIP: mmp test_5 failover pair is needed [ 6893.416643] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6894.022400] Lustre: DEBUG MARKER: rc=0; [ 6894.022400] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6894.022400] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6894.022400] echo $(hostname -s): $val; [ 6894.022400] rc=$val; [ 6894.022400] fi; [ 6894.022400] exit $rc [ 6894.583898] Lustre: DEBUG MARKER: dmesg [ 6895.083581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 6: mount after clean unmount ============================================================= 19:33:50 \(1491248030\) [ 6895.244025] Lustre: DEBUG MARKER: == mmp test 6: mount after clean unmount ============================================================= 19:33:50 (1491248030) [ 6895.429871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_6 failover pair is needed [ 6895.578776] Lustre: DEBUG MARKER: SKIP: mmp test_6 failover pair is needed [ 6895.753846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6896.355546] Lustre: DEBUG MARKER: rc=0; [ 6896.355546] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6896.355546] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6896.355546] echo $(hostname -s): $val; [ 6896.355546] rc=$val; [ 6896.355546] fi; [ 6896.355546] exit $rc [ 6896.919531] Lustre: DEBUG MARKER: dmesg [ 6897.454638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 7: mount after reboot ==================================================================== 19:33:52 \(1491248032\) [ 6897.604391] Lustre: DEBUG MARKER: == mmp test 7: mount after reboot ==================================================================== 19:33:52 (1491248032) [ 6897.786720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_7 failover pair is needed [ 6897.944815] Lustre: DEBUG MARKER: SKIP: mmp test_7 failover pair is needed [ 6898.118163] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6898.705500] Lustre: DEBUG MARKER: rc=0; [ 6898.705500] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6898.705500] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6898.705500] echo $(hostname -s): $val; [ 6898.705500] rc=$val; [ 6898.705500] fi; [ 6898.705500] exit $rc [ 6899.266787] Lustre: DEBUG MARKER: dmesg [ 6899.756540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 8: mount during e2fsck =================================================================== 19:33:54 \(1491248034\) [ 6899.904129] Lustre: DEBUG MARKER: == mmp test 8: mount during e2fsck =================================================================== 19:33:54 (1491248034) [ 6900.036081] Lustre: DEBUG MARKER: debugfs -c -R dump_mmp /dev/lvm-Role_MDS/P1 2>/tmp/mmp.debugfs.msg [ 6900.284491] Lustre: DEBUG MARKER: tune2fs -E mmp_update_interval=30 /dev/lvm-Role_MDS/P1 [ 6911.582716] Lustre: DEBUG MARKER: e2fsck -fy /dev/lvm-Role_MDS/P1 [ 6916.581275] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6918.501767] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6918.747069] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6918.992425] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6919.137465] LustreError: 20776:0:(osd_handler.c:6856:osd_mount()) lustre-MDT0000-osd: can't mount /dev/mapper/lvm--Role_MDS-P1: -16 [ 6919.141306] LustreError: 20776:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000-osd failed (-16) [ 6919.143551] LustreError: 20776:0:(obd_mount.c:199:lustre_start_simple()) lustre-MDT0000-osd setup error -16 [ 6919.146141] LustreError: 20776:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -16 [ 6919.150149] LustreError: 20776:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-16) [ 6919.272943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6973.201894] Lustre: DEBUG MARKER: tune2fs -E mmp_update_interval=5 /dev/lvm-Role_MDS/P1 [ 7169.577605] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7170.179077] Lustre: DEBUG MARKER: rc=0; [ 7170.179077] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7170.179077] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7170.179077] echo $(hostname -s): $val; [ 7170.179077] rc=$val; [ 7170.179077] fi; [ 7170.179077] exit $rc [ 7170.739355] Lustre: DEBUG MARKER: dmesg [ 7171.260120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 9: mount after aborted e2fsck ============================================================ 19:38:26 \(1491248306\) [ 7171.407740] Lustre: DEBUG MARKER: == mmp test 9: mount after aborted e2fsck ============================================================ 19:38:26 (1491248306) [ 7171.547263] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7171.803897] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7172.050168] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7172.293372] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7172.448064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.465767] Lustre: MGS: Connection restored to c3b7cba8-f1eb-55ef-53de-b2087634f089 (at 0@lo) [ 7172.468123] Lustre: Skipped 75 previous similar messages [ 7172.506877] LustreError: 21583:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 7172.517458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.662609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7172.919381] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7173.424683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7173.430936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7173.587682] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7173.587683] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7173.733109] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7173.987319] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7174.237164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7174.493896] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7174.748196] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7176.825996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7176.980710] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7177.513060] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491248308/real 1491248308] req@ffff880036f7de00 x1563680146556608/t0(0) o8->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491248313 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7177.520806] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 7177.878204] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7178.133789] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7178.263619] Lustre: Failing over lustre-MDT0000 [ 7180.949440] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.954786] LustreError: Skipped 14 previous similar messages [ 7184.372706] Lustre: server umount lustre-MDT0000 complete [ 7184.498340] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7191.540170] Lustre: DEBUG MARKER: /usr/lib64/lustre/tests/mmp_mark.sh /dev/lvm-Role_MDS/P1 [ 7203.894869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_MDS\/P1 on mds1... [ 7204.042376] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_MDS/P1 on mds1... [ 7204.172690] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7204.427847] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7204.679279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7204.926920] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7205.070101] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:312: fsck is running on the filesystem [ 7205.072881] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:312: MMP failure info: last update time: 1491248338, last update node: trevis-41vm7.trevis.hpdd.intel.com, last update device: /dev/lvm-Role_MDS/P1 [ 7205.072881] [ 7205.080503] LustreError: 23233:0:(osd_handler.c:6856:osd_mount()) lustre-MDT0000-osd: can't mount /dev/mapper/lvm--Role_MDS-P1: -22 [ 7205.085213] LustreError: 23233:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7205.087950] LustreError: 23233:0:(obd_mount.c:199:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7205.090675] LustreError: 23233:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -22 [ 7205.095416] LustreError: 23233:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-22) [ 7205.221840] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7205.467056] Lustre: DEBUG MARKER: tune2fs -f -E clear-mmp /dev/lvm-Role_MDS/P1 [ 7218.434313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_OSS\/P1 on ost1... [ 7218.599431] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_OSS/P1 on ost1... [ 7220.094728] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7220.686888] Lustre: DEBUG MARKER: rc=0; [ 7220.686888] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7220.686888] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7220.686888] echo $(hostname -s): $val; [ 7220.686888] rc=$val; [ 7220.686888] fi; [ 7220.686888] exit $rc [ 7221.247084] Lustre: DEBUG MARKER: dmesg [ 7221.729050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 10: e2fsck with mounted filesystem ======================================================= 19:39:16 \(1491248356\) [ 7221.875181] Lustre: DEBUG MARKER: == mmp test 10: e2fsck with mounted filesystem ======================================================= 19:39:16 (1491248356) [ 7222.055362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_MDS\/P1 on mds1... [ 7222.210111] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_MDS/P1 on mds1... [ 7222.338140] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7222.593403] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7222.839646] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7223.083466] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7223.230138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.440970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7223.694580] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7224.177456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7224.178810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7224.332917] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7224.334865] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7224.475991] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7224.730636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7224.988252] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7225.254473] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7225.509340] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7225.783336] Lustre: DEBUG MARKER: e2fsck -fn /dev/lvm-Role_MDS/P1 [ 7226.207153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_OSS\/P1 on ost1... [ 7226.363477] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_OSS/P1 on ost1... [ 7228.345174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7228.564579] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7229.753406] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7230.011900] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7230.142189] Lustre: Failing over lustre-MDT0000 [ 7236.367946] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7243.511840] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7244.111029] Lustre: DEBUG MARKER: rc=0; [ 7244.111029] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7244.111029] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7244.111029] echo $(hostname -s): $val; [ 7244.111029] rc=$val; [ 7244.111029] fi; [ 7244.111029] exit $rc [ 7244.678935] Lustre: DEBUG MARKER: dmesg [ 7245.171638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/lvm-Role_MDS\/P1 on mds1 manually... [ 7245.322210] Lustre: DEBUG MARKER: Disabling MMP on /dev/lvm-Role_MDS/P1 on mds1 manually... [ 7245.450500] Lustre: DEBUG MARKER: tune2fs -O ^mmp /dev/lvm-Role_MDS/P1 [ 7256.731277] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 7257.030653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/lvm-Role_OSS\/P1 on ost1 manually... [ 7257.178829] Lustre: DEBUG MARKER: Disabling MMP on /dev/lvm-Role_OSS/P1 on ost1 manually... [ 7268.892139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration 413 sec =============================================================== 19:40:04 \(1491248404\) [ 7269.039054] Lustre: DEBUG MARKER: == mmp test complete, duration 413 sec =============================================================== 19:40:04 (1491248404) [ 7271.073357] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 7271.073357] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 7271.073357] if [ $running -ne $mpts ]; then [ 7271.073357] echo $(hostname) env are INSANE!; [ 7271.073357] exit 1; [ 7271.073357] fi [ 7271.368447] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 7271.368447] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 7271.368447] if [ $running -ne $mpts ]; then [ 7271.368447] echo $(hostname) env are INSANE!; [ 7271.368447] exit 1; [ 7271.368447] fi [ 7272.329575] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7272.585900] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7272.831551] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7273.076256] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7273.229372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.439116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7273.693157] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7274.178161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7274.180998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7274.337140] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7274.337275] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7274.479685] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7274.731776] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7274.982774] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7275.239726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7275.496527] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7277.583198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7277.738998] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7280.538068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7280.700655] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7283.531098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7283.718438] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7286.509796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7286.660845] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7289.516616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7289.723703] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7292.504382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7292.665449] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7295.577125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7295.740314] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7297.906156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7298.067975] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7299.494392] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 7299.804173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 7299.962131] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7300.091353] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 7305.435239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: conf-sanity ============----- Mon Apr 3 19:40:40 UTC 2017 [ 7305.598200] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Apr 3 19:40:40 UTC 2017 [ 7305.906409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test complete, duration -o sec ======================================================== 19:40:41 \(1491248441\) [ 7306.055255] Lustre: DEBUG MARKER: == conf-sanity test complete, duration -o sec ======================================================== 19:40:41 (1491248441) [ 7306.410743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 32newtarball [ 7306.563145] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 7306.738341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 45 69 [ 7306.892871] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 [ 7307.712575] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7307.966350] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7308.303050] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007843c300 x1563680146643888/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 7308.309227] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 7309.644696] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 7316.279656] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7322.290469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 7322.290469] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 7322.290469] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 7322.601615] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7323.155978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 7323.165451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 7323.223946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 7323.250519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 7323.435848] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 7323.450052] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 7323.465077] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 7323.562828] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 7323.835798] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7324.097327] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7324.370495] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [ 7324.564949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7330.985539] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7331.243372] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7331.490454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7331.736090] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7331.889348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7331.927995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.957017] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 7331.959447] Lustre: Skipped 5 previous similar messages [ 7331.977864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 7331.980073] Lustre: Skipped 1 previous similar message [ 7331.995470] Lustre: lustre-MDT0000: new disk, initializing [ 7332.021588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7332.180172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7332.436213] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7332.921049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7332.926921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7333.079983] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7333.079985] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7333.218108] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7333.468286] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7333.723789] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7333.977055] Lustre: DEBUG MARKER: sync; sync; sync [ 7339.246221] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7339.495629] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7341.674612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7341.835626] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7348.056680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 7348.056680] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 7348.056680] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 7348.334404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 7348.334404] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 7348.334404] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 7348.606425] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 7348.856205] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 7349.107663] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7349.600794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7349.607334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7349.761324] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7349.763894] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7349.946256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7349.947285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7350.108761] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7350.111470] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7350.751617] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7350.754544] LustreError: Skipped 6 previous similar messages [ 7350.757012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7350.762226] Lustre: Skipped 2 previous similar messages [ 7357.107489] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7357.361194] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7363.684155] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7364.009997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 0: single mount setup ============================================================ 19:41:39 \(1491248499\) [ 7364.162221] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ============================================================ 19:41:39 (1491248499) [ 7364.301449] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7364.561342] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7364.807983] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7365.052769] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7365.198519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.422830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7365.678492] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7366.161393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7366.164706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7366.317480] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7366.320018] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7366.459846] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7366.708267] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7366.957996] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7367.211602] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7367.460312] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7369.586515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7369.746056] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7380.731762] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 10.9.5.244@tcp failed: rc = -107 [ 7380.734651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7383.967783] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7384.227383] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7390.593916] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7397.056825] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7397.649364] Lustre: DEBUG MARKER: rc=0; [ 7397.649364] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7397.649364] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7397.649364] echo $(hostname -s): $val; [ 7397.649364] rc=$val; [ 7397.649364] fi; [ 7397.649364] exit $rc [ 7398.210661] Lustre: DEBUG MARKER: dmesg [ 7399.519239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 1: start up ost twice \(should return errors\) ===================================== 19:42:13 \(1491248533\) [ 7399.686534] Lustre: DEBUG MARKER: == conf-sanity test 1: start up ost twice (should return errors) ===================================== 19:42:13 (1491248533) [ 7399.822731] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7401.745778] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7401.994664] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7402.241222] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7402.393962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.602619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7402.853577] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7403.331893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7403.355789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7403.517946] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7403.531072] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7403.668383] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7403.938592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7404.206081] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7404.474593] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7404.736397] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7406.834329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7406.992955] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7416.095643] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7416.098296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7420.968033] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7421.217615] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7427.558343] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7433.274385] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7433.870102] Lustre: DEBUG MARKER: rc=0; [ 7433.870102] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7433.870102] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7433.870102] echo $(hostname -s): $val; [ 7433.870102] rc=$val; [ 7433.870102] fi; [ 7433.870102] exit $rc [ 7434.439651] Lustre: DEBUG MARKER: dmesg [ 7435.689789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 2: start up mds twice \(should return err\) ======================================== 19:42:50 \(1491248570\) [ 7435.845342] Lustre: DEBUG MARKER: == conf-sanity test 2: start up mds twice (should return err) ======================================== 19:42:50 (1491248570) [ 7435.989535] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7437.914297] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7438.161619] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7438.405625] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7438.558137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.633269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.635660] Lustre: Skipped 5 previous similar messages [ 7438.783617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7439.035184] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7439.522592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7439.525199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7439.677767] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7439.678083] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7439.817594] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7440.087610] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7440.339043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7440.595141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7440.849607] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7441.112916] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7441.380735] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7441.633566] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7441.887481] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7442.153417] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7443.630113] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491248574/real 1491248574] req@ffff88005d9b1500 x1563680146646640/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491248579 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7443.636954] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 28 previous similar messages [ 7444.239734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7444.396089] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7451.327638] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7451.330209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7457.408085] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7457.659282] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7463.874820] Lustre: server umount lustre-MDT0000 complete [ 7463.877228] Lustre: Skipped 5 previous similar messages [ 7464.009936] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7470.678464] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7471.339233] Lustre: DEBUG MARKER: rc=0; [ 7471.339233] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7471.339233] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7471.339233] echo $(hostname -s): $val; [ 7471.339233] rc=$val; [ 7471.339233] fi; [ 7471.339233] exit $rc [ 7471.913244] Lustre: DEBUG MARKER: dmesg [ 7473.209884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 3: mount client twice \(should return err\) ======================================== 19:43:27 \(1491248607\) [ 7473.361923] Lustre: DEBUG MARKER: == conf-sanity test 3: mount client twice (should return err) ======================================== 19:43:27 (1491248607) [ 7473.503009] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7475.427395] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7475.673999] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7475.921216] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7476.074874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.289349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7476.540862] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7477.026761] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7477.029065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7477.187671] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7477.188544] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7477.326798] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7477.579177] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7477.837786] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7478.099379] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7478.357274] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7480.448067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7480.608417] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7489.599624] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7489.602152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7494.812340] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7495.065048] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7501.485436] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7507.909410] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7508.513802] Lustre: DEBUG MARKER: rc=0; [ 7508.513802] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7508.513802] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7508.513802] echo $(hostname -s): $val; [ 7508.513802] rc=$val; [ 7508.513802] fi; [ 7508.513802] exit $rc [ 7509.083872] Lustre: DEBUG MARKER: dmesg [ 7510.426998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 19:44:04 \(1491248644\) [ 7510.576353] Lustre: DEBUG MARKER: == conf-sanity test 4: force cleanup ost, then cleanup =============================================== 19:44:04 (1491248644) [ 7510.709927] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7512.630049] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7512.878532] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7513.126686] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7513.283201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.501693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7513.755612] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7514.241245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7514.242231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7514.398038] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7514.404880] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7514.540765] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7514.791804] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7515.041882] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7515.297311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7515.547453] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7517.647997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7517.797082] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7525.903637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7542.450755] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7542.702258] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7555.013020] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7560.143250] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7560.728383] Lustre: DEBUG MARKER: rc=0; [ 7560.728383] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7560.728383] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7560.728383] echo $(hostname -s): $val; [ 7560.728383] rc=$val; [ 7560.728383] fi; [ 7560.728383] exit $rc [ 7561.309610] Lustre: DEBUG MARKER: dmesg [ 7562.583483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 19:44:56 \(1491248696\) [ 7562.730665] Lustre: DEBUG MARKER: == conf-sanity test 5a: force cleanup mds, then cleanup ============================================== 19:44:56 (1491248696) [ 7562.871926] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7564.790145] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7565.038119] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7565.289847] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7565.498640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.739155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7565.990528] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7566.471407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7566.475844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7566.630835] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7566.634473] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7566.771723] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7567.027220] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7567.277867] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7567.534899] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7567.789417] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7569.889836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7570.042731] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7572.307265] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7572.560278] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7574.017279] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 7574.019671] Lustre: Skipped 6 previous similar messages [ 7575.599063] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800784ebc00 x1563680146649168/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 7575.604091] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 7580.872044] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.876837] LustreError: Skipped 8 previous similar messages [ 7584.901867] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7601.983357] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7602.229516] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7608.361131] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7608.973819] Lustre: DEBUG MARKER: rc=0; [ 7608.973819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7608.973819] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7608.973819] echo $(hostname -s): $val; [ 7608.973819] rc=$val; [ 7608.973819] fi; [ 7608.973819] exit $rc [ 7609.544323] Lustre: DEBUG MARKER: dmesg [ 7610.794269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5b: Try to start a client with no MGS \(should return errs\) ======================= 19:45:45 \(1491248745\) [ 7610.942515] Lustre: DEBUG MARKER: == conf-sanity test 5b: Try to start a client with no MGS (should return errs) ======================= 19:45:45 (1491248745) [ 7671.596422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7671.753459] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7710.522741] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7710.767636] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7715.990512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7716.592956] Lustre: DEBUG MARKER: rc=0; [ 7716.592956] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7716.592956] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7716.592956] echo $(hostname -s): $val; [ 7716.592956] rc=$val; [ 7716.592956] fi; [ 7716.592956] exit $rc [ 7717.173859] Lustre: DEBUG MARKER: dmesg [ 7718.442130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5c: cleanup after failed mount \(bug 2712\) \(should return errs\) =================== 19:47:32 \(1491248852\) [ 7718.594441] Lustre: DEBUG MARKER: == conf-sanity test 5c: cleanup after failed mount (bug 2712) (should return errs) =================== 19:47:32 (1491248852) [ 7718.731042] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7720.649080] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7720.899757] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7721.145082] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7721.289712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.513727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7721.767296] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7722.257447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7722.270035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7722.418619] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7722.422778] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7722.558216] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7722.810450] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7723.061724] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7723.320279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7723.584982] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7725.690389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7725.844894] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7727.703405] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7727.955147] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7734.284102] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7740.099311] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7740.718147] Lustre: DEBUG MARKER: rc=0; [ 7740.718147] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7740.718147] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7740.718147] echo $(hostname -s): $val; [ 7740.718147] rc=$val; [ 7740.718147] fi; [ 7740.718147] exit $rc [ 7741.303151] Lustre: DEBUG MARKER: dmesg [ 7742.594947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5d: mount with ost down ========================================================== 19:47:56 \(1491248876\) [ 7742.744359] Lustre: DEBUG MARKER: == conf-sanity test 5d: mount with ost down ========================================================== 19:47:56 (1491248876) [ 7803.361940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7803.521269] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7804.425773] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7804.683611] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7804.933166] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7805.180028] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7805.335370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7805.357899] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [ 7805.360292] Lustre: Skipped 66 previous similar messages [ 7805.550118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7805.804807] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7806.308372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7806.308379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7806.461500] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7806.464420] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7806.603854] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7806.853962] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7807.104549] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7807.358126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7807.609666] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7810.399732] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7810.402308] LustreError: Skipped 1 previous similar message [ 7810.404417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7825.372290] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7825.633706] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7837.946439] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7843.177963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7843.769324] Lustre: DEBUG MARKER: rc=0; [ 7843.769324] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7843.769324] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7843.769324] echo $(hostname -s): $val; [ 7843.769324] rc=$val; [ 7843.769324] fi; [ 7843.769324] exit $rc [ 7844.338099] Lustre: DEBUG MARKER: dmesg [ 7845.624395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5e: delayed connect, don\'t crash \(bug 10268\) ===================================== 19:49:39 \(1491248979\) [ 7845.790133] Lustre: DEBUG MARKER: == conf-sanity test 5e: delayed connect, don't crash (bug 10268) ===================================== 19:49:39 (1491248979) [ 7845.928215] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7847.845204] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7848.091446] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7848.337803] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7848.492681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.706143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7848.959531] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7849.529902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7849.529943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7849.690844] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7849.693436] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7849.828703] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7850.080117] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7850.331017] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7850.584768] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7850.838304] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7852.950112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7853.105527] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7871.064888] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7871.318484] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7877.661605] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7883.359458] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7883.963426] Lustre: DEBUG MARKER: rc=0; [ 7883.963426] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7883.963426] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7883.963426] echo $(hostname -s): $val; [ 7883.963426] rc=$val; [ 7883.963426] fi; [ 7883.963426] exit $rc [ 7884.540897] Lustre: DEBUG MARKER: dmesg [ 7885.805783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5f: mds down, cleanup after failed mount \(bug 2712\) ============================== 19:50:20 \(1491249020\) [ 7885.957514] Lustre: DEBUG MARKER: == conf-sanity test 5f: mds down, cleanup after failed mount (bug 2712) ============================== 19:50:20 (1491249020) [ 7886.155288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_5f combined mgs and mds [ 7886.304497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_5f combined mgs and mds [ 7886.481940] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7887.086936] Lustre: DEBUG MARKER: rc=0; [ 7887.086936] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7887.086936] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7887.086936] echo $(hostname -s): $val; [ 7887.086936] rc=$val; [ 7887.086936] fi; [ 7887.086936] exit $rc [ 7887.656875] Lustre: DEBUG MARKER: dmesg [ 7888.155943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 5g: handle missing debugfs ======================================================= 19:50:23 \(1491249023\) [ 7888.311545] Lustre: DEBUG MARKER: == conf-sanity test 5g: handle missing debugfs ======================================================= 19:50:23 (1491249023) [ 7888.547439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7889.142512] Lustre: DEBUG MARKER: rc=0; [ 7889.142512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7889.142512] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7889.142512] echo $(hostname -s): $val; [ 7889.142512] rc=$val; [ 7889.142512] fi; [ 7889.142512] exit $rc [ 7889.707580] Lustre: DEBUG MARKER: dmesg [ 7890.218476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 6: manual umount, then mount again =============================================== 19:50:25 \(1491249025\) [ 7890.372401] Lustre: DEBUG MARKER: == conf-sanity test 6: manual umount, then mount again =============================================== 19:50:25 (1491249025) [ 7890.509119] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7890.778620] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7891.033623] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7891.278798] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7891.431523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7891.640447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7891.891386] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7892.375164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7892.375692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7892.533273] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7892.533708] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7892.673673] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7892.936220] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7893.186188] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7893.443189] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7893.695991] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7895.772348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7895.927969] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7904.575638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7904.580164] Lustre: Skipped 1 previous similar message [ 7910.168092] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7910.420372] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7916.761543] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7922.527156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7923.122868] Lustre: DEBUG MARKER: rc=0; [ 7923.122868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7923.122868] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7923.122868] echo $(hostname -s): $val; [ 7923.122868] rc=$val; [ 7923.122868] fi; [ 7923.122868] exit $rc [ 7923.692091] Lustre: DEBUG MARKER: dmesg [ 7925.000115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 7: manual umount, then cleanup =================================================== 19:50:59 \(1491249059\) [ 7925.226443] Lustre: DEBUG MARKER: == conf-sanity test 7: manual umount, then cleanup =================================================== 19:50:59 (1491249059) [ 7925.368407] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7927.287640] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7927.538747] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7927.783472] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7927.940598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.154524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7928.416158] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7928.898348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7928.898863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7929.057225] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7929.058469] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7929.198843] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7929.453371] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7929.702726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7929.960736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7930.212086] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7932.326774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7932.485041] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7941.519630] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 7941.522197] LustreError: Skipped 2 previous similar messages [ 7946.522188] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7946.771588] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7953.205794] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7958.754877] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7959.346663] Lustre: DEBUG MARKER: rc=0; [ 7959.346663] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7959.346663] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7959.346663] echo $(hostname -s): $val; [ 7959.346663] rc=$val; [ 7959.346663] fi; [ 7959.346663] exit $rc [ 7959.921244] Lustre: DEBUG MARKER: dmesg [ 7961.185697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 8: double mount setup ============================================================ 19:51:35 \(1491249095\) [ 7961.345151] Lustre: DEBUG MARKER: == conf-sanity test 8: double mount setup ============================================================ 19:51:35 (1491249095) [ 7961.479682] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 7963.398347] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7963.646466] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7963.891311] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7964.046271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.121853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.124547] Lustre: Skipped 8 previous similar messages [ 7964.277438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7964.530381] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 7965.020536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7965.023853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7965.180058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7965.180739] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7965.319995] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7965.568499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7965.818109] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7966.072951] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7966.324651] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7968.427178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7968.578957] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 7970.112040] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491249099/real 1491249099] req@ffff880059831500 x1563680146652672/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491249104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7970.119126] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 7982.999384] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7983.250688] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 7989.485362] Lustre: server umount lustre-MDT0000 complete [ 7989.487542] Lustre: Skipped 8 previous similar messages [ 7989.612195] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7996.218100] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7996.861713] Lustre: DEBUG MARKER: rc=0; [ 7996.861713] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7996.861713] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7996.861713] echo $(hostname -s): $val; [ 7996.861713] rc=$val; [ 7996.861713] fi; [ 7996.861713] exit $rc [ 7997.432769] Lustre: DEBUG MARKER: dmesg [ 7998.678741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 19:52:13 \(1491249133\) [ 7998.828849] Lustre: DEBUG MARKER: == conf-sanity test 9: test ptldebug and subsystem for mkfs ========================================== 19:52:13 (1491249133) [ 8059.420610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8059.574808] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8062.304163] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8062.893081] Lustre: DEBUG MARKER: rc=0; [ 8062.893081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8062.893081] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8062.893081] echo $(hostname -s): $val; [ 8062.893081] rc=$val; [ 8062.893081] fi; [ 8062.893081] exit $rc [ 8063.458127] Lustre: DEBUG MARKER: dmesg [ 8063.957753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 17: Verify failed mds_postsetup won\'t fail assertion \(2936\) \(should return errs\) ====================================================================================================== 19:53:19 \(1491249199\) [ 8064.115625] Lustre: DEBUG MARKER: == conf-sanity test 17: Verify failed mds_postsetup won't fail assertion (2936) (should return errs) ====================================================================================================== 19:53:19 (1491249199) [ 8064.256402] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8064.512697] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8064.757493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8065.002400] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8065.144755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.354911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8065.607544] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8066.085715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8066.091395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8066.240392] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8066.243555] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8066.379545] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8066.630090] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8066.884326] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8067.137535] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8067.390995] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8069.517746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8069.671415] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8077.663643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8077.668512] Lustre: Skipped 2 previous similar messages [ 8083.906189] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8084.156525] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8090.529436] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8096.932728] Lustre: DEBUG MARKER: debugfs -w -R 'unlink CONFIGS/lustre-MDT0000' /dev/lvm-Role_MDS/P1 || return 0 [ 8158.480715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8158.633482] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8159.541156] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8159.797474] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8160.051420] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8160.312817] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8160.477695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.520045] LustreError: 15c-8: MGC10.9.5.243@tcp: The configuration from log 'lustre-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. [ 8160.527423] LustreError: 25748:0:(obd_mount_server.c:1351:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 8160.545366] LustreError: 25748:0:(obd_mount_server.c:1840:server_fill_super()) Unable to start targets: -2 [ 8160.548151] LustreError: 25748:0:(obd_mount_server.c:1554:server_put_super()) no obd lustre-MDT0000 [ 8166.560469] LustreError: 25748:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-2) [ 8166.700338] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8167.530724] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8167.777296] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8171.879752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8171.879752] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8171.879752] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8172.195868] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8172.774892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8172.798520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8172.810398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8172.811959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8173.071177] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8173.072018] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8173.079972] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8173.107665] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8173.404107] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8173.650147] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8173.928728] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [ 8174.118387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8180.696086] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8180.952547] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8181.200924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8181.445467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8181.589658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8181.602504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8181.636943] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 8181.639283] Lustre: Skipped 4 previous similar messages [ 8181.659741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 8181.661881] Lustre: Skipped 1 previous similar message [ 8181.677465] Lustre: lustre-MDT0000: new disk, initializing [ 8181.705604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8181.868232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8182.119626] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8182.604386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8182.604387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8182.771833] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8182.773460] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8182.912732] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8183.165407] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8183.418453] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8183.670961] Lustre: DEBUG MARKER: sync; sync; sync [ 8188.977389] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8189.229624] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8191.432426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8191.594153] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8197.895258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8197.895258] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8197.895258] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8198.161377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8198.161377] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8198.161377] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8198.425449] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8198.676914] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8198.930929] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8199.416861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8199.418868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8199.581047] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8199.581497] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8199.772166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8199.778927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8199.969132] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8199.976417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8200.639712] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8200.642546] LustreError: Skipped 3 previous similar messages [ 8202.927619] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8203.181784] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8209.506153] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8209.804320] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8210.419098] Lustre: DEBUG MARKER: rc=0; [ 8210.419098] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8210.419098] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8210.419098] echo $(hostname -s): $val; [ 8210.419098] rc=$val; [ 8210.419098] fi; [ 8210.419098] exit $rc [ 8211.001820] Lustre: DEBUG MARKER: dmesg [ 8211.519915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 18: check mkfs creates large journals ============================================ 19:55:46 \(1491249346\) [ 8211.697223] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ============================================ 19:55:46 (1491249346) [ 8211.908156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark use STORED_MDSSIZE=2097152 [ 8212.077433] Lustre: DEBUG MARKER: use STORED_MDSSIZE=2097152 [ 8212.201634] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8212.444206] Lustre: DEBUG MARKER: dd if=/dev/lvm-Role_MDS/P1 of=/dev/null bs=1k count=1 skip=2000000 2>&1 [ 8213.286476] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8213.531718] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8217.373397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8217.373397] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8217.373397] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8217.692573] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8218.277627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8218.295395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8218.304396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8218.310777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8218.545930] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8218.554844] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8218.559020] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8218.614967] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8218.887892] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8219.134573] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8219.409960] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-E lazy_itable_ [ 8219.617294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8226.388280] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8226.646237] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8226.894155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8227.140973] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8227.284222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8227.297243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.332244] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 8227.334394] Lustre: Skipped 4 previous similar messages [ 8227.350382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 8227.352403] Lustre: Skipped 1 previous similar message [ 8227.367807] Lustre: lustre-MDT0000: new disk, initializing [ 8227.424234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8227.600436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8227.857947] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8228.359955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8228.365497] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8228.517230] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8228.520565] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8228.655211] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8228.904287] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8229.162698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8229.420711] Lustre: DEBUG MARKER: sync; sync; sync [ 8234.697096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8234.948797] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8237.217983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8237.383713] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8243.573167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8243.573167] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8243.573167] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8243.840041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8243.840041] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8243.840041] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8244.105042] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8244.357535] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8244.609863] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8245.104649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8245.111417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8245.295942] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8245.311292] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8245.535166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8245.545774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8245.701952] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8245.702288] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8246.712432] LustreError: 32360:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -107 [ 8253.150909] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8253.398879] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8259.722289] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8259.985617] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8260.241463] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8260.488872] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8260.735627] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8260.892179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.103092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8261.353671] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8261.835205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8261.837185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8261.996528] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8261.999368] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8262.136895] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8262.385397] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8262.635867] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8262.897289] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8263.163614] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8265.268399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8265.420488] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8272.720167] Lustre: DEBUG MARKER: debugfs -c -R 'stat <8>' /dev/lvm-Role_MDS/P1 [ 8273.040377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Success: mkfs creates large journals. Size: 81M [ 8273.196371] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 81M [ 8289.590480] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8289.854648] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8296.475348] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8303.331932] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8303.577096] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8307.581361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8307.581361] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8307.581361] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8307.947698] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8308.597578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8308.698767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8308.771597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8308.779528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8309.016225] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 8309.106496] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 8309.124273] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8309.174308] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 8311.803370] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8312.051019] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8312.327836] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [ 8312.516971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.040547] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8319.298541] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8319.547230] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8319.791832] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8319.944513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.985670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8320.020003] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 8320.022251] Lustre: Skipped 4 previous similar messages [ 8320.042694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 8320.044733] Lustre: Skipped 1 previous similar message [ 8320.060143] Lustre: lustre-MDT0000: new disk, initializing [ 8320.090076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8320.252372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8320.516497] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8321.021674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8321.023519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8321.181764] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8321.183735] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8321.320941] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8321.575260] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8321.834731] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8322.091233] Lustre: DEBUG MARKER: sync; sync; sync [ 8327.361580] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8327.613751] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8329.792546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8329.948742] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8336.141577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8336.141577] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8336.141577] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8336.421410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8336.421410] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8336.421410] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8336.692797] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8336.953504] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8337.207420] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8337.688506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8337.688875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8337.847300] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8337.847944] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8338.031774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.032130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.187601] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.201178] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8340.927750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8340.933171] Lustre: Skipped 3 previous similar messages [ 8345.187692] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8345.437914] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8351.762904] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8352.068061] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8352.673081] Lustre: DEBUG MARKER: rc=0; [ 8352.673081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8352.673081] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8352.673081] echo $(hostname -s): $val; [ 8352.673081] rc=$val; [ 8352.673081] fi; [ 8352.673081] exit $rc [ 8353.245575] Lustre: DEBUG MARKER: dmesg [ 8353.764118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 19a: start\/stop MDS without OSTs ================================================= 19:58:08 \(1491249488\) [ 8353.917697] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ================================================= 19:58:08 (1491249488) [ 8354.053665] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8354.310081] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8354.558633] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8354.807363] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8354.962261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.169929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8355.429329] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8355.913693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8355.913871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8356.070466] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8356.071498] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8356.208839] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8356.460538] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8356.710995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8356.967361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8357.218227] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8357.475562] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8357.725045] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8364.036710] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8364.354230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8364.950249] Lustre: DEBUG MARKER: rc=0; [ 8364.950249] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8364.950249] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8364.950249] echo $(hostname -s): $val; [ 8364.950249] rc=$val; [ 8364.950249] fi; [ 8364.950249] exit $rc [ 8365.537974] Lustre: DEBUG MARKER: dmesg [ 8366.078674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 19b: start\/stop OSTs without MDS ================================================= 19:58:21 \(1491249501\) [ 8366.238657] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ================================================= 19:58:21 (1491249501) [ 8425.298592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8425.445378] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8429.011847] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8429.627240] Lustre: DEBUG MARKER: rc=0; [ 8429.627240] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8429.627240] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8429.627240] echo $(hostname -s): $val; [ 8429.627240] rc=$val; [ 8429.627240] fi; [ 8429.627240] exit $rc [ 8430.225302] Lustre: DEBUG MARKER: dmesg [ 8430.831534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 20: remount ro,rw mounts work and doesn\'t break \/etc\/mtab ======================== 19:59:25 \(1491249565\) [ 8430.994543] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ======================== 19:59:25 (1491249565) [ 8431.140652] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8431.404417] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8431.656653] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8431.905876] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8432.065293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.083528] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [ 8432.085604] Lustre: Skipped 57 previous similar messages [ 8432.278278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8432.530856] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8433.078642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8433.112359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8433.264599] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8433.266195] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8433.408800] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8433.667539] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8433.948823] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8434.205964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8434.459075] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8436.657492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8436.821330] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8443.136411] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8443.391912] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8445.763068] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 8448.111080] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007843cf00 x1563680146658864/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8455.708850] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8463.111465] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8463.743169] Lustre: DEBUG MARKER: rc=0; [ 8463.743169] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8463.743169] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8463.743169] echo $(hostname -s): $val; [ 8463.743169] rc=$val; [ 8463.743169] fi; [ 8463.743169] exit $rc [ 8464.325969] Lustre: DEBUG MARKER: dmesg [ 8464.902719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 20:00:00 \(1491249600\) [ 8465.087631] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ======================================== 20:00:00 (1491249600) [ 8465.250869] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8465.512093] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8465.774255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8466.038490] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8466.185571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.403160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8466.657467] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8467.155438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8467.157328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8467.313213] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8467.322122] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8467.458782] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8467.728390] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8467.982965] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8468.241362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8468.495040] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8470.722851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8470.963822] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8471.998357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8471.998357] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8471.998357] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8472.271919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8472.271919] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8472.271919] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8472.545852] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8472.797505] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8473.057258] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8473.544318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8473.546244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8473.704835] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8473.706790] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8473.891133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8473.905030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8474.062888] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8474.063863] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8477.114479] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8477.370149] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8483.700176] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8484.003394] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8484.629264] Lustre: DEBUG MARKER: rc=0; [ 8484.629264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8484.629264] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8484.629264] echo $(hostname -s): $val; [ 8484.629264] rc=$val; [ 8484.629264] fi; [ 8484.629264] exit $rc [ 8485.237801] Lustre: DEBUG MARKER: dmesg [ 8485.814169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 20:00:20 \(1491249620\) [ 8485.970113] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ======================================== 20:00:20 (1491249620) [ 8545.024609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8545.211164] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8546.135653] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8546.401903] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8546.661298] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8546.917792] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8547.072964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.284146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8547.540495] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8548.045235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8548.045757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8548.203618] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8548.207115] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8548.344293] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8548.595537] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8548.846656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8549.102443] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8549.354676] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8549.612394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8549.612394] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8549.612394] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8549.891098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8549.891098] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8549.891098] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8550.161360] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8550.418284] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8550.674158] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8551.209336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8551.348146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8551.499116] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8551.499375] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8551.695875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8551.724803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8551.878646] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8551.878650] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8552.022136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8552.275048] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8553.091129] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.096357] LustreError: Skipped 1 previous similar message [ 8558.602564] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8560.389074] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8560.999210] Lustre: DEBUG MARKER: rc=0; [ 8560.999210] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8560.999210] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8560.999210] echo $(hostname -s): $val; [ 8560.999210] rc=$val; [ 8560.999210] fi; [ 8560.999210] exit $rc [ 8561.570952] Lustre: DEBUG MARKER: dmesg [ 8562.119571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 20:01:37 \(1491249697\) [ 8562.281855] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last =================================== 20:01:37 (1491249697) [ 8621.471760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8621.634801] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8622.568987] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8622.831187] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8623.084080] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8623.329443] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8623.486586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.553564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8623.556142] Lustre: Skipped 9 previous similar messages [ 8623.706817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8623.966812] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8624.454334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8624.464828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8624.619942] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8624.621698] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8624.763050] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8625.023848] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8625.295199] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8625.560769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8625.811711] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8630.268652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8630.425023] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8639.391453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8639.391453] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8639.391453] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8639.663960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8639.663960] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8639.663960] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8639.935304] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8640.189368] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8640.446058] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8640.967660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 8640.969334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 8641.130228] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 8641.131907] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 8641.323604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8641.323699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8641.482430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8641.495186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8651.995598] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8652.265938] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8658.435049] Lustre: 18697:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491249788/real 1491249788] req@ffff880062ad4900 x1563680146661296/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491249794 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8658.442927] Lustre: 18697:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 8658.482584] Lustre: server umount lustre-MDT0000 complete [ 8658.485031] Lustre: Skipped 10 previous similar messages [ 8658.611634] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8658.878416] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8659.129690] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8659.391638] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [ 8659.558205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8661.652253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8662.260544] Lustre: DEBUG MARKER: rc=0; [ 8662.260544] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8662.260544] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8662.260544] echo $(hostname -s): $val; [ 8662.260544] rc=$val; [ 8662.260544] fi; [ 8662.260544] exit $rc [ 8662.833497] Lustre: DEBUG MARKER: dmesg [ 8663.359140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 21d: start mgs then ost and then mds ============================================= 20:03:18 \(1491249798\) [ 8663.511130] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ============================================= 20:03:18 (1491249798) [ 8663.719938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_21d need separate mgs device [ 8663.872720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 8664.059443] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8664.681745] Lustre: DEBUG MARKER: rc=0; [ 8664.681745] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8664.681745] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8664.681745] echo $(hostname -s): $val; [ 8664.681745] rc=$val; [ 8664.681745] fi; [ 8664.681745] exit $rc [ 8665.285629] Lustre: DEBUG MARKER: dmesg [ 8665.839483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 21e: separate MGS and MDS ======================================================== 20:03:20 \(1491249800\) [ 8666.010239] Lustre: DEBUG MARKER: == conf-sanity test 21e: separate MGS and MDS ======================================================== 20:03:20 (1491249800) [ 8671.013537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8671.165174] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8672.365187] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 8672.620434] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8672.876081] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.9.5.244@tcp --fsname=test1234 --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsopti [ 8673.129742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8678.583038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8678.740526] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8687.289101] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [ 8687.552694] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [ 8687.805028] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [ 8688.055546] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [ 8688.245192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8688.256455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.308991] Lustre: ctl-test1234-MDT0000: No data found on store. Initialize space [ 8688.311244] Lustre: Skipped 1 previous similar message [ 8688.327068] Lustre: test1234-MDT0000: new disk, initializing [ 8688.404539] Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8688.609638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8688.868154] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8689.382362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8689.382436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8689.547967] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8689.547968] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8689.692923] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8689.955690] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8690.237801] Lustre: DEBUG MARKER: sync; sync; sync [ 8695.559051] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [ 8696.010424] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 8696.269582] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-fs2mds [ 8698.383111] LustreError: 4493:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007843ea00 x1563680146661872/t0(0) o13->test1234-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8700.552605] Lustre: test1234-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 8702.628833] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8711.761728] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8712.435238] Lustre: DEBUG MARKER: rc=0; [ 8712.435238] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8712.435238] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8712.435238] echo $(hostname -s): $val; [ 8712.435238] rc=$val; [ 8712.435238] fi; [ 8712.435238] exit $rc [ 8713.035207] Lustre: DEBUG MARKER: dmesg [ 8713.589233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 22: start a client before osts \(should return errs\) ============================== 20:04:08 \(1491249848\) [ 8713.759745] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ============================== 20:04:08 (1491249848) [ 8713.914778] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8714.179200] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8714.441555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8714.696020] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8714.854606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.890985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8714.898635] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 8714.900988] Lustre: Skipped 4 previous similar messages [ 8715.086424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8715.342207] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8715.834899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8715.839657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8715.993027] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8715.997497] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8716.134415] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8716.388851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8716.651703] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8716.912534] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8717.170241] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8718.618618] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8719.876101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8720.029199] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8720.941541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8720.941541] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8720.941541] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8721.210855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8721.210855] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8721.210855] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8721.486406] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8721.743735] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8722.003048] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8722.499366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8722.504177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8722.663888] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8722.664340] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8722.852781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8722.854897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8723.019026] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8723.019102] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8727.375827] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8727.378669] LustreError: Skipped 6 previous similar messages [ 8742.507188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8742.671608] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8743.663183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8743.663183] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8743.663183] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8743.944900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8743.944900] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8743.944900] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8744.227539] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 8744.490872] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 8744.758131] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8745.279441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8745.297582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8745.457512] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8745.481527] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8750.056950] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8750.689723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 8750.690155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 8750.848180] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 8750.852784] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 8752.757520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069887800.ost_server_uuid 40 [ 8752.922196] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069887800.ost_server_uuid 40 [ 8753.117211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069887800.ost_server_uuid in FULL state after 0 sec [ 8753.277413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069887800.ost_server_uuid in FULL state after 0 sec [ 8753.308242] LustreError: 22046:0:(ldlm_lockd.c:698:ldlm_handle_ast_error()) ### client (nid 10.9.5.237@tcp) returned error from blocking AST (req@ffff880059830900 x1563680146663024 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff880060e72800/0xaf2ce85aa2366910 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.5.237@tcp remote: 0x2deff34e44187704 expref: 6 pid: 22046 timeout: 4303520604 lvb_type: 0 [ 8753.320080] LustreError: 138-a: lustre-MDT0000: A client on nid 10.9.5.237@tcp was evicted due to a lock blocking callback time out: rc -107 [ 8757.997488] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8758.316751] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8766.965036] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8773.678372] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8774.297574] Lustre: DEBUG MARKER: rc=0; [ 8774.297574] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8774.297574] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8774.297574] echo $(hostname -s): $val; [ 8774.297574] rc=$val; [ 8774.297574] fi; [ 8774.297574] exit $rc [ 8774.881477] Lustre: DEBUG MARKER: dmesg [ 8776.158130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 20:05:10 \(1491249910\) [ 8776.315458] Lustre: DEBUG MARKER: == conf-sanity test 23a: interrupt client during recovery mount delay ================================ 20:05:10 (1491249910) [ 8776.456147] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8778.438056] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8778.707123] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8778.959233] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8779.314207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.949087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8780.210344] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8780.725524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8780.725540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8780.881624] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8780.895344] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8781.039632] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8781.299227] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8781.563558] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8781.843326] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8782.109025] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8784.770426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8785.015569] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8792.262532] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8792.525970] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8792.658591] Lustre: Failing over lustre-MDT0000 [ 8798.875584] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8809.165876] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8809.437210] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8809.701193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8809.952993] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8810.101173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8810.309083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8810.577318] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8811.062186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8811.064631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8811.222953] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8811.222990] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8811.378159] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8811.637702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8811.895575] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8812.154353] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8812.406807] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8812.570165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8812.572754] Lustre: lustre-MDT0000: Denying connection for new client f2e40c59-31dc-845a-36b0-43a52b82a0f7(at 10.9.5.237@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8817.570234] Lustre: lustre-MDT0000: Denying connection for new client f2e40c59-31dc-845a-36b0-43a52b82a0f7(at 10.9.5.237@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 8818.754358] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8819.027350] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8819.157038] LustreError: 28599:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 8819.159775] Lustre: 27582:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 8819.162373] Lustre: lustre-MDT0000: Recovery over after 0:06, of 1 clients 0 recovered and 1 was evicted. [ 8819.173910] LustreError: 27573:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007843c000 x1563680146664320/t0(0) o5->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [ 8819.179088] LustreError: 27573:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 8831.412309] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8838.539992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8839.227079] Lustre: DEBUG MARKER: rc=0; [ 8839.227079] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8839.227079] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8839.227079] echo $(hostname -s): $val; [ 8839.227079] rc=$val; [ 8839.227079] fi; [ 8839.227079] exit $rc [ 8839.836083] Lustre: DEBUG MARKER: dmesg [ 8840.844371] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8841.102461] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8847.879431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 20:06:22 \(1491249982\) [ 8848.033118] Lustre: DEBUG MARKER: == conf-sanity test 23b: Simulate -EINTR during mount ================================================ 20:06:22 (1491249982) [ 8848.181342] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8850.106302] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8850.358117] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8850.607523] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8850.855429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.482897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8851.740360] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8853.265846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8853.270884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8853.425963] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8853.427119] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8853.567767] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8853.822701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8854.089659] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8854.348173] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8854.604540] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8856.866719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8857.025610] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8865.247830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8865.252540] Lustre: Skipped 5 previous similar messages [ 8870.307631] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8870.563456] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8876.929251] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8882.520856] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8883.156006] Lustre: DEBUG MARKER: rc=0; [ 8883.156006] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8883.156006] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8883.156006] echo $(hostname -s): $val; [ 8883.156006] rc=$val; [ 8883.156006] fi; [ 8883.156006] exit $rc [ 8883.758275] Lustre: DEBUG MARKER: dmesg [ 8885.070175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 20:06:59 \(1491250019\) [ 8885.241612] Lustre: DEBUG MARKER: == conf-sanity test 24a: Multiple MDTs on a single node ============================================== 20:06:59 (1491250019) [ 8885.423773] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 8885.683218] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8885.944518] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsopti [ 8886.162036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8887.237394] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8889.179375] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8889.440106] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8889.691885] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8889.848575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8890.111764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8890.373865] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8890.897401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8890.898272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8891.059213] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8891.060261] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8891.210096] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8891.475841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8891.738956] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8892.011491] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8892.282925] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8894.483700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8894.665533] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8901.923444] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [ 8902.193947] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [ 8902.455967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [ 8902.723747] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [ 8902.878240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8902.891491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.908616] Lustre: Setting parameter test1234-MDT0000-mdtlov.lov.stripesize in log test1234-MDT0000 [ 8902.911100] Lustre: Skipped 4 previous similar messages [ 8902.920553] Lustre: ctl-test1234-MDT0000: No data found on store. Initialize space [ 8902.922958] Lustre: Skipped 1 previous similar message [ 8902.938953] Lustre: test1234-MDT0000: new disk, initializing [ 8902.979698] Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8903.137612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8903.401652] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8903.915716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8903.922401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8904.073972] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8904.078528] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8904.218251] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8904.473525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8904.730500] Lustre: DEBUG MARKER: sync; sync; sync [ 8910.048709] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [ 8912.287426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8912.458883] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8934.284844] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 8934.542508] Lustre: DEBUG MARKER: umount -d /mnt/lustre-fs2mds [ 8934.670122] Lustre: Failing over test1234-MDT0000 [ 8934.878179] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8945.169916] Lustre: DEBUG MARKER: hostname [ 8945.486135] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [ 8945.742733] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [ 8945.999772] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [ 8946.340991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.375332] Lustre: test1234-MDT0000: Not available for connect from 10.9.5.244@tcp (not set up) [ 8946.675081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8946.938115] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 8947.448630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8947.461642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8947.623347] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8947.626835] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8947.766388] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8948.023572] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8948.282367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [ 8954.061139] Lustre: test1234-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 8954.092606] Lustre: test1234-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 8961.516273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8961.672650] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 8962.737313] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8962.993654] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 8965.588982] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 8965.823139] LustreError: 4494:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007843d200 x1563680146667728/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8969.434271] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8969.697374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n devices [ 8975.977098] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 8976.233376] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-fs2mds [ 8982.697786] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8990.618593] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8990.873088] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8996.948910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8997.582095] Lustre: DEBUG MARKER: rc=0; [ 8997.582095] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8997.582095] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8997.582095] echo $(hostname -s): $val; [ 8997.582095] rc=$val; [ 8997.582095] fi; [ 8997.582095] exit $rc [ 8998.184041] Lustre: DEBUG MARKER: dmesg [ 8999.505540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 24b: Multiple MGSs on a single node \(should return err\) ========================== 20:08:53 \(1491250133\) [ 8999.655332] Lustre: DEBUG MARKER: == conf-sanity test 24b: Multiple MGSs on a single node (should return err) ========================== 20:08:53 (1491250133) [ 8999.830363] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 9000.076757] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9000.330623] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsopti [ 9000.501677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9000.659067] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9002.617195] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9002.869173] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9003.119193] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9003.279372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.505830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9003.765107] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9004.319390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9004.319398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9004.482362] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9004.483993] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9004.622203] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9004.891873] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9005.160211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9005.428435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9005.688322] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9007.918390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9008.079255] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9015.412617] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [ 9015.681238] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [ 9015.949129] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [ 9016.215191] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [ 9016.547825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9016.662569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.674470] LustreError: 15d-9: The MGS service was already started from server [ 9016.676965] LustreError: 5436:0:(obd_mount_server.c:1554:server_put_super()) no obd lustre2-MDT0000 [ 9016.679470] LustreError: 5436:0:(obd_mount_server.c:135:server_deregister_mount()) lustre2-MDT0000 not registered [ 9016.697294] LustreError: 5436:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-114) [ 9016.902637] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [ 9017.161375] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9020.411632] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9020.666971] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9027.330252] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9032.813741] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9033.460966] Lustre: DEBUG MARKER: rc=0; [ 9033.460966] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9033.460966] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9033.460966] echo $(hostname -s): $val; [ 9033.460966] rc=$val; [ 9033.460966] fi; [ 9033.460966] exit $rc [ 9034.053821] Lustre: DEBUG MARKER: dmesg [ 9035.389344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 25: Verify modules are referenced ================================================ 20:09:29 \(1491250169\) [ 9035.538656] Lustre: DEBUG MARKER: == conf-sanity test 25: Verify modules are referenced ================================================ 20:09:29 (1491250169) [ 9035.686978] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9037.715804] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9037.975647] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9038.224548] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9038.385975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.405060] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [ 9038.408235] Lustre: Skipped 65 previous similar messages [ 9038.828743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9039.091506] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9039.593301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9039.642288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9039.814819] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9039.814844] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9039.967511] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9040.236800] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9040.506386] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9040.775667] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9041.032313] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9043.192840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9043.382512] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9057.751828] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9058.008703] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9064.504823] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9070.910805] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9071.555912] Lustre: DEBUG MARKER: rc=0; [ 9071.555912] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9071.555912] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9071.555912] echo $(hostname -s): $val; [ 9071.555912] rc=$val; [ 9071.555912] fi; [ 9071.555912] exit $rc [ 9072.168625] Lustre: DEBUG MARKER: dmesg [ 9073.483877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 26: MDT startup failure cleans LOV \(should return errs\) ========================== 20:10:07 \(1491250207\) [ 9073.642859] Lustre: DEBUG MARKER: == conf-sanity test 26: MDT startup failure cleans LOV (should return errs) ========================== 20:10:07 (1491250207) [ 9075.451486] Lustre: DEBUG MARKER: lsmod | grep -q lustre || modprobe lustre [ 9076.064491] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000135 [ 9076.323458] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9076.598650] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9076.856288] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9077.107433] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9077.253948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.309502] Lustre: *** cfs_fail_loc=135, val=0*** [ 9077.339394] LustreError: 8417:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000 failed (-2) [ 9077.341728] LustreError: 8417:0:(obd_config.c:1709:class_config_llog_handler()) MGC10.9.5.243@tcp: cfg command failed: rc = -2 [ 9077.345732] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 9077.345732] [ 9077.349930] LustreError: 15c-8: MGC10.9.5.243@tcp: The configuration from log 'lustre-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. [ 9077.356376] LustreError: 8373:0:(obd_mount_server.c:1351:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 9077.358889] LustreError: 8373:0:(obd_mount_server.c:1840:server_fill_super()) Unable to start targets: -2 [ 9077.361235] LustreError: 8373:0:(obd_config.c:621:class_cleanup()) Device 5 not setup [ 9083.389169] LustreError: 8373:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-2) [ 9083.539108] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9083.814325] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9084.104183] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9084.358683] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9084.611378] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9085.556963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9086.482887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9086.743309] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9087.237362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9087.245265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9087.401037] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9087.404108] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9087.547553] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9087.802182] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9088.057213] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9088.321030] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9088.577345] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9088.848396] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9089.106496] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9095.437171] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9100.107342] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9100.713528] Lustre: DEBUG MARKER: rc=0; [ 9100.713528] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9100.713528] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9100.713528] echo $(hostname -s): $val; [ 9100.713528] rc=$val; [ 9100.713528] fi; [ 9100.713528] exit $rc [ 9101.288624] Lustre: DEBUG MARKER: dmesg [ 9102.596164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 20:10:36 \(1491250236\) [ 9102.754334] Lustre: DEBUG MARKER: == conf-sanity test 27a: Reacquire MGS lock if OST started first ===================================== 20:10:36 (1491250236) [ 9163.910425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9164.070682] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9164.989864] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9165.263328] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9165.515719] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9165.778747] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9165.939186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.163976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9166.420581] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9166.913973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9166.918023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9167.078059] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9167.079125] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9167.227813] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9167.507435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9167.776059] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9168.035574] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9168.290433] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9168.817638] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.ost.client_cache_seconds='35' [ 9168.947446] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 9168.949795] Lustre: Skipped 4 previous similar messages [ 9182.767856] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9183.024748] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9189.352459] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9194.482100] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9195.108711] Lustre: DEBUG MARKER: rc=0; [ 9195.108711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9195.108711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9195.108711] echo $(hostname -s): $val; [ 9195.108711] rc=$val; [ 9195.108711] fi; [ 9195.108711] exit $rc [ 9195.706192] Lustre: DEBUG MARKER: dmesg [ 9197.007464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 20:12:11 \(1491250331\) [ 9197.155796] Lustre: DEBUG MARKER: == conf-sanity test 27b: Reacquire MGS lock after failover =========================================== 20:12:11 (1491250331) [ 9197.305721] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9199.254964] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9199.505953] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9199.757784] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9199.915205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9200.128646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9200.383282] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9200.867255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9200.871070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9201.024433] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9201.025935] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9201.168969] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9201.423057] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9201.692702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9201.967736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9202.228695] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9204.364160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9204.521135] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9211.744319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n devices [ 9212.021836] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9212.278599] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 9212.412693] Lustre: Failing over lustre-MDT0000 [ 9213.486972] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.491654] LustreError: Skipped 9 previous similar messages [ 9218.649632] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9228.961436] Lustre: DEBUG MARKER: hostname [ 9229.265395] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9229.514372] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9229.760786] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9229.916663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.991550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.994043] Lustre: Skipped 13 previous similar messages [ 9230.140822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9230.398570] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9230.918907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9230.924511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9231.077986] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9231.079440] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9231.217122] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9231.467995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9231.723471] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9231.994081] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9232.251883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9232.516836] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-MDT0000.mdt.identity_acquire_expire='35' [ 9232.644408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 9232.774986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9233.031985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9234.287031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9235.544303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9236.806078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9237.374251] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 9237.392350] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 9238.080570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9239.340802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.identity_acquire_expire [ 9239.607958] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-MDT0000.mdc.max_rpcs_in_flight='13' [ 9244.413791] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9244.669148] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9253.010356] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9258.894756] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9259.530463] Lustre: DEBUG MARKER: rc=0; [ 9259.530463] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9259.530463] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9259.530463] echo $(hostname -s): $val; [ 9259.530463] rc=$val; [ 9259.530463] fi; [ 9259.530463] exit $rc [ 9260.128436] Lustre: DEBUG MARKER: dmesg [ 9261.417146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 28: permanent parameter setting ================================================== 20:13:15 \(1491250395\) [ 9261.575028] Lustre: DEBUG MARKER: == conf-sanity test 28: permanent parameter setting ================================================== 20:13:15 (1491250395) [ 9261.719483] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9263.670326] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9263.921312] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9264.174572] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9264.320531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.535078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9264.798888] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9265.299287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9265.299347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9265.460826] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9265.461828] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9265.609858] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9265.873827] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9266.132954] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9266.405754] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9266.681633] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9268.910488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9269.085719] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9269.385096] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491250400/real 1491250400] req@ffff880069340300 x1563680146672736/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491250405 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9269.392189] Lustre: 4492:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [ 9271.386773] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9271.523775] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 9271.526306] Lustre: Skipped 1 previous similar message [ 9277.791374] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9285.338629] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9299.641563] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9299.904563] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9306.112584] Lustre: server umount lustre-MDT0000 complete [ 9306.114787] Lustre: Skipped 16 previous similar messages [ 9306.243071] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9312.738322] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9313.378425] Lustre: DEBUG MARKER: rc=0; [ 9313.378425] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9313.378425] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9313.378425] echo $(hostname -s): $val; [ 9313.378425] rc=$val; [ 9313.378425] fi; [ 9313.378425] exit $rc [ 9313.984728] Lustre: DEBUG MARKER: dmesg [ 9315.340401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 28a: set symlink parameters permanently with conf_param ========================== 20:14:09 \(1491250449\) [ 9315.507343] Lustre: DEBUG MARKER: == conf-sanity test 28a: set symlink parameters permanently with conf_param ========================== 20:14:09 (1491250449) [ 9315.928740] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9317.867247] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9318.126202] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9318.377634] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9318.538376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.750541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9319.007277] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9319.584077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9319.585844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9319.748112] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9319.750030] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9319.890651] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9320.156173] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9320.419612] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9320.678633] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9320.944773] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9323.688230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9323.909393] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9326.682693] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.ost.writethrough_cache_enable='0' [ 9332.784119] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osd.writethrough_cache_enable='1' [ 9340.508170] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.ost.client_cache_seconds='70' [ 9340.636608] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 9340.639037] Lustre: Skipped 4 previous similar messages [ 9349.122200] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.ost.client_cache_seconds='35' [ 9355.468183] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osd.auto_scrub='0' [ 9360.287615] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osd.auto_scrub='1' [ 9369.759891] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 9369.762435] LustreError: Skipped 8 previous similar messages [ 9373.236575] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9373.507343] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9380.161226] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9385.712949] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9386.325402] Lustre: DEBUG MARKER: rc=0; [ 9386.325402] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9386.325402] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9386.325402] echo $(hostname -s): $val; [ 9386.325402] rc=$val; [ 9386.325402] fi; [ 9386.325402] exit $rc [ 9386.894908] Lustre: DEBUG MARKER: dmesg [ 9388.209357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 29: permanently remove an OST ==================================================== 20:15:22 \(1491250522\) [ 9388.391354] Lustre: DEBUG MARKER: == conf-sanity test 29: permanently remove an OST ==================================================== 20:15:22 (1491250522) [ 9388.532373] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9390.620117] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9390.892434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9391.147727] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9391.293784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.540417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9391.804834] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9392.316344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9392.316367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9392.480289] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9392.493398] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9392.638777] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9392.894103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9393.152891] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9393.421896] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9393.679784] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9396.139426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9396.383696] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9405.322495] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9406.500837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9406.679403] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9417.658222] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0001.osc.active='0' [ 9417.793518] Lustre: Permanently deactivating lustre-OST0001 [ 9423.062736] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9427.084191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 9427.084191] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 9427.084191] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 9427.372318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 9427.372318] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 9427.372318] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 9427.653651] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 9432.912605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.active [ 9433.368790] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0001.osc.active='1' [ 9433.503544] Lustre: Permanently reactivating lustre-OST0001 [ 9453.802781] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9454.055342] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9460.395338] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9466.213689] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9466.472595] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9466.757487] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [ 9466.927368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9470.790662] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9472.186521] Lustre: DEBUG MARKER: rc=0; [ 9472.186521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9472.186521] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9472.186521] echo $(hostname -s): $val; [ 9472.186521] rc=$val; [ 9472.186521] fi; [ 9472.186521] exit $rc [ 9472.784331] Lustre: DEBUG MARKER: dmesg [ 9474.080059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 20:16:48 \(1491250608\) [ 9474.232167] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and conf_param deletion ===================================== 20:16:48 (1491250608) [ 9474.368286] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9476.282343] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9476.530736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9476.778316] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9476.932742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.969172] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9476.976365] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 9476.978599] Lustre: Skipped 7 previous similar messages [ 9477.163835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9477.414668] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9477.904297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9477.907126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9478.064206] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9478.065929] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9478.206157] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9478.457692] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9478.708581] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9478.964534] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9479.214829] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9480.422257] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9481.409981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9481.580834] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9488.860529] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='1' [ 9498.256183] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9505.636369] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9515.029271] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9520.372976] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='5' [ 9529.760056] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9534.095928] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9541.454991] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9547.799669] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='1' [ 9554.147144] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9563.518264] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9568.856487] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9574.213349] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='5' [ 9581.576124] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9587.936768] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9594.301113] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9602.699883] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='1' [ 9613.112694] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='2' [ 9621.491977] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='3' [ 9629.868347] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='4' [ 9637.243536] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.max_read_ahead_whole_mb='5' [ 9645.629929] Lustre: MGS: Connection restored to f12920f1-3d5b-1e78-d777-e27fb9937371 (at 10.9.5.237@tcp) [ 9645.632075] Lustre: Skipped 51 previous similar messages [ 9645.804708] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param -d lustre.llite.max_read_ahead_whole_mb [ 9648.719991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9648.723745] Lustre: Skipped 9 previous similar messages [ 9653.180738] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9653.430689] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9659.772091] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9665.652684] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9666.283830] Lustre: DEBUG MARKER: rc=0; [ 9666.283830] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9666.283830] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9666.283830] echo $(hostname -s): $val; [ 9666.283830] rc=$val; [ 9666.283830] fi; [ 9666.283830] exit $rc [ 9666.881855] Lustre: DEBUG MARKER: dmesg [ 9668.166359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 30b: Remove failover nids ======================================================== 20:20:02 \(1491250802\) [ 9668.322834] Lustre: DEBUG MARKER: == conf-sanity test 30b: Remove failover nids ======================================================== 20:20:02 (1491250802) [ 9668.466451] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9670.407570] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9670.658314] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9670.919775] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9671.069381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.293413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9671.558061] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9672.050133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9672.055098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9672.214247] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9672.232925] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9672.374042] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9672.630255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9672.882292] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9673.143264] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9673.396417] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9675.587809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9675.807228] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [ 9683.350169] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.failover.node='10.9.5.8@tcp' [ 9685.678980] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param -d lustre-OST0000.failover.node [ 9693.063288] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9693.320814] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 9699.666482] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9705.478845] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9706.097474] Lustre: DEBUG MARKER: rc=0; [ 9706.097474] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9706.097474] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9706.097474] echo $(hostname -s): $val; [ 9706.097474] rc=$val; [ 9706.097474] fi; [ 9706.097474] exit $rc [ 9706.680545] Lustre: DEBUG MARKER: dmesg [ 9707.993860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 31: Connect to non-existent node \(shouldn\'t crash\) =============================== 20:20:42 \(1491250842\) [ 9708.156900] Lustre: DEBUG MARKER: == conf-sanity test 31: Connect to non-existent node (shouldn't crash) =============================== 20:20:42 (1491250842) [ 9745.875024] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9746.124919] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9750.579035] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9751.175517] Lustre: DEBUG MARKER: rc=0; [ 9751.175517] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9751.175517] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9751.175517] echo $(hostname -s): $val; [ 9751.175517] rc=$val; [ 9751.175517] fi; [ 9751.175517] exit $rc [ 9751.758729] Lustre: DEBUG MARKER: dmesg [ 9753.104717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 32a: Upgrade \(not live\) ========================================================== 20:21:27 \(1491250887\) [ 9753.272706] Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) ========================================================== 20:21:27 (1491250887) [ 9753.406166] Lustre: DEBUG MARKER: which tunefs.lustre [ 9753.666117] Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2' [ 9755.604105] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [ 9755.850812] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [ 9756.095150] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_4-ldiskfs.tar.bz2 -S -C /tmp/t32 [ 9756.898951] Lustre: DEBUG MARKER: cat /tmp/t32/commit [ 9757.143679] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [ 9757.388637] Lustre: DEBUG MARKER: cat /tmp/t32/arch [ 9757.641364] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [ 9757.886559] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [ 9758.373596] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [ 9758.616371] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [ 9759.401586] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [ 9759.643826] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [ 9759.887194] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [ 9760.147180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9760.626173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [ 9760.636417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [ 9760.789185] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [ 9760.802182] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [ 9760.932447] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9761.413580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9761.413597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9761.567335] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9761.567401] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9761.904700] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [ 9765.600441] loop: module loaded [ 9765.917028] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.039572] Lustre: 30993:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [ 9766.260696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [ 9766.527402] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [ 9766.808143] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [ 9766.962129] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [ 9766.964757] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9767.516742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [ 9767.768310] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [ 9767.896924] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [ 9767.899301] Lustre: Skipped 26 previous similar messages [ 9768.032083] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [ 9768.285411] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [ 9768.537857] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [ 9768.788645] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [ 9771.450162] LustreError: 30975:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [ 9771.452717] Lustre: 30975:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [ 9771.456991] LustreError: 30975:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [ 9771.462517] Lustre: 31723:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [ 9771.466915] LustreError: 31723:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [ 9773.040917] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [ 9773.296691] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [ 9773.550842] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [ 9793.679037] LustreError: 31946:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [ 9793.684020] LustreError: 31946:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8800572d4f00 x1563680146688688/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 1 dl 1491250906 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [ 9793.691767] LustreError: 31946:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [ 9813.696071] LustreError: 31946:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [ 9813.700928] LustreError: 31946:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8800572d4f00 x1563680146688688/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 1 dl 1491250906 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [ 9813.708775] LustreError: 31946:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [ 9819.019745] Lustre: Failing over t32fs-MDT0000 [ 9819.205019] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [ 9825.499885] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9826.074502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [ 9826.074503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [ 9826.241535] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [ 9826.242091] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [ 9831.028026] LNet: 32598:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9835.030024] LNet: 32598:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9843.032026] LNet: 32598:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9859.034027] LNet: 32598:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9891.036027] LNet: 32598:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9895.038045] LNet: Removed LNI 10.9.5.243@tcp [ 9895.961203] LNet: 32639:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [ 9896.521949] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 9896.568827] alg: No test for adler32 (adler32-zlib) [ 9896.571414] alg: No test for crc32 (crc32-table) [ 9897.565142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [ 9897.569678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [ 9897.742374] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [ 9897.743374] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [ 9897.981928] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9898.541235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [ 9898.546944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [ 9898.742889] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [ 9898.751325] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [ 9898.798370] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [ 9898.922697] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 9898.925034] LNet: Accept all, port 7988 [ 9900.629030] Lustre: 1170:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [ 9900.633744] Lustre: 1170:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [ 9900.636185] Key type lgssc registered [ 9900.821892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9900.969705] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [ 9901.242570] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [ 9901.404440] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.420904] Lustre: MGS: Connection restored to dedc1fc0-4f48-4e20-8f81-df0b9cabf83d (at 0@lo) [ 9901.506755] Lustre: 1441:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [ 9901.514129] LustreError: 1441:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [ 9901.539734] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.708473] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [ 9906.520056] Lustre: 1159:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491251037/real 0] req@ffff880036c30300 x1563690044227744/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.10.4.134@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251042 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9907.843079] LustreError: 1535:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [ 9907.847298] LustreError: 1535:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880068f50000 x1563690044227920/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251042 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [ 9907.854098] LustreError: 1535:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [ 9927.858061] LustreError: 1535:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [ 9927.862639] LustreError: 1535:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880068f50000 x1563690044227920/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251042 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [ 9927.870177] LustreError: 1535:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [ 9947.875071] LustreError: 1535:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [ 9947.880031] LustreError: 1535:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880068f50000 x1563690044227920/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251042 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [ 9947.888085] LustreError: 1535:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [ 9957.898623] Lustre: Failing over t32fs-MDT0000 [ 9963.932042] Lustre: 1535:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251093/real 1491251093] req@ffff880068f50000 x1563690044228240/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251099 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9963.963180] Lustre: server umount t32fs-MDT0000 complete [ 9964.087429] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [ 9964.455489] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [ 9964.703033] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [ 9964.945919] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_7-ldiskfs.tar.bz2 -S -C /tmp/t32 [ 9965.605862] Lustre: DEBUG MARKER: cat /tmp/t32/commit [ 9965.849961] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [ 9966.096171] Lustre: DEBUG MARKER: cat /tmp/t32/arch [ 9966.339463] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [ 9967.050249] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [ 9967.294593] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [ 9967.548654] Lustre: DEBUG MARKER: cat /tmp/t32/blimit [ 9967.793184] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [ 9968.038380] Lustre: DEBUG MARKER: cat /tmp/t32/ilimit [ 9968.837522] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [ 9969.079958] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [ 9969.322291] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [ 9969.578409] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt2 [ 9969.849531] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9970.344384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [ 9970.345471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [ 9970.494644] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [ 9970.502688] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [ 9970.635862] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9971.114485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9971.120141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9971.270743] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9971.272962] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [ 9971.401300] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [ 9971.539935] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.553714] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [ 9971.628695] Lustre: 3762:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [ 9971.631193] Lustre: 3762:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [ 9971.634706] Lustre: 1159:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251107/real 1491251107] req@ffff880036c30c00 x1563690044228416/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.100.4.87@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251112 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [ 9971.666939] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.850671] Lustre: DEBUG MARKER: mount -t lustre -o mgsnode=10.9.5.243@tcp,loop,exclude=t32fs-OST0000 /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 [ 9971.994397] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.292337] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.t32fs*.enable_remote_dir=1 [ 9972.558266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [ 9972.824831] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [ 9973.099027] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [ 9973.242086] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [ 9973.244700] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.393429] Lustre: t32fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9973.395797] Lustre: Skipped 1 previous similar message [ 9973.558349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [ 9973.812512] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [ 9973.939422] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [ 9974.071422] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [ 9974.339784] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [ 9974.471322] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [ 9974.473901] Lustre: Skipped 2 previous similar messages [ 9974.600415] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [ 9974.854867] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [ 9976.376056] Lustre: 1159:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251107/real 1491251107] req@ffff880036c30900 x1563690044228400/t0(0) o38->t32fs-MDT0001-osp-MDT0000@10.100.4.87@tcp:24/4 lens 520/544 e 0 to 1 dl 1491251112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9976.383873] Lustre: 1159:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 9980.107448] LustreError: 3744:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [ 9980.110198] Lustre: 3744:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [ 9980.114937] LustreError: 3744:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [ 9980.121070] Lustre: 4672:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0001 (on exclusion list) [ 9980.123906] Lustre: 4672:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 1 previous similar message [ 9980.126618] LustreError: 4672:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0001 unknown param max_dirty_mb=15 [ 9981.107708] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [ 9981.235948] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [ 9981.364851] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [ 9981.629037] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt1 [ 9981.758969] Lustre: Failing over t32fs-MDT0001 [ 9981.761544] LustreError: 3875:0:(osp_object.c:527:osp_attr_get()) t32fs-MDT0000-osp-MDT0001:osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [ 9981.766449] LustreError: 3875:0:(lod_sub_object.c:959:lod_sub_prep_llog()) t32fs-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [ 9981.771294] LustreError: 3875:0:(lod_dev.c:419:lod_sub_recovery_thread()) t32fs-MDT0000-osp-MDT0001 getting update log failed: rc = -5 [ 9981.864385] Lustre: server umount t32fs-MDT0001 complete [ 9981.990352] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [ 9982.434182] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [ 9988.568030] Lustre: 5043:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251118/real 1491251118] req@ffff880036c32400 x1563690044230432/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251124 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9988.575782] Lustre: 5043:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9988.599965] Lustre: server umount t32fs-OST0000 complete [ 9988.602334] Lustre: Skipped 1 previous similar message [ 9988.736134] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 9989.215838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [ 9989.227695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [ 9989.377039] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [ 9989.381542] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [ 9989.557850] Key type lgssc unregistered [ 9989.560242] Lustre: 5618:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [ 9993.563054] LNet: 5634:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [ 9997.565050] LNet: 5634:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10005.567035] LNet: 5634:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10021.569039] LNet: 5634:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10030.571156] LNet: Removed LNI 10.9.5.243@tcp [10030.870245] LNet: 5675:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10031.364401] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10031.369188] alg: No test for adler32 (adler32-zlib) [10031.371651] alg: No test for crc32 (crc32-table) [10032.284323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10032.284370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10032.437236] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10032.440213] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10032.679249] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10033.158767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10033.161872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10033.310353] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10033.312739] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10033.361211] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10033.381245] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10033.383461] LNet: Accept all, port 7988 [10034.894034] Lustre: 6591:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10034.898528] Lustre: 6591:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10034.901143] Key type lgssc registered [10034.968118] Lustre: Echo OBD driver; http://www.lustre.org/ [10035.104063] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10035.360658] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10035.510742] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.525561] Lustre: MGS: Connection restored to e801e238-777f-7680-a5db-fd3403965150 (at 0@lo) [10035.630169] Lustre: 6862:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10035.633524] Lustre: 6580:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251171/real 1491251171] req@ffff8800373f4600 x1563690185785520/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.100.4.87@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251176 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10035.633535] LustreError: 6862:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10035.650133] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10035.815201] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10035.943705] Lustre: Failing over t32fs-MDT0000 [10035.945876] LustreError: 6879:0:(osp_object.c:527:osp_attr_get()) t32fs-MDT0001-osp-MDT0000:osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [10035.950328] LustreError: 6879:0:(lod_sub_object.c:959:lod_sub_prep_llog()) t32fs-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [10035.955074] LustreError: 6879:0:(lod_dev.c:419:lod_sub_recovery_thread()) t32fs-MDT0001-osp-MDT0000 getting update log failed: rc = -5 [10041.995042] Lustre: 6958:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251171/real 1491251171] req@ffff880063bbc600 x1563690185785840/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251177 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10042.002591] Lustre: 6958:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10042.072829] Lustre: server umount t32fs-MDT0000 complete [10042.197432] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10042.565383] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10042.810622] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10043.053814] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10043.619312] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10043.864124] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10044.288419] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10044.539220] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10044.791650] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10045.034341] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10045.281815] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10046.072684] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10046.320369] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10046.567899] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10046.840281] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10047.326732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10047.333885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10047.485136] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10047.486945] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10047.620754] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10048.104139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10048.104449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10048.256021] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10048.270828] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10048.400624] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10048.540899] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10048.557037] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10048.697642] Lustre: 8947:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10048.700196] Lustre: 8947:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10048.708468] Lustre: 6580:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251184/real 1491251184] req@ffff880063bbc600 x1563690185786176/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251189 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10048.736109] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.919762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10049.172344] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10049.454885] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10049.597696] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10049.600316] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [10049.636053] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10049.883915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10050.146083] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10050.274294] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10050.402827] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10050.654753] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10050.785196] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10050.787610] Lustre: Skipped 1 previous similar message [10050.913158] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10051.163369] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10053.643047] Lustre: 6580:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251184/real 1491251184] req@ffff880063bbc300 x1563690185786016/t0(0) o8->t32fs-OST0000-osc-MDT0000@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251189 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10053.650531] Lustre: 6580:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10056.519117] LustreError: 8928:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [10056.521930] Lustre: 8928:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [10056.526840] LustreError: 8928:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [10056.532880] Lustre: 9675:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [10056.535779] LustreError: 9675:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10057.417175] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10057.543783] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10057.672752] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10057.927563] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10058.056390] Lustre: Failing over t32fs-MDT0000 [10058.148799] Lustre: server umount t32fs-MDT0000 complete [10058.275181] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10064.407041] Lustre: 9972:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251194/real 1491251194] req@ffff880063bbea00 x1563690185787264/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251200 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10064.547208] Lustre: server umount t32fs-OST0000 complete [10064.680600] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10065.154641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10065.167615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10065.316831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10065.319900] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10065.492130] Key type lgssc unregistered [10065.494432] Lustre: 10547:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10069.401052] LNet: Removed LNI 10.9.5.243@tcp [10069.743370] LNet: 10601:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10070.237758] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10070.242151] alg: No test for adler32 (adler32-zlib) [10070.244558] alg: No test for crc32 (crc32-table) [10071.157305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10071.158133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10071.313405] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10071.313406] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10071.555650] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10072.028841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10072.028968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10072.180547] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10072.180879] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10072.230773] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10072.251789] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10072.254118] LNet: Accept all, port 7988 [10073.765029] Lustre: 11517:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10073.769779] Lustre: 11517:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10073.772324] Key type lgssc registered [10073.838704] Lustre: Echo OBD driver; http://www.lustre.org/ [10073.973139] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10074.229779] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10074.378568] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10074.392414] Lustre: MGS: Connection restored to 5b361eed-6cfb-6894-155a-72a619b09239 (at 0@lo) [10074.452516] Lustre: 11789:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10074.459620] LustreError: 11789:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10074.466118] Lustre: 11506:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251210/real 1491251210] req@ffff88005eb74600 x1563690225631568/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251215 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10074.478829] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10074.640277] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10074.767033] Lustre: Failing over t32fs-MDT0000 [10080.800053] Lustre: 11883:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251210/real 1491251210] req@ffff88005eb74300 x1563690225631696/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251216 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10080.883016] Lustre: server umount t32fs-MDT0000 complete [10081.007721] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10081.381402] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10081.627191] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10081.872121] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk1_8_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10082.271526] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10082.574744] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10082.821987] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10083.063552] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10083.305179] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10083.548991] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10083.790153] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10084.586361] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10084.829313] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10085.071943] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10085.336787] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10085.846820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10085.847662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10086.004056] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10086.004057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10086.137681] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10086.620414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10086.624172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10086.775764] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10086.775765] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10086.905640] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10087.047337] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10087.051655] Lustre: 13828:0:(osd_handler.c:6906:osd_mount()) t32fs-MDT0000-osd: device /dev/loop0 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [10087.071868] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10087.087928] Lustre: 13871:0:(mgs_llog.c:261:mgs_fsdb_handler()) MDT using 1.8 OSC name scheme [10087.109411] Lustre: 13872:0:(obd_config.c:1539:class_config_llog_handler()) For 1.8 interoperability, rename obd type from mds to mdt [10087.149412] Lustre: 13872:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10087.151920] Lustre: 13872:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10087.160560] Lustre: 11506:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251222/real 1491251222] req@ffff88005eb74f00 x1563690225632016/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.129@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251227 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10087.180668] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.198835] Lustre: ctl-t32fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10087.368101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10087.632923] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10087.909153] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10088.062025] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10088.064871] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [10088.101706] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10088.315309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10088.571082] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10088.699602] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10088.831393] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10089.085772] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10089.215556] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10089.218257] Lustre: Skipped 1 previous similar message [10089.344091] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10089.594538] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10092.109042] Lustre: 11506:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251222/real 1491251222] req@ffff88005eb74300 x1563690225631856/t0(0) o8->t32fs-OST0000-osc@192.168.203.129@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10092.117058] Lustre: 11506:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10093.937024] Lustre: 14602:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc (on exclusion list) [10093.939904] LustreError: 14602:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc unknown param max_dirty_mb=15 [10095.846871] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10095.974815] Lustre: Modifying parameter t32fs-mdtlov.lov.stripesize in log t32fs-MDT0000 [10096.103708] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10096.357826] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10096.487074] Lustre: Failing over t32fs-MDT0000 [10096.639880] Lustre: server umount t32fs-MDT0000 complete [10096.765375] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10102.897049] Lustre: 14899:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251232/real 1491251232] req@ffff88005eb75800 x1563690225633056/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251238 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10102.917075] Lustre: server umount t32fs-OST0000 complete [10103.050974] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10103.588254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10103.590444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10103.746788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10103.747281] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10103.928151] Key type lgssc unregistered [10103.930421] Lustre: 15475:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10107.890077] LNet: Removed LNI 10.9.5.243@tcp [10108.206368] LNet: 15528:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10108.703481] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10108.708186] alg: No test for adler32 (adler32-zlib) [10108.710531] alg: No test for crc32 (crc32-table) [10109.628391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10109.634279] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10109.785748] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10109.797685] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10110.040961] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10110.516258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10110.519333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10110.666214] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10110.666249] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10110.717955] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10110.738238] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10110.740614] LNet: Accept all, port 7988 [10112.252058] Lustre: 16444:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10112.256828] Lustre: 16444:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10112.259325] Key type lgssc registered [10112.326847] Lustre: Echo OBD driver; http://www.lustre.org/ [10112.463628] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10112.733453] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10112.885754] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10112.890344] Lustre: 16666:0:(osd_handler.c:6906:osd_mount()) t32fs-MDT0000-osd: device /dev/loop0 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [10112.909054] Lustre: MGS: Connection restored to ff11e97c-b1e8-9caa-0e0e-832490a075be (at 0@lo) [10112.929652] Lustre: 16715:0:(mgs_llog.c:261:mgs_fsdb_handler()) MDT using 1.8 OSC name scheme [10112.941965] Lustre: 16716:0:(obd_config.c:1539:class_config_llog_handler()) For 1.8 interoperability, rename obd type from mds to mdt [10112.963682] Lustre: 16716:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10112.970745] LustreError: 16716:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc unknown param max_dirty_mb=15 [10112.977613] Lustre: 16433:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251248/real 1491251248] req@ffff88005dd38000 x1563690266526016/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.129@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251253 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10112.989292] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.144052] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10113.269785] Lustre: Failing over t32fs-MDT0000 [10119.303049] Lustre: 16810:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251248/real 1491251248] req@ffff88005dd38600 x1563690266526144/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251254 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10119.413495] Lustre: server umount t32fs-MDT0000 complete [10119.539184] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10119.914241] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10120.156970] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10120.399805] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_3_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10120.883389] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10121.126582] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10121.371560] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10121.614912] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10121.866072] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10122.340237] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10122.586668] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10123.367425] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10123.613861] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10123.859969] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10124.130635] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10124.628447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10124.628903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10124.782432] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10124.785788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10124.933464] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10125.433468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10125.434272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10125.586622] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10125.589495] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10125.718634] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10125.863448] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10125.881386] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10125.952440] Lustre: 18797:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10125.954981] Lustre: 18797:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10125.982210] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10126.160233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10126.412796] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10126.674483] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10126.817428] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10126.819959] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.855610] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10127.128458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10127.382305] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10127.509729] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10127.640321] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10127.890389] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10128.017956] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10128.020303] Lustre: Skipped 1 previous similar message [10128.145490] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10128.396081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10130.863045] Lustre: 16433:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491251261/real 0] req@ffff88005dd38900 x1563690266526480/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251266 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10133.634549] LustreError: 18778:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [10133.637199] Lustre: 18778:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [10133.641830] LustreError: 18778:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [10133.648390] Lustre: 19524:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [10133.655585] LustreError: 19524:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10134.649407] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10134.777427] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10134.907054] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10135.161695] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10155.288048] LustreError: 19747:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10155.292822] LustreError: 19747:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff88005dd38900 x1563690266526480/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251266 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [10155.300423] LustreError: 19747:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10175.305040] LustreError: 19747:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10175.310082] LustreError: 19747:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff88005dd38900 x1563690266526480/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251266 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [10175.318113] LustreError: 19747:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10178.738781] Lustre: Failing over t32fs-MDT0000 [10178.803447] Lustre: server umount t32fs-MDT0000 complete [10178.931160] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10185.064042] Lustre: 19829:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251314/real 1491251314] req@ffff880038fde100 x1563690266527920/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251320 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10185.072414] Lustre: 19829:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [10185.101637] Lustre: server umount t32fs-OST0000 complete [10185.238231] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10185.714760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10185.719130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10185.873471] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10185.875985] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10186.052617] Key type lgssc unregistered [10186.055206] Lustre: 20405:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10190.738072] LNet: 20421:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10194.741028] LNet: 20421:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10202.743022] LNet: 20421:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10218.745028] LNet: 20421:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10250.747025] LNet: 20421:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10254.749047] LNet: Removed LNI 10.9.5.243@tcp [10255.037069] LNet: 20462:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10255.551529] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10255.556037] alg: No test for adler32 (adler32-zlib) [10255.558504] alg: No test for crc32 (crc32-table) [10256.467488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10256.483909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10256.639441] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10256.642798] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10256.884630] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10257.353309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10257.356359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10257.507264] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10257.507266] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10257.557994] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10257.578754] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10257.580972] LNet: Accept all, port 7988 [10259.092047] Lustre: 21379:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10259.096541] Lustre: 21379:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10259.098904] Key type lgssc registered [10259.165379] Lustre: Echo OBD driver; http://www.lustre.org/ [10259.299974] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10259.557270] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10259.705967] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.721429] Lustre: MGS: Connection restored to 94816021-2714-a5f5-80a6-16c1c193aca0 (at 0@lo) [10259.779705] Lustre: 21651:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10259.787213] LustreError: 21651:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10259.802014] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10259.969561] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10264.793055] Lustre: 21368:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491251395/real 0] req@ffff880063c1c000 x1563690420666544/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.10.4.4@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251400 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10266.106039] LustreError: 21745:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10266.110214] LustreError: 21745:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880063c1c300 x1563690420666720/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251400 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [10266.117030] LustreError: 21745:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10286.121044] LustreError: 21745:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10286.125664] LustreError: 21745:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880063c1c300 x1563690420666720/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251400 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [10286.133177] LustreError: 21745:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10306.138046] LustreError: 21745:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10306.143016] LustreError: 21745:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880063c1c300 x1563690420666720/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251400 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [10306.151060] LustreError: 21745:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10312.578619] Lustre: Failing over t32fs-MDT0000 [10318.612042] Lustre: 21745:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251448/real 1491251448] req@ffff880038fc8600 x1563690420667024/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251454 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10318.656623] Lustre: server umount t32fs-MDT0000 complete [10318.782395] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10319.110065] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10319.730508] Lustre: DEBUG MARKER: rc=0; [10319.730508] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10319.730508] if [[ $? -eq 0 && $val -ne 0 ]]; then [10319.730508] echo $(hostname -s): $val; [10319.730508] rc=$val; [10319.730508] fi; [10319.730508] exit $rc [10320.306144] Lustre: DEBUG MARKER: dmesg [10320.943515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 32b: Upgrade with writeconf ====================================================== 20:30:56 \(1491251456\) [10321.111875] Lustre: DEBUG MARKER: == conf-sanity test 32b: Upgrade with writeconf ====================================================== 20:30:56 (1491251456) [10321.246658] Lustre: DEBUG MARKER: which tunefs.lustre [10321.506864] Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2' [10321.856348] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10322.100749] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10322.346632] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_4-ldiskfs.tar.bz2 -S -C /tmp/t32 [10322.853683] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10323.100599] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10323.344864] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10323.901152] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10324.145278] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10324.390257] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10324.642676] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10325.437904] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10325.680522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10325.922429] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10326.177041] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt [10326.451226] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10326.923911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10326.926612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10327.079673] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10327.080235] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10327.213602] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10327.697870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10327.703068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10327.853398] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10327.853399] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10327.984738] Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt [10328.127073] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.141401] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10328.157823] Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10328.167263] Lustre: Setting parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10328.222989] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10328.383365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10328.636098] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10328.900952] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost [10329.186679] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost [10329.342086] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10329.344645] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10329.371608] Lustre: MGS: Regenerating t32fs-OST0000 log by user request: rc = 0 [10329.399692] Lustre: t32fs-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10329.412173] Lustre: t32fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.598119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10329.856136] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10329.984592] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10329.986902] Lustre: Skipped 4 previous similar messages [10330.127770] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10330.383098] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10330.638422] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10330.898368] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10334.399485] Lustre: t32fs-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10335.801120] LustreError: 25101:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10335.806081] Lustre: t32fs-OST0000: deleting orphan objects from 0x0:394 to 0x0:449 [10337.156625] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10337.286168] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10337.288700] Lustre: Skipped 2 previous similar messages [10337.417353] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10337.727569] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10338.023751] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt=ug [10338.276282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10338.528843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10339.783750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10341.042996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10341.305614] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost=ug [10341.573226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10341.827780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10343.080800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10344.343712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10345.618496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10346.887906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10348.154561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10349.421854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10350.684254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10351.954481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10358.529793] Lustre: DEBUG MARKER: test -f /tmp/t32/list [10358.789922] Lustre: DEBUG MARKER: cat /tmp/t32/list [10364.571808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*.force_sync=1 [10364.904185] Lustre: DEBUG MARKER: test -f /tmp/t32/sha1sums [10365.171466] Lustre: DEBUG MARKER: cat /tmp/t32/sha1sums [10366.811317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10366.811317] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10366.811317] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10367.078437] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=14 [10367.207189] Lustre: Modifying parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10367.209126] Lustre: Skipped 2 previous similar messages [10373.698207] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10373.827570] Lustre: Failing over t32fs-MDT0000 [10373.919276] Lustre: server umount t32fs-MDT0000 complete [10374.042990] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10380.167049] Lustre: 27181:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251509/real 1491251509] req@ffff880003641b00 x1563690420670400/t0(0) o39->t32fs-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1491251515 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10380.173385] Lustre: Failing over t32fs-OST0000 [10386.181050] Lustre: 27181:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251515/real 1491251515] req@ffff880003641b00 x1563690420670496/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251521 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10386.251568] Lustre: server umount t32fs-OST0000 complete [10386.385301] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10386.766456] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10387.012240] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10387.255369] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_7-ldiskfs.tar.bz2 -S -C /tmp/t32 [10387.717477] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10387.962072] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10388.210710] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10389.025049] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10389.276567] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10389.530757] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10389.778125] Lustre: DEBUG MARKER: cat /tmp/t32/blimit [10390.036097] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10390.288292] Lustre: DEBUG MARKER: cat /tmp/t32/ilimit [10391.074447] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10391.323485] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10391.572985] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10391.839715] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt2 [10392.100086] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt [10392.377099] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt2 [10392.672209] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10393.144542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10393.152374] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10393.301386] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10393.302788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10393.438212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10393.914930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10393.917555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10394.069784] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10394.070283] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10394.200570] Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt [10394.339734] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.356320] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10394.358504] Lustre: Skipped 4 previous similar messages [10394.370631] Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10394.377746] Lustre: Setting parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10394.468175] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10394.642288] Lustre: DEBUG MARKER: mount -t lustre -o mgsnode=10.9.5.243@tcp,loop,writeconf /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 [10394.784985] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.798415] Lustre: MGS: Regenerating t32fs-MDT0001 log by user request: rc = 0 [10395.011644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.t32fs*.enable_remote_dir=1 [10395.266714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10395.519702] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10395.782840] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost [10396.064163] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost [10396.210081] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [10396.212807] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10396.237516] Lustre: MGS: Regenerating t32fs-OST0000 log by user request: rc = 0 [10396.420177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10396.676116] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10396.928758] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10397.185305] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10397.440991] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10397.693320] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10402.581806] LustreError: 30594:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0001 unknown param max_dirty_mb=15 [10402.586914] Lustre: t32fs-OST0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10402.589340] Lustre: Skipped 5 previous similar messages [10402.647177] Lustre: t32fs-OST0000: deleting orphan objects from 0x200000400:634 to 0x200000400:705 [10402.657917] Lustre: t32fs-OST0000: deleting orphan objects from 0x0:642 to 0x0:705 [10403.952163] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10404.080122] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10404.082671] Lustre: Skipped 13 previous similar messages [10404.215275] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10404.512912] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10404.812661] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt=ug [10405.070573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10405.324510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10406.588973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10407.865239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10409.124066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10410.376211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10411.633167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10411.889934] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost=ug [10412.157591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10412.413154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10413.672154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10414.939353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10416.196205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10417.458027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10418.716927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10425.468301] Lustre: DEBUG MARKER: test -f /tmp/t32/list [10425.737296] Lustre: DEBUG MARKER: test -f /tmp/t32/list2 [10425.998026] Lustre: DEBUG MARKER: cat /tmp/t32/list2 [10431.755391] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*.force_sync=1 [10432.050193] Lustre: DEBUG MARKER: test -f /tmp/t32/sha1sums [10432.316439] Lustre: DEBUG MARKER: cat /tmp/t32/sha1sums [10434.046443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10434.046443] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10434.046443] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10434.330258] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=14 [10434.468378] Lustre: Modifying parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10434.470298] Lustre: Skipped 2 previous similar messages [10443.014175] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10443.147379] Lustre: Failing over t32fs-MDT0000 [10443.230438] Lustre: server umount t32fs-MDT0000 complete [10443.358697] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt1 [10443.487411] LustreError: 11-0: t32fs-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 [10443.695894] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10449.823073] Lustre: 369:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251579/real 1491251579] req@ffff880079309e00 x1563690420676288/t0(0) o39->t32fs-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1491251585 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10454.653078] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10455.829769] Lustre: Failing over t32fs-OST0000 [10455.831355] Lustre: Skipped 1 previous similar message [10455.861728] Lustre: server umount t32fs-OST0000 complete [10455.863526] Lustre: Skipped 1 previous similar message [10455.997647] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10456.376871] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10456.624574] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10456.864778] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10457.329030] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10457.580495] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10457.834990] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10458.086806] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10458.334360] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10458.653323] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10458.898914] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10459.690744] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10459.938864] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10460.199284] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10460.464428] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt [10460.766694] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10461.244311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10461.244316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10461.401423] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10461.404550] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10461.536057] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10462.007806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10462.017360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10462.164240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10462.166150] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10462.300634] Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt [10462.442626] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10462.455418] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10462.457594] Lustre: Skipped 6 previous similar messages [10462.473371] Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10462.565203] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10462.567461] Lustre: Skipped 2 previous similar messages [10462.745329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10462.992897] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10463.253562] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost [10463.592655] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost [10463.738635] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10463.741177] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [10463.769166] Lustre: MGS: Regenerating t32fs-OST0000 log by user request: rc = 0 [10463.799870] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10464.021382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10464.272151] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10464.525218] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10464.776357] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10465.030218] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10465.284692] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10468.552811] LustreError: 3366:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10468.557204] LustreError: 3366:0:(obd_config.c:1387:class_process_proc_param()) Skipped 1 previous similar message [10468.558498] LustreError: 3368:0:(osp_sync.c:731:osp_sync_new_setattr_job()) t32fs-OST0000-osc-MDT0000: invalid setattr record, lsr_valid:4294967352 [10468.560568] Lustre: t32fs-OST0000: deleting orphan objects from 0x0:542 to 0x0:609 [10469.539581] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10469.668573] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10469.671209] Lustre: Skipped 7 previous similar messages [10469.802109] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10470.105515] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10470.404214] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt=ug [10470.658892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10470.922364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10472.185537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10473.448120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10474.712162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10475.977024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10476.237778] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost=ug [10476.500674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10476.754873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10478.013269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10479.288693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10480.542488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10481.802463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10483.058467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10488.634946] Lustre: DEBUG MARKER: test -f /tmp/t32/list [10488.885524] Lustre: DEBUG MARKER: cat /tmp/t32/list [10494.817116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*.force_sync=1 [10495.142695] Lustre: DEBUG MARKER: test -f /tmp/t32/sha1sums [10495.399597] Lustre: DEBUG MARKER: cat /tmp/t32/sha1sums [10497.042628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10497.042628] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10497.042628] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10497.315292] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=14 [10508.012041] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10508.145283] Lustre: Failing over t32fs-MDT0000 [10508.287509] Lustre: server umount t32fs-MDT0000 complete [10508.412076] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10514.542081] Lustre: 5374:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251644/real 1491251644] req@ffff88005b8e7000 x1563690420680448/t0(0) o39->t32fs-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1491251650 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10514.548992] Lustre: 5374:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [10520.742754] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10521.134374] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10521.395155] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10521.642258] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk1_8_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10522.022984] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10522.385184] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10522.646255] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10522.895023] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10523.145375] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10523.400151] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10523.655075] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10524.461035] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10524.711251] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10524.966249] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10525.231720] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt [10525.563098] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10526.063766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10526.066754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10526.219209] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10526.225685] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10526.361530] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10526.851353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10526.856061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10527.009045] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10527.009940] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10527.142672] Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt [10527.285369] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10527.289313] Lustre: 7412:0:(osd_handler.c:6906:osd_mount()) t32fs-MDT0000-osd: device /dev/loop0 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [10527.308601] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10527.312526] Lustre: Skipped 6 previous similar messages [10527.327754] Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10527.417193] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10527.419605] Lustre: Skipped 1 previous similar message [10527.437122] Lustre: ctl-t32fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10527.603863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10527.882261] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10528.144392] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost [10528.598228] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost [10528.746947] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10528.749635] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [10528.775478] Lustre: MGS: Regenerating t32fs-OST0000 log by user request: rc = 0 [10528.803818] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10529.021787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10529.284031] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10529.564383] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10529.840291] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10530.097391] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10530.361238] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10535.905753] LustreError: 8284:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10535.911486] Lustre: t32fs-OST0000: deleting orphan objects from 0x0:589 to 0x0:609 [10536.626442] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10536.778432] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10536.781025] Lustre: Skipped 10 previous similar messages [10536.933532] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10537.261958] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10537.598409] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt=ug [10537.862501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10538.116293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10539.377966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10540.638830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10541.898708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10542.158337] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost=ug [10542.420580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10542.680327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10543.943054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10545.214828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10546.480927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10547.747418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10553.440611] Lustre: DEBUG MARKER: test -f /tmp/t32/list [10553.709741] Lustre: DEBUG MARKER: cat /tmp/t32/list [10559.659928] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*.force_sync=1 [10559.978042] Lustre: DEBUG MARKER: test -f /tmp/t32/sha1sums [10560.235298] Lustre: DEBUG MARKER: cat /tmp/t32/sha1sums [10561.781731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10561.781731] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10561.781731] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10562.050075] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=14 [10570.690822] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10570.824822] Lustre: Failing over t32fs-MDT0000 [10570.826466] Lustre: Skipped 1 previous similar message [10570.910409] LustreError: 11-0: t32fs-MDT0000-lwp-OST0000: operation obd_ping to node 0@lo failed: rc = -107 [10570.913616] LustreError: Skipped 1 previous similar message [10570.915683] Lustre: t32fs-MDT0000-lwp-OST0000: Connection to t32fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10570.934165] Lustre: server umount t32fs-MDT0000 complete [10570.935912] Lustre: Skipped 1 previous similar message [10571.060732] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10577.201058] Lustre: 10143:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251706/real 1491251706] req@ffff88007b58b900 x1563690420683952/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251712 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10577.206829] Lustre: 10143:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [10577.406030] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10577.803130] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10578.071771] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10578.326407] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_3_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10578.834507] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10579.080254] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10579.327574] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10579.571417] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10579.841369] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10580.086273] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10580.332424] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10581.131657] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10581.378543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10581.624153] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10581.881171] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt [10582.244487] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10582.724158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10582.724180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10582.878802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10582.882181] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10583.017790] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10583.499574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10583.499880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10583.650830] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10583.651286] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10583.785678] Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt [10583.927898] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10583.958251] Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10584.048033] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10584.050358] Lustre: Skipped 1 previous similar message [10584.226099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10584.476718] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10584.739285] Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost [10585.073217] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost [10585.222114] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10585.224762] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.251615] Lustre: MGS: Regenerating t32fs-OST0000 log by user request: rc = 0 [10585.279309] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10585.503204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10585.755575] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10586.008220] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10586.263657] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10586.519788] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10586.780842] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10592.768995] LustreError: 13052:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10592.773728] Lustre: t32fs-OST0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10592.776050] Lustre: Skipped 9 previous similar messages [10592.779350] Lustre: t32fs-OST0000: deleting orphan objects from 0x0:394 to 0x0:449 [10593.033781] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10593.287235] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10593.588164] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10593.885821] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt=ug [10594.143286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10594.399280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10595.656988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10596.909997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10598.165743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10599.419455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled [10599.676316] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost=ug [10599.932203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10600.185945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10601.438768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10602.694744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10603.954199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10605.214704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10606.473493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10607.733326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10608.994401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled [10614.769367] Lustre: DEBUG MARKER: test -f /tmp/t32/list [10615.022953] Lustre: DEBUG MARKER: cat /tmp/t32/list [10620.757378] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*.force_sync=1 [10621.079562] Lustre: DEBUG MARKER: test -f /tmp/t32/sha1sums [10621.329803] Lustre: DEBUG MARKER: cat /tmp/t32/sha1sums [10622.846776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10622.846776] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10622.846776] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10623.119821] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=14 [10633.789269] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10633.927559] Lustre: Failing over t32fs-MDT0000 [10633.929290] Lustre: Skipped 1 previous similar message [10634.060483] Lustre: server umount t32fs-MDT0000 complete [10634.062065] Lustre: Skipped 1 previous similar message [10634.194551] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10640.320104] Lustre: 15208:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251769/real 1491251769] req@ffff880039dc5200 x1563690420687808/t0(0) o39->t32fs-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1491251775 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10646.488490] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10646.828779] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10647.474402] Lustre: DEBUG MARKER: rc=0; [10647.474402] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10647.474402] if [[ $? -eq 0 && $val -ne 0 ]]; then [10647.474402] echo $(hostname -s): $val; [10647.474402] rc=$val; [10647.474402] fi; [10647.474402] exit $rc [10648.078519] Lustre: DEBUG MARKER: dmesg [10648.648493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 32c: dne upgrade test ============================================================ 20:36:23 \(1491251783\) [10648.823165] Lustre: DEBUG MARKER: == conf-sanity test 32c: dne upgrade test ============================================================ 20:36:23 (1491251783) [10649.007660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_32c needs \>= 2 MDTs [10649.159968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_32c needs >= 2 MDTs [10649.344435] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10649.954903] Lustre: DEBUG MARKER: rc=0; [10649.954903] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10649.954903] if [[ $? -eq 0 && $val -ne 0 ]]; then [10649.954903] echo $(hostname -s): $val; [10649.954903] rc=$val; [10649.954903] fi; [10649.954903] exit $rc [10650.529949] Lustre: DEBUG MARKER: dmesg [10651.080215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 32d: convert ff test ============================================================= 20:36:26 \(1491251786\) [10651.226769] Lustre: DEBUG MARKER: == conf-sanity test 32d: convert ff test ============================================================= 20:36:26 (1491251786) [10651.358229] Lustre: DEBUG MARKER: which tunefs.lustre [10651.602614] Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2' [10651.953799] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10652.204673] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10652.449376] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_4-ldiskfs.tar.bz2 -S -C /tmp/t32 [10652.895986] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10653.145988] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10654.595218] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10654.841494] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10655.086051] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10655.326159] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10655.567474] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10656.348396] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10656.589005] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10656.831149] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10657.094316] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10657.572679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10657.572683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10657.727095] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10657.728803] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10657.860900] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10658.428687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10658.435056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10658.649884] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10658.652610] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10658.779609] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10658.918268] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.082447] Lustre: 18005:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10659.084793] Lustre: 18005:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10659.113414] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10659.115601] Lustre: Skipped 1 previous similar message [10659.302199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10659.549911] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10659.809311] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10659.955118] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10659.957848] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10660.209746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10660.465876] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10660.719376] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10660.970944] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10661.225674] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10661.480339] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10665.109054] LustreError: 17987:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [10665.111515] Lustre: 17987:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [10665.115658] LustreError: 17987:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [10665.121115] Lustre: 18730:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [10665.125390] LustreError: 18730:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10665.734836] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10665.865146] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10665.867772] Lustre: Skipped 18 previous similar messages [10665.997591] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10666.251195] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10686.379084] LustreError: 18954:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10686.383856] LustreError: 18954:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880039dc5e00 x1563690420688240/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251799 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [10686.391375] LustreError: 18954:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10706.396074] LustreError: 18954:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10706.400950] LustreError: 18954:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880039dc5e00 x1563690420688240/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 1 dl 1491251799 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [10706.408727] LustreError: 18954:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10720.578835] Lustre: Failing over t32fs-MDT0000 [10720.581280] Lustre: Skipped 1 previous similar message [10720.657068] Lustre: server umount t32fs-MDT0000 complete [10720.659599] Lustre: Skipped 1 previous similar message [10720.793768] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10726.934034] Lustre: 19035:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251856/real 1491251856] req@ffff880039dc5b00 x1563690420689680/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251862 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10726.942029] Lustre: 19035:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [10727.088273] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10727.578543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10727.581474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10727.736288] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10727.738721] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10727.913842] Key type lgssc unregistered [10727.916325] Lustre: 19610:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10731.588025] LNet: 19626:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10735.590025] LNet: 19626:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10743.592028] LNet: 19626:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10759.594027] LNet: 19626:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10787.596121] LNet: Removed LNI 10.9.5.243@tcp [10787.887575] LNet: 19667:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10788.398493] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10788.403501] alg: No test for adler32 (adler32-zlib) [10788.405958] alg: No test for crc32 (crc32-table) [10789.328366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10789.333353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10789.483418] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10789.484206] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10789.731595] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10790.207754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10790.218146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10790.367025] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10790.372203] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10790.419430] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10790.439261] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10790.441465] LNet: Accept all, port 7988 [10791.952040] Lustre: 20585:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10791.956523] Lustre: 20585:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10791.958968] Key type lgssc registered [10792.030790] Lustre: Echo OBD driver; http://www.lustre.org/ [10792.166327] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10792.422876] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10792.570909] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.585919] Lustre: MGS: Connection restored to 60e35035-7a18-5fea-3881-ea1e0113afe4 (at 0@lo) [10792.642639] Lustre: 20856:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10792.649604] LustreError: 20856:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10792.677256] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10792.839513] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10797.656071] Lustre: 20574:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491251928/real 0] req@ffff8800797a4000 x1563690979557536/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.10.4.134@tcp:28/4 lens 520/544 e 0 to 1 dl 1491251933 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10812.967034] LustreError: 20950:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10812.971309] LustreError: 20950:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8800797a4300 x1563690979557712/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251933 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [10812.978213] LustreError: 20950:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10832.982024] LustreError: 20950:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [10832.986632] LustreError: 20950:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8800797a4300 x1563690979557712/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.134@tcp:12/10 lens 520/544 e 0 to 0 dl 1491251933 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [10832.994169] LustreError: 20950:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [10849.439646] Lustre: Failing over t32fs-MDT0000 [10855.473035] Lustre: 20950:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251985/real 1491251985] req@ffff8800797a4600 x1563690979558032/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491251991 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10855.512780] Lustre: server umount t32fs-MDT0000 complete [10855.637137] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10856.009335] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10856.261466] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10856.506193] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_7-ldiskfs.tar.bz2 -S -C /tmp/t32 [10856.949338] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10857.836074] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10858.083865] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10858.346108] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10858.603158] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10858.847994] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10859.097272] Lustre: DEBUG MARKER: cat /tmp/t32/blimit [10859.345200] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10859.588809] Lustre: DEBUG MARKER: cat /tmp/t32/ilimit [10860.392084] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10860.650466] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10860.903783] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10861.160038] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt2 [10861.421386] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10861.899821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10861.903631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10862.054555] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10862.058051] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10862.191569] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10862.669484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10862.675269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10862.827839] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10862.828942] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10862.960507] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10863.100532] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.114794] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10863.183708] Lustre: 23171:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10863.186163] Lustre: 23171:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10863.190123] Lustre: 20574:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491251998/real 1491251998] req@ffff8800797a5200 x1563690979558208/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.100.4.87@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252003 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10863.207770] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10863.377315] Lustre: DEBUG MARKER: mount -t lustre -o mgsnode=10.9.5.243@tcp,loop,exclude=t32fs-OST0000 /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 [10863.520646] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.833206] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.t32fs*.enable_remote_dir=1 [10864.086363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10864.339865] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10864.603360] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10864.746072] LDISKFS-fs (loop2): file extents enabled, maximum tree depth=5 [10864.748737] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10864.808703] Lustre: t32fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.811163] Lustre: Skipped 1 previous similar message [10864.986858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10865.238739] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10865.366437] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10865.497949] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10865.762317] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10865.893448] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10865.896044] Lustre: Skipped 2 previous similar messages [10866.022979] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10866.280066] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10867.791054] Lustre: 20574:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491251998/real 1491251998] req@ffff8800797a4600 x1563690979558192/t0(0) o38->t32fs-MDT0001-osp-MDT0000@10.100.4.87@tcp:24/4 lens 520/544 e 0 to 1 dl 1491252003 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10867.799069] Lustre: 20574:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [10869.218682] LustreError: 23153:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [10869.221420] Lustre: 23153:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [10869.226162] LustreError: 23153:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [10869.232204] Lustre: 24081:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0001 (on exclusion list) [10869.236955] Lustre: 24081:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 1 previous similar message [10869.239703] LustreError: 24081:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0001 unknown param max_dirty_mb=15 [10869.791539] Lustre: t32fs-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10870.536832] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10870.668822] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10870.799608] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10871.055492] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt1 [10871.186530] Lustre: Failing over t32fs-MDT0001 [10871.189074] LustreError: 23285:0:(osp_object.c:527:osp_attr_get()) t32fs-MDT0000-osp-MDT0001:osp_attr_get update error [0x200000009:0x0:0x0]: rc = -5 [10871.194053] LustreError: 23285:0:(lod_sub_object.c:959:lod_sub_prep_llog()) t32fs-MDT0001-mdtlov: can't get id from catalogs: rc = -5 [10871.198971] LustreError: 23285:0:(lod_dev.c:419:lod_sub_recovery_thread()) t32fs-MDT0000-osp-MDT0001 getting update log failed: rc = -5 [10871.301161] Lustre: server umount t32fs-MDT0001 complete [10871.430872] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10871.838890] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10877.982049] Lustre: 24455:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252007/real 1491252007] req@ffff8800797a7c00 x1563690979560256/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252013 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10877.989886] Lustre: 24455:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10878.060476] Lustre: server umount t32fs-OST0000 complete [10878.062974] Lustre: Skipped 1 previous similar message [10878.197161] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10878.674658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10878.676545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10878.837068] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10878.838715] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10879.018318] Key type lgssc unregistered [10879.020748] Lustre: 25031:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10882.974039] LNet: 25047:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10886.976033] LNet: 25047:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10894.978026] LNet: 25047:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10910.980047] LNet: 25047:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [10920.982042] LNet: Removed LNI 10.9.5.243@tcp [10921.239322] LNet: 25087:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10921.736891] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10921.741210] alg: No test for adler32 (adler32-zlib) [10921.743592] alg: No test for crc32 (crc32-table) [10922.658403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10922.658404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10922.814219] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10922.816754] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10923.065224] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10923.547305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10923.552329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10923.702325] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10923.703531] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10923.751185] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10923.771107] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10923.773352] LNet: Accept all, port 7988 [10925.284060] Lustre: 26005:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10925.288760] Lustre: 26005:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10925.291153] Key type lgssc registered [10925.363744] Lustre: Echo OBD driver; http://www.lustre.org/ [10925.499384] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10925.759479] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10925.908855] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.923845] Lustre: MGS: Connection restored to 23481ee5-e4c2-39cf-143a-74f3359dc734 (at 0@lo) [10925.984919] Lustre: 26277:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10925.988423] LustreError: 26277:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10925.988438] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491252061/real 1491252061] req@ffff880060c24600 x1563691119018160/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.100.4.87@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252066 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10926.007940] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10926.172135] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10926.299697] Lustre: Failing over t32fs-MDT0000 [10926.301868] LustreError: 26294:0:(osp_object.c:527:osp_attr_get()) t32fs-MDT0001-osp-MDT0000:osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [10926.306184] LustreError: 26294:0:(lod_sub_object.c:959:lod_sub_prep_llog()) t32fs-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [10926.310551] LustreError: 26294:0:(lod_dev.c:419:lod_sub_recovery_thread()) t32fs-MDT0001-osp-MDT0000 getting update log failed: rc = -5 [10932.347045] Lustre: 26373:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252061/real 1491252061] req@ffff880060c24900 x1563691119018480/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252067 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10932.354686] Lustre: 26373:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10932.388499] Lustre: server umount t32fs-MDT0000 complete [10932.513409] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10932.883340] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10933.132185] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10933.382279] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10933.848900] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10934.102492] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10934.346381] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10934.593083] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10935.546383] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [10935.798275] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [10936.041857] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [10936.831902] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [10937.074328] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [10937.319269] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10937.581929] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10938.070391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10938.071325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [10938.226055] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10938.226394] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [10938.362105] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10938.844585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10938.848163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10938.999279] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10939.003206] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [10939.132019] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10939.272562] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10939.289222] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [10939.359075] Lustre: 28363:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10939.361636] Lustre: 28363:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [10939.370192] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491252075/real 1491252075] req@ffff88005dd70300 x1563691119018816/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252080 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10939.395566] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10939.569092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [10939.822706] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [10940.084746] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [10940.228427] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [10940.231003] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [10940.267283] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [10940.503768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [10940.755090] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [10940.882718] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [10941.010587] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [10941.264253] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [10941.391464] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [10941.393846] Lustre: Skipped 1 previous similar message [10941.522514] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [10941.773109] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [10944.950979] LustreError: 28344:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [10944.953632] Lustre: 28344:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [10944.958252] LustreError: 28344:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [10944.964181] Lustre: 29092:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [10944.968942] LustreError: 29092:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10945.275039] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252075/real 1491252075] req@ffff88005dd70900 x1563691119018656/t0(0) o8->t32fs-OST0000-osc-MDT0000@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10945.283227] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10945.286193] Lustre: t32fs-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [10945.288735] Lustre: Skipped 1 previous similar message [10946.026310] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [10946.155411] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [10946.287488] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [10946.545153] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M t32fs-OST0000 [10946.799242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [10946.799242] awk '/^status/ { print $2 }' [10947.054962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [10947.054962] awk '/^status/ { print $2 }' [10947.309934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub [10947.560939] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10947.693296] Lustre: Failing over t32fs-MDT0000 [10947.778333] Lustre: server umount t32fs-MDT0000 complete [10947.906844] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [10954.034041] Lustre: 29688:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252083/real 1491252083] req@ffff88005dd73600 x1563691119019840/t0(0) o39->t32fs-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1491252089 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10961.005053] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.009986] LustreError: 28344:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [10961.012823] Lustre: Failing over t32fs-OST0000 [10967.012042] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252096/real 1491252096] req@ffff88005dd70600 x1563691119019904/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491252102 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10967.020002] Lustre: 25994:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10981.012083] LustreError: 28344:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [10981.040237] Lustre: server umount t32fs-OST0000 complete [10981.174193] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10981.664049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10981.665245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [10981.818245] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10981.829489] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [10982.005239] Key type lgssc unregistered [10982.007665] Lustre: 30266:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [10986.015068] LNet: Removed LNI 10.9.5.243@tcp [10986.270283] LNet: 30320:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [10986.787873] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [10986.792535] alg: No test for adler32 (adler32-zlib) [10986.795024] alg: No test for crc32 (crc32-table) [10987.787604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10987.789899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10988.002585] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10988.005369] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [10988.250707] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [10988.726643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10988.728674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10988.879994] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10988.880050] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [10988.929775] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [10988.950266] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [10988.952635] LNet: Accept all, port 7988 [10990.463024] Lustre: 31236:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [10990.467868] Lustre: 31236:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [10990.470430] Key type lgssc registered [10990.537879] Lustre: Echo OBD driver; http://www.lustre.org/ [10990.675750] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [10990.938699] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [10991.089867] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [10991.103977] Lustre: MGS: Connection restored to a5f4d183-b119-80f4-67aa-a019d9b952da (at 0@lo) [10991.161119] Lustre: 31508:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [10991.168173] LustreError: 31508:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [10991.174845] Lustre: 31225:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491252126/real 1491252126] req@ffff880060bd8300 x1563691187175760/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252131 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10991.188016] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10991.352772] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [10991.480901] Lustre: Failing over t32fs-MDT0000 [10997.514035] Lustre: 31601:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252127/real 1491252127] req@ffff880060bd8000 x1563691187175888/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252133 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10997.612870] Lustre: server umount t32fs-MDT0000 complete [10997.737660] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [10998.111688] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [10998.361199] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [10998.605979] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk1_8_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [10998.939259] Lustre: DEBUG MARKER: cat /tmp/t32/commit [10999.183539] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [10999.431199] Lustre: DEBUG MARKER: cat /tmp/t32/arch [10999.677034] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [10999.925117] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [11000.172789] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [11000.417191] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [11001.203870] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [11001.448910] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [11001.693361] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [11001.958143] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11002.435594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [11002.440732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [11002.611286] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [11002.619235] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [11002.752723] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11003.232768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11003.243982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11003.391026] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11003.403569] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11003.534268] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [11003.679806] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [11003.684250] Lustre: 1161:0:(osd_handler.c:6906:osd_mount()) t32fs-MDT0000-osd: device /dev/loop0 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [11003.705559] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11003.721604] Lustre: 1204:0:(mgs_llog.c:261:mgs_fsdb_handler()) MDT using 1.8 OSC name scheme [11003.743236] Lustre: 1205:0:(obd_config.c:1539:class_config_llog_handler()) For 1.8 interoperability, rename obd type from mds to mdt [11003.781684] Lustre: 1205:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [11003.784162] Lustre: 1205:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [11003.792252] Lustre: 31225:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491252139/real 1491252139] req@ffff880079c74300 x1563691187176208/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.129@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252144 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [11003.810100] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11003.829946] Lustre: ctl-t32fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11003.991970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [11004.242230] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [11004.500300] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [11004.643650] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [11004.646392] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,,no_mbcache,nodelalloc [11004.681564] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [11004.909681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [11005.160947] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [11005.289524] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [11005.418501] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [11005.672412] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [11005.800317] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [11005.802993] Lustre: Skipped 1 previous similar message [11005.928345] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [11006.192179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [11008.689034] Lustre: 31225:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252139/real 1491252139] req@ffff880060bd8900 x1563691187176048/t0(0) o8->t32fs-OST0000-osc@192.168.203.129@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252144 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11008.697155] Lustre: 31225:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11010.020709] Lustre: 1934:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc (on exclusion list) [11010.023551] LustreError: 1934:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc unknown param max_dirty_mb=15 [11010.446250] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [11010.573719] Lustre: Modifying parameter t32fs-mdtlov.lov.stripesize in log t32fs-MDT0000 [11010.703927] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [11010.962199] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M t32fs-OST0000 [11011.222903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [11011.222903] awk '/^status/ { print $2 }' [11011.484458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [11011.484458] awk '/^status/ { print $2 }' [11011.750473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub [11012.001131] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [11012.131145] Lustre: Failing over t32fs-MDT0000 [11012.280744] Lustre: server umount t32fs-MDT0000 complete [11012.407625] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [11018.544044] Lustre: 2530:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252148/real 1491252148] req@ffff880079c74900 x1563691187177248/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252154 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11018.607202] Lustre: server umount t32fs-OST0000 complete [11018.739057] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11019.222998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [11019.223067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [11019.385700] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [11019.386661] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [11019.563734] Key type lgssc unregistered [11019.565952] Lustre: 3110:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [11024.534050] LNet: Removed LNI 10.9.5.243@tcp [11024.776080] LNet: 3163:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [11025.269054] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [11025.273410] alg: No test for adler32 (adler32-zlib) [11025.275740] alg: No test for crc32 (crc32-table) [11026.188628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11026.191556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11026.342523] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11026.342743] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11026.592090] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11027.075552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11027.083101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11027.229855] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11027.245734] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11027.292121] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [11027.312197] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [11027.314444] LNet: Accept all, port 7988 [11028.825067] Lustre: 4080:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [11028.829842] Lustre: 4080:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [11028.832280] Key type lgssc registered [11028.900381] Lustre: Echo OBD driver; http://www.lustre.org/ [11029.043180] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [11029.313212] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [11029.473977] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,user_xattr,acl,no_mbcache,nodelalloc [11029.478555] Lustre: 4301:0:(osd_handler.c:6906:osd_mount()) t32fs-MDT0000-osd: device /dev/loop0 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [11029.496860] Lustre: MGS: Connection restored to b059cf10-b260-8505-d652-907bee8c6f3b (at 0@lo) [11029.518029] Lustre: 4350:0:(mgs_llog.c:261:mgs_fsdb_handler()) MDT using 1.8 OSC name scheme [11029.531173] Lustre: 4351:0:(obd_config.c:1539:class_config_llog_handler()) For 1.8 interoperability, rename obd type from mds to mdt [11029.552450] Lustre: 4351:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [11029.559698] LustreError: 4351:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc unknown param max_dirty_mb=15 [11029.566498] Lustre: 4069:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1491252165/real 1491252165] req@ffff880057274300 x1563691227021632/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@192.168.203.129@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252170 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [11029.579309] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11029.738269] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [11029.868381] Lustre: Failing over t32fs-MDT0000 [11035.902067] Lustre: 4445:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252165/real 1491252165] req@ffff880065fbc000 x1563691227021760/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252171 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11035.938409] Lustre: server umount t32fs-MDT0000 complete [11036.065779] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [11036.454643] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [11036.698222] Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost [11036.944712] Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_3_up_2_5-ldiskfs.tar.bz2 -S -C /tmp/t32 [11037.428347] Lustre: DEBUG MARKER: cat /tmp/t32/commit [11037.673688] Lustre: DEBUG MARKER: cat /tmp/t32/kernel [11037.918834] Lustre: DEBUG MARKER: cat /tmp/t32/arch [11038.167334] Lustre: DEBUG MARKER: cat /tmp/t32/bspace [11038.485680] Lustre: DEBUG MARKER: cat /tmp/t32/ispace [11038.732567] Lustre: DEBUG MARKER: test -f /tmp/t32/blimit [11038.977932] Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit [11039.777182] Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 [11040.021443] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 [11040.269662] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [11040.533221] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11041.019776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [11041.031939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lctl device_list [11041.183529] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [11041.187678] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lctl device_list [11041.318847] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11041.796413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11041.796455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11041.967475] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11041.968441] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing losetup -a [11042.098732] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [11042.238236] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.258897] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11042.305799] Lustre: 6435:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [11042.308336] Lustre: 6435:0:(obd_mount.c:893:lustre_check_exclusion()) Skipped 2 previous similar messages [11042.327052] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.496412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid [11042.744228] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost [11043.003878] Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.9.5.243@tcp /tmp/t32/ost /tmp/t32/mnt/ost [11043.147167] LDISKFS-fs (loop1): file extents enabled, maximum tree depth=5 [11043.149687] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11043.184811] Lustre: srv-t32fs-OST0000: No data found on store. Initialize space [11043.400799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid [11043.653528] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 [11043.781884] Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client [11043.911955] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.9.5.243@tcp [11044.165815] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 [11044.302449] Lustre: Setting parameter t32fs-MDT0000-mdc.mdc.max_rpcs_in_flight in log t32fs-client [11044.304814] Lustre: Skipped 1 previous similar message [11044.439796] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.9.5.243@tcp [11044.701159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop [11047.192035] Lustre: 4069:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491252177/real 0] req@ffff880057274f00 x1563691227022096/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252182 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11049.200143] LustreError: 6416:0:(mgc_request.c:1571:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 10.9.5.243@tcp [11049.202762] Lustre: 6416:0:(mgc_request.c:1801:mgc_process_recover_nodemap_log()) MGC10.9.5.243@tcp: error processing recovery log t32fs-mdtir: rc = -2 [11049.207347] LustreError: 6416:0:(mgc_request.c:2131:mgc_process_log()) MGC10.9.5.243@tcp: recover log t32fs-mdtir failed, not fatal: rc = -2 [11049.213571] Lustre: 7164:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000-osc-MDT0000 (on exclusion list) [11049.216462] LustreError: 7164:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [11050.961897] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M [11051.096388] Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 [11051.233821] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 [11051.496376] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M t32fs-OST0000 [11051.756171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [11051.756171] awk '/^status/ { print $2 }' [11052.021820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub | [11052.021820] awk '/^status/ { print $2 }' [11052.287419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.oi_scrub [11052.562336] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [11072.697061] LustreError: 7686:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [11072.703374] LustreError: 7686:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880057274f00 x1563691227022096/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252182 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [11072.710950] LustreError: 7686:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [11092.716073] LustreError: 7686:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [11092.722459] LustreError: 7686:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880057274f00 x1563691227022096/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 1 dl 1491252182 ref 2 fl UnregRPC:EXN/0/ffffffff rc -5/-1 [11092.730430] LustreError: 7686:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [11095.312881] Lustre: Failing over t32fs-MDT0000 [11095.468475] Lustre: server umount t32fs-MDT0000 complete [11095.603351] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/ost [11101.739071] Lustre: 7767:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252231/real 1491252231] req@ffff880057276100 x1563691227023536/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252237 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11101.747323] Lustre: 7767:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [11101.772134] Lustre: server umount t32fs-OST0000 complete [11101.906096] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11102.384430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [11102.396547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [11102.545327] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [11102.567494] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [11102.742361] Key type lgssc unregistered [11102.744836] Lustre: 8344:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [11106.322046] LNet: 8360:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [11110.324062] LNet: 8360:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [11118.326048] LNet: 8360:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [11134.328042] LNet: 8360:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [11166.330044] LNet: 8360:0:(socklnd.c:2566:ksocknal_shutdown()) waiting for 1 peers to disconnect [11171.332044] LNet: Removed LNI 10.9.5.243@tcp [11171.539776] LNet: 8401:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [11172.037221] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [11172.042179] alg: No test for adler32 (adler32-zlib) [11172.044595] alg: No test for crc32 (crc32-table) [11172.962501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11172.967958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11173.111829] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11173.121902] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [11173.366383] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11173.926487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11173.928214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11174.086276] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11174.086314] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules [11174.135799] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [11174.155729] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [11174.157943] LNet: Accept all, port 7988 [11175.668029] Lustre: 9318:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [11175.672533] Lustre: 9318:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [11175.674999] Key type lgssc registered [11175.742695] Lustre: Echo OBD driver; http://www.lustre.org/ [11175.878628] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt [11176.136195] Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt [11176.285518] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11176.303875] Lustre: MGS: Connection restored to 85faf3f5-6856-e5c0-1482-ff6a09301859 (at 0@lo) [11176.350213] Lustre: 9590:0:(obd_mount.c:893:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) [11176.356959] LustreError: 9590:0:(obd_config.c:1387:class_process_proc_param()) osc: t32fs-OST0000-osc-MDT0000 unknown param max_dirty_mb=15 [11176.368837] Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11176.520261] Lustre: DEBUG MARKER: umount -d /tmp/t32/mnt/mdt [11180.363038] Lustre: 9307:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1491252311/real 0] req@ffff880064574000 x1563691381162160/t0(0) o8->t32fs-OST0000-osc-MDT0000@10.10.4.4@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252316 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11182.645047] LustreError: 9684:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [11182.649187] LustreError: 9684:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880064574300 x1563691381162336/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491252317 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [11182.656008] LustreError: 9684:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [11202.660043] LustreError: 9684:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [11202.664662] LustreError: 9684:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880064574300 x1563691381162336/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491252317 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [11202.672161] LustreError: 9684:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [11222.677028] LustreError: 9684:0:(import.c:334:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: rc = -110 waiting for callback (1 != 0) [11222.681977] LustreError: 9684:0:(import.c:360:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880064574300 x1563691381162336/t0(0) o38->t32fs-MDT0000-lwp-MDT0000@10.10.4.4@tcp:12/10 lens 520/544 e 0 to 0 dl 1491252317 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1 [11222.689998] LustreError: 9684:0:(import.c:374:ptlrpc_invalidate_import()) t32fs-MDT0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out. [11229.155908] Lustre: Failing over t32fs-MDT0000 [11235.191035] Lustre: 9684:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252364/real 1491252364] req@ffff880064574000 x1563691381162640/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252370 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11235.234500] Lustre: server umount t32fs-MDT0000 complete [11235.359517] Lustre: DEBUG MARKER: rm -rf /tmp/t32 [11235.691247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11236.302719] Lustre: DEBUG MARKER: rc=0; [11236.302719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11236.302719] if [[ $? -eq 0 && $val -ne 0 ]]; then [11236.302719] echo $(hostname -s): $val; [11236.302719] rc=$val; [11236.302719] fi; [11236.302719] exit $rc [11236.890579] Lustre: DEBUG MARKER: dmesg [11237.656707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 33a: Mount ost with a large index number ========================================= 20:46:12 \(1491252372\) [11237.809112] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================= 20:46:12 (1491252372) [11238.003636] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [11238.253301] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11238.506942] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsopti [11238.679417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11239.687919] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [11239.952862] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [11240.213688] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [11240.460766] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [11240.625783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11240.667672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.682975] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [11240.698363] Lustre: Setting parameter test-123-MDT0000-mdtlov.lov.stripesize in log test-123-MDT0000 [11240.712667] Lustre: ctl-test-123-MDT0000: No data found on store. Initialize space [11240.728416] Lustre: test-123-MDT0000: new disk, initializing [11240.745167] Lustre: test-123-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11240.751400] Lustre: ctl-test-123-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11240.916856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11241.183695] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11241.709417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11241.713167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11241.867661] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11241.867662] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11242.010593] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11242.266264] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11242.520784] Lustre: DEBUG MARKER: sync; sync; sync [11245.741439] Lustre: test-123-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [11247.845805] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [11249.013619] Lustre: MGS: Connection restored to a1c50471-46a6-79d1-38e5-2f89e45a7473 (at 10.9.5.244@tcp) [11250.051954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11250.205130] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11251.106620] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param test-123.sys.timeout=200 [11254.911733] LustreError: 11-0: test-123-OST1fff-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11254.914466] Lustre: test-123-OST1fff-osc-MDT0000: Connection to test-123-OST1fff (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11258.284317] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [11258.539740] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-fs2mds [11265.729103] Lustre: 11753:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252394/real 1491252394] req@ffff880056f80c00 x1563691381163824/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252401 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11265.816853] Lustre: server umount test-123-MDT0000 complete [11265.952151] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11266.748748] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11266.993077] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11272.828907] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11273.445430] Lustre: DEBUG MARKER: rc=0; [11273.445430] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11273.445430] if [[ $? -eq 0 && $val -ne 0 ]]; then [11273.445430] echo $(hostname -s): $val; [11273.445430] rc=$val; [11273.445430] fi; [11273.445430] exit $rc [11274.026856] Lustre: DEBUG MARKER: dmesg [11274.491581] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [11274.740769] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11276.342845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 33b: Drop cancel during umount =================================================== 20:46:50 \(1491252410\) [11276.497053] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount =================================================== 20:46:50 (1491252410) [11276.634596] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11278.558731] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11278.810195] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11279.107258] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11279.310960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.334211] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11279.336456] Lustre: Skipped 3 previous similar messages [11279.426443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11279.575936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11279.829690] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11280.323069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11280.323900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11280.485849] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11280.487459] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11280.626685] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11280.883614] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11281.135177] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11281.393267] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11281.645940] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11283.744707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11283.898323] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11289.422132] Lustre: 9307:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252415/real 1491252415] req@ffff880056f80000 x1563691381163984/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252425 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11289.708418] Lustre: lustre-MDT0000: Connection restored to 6cdd0090-865a-5266-dc2a-ab4694820936 (at 10.9.5.237@tcp) [11289.710912] Lustre: Skipped 4 previous similar messages [11294.447636] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11294.450313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11298.136865] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11298.389076] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11304.570039] Lustre: 13867:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252434/real 1491252434] req@ffff880034bc0300 x1563691381164512/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252440 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11304.616447] Lustre: server umount lustre-MDT0000 complete [11304.748517] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11311.268833] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11311.870636] Lustre: DEBUG MARKER: rc=0; [11311.870636] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11311.870636] if [[ $? -eq 0 && $val -ne 0 ]]; then [11311.870636] echo $(hostname -s): $val; [11311.870636] rc=$val; [11311.870636] fi; [11311.870636] exit $rc [11312.453323] Lustre: DEBUG MARKER: dmesg [11313.763723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 34a: umount with opened file should be fail ====================================== 20:47:28 \(1491252448\) [11313.917417] Lustre: DEBUG MARKER: == conf-sanity test 34a: umount with opened file should be fail ====================================== 20:47:28 (1491252448) [11314.057982] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11315.977609] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11316.228567] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11316.479614] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11316.628669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11316.649197] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11316.698860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11316.847278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11317.101756] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11317.587041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11317.588782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11317.747377] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11317.749341] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11317.887803] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11318.147366] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11318.399414] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11318.654157] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11318.910471] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11321.048917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11321.205065] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11321.695114] Lustre: 9307:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252452/real 1491252452] req@ffff880034bc0300 x1563691381164672/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491252457 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11331.359814] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11331.362404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11336.594315] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11336.846662] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11343.034045] Lustre: 15677:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252472/real 1491252472] req@ffff880064577300 x1563691381165232/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252478 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11343.142552] Lustre: server umount lustre-MDT0000 complete [11343.273041] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11349.503843] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11350.131775] Lustre: DEBUG MARKER: rc=0; [11350.131775] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11350.131775] if [[ $? -eq 0 && $val -ne 0 ]]; then [11350.131775] echo $(hostname -s): $val; [11350.131775] rc=$val; [11350.131775] fi; [11350.131775] exit $rc [11350.716194] Lustre: DEBUG MARKER: dmesg [11352.028078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 34b: force umount with failed mds should be normal =============================== 20:48:06 \(1491252486\) [11352.185868] Lustre: DEBUG MARKER: == conf-sanity test 34b: force umount with failed mds should be normal =============================== 20:48:06 (1491252486) [11352.323109] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11354.240163] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11354.489161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11354.735122] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11354.890861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11354.912040] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11354.914367] Lustre: Skipped 5 previous similar messages [11354.958579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11355.107327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11355.373660] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11355.859426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11355.860934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11356.016689] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11356.019998] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11356.158901] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11356.411060] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11356.663274] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11356.928667] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11357.193335] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11359.323730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11359.478759] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11366.720619] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11366.969847] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11368.453591] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11369.151090] LustreError: 9309:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880034bc0300 x1563691381165808/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11375.307887] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.442897] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11379.132032] Lustre: 17490:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252508/real 1491252508] req@ffff880064577900 x1563691381165904/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252514 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11379.139418] Lustre: 17490:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11379.224076] Lustre: server umount lustre-MDT0000 complete [11379.349929] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11396.490052] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11396.739217] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11401.807987] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11402.405687] Lustre: DEBUG MARKER: rc=0; [11402.405687] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11402.405687] if [[ $? -eq 0 && $val -ne 0 ]]; then [11402.405687] echo $(hostname -s): $val; [11402.405687] rc=$val; [11402.405687] fi; [11402.405687] exit $rc [11402.992731] Lustre: DEBUG MARKER: dmesg [11404.288202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 34c: force umount with failed ost should be normal =============================== 20:48:58 \(1491252538\) [11404.433781] Lustre: DEBUG MARKER: == conf-sanity test 34c: force umount with failed ost should be normal =============================== 20:48:58 (1491252538) [11404.573707] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11406.494803] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11406.743100] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11406.987960] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11407.165688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.235056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11407.382948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11407.632677] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11408.117590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11408.118709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11408.274670] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11408.277035] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11408.413102] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11408.665684] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11408.920375] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11409.177855] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11409.429642] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11411.581417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11411.738125] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11419.391640] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11419.394212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11436.357943] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11436.611873] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11448.775046] Lustre: 19453:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252578/real 1491252578] req@ffff880064575b00 x1563691381166704/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252584 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11448.782118] Lustre: 19453:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11448.810702] Lustre: server umount lustre-MDT0000 complete [11448.935391] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11455.084830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11455.687837] Lustre: DEBUG MARKER: rc=0; [11455.687837] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11455.687837] if [[ $? -eq 0 && $val -ne 0 ]]; then [11455.687837] echo $(hostname -s): $val; [11455.687837] rc=$val; [11455.687837] fi; [11455.687837] exit $rc [11456.266453] Lustre: DEBUG MARKER: dmesg [11457.577623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 35a: Reconnect to the last active server first =================================== 20:49:51 \(1491252591\) [11457.750565] Lustre: DEBUG MARKER: == conf-sanity test 35a: Reconnect to the last active server first =================================== 20:49:51 (1491252591) [11457.885782] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11459.804794] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11460.050403] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11460.300245] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11460.443871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.465273] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11460.467598] Lustre: Skipped 11 previous similar messages [11460.515352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11460.665386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11460.920086] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11461.402505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11461.414568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11461.567648] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11461.569209] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11461.707057] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11461.958234] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11462.209624] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11462.464943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11462.715996] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11464.788728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11464.947559] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11472.214397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set up a fake failnode for the MDS [11472.370886] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [11472.518770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n devices [11472.779961] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-MDT0000.failover.node=127.0.0.2@tcp [11473.113618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for RECONNECT_INTERVAL seconds \(10s\) [11473.265904] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [11483.463539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity.sh test_35a 2017-04-0320h50m18s [11483.621241] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2017-04-0320h50m18s [11483.815353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Stopping the MDT: lustre-MDT0000 [11483.965228] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [11484.106716] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11484.356956] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11485.784365] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [11487.999084] LustreError: 9309:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064574600 x1563691381167712/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11488.929071] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11496.592773] Lustre: server umount lustre-MDT0000 complete [11496.718510] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11497.043957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Restarting the MDT: lustre-MDT0000 [11497.188868] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [11497.332372] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11497.601983] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11497.852104] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11498.112510] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11498.270182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.347390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11498.502426] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11498.756130] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11499.259592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11499.259674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11499.416648] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11499.424510] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11499.560116] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11499.813096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11500.065771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11500.327925] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11500.588972] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11500.896465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for df \(9811\) ... [11501.049745] Lustre: DEBUG MARKER: Wait for df (9811) ... [11502.999764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [11503.154760] Lustre: DEBUG MARKER: done [11508.367677] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11508.370147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11510.296404] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11510.561983] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11516.900353] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11522.738478] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11522.987914] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11523.243150] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [11523.397228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11525.365567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11525.996695] Lustre: DEBUG MARKER: rc=0; [11525.996695] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11525.996695] if [[ $? -eq 0 && $val -ne 0 ]]; then [11525.996695] echo $(hostname -s): $val; [11525.996695] rc=$val; [11525.996695] fi; [11525.996695] exit $rc [11526.582238] Lustre: DEBUG MARKER: dmesg [11527.931436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 20:51:02 \(1491252662\) [11528.095495] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ================= 20:51:02 (1491252662) [11528.249467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11530.170913] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11530.428586] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11530.675094] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11530.830371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.864877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11530.872666] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11530.874858] Lustre: Skipped 4 previous similar messages [11531.059537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11531.311896] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11531.797513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11531.803746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11531.953795] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11531.959421] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11532.095765] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11532.347812] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11532.601560] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11532.857083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11533.106590] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11534.303876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11535.234666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11535.394167] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11542.613722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [11542.923063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity.sh test_35b 2017-04-0320h51m18s [11543.072313] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2017-04-0320h51m18s [11543.254108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set up a fake failnode for the MDS [11543.406750] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [11543.538456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n devices [11543.791846] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-MDT0000.failover.node=127.0.0.2@tcp [11544.051319] Lustre: DEBUG MARKER: lctl get_param -n at_max [11544.301659] Lustre: DEBUG MARKER: lctl get_param -n at_max [11544.595086] Lustre: DEBUG MARKER: lctl set_param at_max=0 [11544.926123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Injecting EBUSY on MDS [11545.072667] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [11545.202547] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000136 [11545.512931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Creating a test file and stat it [11545.662573] Lustre: DEBUG MARKER: Creating a test file and stat it [11545.675543] LustreError: 24306:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [11565.686920] Lustre: lustre-MDT0000: Client eee86bf8-7a10-99dd-a27e-50fca3526794 (at 10.9.5.237@tcp) reconnecting [11585.678039] LustreError: 24306:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [11585.862711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Stop injecting EBUSY on MDS [11586.018146] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [11586.148247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [11586.485715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [11586.635778] Lustre: DEBUG MARKER: done [11586.812913] Lustre: DEBUG MARKER: lctl set_param at_max=600 [11590.319675] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11590.322111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11594.052786] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11594.301289] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11600.486033] Lustre: 26538:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491252730/real 1491252730] req@ffff880064577000 x1563691381170064/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491252736 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11600.492677] Lustre: 26538:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [11600.532136] Lustre: server umount lustre-MDT0000 complete [11600.534159] Lustre: Skipped 1 previous similar message [11600.658895] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11606.048416] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11606.292155] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11606.551597] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [11606.698230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11608.598272] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11609.220924] Lustre: DEBUG MARKER: rc=0; [11609.220924] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11609.220924] if [[ $? -eq 0 && $val -ne 0 ]]; then [11609.220924] echo $(hostname -s): $val; [11609.220924] rc=$val; [11609.220924] fi; [11609.220924] exit $rc [11609.791556] Lustre: DEBUG MARKER: dmesg [11611.120664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 20:52:25 \(1491252745\) [11611.271275] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ====================== 20:52:25 (1491252745) [11611.448721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_36 remote OST [11611.594576] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [11611.807873] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11612.423221] Lustre: DEBUG MARKER: rc=0; [11612.423221] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11612.423221] if [[ $? -eq 0 && $val -ne 0 ]]; then [11612.423221] echo $(hostname -s): $val; [11612.423221] rc=$val; [11612.423221] fi; [11612.423221] exit $rc [11612.991346] Lustre: DEBUG MARKER: dmesg [11613.546538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 37: verify set tunables works for symlink device ================================= 20:52:28 \(1491252748\) [11613.702707] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ================================= 20:52:28 (1491252748) [11613.843773] Lustre: DEBUG MARKER: rm -f /tmp/sym_mdt.img [11614.089675] Lustre: DEBUG MARKER: ln -s /dev/lvm-Role_MDS/P1 /tmp/sym_mdt.img [11614.467267] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11614.714828] Lustre: DEBUG MARKER: mount -v -t lustre /tmp/sym_mdt.img /mnt/lustre-mds1 [11614.866170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11614.888047] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11614.890181] Lustre: Skipped 16 previous similar messages [11614.901663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11614.908878] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11614.911030] Lustre: Skipped 4 previous similar messages [11614.941106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11614.943282] Lustre: Skipped 1 previous similar message [11615.098568] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 && rm -f /tmp/sym_mdt.img [11615.227075] Lustre: Failing over lustre-MDT0000 [11621.473413] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11622.065418] Lustre: DEBUG MARKER: rc=0; [11622.065418] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11622.065418] if [[ $? -eq 0 && $val -ne 0 ]]; then [11622.065418] echo $(hostname -s): $val; [11622.065418] rc=$val; [11622.065418] fi; [11622.065418] exit $rc [11622.633550] Lustre: DEBUG MARKER: dmesg [11623.199400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 20:52:38 \(1491252758\) [11623.348489] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data =========================== 20:52:38 (1491252758) [11623.489262] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11625.441040] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11625.702350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11625.955413] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11626.118779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.330924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11626.584498] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11627.101298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11627.101487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11627.260405] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11627.263168] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11627.405145] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11627.665105] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11627.923308] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11628.182962] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11628.441282] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11629.662306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11630.653698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11630.810040] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11638.449676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying 10 files to \/mnt\/lustre\/d38.conf-sanity [11638.594630] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [11639.228868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osp.*.prealloc_next_id [11639.494493] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11639.744083] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11644.447062] LustreError: 9309:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880034823600 x1563691381171328/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11644.702295] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11652.115678] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11652.446823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark delete lov_objid file on MDS [11652.596492] Lustre: DEBUG MARKER: delete lov_objid file on MDS [11652.747513] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11652.998905] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11653.133258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11653.267779] Lustre: DEBUG MARKER: od -Ax -td8 /mnt/lustre-mds1/lov_objid; rm /mnt/lustre-mds1/lov_objid [11653.695459] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11653.957596] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11654.212568] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11654.461945] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11654.706774] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11654.863018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11655.078224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11655.333213] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11655.819787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11655.827303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11655.979803] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11655.980951] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11656.120038] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11656.370980] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11656.622340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11656.876159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11657.126409] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11657.541935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osp.*.prealloc_next_id [11657.953456] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11658.205664] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11659.935075] LustreError: 9309:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064577c00 x1563691381172048/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11666.707047] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11670.441499] Lustre: server umount lustre-MDT0000 complete [11670.443696] Lustre: Skipped 2 previous similar messages [11670.569679] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11670.840212] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11671.087884] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11671.261324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11671.386781] Lustre: DEBUG MARKER: od -Ax -td8 /mnt/lustre-mds1/lov_objid [11671.634210] Lustre: DEBUG MARKER: dd if=/dev/zero of=/mnt/lustre-mds1/lov_objid.clear count=8 [11671.897844] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11672.174078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11672.433626] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11672.683375] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11672.929781] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11673.082844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11673.293879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11673.547789] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11674.054981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11674.058230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11674.211627] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11674.212043] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11674.349086] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11674.600274] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11674.850270] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11675.107801] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11675.359771] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11675.821776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osp.*.prealloc_next_id [11676.239253] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11676.492516] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11676.682917] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11678.143094] LustreError: 9308:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064575b00 x1563691381172672/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11682.682892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.683122] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.852024] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11689.127279] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11689.373673] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11689.513416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11689.638123] Lustre: DEBUG MARKER: od -Ax -td8 /mnt/lustre-mds1/lov_objid [11689.899126] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11690.214748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark files compared the same [11690.364325] Lustre: DEBUG MARKER: files compared the same [11691.352180] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11691.637657] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11698.113251] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11698.712568] Lustre: DEBUG MARKER: rc=0; [11698.712568] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11698.712568] if [[ $? -eq 0 && $val -ne 0 ]]; then [11698.712568] echo $(hostname -s): $val; [11698.712568] rc=$val; [11698.712568] fi; [11698.712568] exit $rc [11699.288983] Lustre: DEBUG MARKER: dmesg [11700.597557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 20:53:54 \(1491252834\) [11700.760663] Lustre: DEBUG MARKER: == conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages ================== 20:53:54 (1491252834) [11700.907764] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11702.917479] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11703.172115] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11703.421992] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11703.581268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.796729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11704.057111] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11704.552894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11704.552895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11704.715867] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11704.715956] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11704.858339] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11705.114241] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11705.380817] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11705.647509] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11705.904863] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11708.099829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11708.248415] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug +malloc all 4 [11719.519894] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11719.523974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11722.498420] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11722.761156] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11729.100151] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11734.786077] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11735.433752] Lustre: DEBUG MARKER: rc=0; [11735.433752] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11735.433752] if [[ $? -eq 0 && $val -ne 0 ]]; then [11735.433752] echo $(hostname -s): $val; [11735.433752] rc=$val; [11735.433752] fi; [11735.433752] exit $rc [11736.048830] Lustre: DEBUG MARKER: dmesg [11737.478148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 40: race during service thread startup =========================================== 20:54:31 \(1491252871\) [11737.632305] Lustre: DEBUG MARKER: == conf-sanity test 40: race during service thread startup =========================================== 20:54:31 (1491252871) [11797.910534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11798.074418] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11798.971551] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000706 [11799.241664] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11799.503505] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11799.754471] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11800.001109] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11800.146202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.162403] Lustre: *** cfs_fail_loc=706, val=0*** [11800.215564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11800.217904] Lustre: Skipped 4 previous similar messages [11800.367771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11800.631450] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11801.128664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11801.131316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11801.280150] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11801.287515] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11801.422995] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11801.676468] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11801.927970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11802.185698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11802.438436] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11805.215689] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11805.218204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11809.519722] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11809.772708] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11815.966837] Lustre: server umount lustre-MDT0000 complete [11815.968994] Lustre: Skipped 2 previous similar messages [11816.097053] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11820.368924] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11820.974760] Lustre: DEBUG MARKER: rc=0; [11820.974760] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11820.974760] if [[ $? -eq 0 && $val -ne 0 ]]; then [11820.974760] echo $(hostname -s): $val; [11820.974760] rc=$val; [11820.974760] fi; [11820.974760] exit $rc [11821.545577] Lustre: DEBUG MARKER: dmesg [11822.842122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 20:55:57 \(1491252957\) [11822.992613] Lustre: DEBUG MARKER: == conf-sanity test 41a: mount mds with --nosvc and --nomgs ========================================== 20:55:57 (1491252957) [11823.139536] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11823.412095] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11825.333056] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11825.583878] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11825.828214] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o nosvc -n /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11825.969871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.112454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11826.362616] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11826.846314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11826.856948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11827.007477] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11827.007537] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11827.145474] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11827.398197] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11827.646499] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11829.741789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11829.889452] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11830.791846] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11831.062491] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11831.328239] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11831.592329] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o nomgs,force /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11831.988264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11832.242457] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11832.729736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11832.729825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11832.887220] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11832.900195] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11833.037966] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11833.289922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11833.540901] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11833.795639] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11834.050398] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11846.412090] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11846.667758] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11846.993476] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11847.264368] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11847.515796] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11853.846407] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11854.180101] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11854.812543] Lustre: DEBUG MARKER: rc=0; [11854.812543] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11854.812543] if [[ $? -eq 0 && $val -ne 0 ]]; then [11854.812543] echo $(hostname -s): $val; [11854.812543] rc=$val; [11854.812543] fi; [11854.812543] exit $rc [11855.393691] Lustre: DEBUG MARKER: dmesg [11855.931271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 20:56:31 \(1491252991\) [11856.085570] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount =========================== 20:56:31 (1491252991) [11856.223383] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11857.089781] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11857.343420] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11861.814660] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11862.066793] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11865.930855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11865.930855] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11865.930855] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11866.249469] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11866.835951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11866.853865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11866.878246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11866.892556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11867.074715] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11867.117024] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11867.152475] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11867.171878] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11867.483797] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11867.741270] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11868.031659] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [11868.206355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11874.774444] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11875.042787] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11875.294391] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11875.549899] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o nosvc -n /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11875.700179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11875.710379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.727264] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [11875.729320] Lustre: Skipped 29 previous similar messages [11875.854359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11876.114460] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11876.606343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11876.606809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11876.764269] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11876.767808] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11876.907072] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11877.175713] Lustre: DEBUG MARKER: sync; sync; sync [11882.492409] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11882.762877] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11884.933741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11885.097673] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11891.327067] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11891.591756] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11891.848378] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11892.103713] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o nomgs,force /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11892.269358] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11892.271688] Lustre: Skipped 4 previous similar messages [11892.281950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [11892.284104] Lustre: Skipped 1 previous similar message [11892.300046] Lustre: lustre-MDT0000: new disk, initializing [11892.328903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11892.520957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11892.784677] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11893.287869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11893.290855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11893.446613] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11893.447564] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11893.599256] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11893.868191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11894.131985] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11894.402241] Lustre: DEBUG MARKER: sync; sync; sync [11899.688679] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11899.957141] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11907.343681] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [11907.346453] LustreError: Skipped 1 previous similar message [11907.348716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [11907.353623] Lustre: Skipped 1 previous similar message [11912.275763] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11912.537529] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11912.915543] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11913.221988] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11913.492071] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11919.627051] Lustre: 13038:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491253049/real 1491253049] req@ffff880065591500 x1563691381175696/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491253055 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [11919.634193] Lustre: 13038:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [11919.782587] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11920.094725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11920.723719] Lustre: DEBUG MARKER: rc=0; [11920.723719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11920.723719] if [[ $? -eq 0 && $val -ne 0 ]]; then [11920.723719] echo $(hostname -s): $val; [11920.723719] rc=$val; [11920.723719] fi; [11920.723719] exit $rc [11921.323226] Lustre: DEBUG MARKER: dmesg [11921.926405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 41c: concurrent mounts of MDT\/OST should all fail but one ======================== 20:57:37 \(1491253057\) [11922.081111] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ======================== 20:57:37 (1491253057) [11922.217708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11922.217708] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11922.217708] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11923.074925] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11923.345759] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11929.696481] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11932.387442] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11932.650185] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11932.912864] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11933.067351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11933.282654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11933.535979] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11934.030389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11934.034195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11934.200208] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11934.214242] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11934.353060] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11934.614098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11934.870342] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11935.147446] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11935.410223] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11937.586547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11937.740092] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11952.023335] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11952.290404] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11958.633509] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11967.534405] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11968.072494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [11968.147512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [11968.188162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [11968.259180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [11968.305538] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [11968.496833] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [11968.511849] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [11968.535616] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [11972.737374] Lustre: DEBUG MARKER: lsmod | grep -q libcfs [11973.001743] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11973.253271] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000716 [11973.536099] Lustre: DEBUG MARKER: mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11973.543500] Lustre: DEBUG MARKER: mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11973.691924] LustreError: 16114:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 716 sleeping [11973.708093] LustreError: 16132:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 716 waking [11973.710696] LustreError: 16114:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 716 awake, rc=0 [11973.712901] LustreError: 16114:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -114 [11973.717529] LustreError: 16114:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-114) [11973.729562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.944191] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 [11974.687580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing lsmod [11974.843582] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing lsmod [11976.127290] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11976.382538] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11980.718537] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.706297] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11989.801797] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11990.061714] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11990.319129] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11990.571649] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11990.729313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.947109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11991.207094] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [11991.710229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11991.710423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11991.875199] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11991.878236] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11992.017546] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11992.291416] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11992.542851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11992.800774] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11993.062822] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11995.288282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [11995.453442] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12008.509541] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12008.782823] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12015.163733] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12021.822299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12022.492535] Lustre: DEBUG MARKER: rc=0; [12022.492535] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12022.492535] if [[ $? -eq 0 && $val -ne 0 ]]; then [12022.492535] echo $(hostname -s): $val; [12022.492535] rc=$val; [12022.492535] fi; [12022.492535] exit $rc [12023.090559] Lustre: DEBUG MARKER: dmesg [12024.426368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 42: allow client\/server mount\/unmount with invalid config param ================== 20:59:18 \(1491253158\) [12024.595168] Lustre: DEBUG MARKER: == conf-sanity test 42: allow client/server mount/unmount with invalid config param ================== 20:59:18 (1491253158) [12024.731348] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12026.662305] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12026.910453] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12027.162414] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12027.323674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.554707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12027.829160] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12028.337838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12028.348959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12028.501146] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12028.510053] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12028.651313] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12028.904559] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12029.153794] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12029.425249] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12029.693746] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12031.902811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12032.083083] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12039.387477] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.llite.some_wrong_param=10 [12039.515292] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [12039.517760] Lustre: Skipped 4 previous similar messages [12039.844079] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.some_wrong_param=20 [12039.969657] LustreError: 19816:0:(mgs_llog.c:3944:mgs_write_log_param()) err -22 on param 'sys.some_wrong_param=20' [12039.973967] LustreError: 19816:0:(mgs_handler.c:1019:mgs_iocontrol()) MGS: setparam err: rc = -22 [12041.823712] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [12041.826220] LustreError: Skipped 2 previous similar messages [12041.828317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [12041.832873] Lustre: Skipped 2 previous similar messages [12047.038344] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12047.291439] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12053.678938] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12059.187543] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12061.871771] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12062.119506] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12062.375383] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12062.539820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.615607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12062.618086] Lustre: Skipped 6 previous similar messages [12062.773895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12063.042298] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12063.536860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12063.548871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12063.704117] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12063.709317] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12063.845460] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12064.099452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12064.354372] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12064.618793] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12064.872962] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12066.979577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12067.135414] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12081.375387] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12081.649767] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12087.889211] Lustre: server umount lustre-MDT0000 complete [12087.891477] Lustre: Skipped 8 previous similar messages [12088.031865] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12094.449108] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12095.056968] Lustre: DEBUG MARKER: rc=0; [12095.056968] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12095.056968] if [[ $? -eq 0 && $val -ne 0 ]]; then [12095.056968] echo $(hostname -s): $val; [12095.056968] rc=$val; [12095.056968] fi; [12095.056968] exit $rc [12095.673591] Lustre: DEBUG MARKER: dmesg [12097.004335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 43a: check root_squash and nosquash_nids ========================================= 21:00:31 \(1491253231\) [12097.176389] Lustre: DEBUG MARKER: == conf-sanity test 43a: check root_squash and nosquash_nids ========================================= 21:00:31 (1491253231) [12097.325015] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12097.325015] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12097.325015] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12097.662355] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12099.586678] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12099.839069] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12100.087830] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12100.249489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.461886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12100.724732] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12101.219361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12101.231426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12101.385248] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12101.388476] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12101.528127] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12101.779823] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12102.029942] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12102.284920] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12102.539085] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12104.707714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12104.863527] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12112.128508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12112.385496] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.mdt.root_squash='0:0' [12112.526959] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [12112.686266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12112.952799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12113.237330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12113.488273] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.mdt.nosquash_nids='NONE' [12113.741587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12114.005054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12114.378314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12114.634295] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.mdt.root_squash='500:500' [12114.908707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12115.198443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12116.487650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12117.751479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12118.461938] Lustre: lustre-MDT0000: nosquash_nids is cleared [12118.464033] Lustre: lustre-MDT0000: root_squash is set to 500:500 [12119.005433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.root_squash [12120.328499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12120.579501] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-MDTall.mdt.nosquash_nids='2@gni 0@lo 10.9.5.237@tcp 192.168.0.[2,10]@tcp' [12120.838182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12121.099561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12122.353699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12123.612359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12124.868548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12126.146724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12127.407133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12128.508684] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 10.9.5.237@tcp 192.168.0.[2,10]@tcp [12128.658288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.nosquash_nids [12135.899990] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12136.152884] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12142.498624] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12148.233783] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12148.844545] Lustre: DEBUG MARKER: rc=0; [12148.844545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12148.844545] if [[ $? -eq 0 && $val -ne 0 ]]; then [12148.844545] echo $(hostname -s): $val; [12148.844545] rc=$val; [12148.844545] fi; [12148.844545] exit $rc [12149.437987] Lustre: DEBUG MARKER: dmesg [12150.827902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 43b: parse nosquash_nids with commas in expr_list ================================ 21:01:25 \(1491253285\) [12150.986920] Lustre: DEBUG MARKER: == conf-sanity test 43b: parse nosquash_nids with commas in expr_list ================================ 21:01:25 (1491253285) [12151.126029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12151.126029] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12151.126029] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12151.491676] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [12151.772281] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12152.028833] Lustre: DEBUG MARKER: mkfs.lustre --mgs --param=sys.timeout=20 --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-E lazy_itable_init" --fsname=test1234 --param mdt.root_squash=500:500 --param mdt.nosquash_nids=10.9.5.[237,237,237]@tcp --reformat /dev/lvm-Role_MDS/S1 [12152.186044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12152.330459] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [12152.589141] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [12152.847242] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [12153.103606] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [12153.262752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12153.312145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12153.315813] Lustre: 26202:0:(osd_handler.c:6906:osd_mount()) MGS-osd: device /dev/mapper/lvm--Role_MDS-S1 was upgraded from Lustre-1.x without enabling the dirdata feature. If you do not want to downgrade to Lustre-1.x again, you can enable it via 'tune2fs -O dirdata device' [12153.481179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12153.752652] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12154.271956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12154.283164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12154.430837] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12154.440031] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12154.576490] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12154.835138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12155.098548] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [12155.360522] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [12155.620945] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-fs2mds [12161.998224] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12162.334441] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12162.973565] Lustre: DEBUG MARKER: rc=0; [12162.973565] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12162.973565] if [[ $? -eq 0 && $val -ne 0 ]]; then [12162.973565] echo $(hostname -s): $val; [12162.973565] rc=$val; [12162.973565] fi; [12162.973565] exit $rc [12163.588316] Lustre: DEBUG MARKER: dmesg [12164.610436] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12164.857122] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12170.459815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 44: mounted client proc entry exists ============================================= 21:01:44 \(1491253304\) [12170.625401] Lustre: DEBUG MARKER: == conf-sanity test 44: mounted client proc entry exists ============================================= 21:01:44 (1491253304) [12170.762541] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12172.699110] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12172.954367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12173.202762] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12173.348515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12173.407018] Lustre: lustre-MDT0000: root_squash is set to 500:500 [12173.409387] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 10.9.5.237@tcp 192.168.0.[2,10]@tcp [12173.566101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12173.824097] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12174.328351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12174.328868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12174.485397] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12174.487614] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12174.625728] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12174.880384] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12175.131333] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12175.386103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12175.640542] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12177.866475] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12178.067492] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12185.386173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.*.uuid [12192.682981] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12192.932932] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12199.273991] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12205.420122] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12206.073606] Lustre: DEBUG MARKER: rc=0; [12206.073606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12206.073606] if [[ $? -eq 0 && $val -ne 0 ]]; then [12206.073606] echo $(hostname -s): $val; [12206.073606] rc=$val; [12206.073606] fi; [12206.073606] exit $rc [12206.696061] Lustre: DEBUG MARKER: dmesg [12207.994826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_45 skipping SLOW test 45 [12208.154025] Lustre: DEBUG MARKER: SKIP: conf-sanity test_45 skipping SLOW test 45 [12208.355239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 46a: handle ost additional - wide striped file =================================== 21:02:23 \(1491253343\) [12208.509104] Lustre: DEBUG MARKER: == conf-sanity test 46a: handle ost additional - wide striped file =================================== 21:02:23 (1491253343) [12209.246629] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12209.500244] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12213.502313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12213.502313] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12213.502313] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12213.817233] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12214.432933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12214.445071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12214.470679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12214.512385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12214.650552] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12214.717777] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12214.747600] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12214.797769] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12216.683489] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12216.937902] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12217.216090] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [12217.409597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12224.038215] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12224.302346] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12224.558746] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12224.813800] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12224.959626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12224.972970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.007796] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12225.009991] Lustre: Skipped 7 previous similar messages [12225.030557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12225.032577] Lustre: Skipped 1 previous similar message [12225.048127] Lustre: lustre-MDT0000: new disk, initializing [12225.078205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12225.245827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12225.515822] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12226.030024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12226.031133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12226.192688] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12226.196450] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12226.336185] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12226.587864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12226.863448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12227.122155] Lustre: DEBUG MARKER: sync; sync; sync [12232.446684] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12232.705730] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12234.978028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12235.144638] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12241.354310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12241.354310] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12241.354310] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12241.624330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12241.624330] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12241.624330] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12241.891981] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12242.145346] Lustre: DEBUG MARKER: lctl get_param -n at_min [12242.399214] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12242.887311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12242.900072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12243.053554] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12243.056577] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12243.241777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12243.242211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12243.406623] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12243.408823] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12250.395196] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12250.644755] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12256.968423] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12257.237071] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12257.506083] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12257.757500] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12258.010974] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12258.155292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.363084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12258.615625] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12259.100149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12259.115462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12259.265493] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12259.268313] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12259.409503] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12259.663032] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12259.913052] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12260.171171] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12260.422893] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12262.619744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12262.771272] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12263.731907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12263.731907] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12263.731907] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12264.011204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12264.011204] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12264.011204] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12264.275075] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12264.524082] Lustre: DEBUG MARKER: lctl get_param -n at_min [12264.776236] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12265.344649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12265.354335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12265.587355] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12265.592177] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12266.784564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12266.784965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12266.943577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12266.943582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12269.197804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12269.358515] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12277.694670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12277.863869] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12286.157939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12286.332563] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12294.971993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12295.123377] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12303.467354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12303.629123] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12312.097979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12312.256115] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12318.594102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12318.594102] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12318.594102] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12318.867252] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12318.867252] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12318.867252] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12319.136570] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12319.388248] Lustre: DEBUG MARKER: lctl get_param -n at_min [12319.642812] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12320.127304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [12320.132200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [12320.290407] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [12320.294820] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [12320.476331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12320.478917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12320.635750] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12320.635977] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12321.014052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12321.171865] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12321.355255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12321.551915] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12321.687929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12321.687929] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12321.687929] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12321.954409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12321.954409] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12321.954409] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12322.221545] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12322.476492] Lustre: DEBUG MARKER: lctl get_param -n at_min [12322.729691] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12323.217367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [12323.217591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [12323.374447] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [12323.375315] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0002-osc-MDT0000.ost_server_uuid 40 [12323.576333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12323.576371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12323.740325] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12323.742873] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12324.104387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12324.282107] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12324.470421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12324.634257] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12324.775550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12324.775550] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12324.775550] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12325.046658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12325.046658] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12325.046658] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12325.318478] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12325.574099] Lustre: DEBUG MARKER: lctl get_param -n at_min [12325.835719] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12326.339191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [12326.339387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [12326.500188] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [12326.500546] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0003-osc-MDT0000.ost_server_uuid 40 [12326.691838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12326.693312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12326.853066] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12326.865980] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12327.236666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12327.394836] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12327.584540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12327.741413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12327.885336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12327.885336] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12327.885336] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12328.156917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12328.156917] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12328.156917] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12328.428205] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12328.683649] Lustre: DEBUG MARKER: lctl get_param -n at_min [12328.939061] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12329.423701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [12329.425248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [12329.585352] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [12329.587074] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0004-osc-MDT0000.ost_server_uuid 40 [12329.775708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12329.776631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12329.942944] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12329.943944] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12330.314170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12330.472813] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12330.658412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12330.809937] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12330.950952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12330.950952] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12330.950952] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12331.225076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12331.225076] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12331.225076] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12331.498223] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12331.753571] Lustre: DEBUG MARKER: lctl get_param -n at_min [12332.010277] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12332.497477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [12332.499107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [12332.657739] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [12332.664038] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0005-osc-MDT0000.ost_server_uuid 40 [12332.851682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12332.854955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12333.013978] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12333.014777] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12333.381755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12333.545804] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12333.750115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12333.919997] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12334.084266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12334.084266] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12334.084266] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12334.356710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12334.356710] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12334.356710] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12334.630491] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12334.888462] Lustre: DEBUG MARKER: lctl get_param -n at_min [12335.144224] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12335.637955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0006-osc-MDT0000.ost_server_uuid 40 [12335.647405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0006-osc-MDT0000.ost_server_uuid 40 [12335.795554] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0006-osc-MDT0000.ost_server_uuid 40 [12335.801338] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0006-osc-MDT0000.ost_server_uuid 40 [12335.984864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12335.986805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12336.151507] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12336.151526] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12336.526462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uui [12336.701867] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0001-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0002-osc-ffff880069a76800.ost_server_uuid osc.lustre-OST0003 [12336.899512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12337.058635] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880069a76800.ost_server_uuid in FULL state after 0 sec [12337.148159] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12337.151089] Lustre: Skipped 4 previous similar messages [12339.247873] LustreError: 11-0: lustre-OST0006-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [12339.250929] LustreError: Skipped 4 previous similar messages [12339.253566] Lustre: lustre-OST0006-osc-MDT0000: Connection to lustre-OST0006 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [12339.259218] Lustre: Skipped 4 previous similar messages [12345.100228] LustreError: 10199:0:(obd_config.c:1392:class_process_proc_param()) lov.: error writing proc entry 'stripecount': rc = -34 [12385.750559] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12386.004661] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12392.345332] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12393.156284] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12393.406918] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12399.612989] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12399.860113] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12400.115423] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [12400.268397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12402.318561] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12402.960930] Lustre: DEBUG MARKER: rc=0; [12402.960930] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12402.960930] if [[ $? -eq 0 && $val -ne 0 ]]; then [12402.960930] echo $(hostname -s): $val; [12402.960930] rc=$val; [12402.960930] fi; [12402.960930] exit $rc [12403.569147] Lustre: DEBUG MARKER: dmesg [12404.876403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 21:05:39 \(1491253539\) [12405.032649] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ================= 21:05:39 (1491253539) [12405.749701] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12405.996100] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12410.074741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12410.074741] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12410.074741] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12410.386904] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12410.891049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12410.904655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12410.978902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12411.026837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12411.177895] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12411.314240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12411.318837] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12411.345222] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12413.288424] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12413.536663] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12413.827287] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [12414.017674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12420.850724] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12421.105349] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12421.354631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12421.618065] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12421.772407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12421.809120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12421.825408] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [12421.827477] Lustre: Skipped 64 previous similar messages [12421.843393] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12421.845604] Lustre: Skipped 5 previous similar messages [12421.865661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12421.867670] Lustre: Skipped 1 previous similar message [12421.883101] Lustre: lustre-MDT0000: new disk, initializing [12421.913836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12422.078655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12422.331178] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12422.815644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12422.817208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12422.971943] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12422.976235] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12423.111346] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12423.362841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12423.615881] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12423.868521] Lustre: DEBUG MARKER: sync; sync; sync [12429.182126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12429.432954] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12431.762040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12431.930758] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12458.820717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12458.999329] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12459.970714] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12460.219801] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12460.351560] Lustre: Failing over lustre-MDT0000 [12462.868589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.388066] Lustre: 13722:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491253596/real 1491253596] req@ffff880077bf5b00 x1563691381191376/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491253602 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12466.395657] Lustre: 13722:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [12466.555721] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12476.873418] Lustre: DEBUG MARKER: hostname [12477.179635] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12477.432732] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12477.680978] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12477.837474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.057405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12478.314699] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12478.838503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12478.841365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12478.999570] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12479.001761] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12479.140675] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12479.392158] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12479.643913] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12479.896610] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12485.416074] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [12485.433694] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [12492.679913] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12492.945385] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12499.296919] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12505.024222] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12505.642294] Lustre: DEBUG MARKER: rc=0; [12505.642294] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12505.642294] if [[ $? -eq 0 && $val -ne 0 ]]; then [12505.642294] echo $(hostname -s): $val; [12505.642294] rc=$val; [12505.642294] fi; [12505.642294] exit $rc [12506.249345] Lustre: DEBUG MARKER: dmesg [12507.558997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 48: too many acls on file ======================================================== 21:07:21 \(1491253641\) [12507.715362] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ======================================================== 21:07:21 (1491253641) [12507.858235] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12508.104522] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12518.404705] Lustre: DEBUG MARKER: hostname [12520.371883] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12520.618903] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12520.863601] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12521.019614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12521.230242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12521.487377] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12521.981576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12521.985824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12522.143259] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12522.143562] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12522.281751] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12522.531620] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12522.781962] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12523.035710] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12523.828723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12523.846597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12524.063998] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12524.089900] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12526.018481] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12526.270787] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12532.595605] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12536.629650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12536.629650] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12536.629650] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12536.947833] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12537.497461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12537.528307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12537.549056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12537.614137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12537.739612] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12537.743062] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12537.841970] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12537.848298] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12538.163207] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12538.408272] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12538.688690] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [12538.844792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12545.734770] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12545.995961] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12546.244026] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12546.488903] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12546.641696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12546.684689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12546.737750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12546.739999] Lustre: Skipped 1 previous similar message [12546.755976] Lustre: lustre-MDT0000: new disk, initializing [12546.788770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12546.963385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12547.222961] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12547.699166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12547.711288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12547.859754] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12547.872586] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12548.006441] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12548.259115] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12548.511586] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12548.763072] Lustre: DEBUG MARKER: sync; sync; sync [12554.071766] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12554.323819] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12556.651092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12556.812179] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12563.116463] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12563.118891] Lustre: Skipped 9 previous similar messages [12564.212464] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12564.468391] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12565.701455] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [12568.015119] LustreError: 9308:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88003a635200 x1563691381194112/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [12570.700968] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12570.705905] LustreError: Skipped 1 previous similar message [12576.879249] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12587.157327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12587.157327] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12587.157327] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12587.476157] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12588.042269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12588.064986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12588.092786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12588.104317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12588.370730] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12588.379336] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12588.381084] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12588.392878] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12588.705756] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12588.967182] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12589.260038] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [12589.477867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12596.325548] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12596.586439] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12596.836914] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12597.080977] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12597.235150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12597.280145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12597.333247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12597.335490] Lustre: Skipped 1 previous similar message [12597.351256] Lustre: lustre-MDT0000: new disk, initializing [12597.372847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12597.375073] Lustre: Skipped 8 previous similar messages [12597.384547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12597.546165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12597.798242] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12598.296357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12598.296667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12598.454203] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12598.458897] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12598.593453] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12598.845697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12599.099785] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12599.352425] Lustre: DEBUG MARKER: sync; sync; sync [12604.658697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12604.909717] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12607.159418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12607.320104] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12613.642575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12613.642575] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12613.642575] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12613.909637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12613.909637] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12613.909637] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12614.174608] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12614.426257] Lustre: DEBUG MARKER: lctl get_param -n at_min [12614.679203] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12615.168128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12615.169742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12615.324992] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12615.325173] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12615.512927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12615.513701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12615.673202] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12615.673912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12622.674316] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12622.934523] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12629.135231] Lustre: server umount lustre-MDT0000 complete [12629.137685] Lustre: Skipped 9 previous similar messages [12629.271263] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12629.585622] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12630.212779] Lustre: DEBUG MARKER: rc=0; [12630.212779] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12630.212779] if [[ $? -eq 0 && $val -ne 0 ]]; then [12630.212779] echo $(hostname -s): $val; [12630.212779] rc=$val; [12630.212779] fi; [12630.212779] exit $rc [12630.833837] Lustre: DEBUG MARKER: dmesg [12631.364827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================== 21:09:26 \(1491253766\) [12631.514407] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================== 21:09:26 (1491253766) [12632.229941] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12632.478661] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12636.511083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12636.511083] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12636.511083] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12636.822953] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12637.338478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12637.377211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12637.426558] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12637.475111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12637.648613] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12637.707142] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12637.733802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12637.747151] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12638.050737] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12638.296455] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12638.575298] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=sys.ldlm_timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --m [12638.771162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12645.845422] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12646.103229] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12646.354900] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12646.615601] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12646.778264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12646.814179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12646.867703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12646.869736] Lustre: Skipped 1 previous similar message [12646.885231] Lustre: lustre-MDT0000: new disk, initializing [12646.914866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12647.080683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12647.338227] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12647.823991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12647.825437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12647.979958] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12647.981828] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12648.120674] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12648.372353] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12648.622255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12648.874258] Lustre: DEBUG MARKER: sync; sync; sync [12654.206390] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12654.467905] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12656.726745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12656.928511] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12663.301375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm_timeout [12670.830386] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12671.086989] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12677.433895] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12677.748113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12678.363345] Lustre: DEBUG MARKER: rc=0; [12678.363345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12678.363345] if [[ $? -eq 0 && $val -ne 0 ]]; then [12678.363345] echo $(hostname -s): $val; [12678.363345] rc=$val; [12678.363345] fi; [12678.363345] exit $rc [12678.972224] Lustre: DEBUG MARKER: dmesg [12679.544780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================== 21:10:14 \(1491253814\) [12679.698786] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================== 21:10:14 (1491253814) [12680.419900] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12680.667741] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12684.624994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12684.624994] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12684.624994] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12684.931406] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12685.516562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12685.531268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12685.542162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12685.564258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12685.779109] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12685.781348] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12685.800774] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12685.849717] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12686.133155] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12686.381413] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12686.671756] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=sys.ldlm_timeout=19 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --m [12686.862899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12693.788327] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12694.046146] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12694.295557] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12694.541157] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12694.694120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12694.732352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.785897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12694.787941] Lustre: Skipped 1 previous similar message [12694.803613] Lustre: lustre-MDT0000: new disk, initializing [12694.833654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12695.004072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12695.260212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12695.740247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12695.742303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12695.896858] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12695.897417] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12696.037294] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12696.289072] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12696.538115] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12696.798704] Lustre: DEBUG MARKER: sync; sync; sync [12702.083738] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12702.336442] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12704.671682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12704.841488] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12711.237339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm_timeout [12718.752904] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12719.007198] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12725.400500] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12731.592259] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12732.205432] Lustre: DEBUG MARKER: rc=0; [12732.205432] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12732.205432] if [[ $? -eq 0 && $val -ne 0 ]]; then [12732.205432] echo $(hostname -s): $val; [12732.205432] rc=$val; [12732.205432] fi; [12732.205432] exit $rc [12732.791544] Lustre: DEBUG MARKER: dmesg [12734.105912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50a: lazystatfs all servers available ============================================ 21:11:08 \(1491253868\) [12734.253408] Lustre: DEBUG MARKER: == conf-sanity test 50a: lazystatfs all servers available ============================================ 21:11:08 (1491253868) [12734.395194] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12736.324039] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12736.585435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12736.835097] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12736.978058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.186900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12737.441315] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12737.930655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12737.930656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12738.090364] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12738.090365] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12738.229337] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12738.483713] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12738.734493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12738.989752] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12739.240382] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12741.503995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12741.668324] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12760.918212] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12761.171069] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12767.543718] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12774.037215] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12774.680434] Lustre: DEBUG MARKER: rc=0; [12774.680434] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12774.680434] if [[ $? -eq 0 && $val -ne 0 ]]; then [12774.680434] echo $(hostname -s): $val; [12774.680434] rc=$val; [12774.680434] fi; [12774.680434] exit $rc [12775.279098] Lustre: DEBUG MARKER: dmesg [12776.617812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50b: lazystatfs all servers down ================================================= 21:11:50 \(1491253910\) [12776.771171] Lustre: DEBUG MARKER: == conf-sanity test 50b: lazystatfs all servers down ================================================= 21:11:50 (1491253910) [12776.914887] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12778.860913] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12779.121638] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12779.384373] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12779.567562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.785212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12780.044016] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12780.534209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12780.547787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12780.701659] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12780.701660] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12780.842725] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12781.105340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12781.357966] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12781.646338] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12781.910134] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12784.192592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12784.373230] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12798.546955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12798.546955] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12798.546955] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12798.816097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12798.816097] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12798.816097] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12799.092334] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12799.362932] Lustre: DEBUG MARKER: lctl get_param -n at_min [12799.637664] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12800.157317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12800.183783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12800.346825] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12800.349062] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12800.531232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12800.533552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12800.692410] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12800.692706] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12812.974932] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12813.234648] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12825.576173] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12825.920488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12826.552598] Lustre: DEBUG MARKER: rc=0; [12826.552598] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12826.552598] if [[ $? -eq 0 && $val -ne 0 ]]; then [12826.552598] echo $(hostname -s): $val; [12826.552598] rc=$val; [12826.552598] fi; [12826.552598] exit $rc [12827.176143] Lustre: DEBUG MARKER: dmesg [12827.739844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50c: lazystatfs one server down ================================================== 21:12:42 \(1491253962\) [12827.917349] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ================================================== 21:12:42 (1491253962) [12828.067045] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12828.351609] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12828.601680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12828.850974] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12829.007592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12829.239818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12829.511225] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12830.031584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12830.037174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12830.193961] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12830.195247] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12830.334853] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12830.587865] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12830.843657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12831.102169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12831.371347] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12833.615831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12833.768483] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12836.816004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12836.993960] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12850.194701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12850.194701] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12850.194701] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12850.482642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12850.482642] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12850.482642] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12850.749602] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12851.013759] Lustre: DEBUG MARKER: lctl get_param -n at_min [12851.285338] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12851.782091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12851.791728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12851.955951] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12851.962176] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12852.152669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12852.154233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12852.312911] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12852.314168] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12862.191704] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [12862.194617] LustreError: Skipped 15 previous similar messages [12862.197070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [12862.202718] Lustre: Skipped 14 previous similar messages [12864.411455] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12864.676828] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12871.023784] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12871.297361] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12871.550305] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12871.805978] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [12871.951408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12873.800674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12874.445485] Lustre: DEBUG MARKER: rc=0; [12874.445485] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12874.445485] if [[ $? -eq 0 && $val -ne 0 ]]; then [12874.445485] echo $(hostname -s): $val; [12874.445485] rc=$val; [12874.445485] fi; [12874.445485] exit $rc [12875.047489] Lustre: DEBUG MARKER: dmesg [12875.620756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50d: lazystatfs client\/server conn race ========================================== 21:13:30 \(1491254010\) [12875.775796] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================== 21:13:30 (1491254010) [12875.919656] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12876.184202] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12876.431904] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12876.676655] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12876.845337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.881270] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12876.889527] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12876.891956] Lustre: Skipped 20 previous similar messages [12877.076715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12877.329811] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12877.813968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12877.815696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12877.970822] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12877.972589] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12878.109432] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12878.358454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12878.608255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12878.864479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12879.117248] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12880.319260] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12881.236123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12881.389394] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12883.304260] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [12884.290522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12884.459719] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12909.307352] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12909.559527] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12915.906767] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12916.172982] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12916.421759] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12916.679078] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [12916.832095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12918.795952] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12919.428812] Lustre: DEBUG MARKER: rc=0; [12919.428812] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12919.428812] if [[ $? -eq 0 && $val -ne 0 ]]; then [12919.428812] echo $(hostname -s): $val; [12919.428812] rc=$val; [12919.428812] fi; [12919.428812] exit $rc [12920.019272] Lustre: DEBUG MARKER: dmesg [12920.627104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50e: normal statfs all servers down ============================================== 21:14:15 \(1491254055\) [12920.811289] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down ============================================== 21:14:15 (1491254055) [12921.570887] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12921.816422] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12925.786032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12925.786032] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12925.786032] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12926.144446] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12926.688271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12926.703903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12926.746849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12926.823212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12926.999679] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12927.039308] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12927.047562] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12927.084535] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12927.379692] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12927.635321] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12927.922566] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [12928.091220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12934.681465] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12934.952218] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12935.232739] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12935.493401] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12935.642607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12935.662448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12935.718508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12935.720568] Lustre: Skipped 1 previous similar message [12935.736079] Lustre: lustre-MDT0000: new disk, initializing [12935.767935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12935.943165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12936.217315] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12936.714370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12936.729678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12936.882827] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12936.889483] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12937.025865] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12937.277103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12937.529211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12937.781016] Lustre: DEBUG MARKER: sync; sync; sync [12943.059531] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12943.324662] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12945.530660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12945.683465] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12951.925350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12951.925350] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12951.925350] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12952.223669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12952.223669] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12952.223669] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12952.508656] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12952.769899] Lustre: DEBUG MARKER: lctl get_param -n at_min [12953.040137] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12953.544905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12953.556339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12953.705207] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12953.711825] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12953.911561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12953.911664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12954.069630] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12954.070511] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12961.118387] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12961.375938] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12967.713392] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12967.975456] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12968.234525] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12968.484723] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12968.736439] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12968.891199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12969.114652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12969.390318] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12969.894716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12969.905732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12970.088172] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12970.089180] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12970.229488] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12970.485161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12970.750098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12971.016218] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12971.285642] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12973.648232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12973.804656] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [12974.736055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12974.736055] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12974.736055] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12975.009632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12975.009632] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12975.009632] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12975.296481] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12975.558764] Lustre: DEBUG MARKER: lctl get_param -n at_min [12975.816290] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12976.305473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12976.320390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12976.493452] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12976.508764] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12977.695514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12977.707652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12977.862790] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12977.878004] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [12984.887976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12984.887976] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12984.887976] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12985.194160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12985.194160] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12985.194160] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12985.467407] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [12985.726741] Lustre: DEBUG MARKER: lctl get_param -n at_min [12985.976425] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [12986.461634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12986.462840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12986.627231] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12986.627367] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12986.817251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12986.822964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12986.978470] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [12986.979243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [13010.108534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13010.279615] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13019.114623] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13019.380555] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13025.724615] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13026.043842] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13026.688886] Lustre: DEBUG MARKER: rc=0; [13026.688886] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13026.688886] if [[ $? -eq 0 && $val -ne 0 ]]; then [13026.688886] echo $(hostname -s): $val; [13026.688886] rc=$val; [13026.688886] fi; [13026.688886] exit $rc [13027.277746] Lustre: DEBUG MARKER: dmesg [13027.942661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50f: normal statfs one server in down ============================================ 21:16:03 \(1491254163\) [13028.095733] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ============================================ 21:16:03 (1491254163) [13028.234733] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13028.503629] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13028.764840] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13029.011266] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13029.168860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.190616] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [13029.194283] Lustre: Skipped 76 previous similar messages [13029.392609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13029.661869] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13030.156319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13030.158225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13030.317295] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13030.319218] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13030.467744] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13030.731361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13031.001832] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13031.259029] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13031.512136] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13033.710622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13033.890618] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13034.835611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13034.835611] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13034.835611] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13035.106798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13035.106798] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13035.106798] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13035.375721] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13035.626024] Lustre: DEBUG MARKER: lctl get_param -n at_min [13035.886881] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13036.392289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13036.394754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13036.556120] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13036.556968] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13039.756827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [13039.764684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [13039.912474] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [13039.918530] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [13042.045848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13042.203508] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13048.521357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13048.521357] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13048.521357] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13048.791866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13048.791866] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13048.791866] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13049.058969] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13049.313695] Lustre: DEBUG MARKER: lctl get_param -n at_min [13049.580128] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13050.068483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13050.073037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13050.228529] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13050.231783] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13050.417256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13050.417329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13050.575298] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13050.575303] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13057.552734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13057.552734] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13057.552734] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13057.826781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13057.826781] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13057.826781] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13058.097488] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13058.355239] Lustre: DEBUG MARKER: lctl get_param -n at_min [13058.609385] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13059.096123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13059.096801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13059.253587] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13059.257725] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state DISCONN osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13059.438733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [13059.442910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [13059.611451] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [13059.612440] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [13082.821385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13082.976189] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13083.895045] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [13108.713736] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13108.980940] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13121.143051] Lustre: 18669:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254250/real 1491254250] req@ffff88003a636400 x1563691381205360/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254256 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13121.151429] Lustre: 18669:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [13121.304036] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13121.566426] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13121.815652] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13122.073785] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [13122.230509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13124.183475] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13124.810772] Lustre: DEBUG MARKER: rc=0; [13124.810772] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13124.810772] if [[ $? -eq 0 && $val -ne 0 ]]; then [13124.810772] echo $(hostname -s): $val; [13124.810772] rc=$val; [13124.810772] fi; [13124.810772] exit $rc [13125.417590] Lustre: DEBUG MARKER: dmesg [13125.978731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50g: deactivated OST should not cause panic ====================================== 21:17:41 \(1491254261\) [13126.125859] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ====================================== 21:17:41 (1491254261) [13126.271354] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13126.531550] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13126.779301] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13127.053916] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13127.228220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13127.264016] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13127.449042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13127.700545] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13128.191501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13128.203379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13128.362737] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13128.362743] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13128.503494] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13128.754569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13129.008068] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13129.275592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13129.533004] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13130.754090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13131.706743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13131.863783] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13140.157191] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13141.201479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13141.372032] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13142.320301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13142.320301] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13142.320301] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13142.606338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13142.606338] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13142.606338] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13142.900383] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13143.154929] Lustre: DEBUG MARKER: lctl get_param -n at_min [13143.410123] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13143.899809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13143.920252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13144.073723] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13144.076297] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [13144.260386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13144.263478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13144.427854] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13144.430472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13144.815338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880060f03000.ost_server_uuid 40 [13144.969556] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff880060f03000.ost_server_uuid 40 [13145.162664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880060f03000.ost_server_uuid in FULL state after 0 sec [13145.325820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880060f03000.ost_server_uuid in FULL state after 0 sec [13145.466132] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0001.osc.active=0 [13145.593237] Lustre: Permanently deactivating lustre-OST0001 [13145.921555] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0001.osc.active=1 [13152.038187] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13152.292640] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13158.641294] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13158.906154] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13159.153172] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13159.407334] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [13159.564728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13161.617955] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13162.229445] Lustre: DEBUG MARKER: rc=0; [13162.229445] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13162.229445] if [[ $? -eq 0 && $val -ne 0 ]]; then [13162.229445] echo $(hostname -s): $val; [13162.229445] rc=$val; [13162.229445] fi; [13162.229445] exit $rc [13162.831783] Lustre: DEBUG MARKER: dmesg [13163.373160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50h: LU-642: activate deactivated OST ============================================ 21:18:18 \(1491254298\) [13163.524077] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ============================================ 21:18:18 (1491254298) [13164.078903] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13164.334850] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13164.581327] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13164.833158] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13164.981282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13165.017277] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13165.204745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13165.457629] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13165.959619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13165.962954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13166.123575] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13166.127157] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13166.261654] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13166.510698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13166.763804] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13167.016649] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13167.277077] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13168.678553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13168.683258] Lustre: Permanently deactivating lustre-OST0000 [13168.685274] Lustre: Skipped 1 previous similar message [13169.649597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13169.808675] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13172.685040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13172.845399] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13174.855034] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13178.856895] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osc.active='1' [13178.984802] Lustre: Permanently reactivating lustre-OST0000 [13198.143926] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13198.410745] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13204.767755] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13211.275771] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13211.888934] Lustre: DEBUG MARKER: rc=0; [13211.888934] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13211.888934] if [[ $? -eq 0 && $val -ne 0 ]]; then [13211.888934] echo $(hostname -s): $val; [13211.888934] rc=$val; [13211.888934] fi; [13211.888934] exit $rc [13212.482268] Lustre: DEBUG MARKER: dmesg [13213.778784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 50i: activate deactivated MDT ==================================================== 21:19:08 \(1491254348\) [13213.936309] Lustre: DEBUG MARKER: == conf-sanity test 50i: activate deactivated MDT ==================================================== 21:19:08 (1491254348) [13214.113443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_50i 1 \< 2, skipping [13214.260973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_50i 1 < 2, skipping [13214.450794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13215.060264] Lustre: DEBUG MARKER: rc=0; [13215.060264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13215.060264] if [[ $? -eq 0 && $val -ne 0 ]]; then [13215.060264] echo $(hostname -s): $val; [13215.060264] rc=$val; [13215.060264] fi; [13215.060264] exit $rc [13215.658312] Lustre: DEBUG MARKER: dmesg [13216.179455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 21:19:11 \(1491254351\) [13216.326622] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ====== 21:19:11 (1491254351) [13217.059140] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13217.305475] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13221.329177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13221.329177] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13221.329177] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13221.659101] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13222.291218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13222.302005] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13222.310955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13222.332988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13222.551116] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13222.555516] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13222.568731] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13222.642975] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13224.525402] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13224.773505] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13225.056553] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13225.250802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13232.064209] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13232.320327] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13232.569206] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13232.816535] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13232.970503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13233.009045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13233.061466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13233.063564] Lustre: Skipped 1 previous similar message [13233.079100] Lustre: lustre-MDT0000: new disk, initializing [13233.101652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13233.103770] Lustre: Skipped 11 previous similar messages [13233.113332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13233.273054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13233.528556] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13234.032405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13234.038740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13234.195261] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13234.196801] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13234.336788] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13234.590301] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13234.839609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13235.089894] Lustre: DEBUG MARKER: sync; sync; sync [13240.404980] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13240.655974] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13243.049976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13243.235863] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13254.676921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x142 [13254.810745] LustreError: 26377:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [13259.069241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13259.226468] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13264.813037] LustreError: 26377:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [13264.817700] LustreError: 26377:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [13274.820030] LustreError: 26377:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [13298.819908] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13299.071026] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13311.271194] Lustre: server umount lustre-MDT0000 complete [13311.273483] Lustre: Skipped 11 previous similar messages [13311.400963] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13316.500507] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13316.749528] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13317.003648] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [13317.157764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13319.188796] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13319.824325] Lustre: DEBUG MARKER: rc=0; [13319.824325] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13319.824325] if [[ $? -eq 0 && $val -ne 0 ]]; then [13319.824325] echo $(hostname -s): $val; [13319.824325] rc=$val; [13319.824325] fi; [13319.824325] exit $rc [13320.426939] Lustre: DEBUG MARKER: dmesg [13321.793329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 52: check recovering objects from lost+found ===================================== 21:20:56 \(1491254456\) [13321.945557] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ===================================== 21:20:56 (1491254456) [13322.083974] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13324.015575] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13324.276101] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13324.531469] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13324.674398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.709740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13324.901766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13325.154931] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13325.637274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13325.651847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13325.806797] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13325.823821] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13325.958159] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13326.218804] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13326.473968] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13326.734955] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13326.997376] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13328.229598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13328.231793] Lustre: Skipped 1 previous similar message [13329.192939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13329.372397] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13351.086348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13351.259182] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13351.929472] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [13360.874616] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13361.142389] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13367.501487] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13373.692704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13374.319976] Lustre: DEBUG MARKER: rc=0; [13374.319976] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13374.319976] if [[ $? -eq 0 && $val -ne 0 ]]; then [13374.319976] echo $(hostname -s): $val; [13374.319976] rc=$val; [13374.319976] fi; [13374.319976] exit $rc [13374.905662] Lustre: DEBUG MARKER: dmesg [13376.267163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 53a: check OSS thread count params =============================================== 21:21:50 \(1491254510\) [13376.424543] Lustre: DEBUG MARKER: == conf-sanity test 53a: check OSS thread count params =============================================== 21:21:50 (1491254510) [13376.571813] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13378.526334] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13378.776314] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13379.023394] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13379.184168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.395200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13379.650713] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13380.150769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13380.153829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13380.311941] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13380.311943] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13380.452718] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13380.704716] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13380.953683] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13381.211462] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13381.463341] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13383.680710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13383.851163] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13402.270516] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13402.521429] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13408.901889] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13415.078422] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13415.648791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13415.679366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13415.695889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13415.741942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13415.949475] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13415.997195] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13415.997428] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13416.022915] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13416.172012] Key type lgssc unregistered [13416.174230] Lustre: 32469:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [13421.561118] LNet: Removed LNI 10.9.5.243@tcp [13421.799104] LNet: 32522:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13423.129847] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13423.134270] alg: No test for adler32 (adler32-zlib) [13423.136661] alg: No test for crc32 (crc32-table) [13424.085235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13424.099507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13424.133589] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13424.322399] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13424.400938] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13424.414860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13424.439968] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13424.571253] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13425.602705] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13428.328242] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13428.918679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13428.927527] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13428.956546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13428.957812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13429.160738] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13429.172245] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13429.176123] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13429.231497] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13429.254327] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13429.256443] LNet: Accept all, port 7988 [13429.279385] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13430.769052] Lustre: 1400:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [13430.773537] Lustre: 1400:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [13430.776053] Key type lgssc registered [13430.844229] Lustre: Echo OBD driver; http://www.lustre.org/ [13432.781363] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13433.047221] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13433.296697] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13433.459429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13433.480488] Lustre: MGS: Connection restored to 20b64449-0e75-887e-a1c8-20981b123b57 (at 0@lo) [13433.533780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13433.681783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13433.934684] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13434.424166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13434.430694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13434.589125] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13434.592054] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13434.738239] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13434.997389] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13435.249504] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13435.503570] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13435.753826] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13437.880245] Lustre: MGS: Connection restored to 2d54f687-89e6-4ac8-8719-8ecf8f3a3369 (at 10.9.5.244@tcp) [13438.531073] Lustre: 1385:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254569/real 1491254569] req@ffff880057274000 x1563693745701024/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254574 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13439.770955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13439.925074] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13440.753854] Lustre: MGS: Connection restored to 46231eba-9247-00d7-1b6e-685279d2fe0e (at 10.9.5.237@tcp) [13440.756543] Lustre: Skipped 2 previous similar messages [13445.327801] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13445.330435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13450.066328] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13450.316244] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13456.497047] Lustre: 2856:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254586/real 1491254586] req@ffff880056f48300 x1563693745701520/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254592 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13456.539167] Lustre: server umount lustre-MDT0000 complete [13456.667145] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13462.152267] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13462.767140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13462.797087] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13462.812691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13462.845186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13463.007065] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13463.078924] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13463.116463] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13463.154788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13463.316448] Key type lgssc unregistered [13463.318803] Lustre: 3633:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [13467.339137] LNet: Removed LNI 10.9.5.243@tcp [13467.539165] LNet: 3687:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13468.554955] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13468.559718] alg: No test for adler32 (adler32-zlib) [13468.562099] alg: No test for crc32 (crc32-table) [13469.614437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13469.628665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13469.637525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13469.638303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13469.893222] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13469.899851] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13469.902288] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13469.926149] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13470.286393] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13473.045692] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13473.651137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13473.684340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13473.743008] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13473.762318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13473.968697] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13473.982463] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13474.016864] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13474.071186] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13474.116317] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13474.136972] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13474.139049] LNet: Accept all, port 7988 [13475.650108] Lustre: 4957:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [13475.654373] Lustre: 4957:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [13475.657191] Key type lgssc registered [13475.727080] Lustre: Echo OBD driver; http://www.lustre.org/ [13475.869171] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13476.126781] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13476.378386] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13476.549656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13476.569617] Lustre: MGS: Connection restored to d5b18917-b637-39d1-42a3-67a173faea52 (at 0@lo) [13476.632762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13476.796051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13477.059376] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13477.560288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13477.572698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13477.721651] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13477.723830] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13477.868167] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13478.124149] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13478.382743] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13478.656672] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13478.919843] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13480.195464] Lustre: MGS: Connection restored to 7c37e4ba-07f8-ad66-be89-ff661313627b (at 10.9.5.244@tcp) [13481.121229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13481.285296] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13481.630159] Lustre: 4946:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254612/real 1491254612] req@ffff880066664000 x1563693792886944/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254617 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13481.637937] Lustre: lustre-MDT0000: Connection restored to d5b18917-b637-39d1-42a3-67a173faea52 (at 0@lo) [13481.640395] Lustre: Skipped 1 previous similar message [13487.136500] Lustre: lustre-MDT0000: Connection restored to 2c04d767-ab00-ab9a-e222-c416dcbdaaf5 (at 10.9.5.237@tcp) [13487.139061] Lustre: Skipped 1 previous similar message [13492.479683] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13492.482353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13495.473055] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13495.722323] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13501.899062] Lustre: 6399:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254631/real 1491254631] req@ffff880077a9ea00 x1563693792887456/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254637 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13501.935881] Lustre: server umount lustre-MDT0000 complete [13502.063740] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13508.498806] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13509.012406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13509.059129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13509.122615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13509.150924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13509.343059] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13509.390156] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13509.402168] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13509.436604] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13509.583846] Key type lgssc unregistered [13509.586152] Lustre: 7176:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [13513.490099] LNet: Removed LNI 10.9.5.243@tcp [13513.667552] LNet: 7229:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13515.020645] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13515.024981] alg: No test for adler32 (adler32-zlib) [13515.028671] alg: No test for crc32 (crc32-table) [13516.022289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13516.022484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13516.089518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13516.113894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13516.340976] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13516.350030] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13516.394677] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13516.410584] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13516.818448] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13517.459967] Lustre: DEBUG MARKER: rc=0; [13517.459967] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13517.459967] if [[ $? -eq 0 && $val -ne 0 ]]; then [13517.459967] echo $(hostname -s): $val; [13517.459967] rc=$val; [13517.459967] fi; [13517.459967] exit $rc [13518.072705] Lustre: DEBUG MARKER: dmesg [13519.372918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 53b: check MDS thread count params =============================================== 21:24:13 \(1491254653\) [13519.529590] Lustre: DEBUG MARKER: == conf-sanity test 53b: check MDS thread count params =============================================== 21:24:13 (1491254653) [13519.669194] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13521.609715] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13521.883070] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13522.141485] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13522.298823] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13522.315079] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13522.317030] LNet: Accept all, port 7988 [13522.329856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13522.349227] Lustre: MGS: Connection restored to 6fcbe812-93f7-de06-6f4b-562a58891ea8 (at 0@lo) [13522.403973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13522.555997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13522.817224] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13523.386790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13523.391950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13523.561349] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13523.564450] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13523.700735] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13523.951817] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13524.200528] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13524.461631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13524.714277] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13525.996333] Lustre: MGS: Connection restored to ee93e11d-e7f9-9553-acbc-7e690e46d701 (at 10.9.5.244@tcp) [13526.943969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13527.099529] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13527.401108] Lustre: 8428:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254658/real 1491254658] req@ffff8800642f4000 x1563693843218592/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254663 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13527.408411] Lustre: lustre-MDT0000: Connection restored to 6fcbe812-93f7-de06-6f4b-562a58891ea8 (at 0@lo) [13527.410837] Lustre: Skipped 1 previous similar message [13532.936768] Lustre: lustre-MDT0000: Connection restored to 2092e46b-3303-4ce3-10ce-2d8a0ce7333b (at 10.9.5.237@tcp) [13532.939734] Lustre: Skipped 1 previous similar message [13534.348998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -N mds.*.*.threads_max 2>/dev/null [13534.619829] Lustre: DEBUG MARKER: lctl get_param -n cpu_partition_table 2>/dev/null| wc -l [13534.925315] Lustre: DEBUG MARKER: lctl get_param -N mds.*.*..threads_min [13535.179229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13535.432163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_max [13535.684296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_started [13535.942993] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mds.MDS.mdt..threads_min=4 [13536.196775] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mds.MDS.mdt..threads_max=79 [13536.448294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13536.699513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_max [13536.955544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13537.208356] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mds.MDS.mdt..threads_max=4 [13537.461852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13537.711334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_max [13537.989883] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mds.MDS.mdt..threads_max=3 [13538.239627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13538.498794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_max [13539.551628] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13539.554077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13545.724480] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13545.975679] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13552.185060] Lustre: 10824:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254681/real 1491254681] req@ffff8800642f5e00 x1563693843219136/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254687 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13552.263220] Lustre: server umount lustre-MDT0000 complete [13552.387473] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13557.377215] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13557.917386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13557.959399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13558.018337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13558.029962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13558.254660] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13558.263434] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13558.293986] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13558.297290] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13562.562047] LNet: Removed LNI 10.9.5.243@tcp [13562.766533] LNet: 11601:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13563.709828] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13563.714535] alg: No test for adler32 (adler32-zlib) [13563.716691] alg: No test for crc32 (crc32-table) [13564.673188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13564.729582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13564.741445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13564.945874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13565.125188] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13565.160176] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13565.175389] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13565.330347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13565.649921] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13568.371067] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13568.961030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13568.990857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13568.991193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13568.998821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13569.207787] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13569.239558] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13569.276228] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13569.276229] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13569.346624] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13569.368143] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13569.370217] LNet: Accept all, port 7988 [13570.881049] Lustre: 12870:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [13570.887126] Lustre: 12870:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [13570.890089] Key type lgssc registered [13570.959869] Lustre: Echo OBD driver; http://www.lustre.org/ [13571.101176] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13571.351883] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13571.602579] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13571.763939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13571.784090] Lustre: MGS: Connection restored to 43c7cf45-f6b3-24f5-0291-fb75a63d6952 (at 0@lo) [13571.843324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13571.988401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13572.242772] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13572.723534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13572.727135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13572.880948] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13572.889721] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13573.026345] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13573.277406] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13573.528102] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13573.793510] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13574.062206] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13575.242912] Lustre: MGS: Connection restored to ba795e9f-04be-bae5-0547-75ac5a730b39 (at 10.9.5.244@tcp) [13576.165936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13576.323628] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13576.840108] Lustre: 12859:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254707/real 1491254707] req@ffff8800689d4000 x1563693893550240/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254712 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13576.847742] Lustre: lustre-MDT0000: Connection restored to 43c7cf45-f6b3-24f5-0291-fb75a63d6952 (at 0@lo) [13576.850181] Lustre: Skipped 1 previous similar message [13582.141764] Lustre: lustre-MDT0000: Connection restored to 17dcdcce-2d06-1f13-a2b5-6f6d9591df79 (at 10.9.5.237@tcp) [13582.144394] Lustre: Skipped 1 previous similar message [13583.584860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_min [13583.836956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_max [13584.093146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mds.MDS.mdt..threads_started [13585.263667] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13585.266321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13591.299403] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13591.559635] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13597.750053] Lustre: 14528:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254727/real 1491254727] req@ffff8800689d4000 x1563693893550736/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254733 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13597.826657] Lustre: server umount lustre-MDT0000 complete [13597.955722] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13603.541919] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13604.106907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13604.123820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13604.127933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13604.221296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13604.364065] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13604.367755] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13604.461299] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13604.481861] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13604.667571] Key type lgssc unregistered [13604.669763] Lustre: 15304:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [13610.282048] LNet: Removed LNI 10.9.5.243@tcp [13610.521289] LNet: 15358:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13611.041361] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13611.046265] alg: No test for adler32 (adler32-zlib) [13611.048687] alg: No test for crc32 (crc32-table) [13612.035175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13612.066930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13612.091431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13612.093655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13612.308736] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13612.317591] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13612.351857] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13612.383276] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13612.741474] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13615.476920] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13616.014378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13616.026897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13616.095356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13616.099733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13616.279312] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13616.311070] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_modules_local [13616.378512] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13616.380580] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing load_modules_local [13616.419289] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13616.440625] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13616.442681] LNet: Accept all, port 7988 [13617.953052] Lustre: 16627:0:(gss_svc_upcall.c:1178:gss_init_svc_upcall()) Init channel is not opened by lsvcgssd, following request might be dropped until lsvcgssd is active [13617.957245] Lustre: 16627:0:(gss_mech_switch.c:72:lgss_mech_register()) Register gssnull mechanism [13617.959439] Key type lgssc registered [13618.048446] Lustre: Echo OBD driver; http://www.lustre.org/ [13618.197384] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13618.458473] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13618.717840] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13618.883231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13618.904851] Lustre: MGS: Connection restored to 1167048b-adec-b962-6d72-201f38755984 (at 0@lo) [13618.958985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13619.106212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13619.360602] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13619.845321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13619.849212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13620.005034] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13620.006117] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13620.145327] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13620.396108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13620.662377] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13620.931679] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13621.191957] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13622.377459] Lustre: MGS: Connection restored to 2b277f4e-6099-03b2-2032-015fb5b9011b (at 10.9.5.244@tcp) [13623.308539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13623.457051] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13623.956053] Lustre: 16616:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254754/real 1491254754] req@ffff880037fa8000 x1563693942833312/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254759 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13623.963682] Lustre: lustre-MDT0000: Connection restored to 1167048b-adec-b962-6d72-201f38755984 (at 0@lo) [13623.966117] Lustre: Skipped 1 previous similar message [13629.274834] Lustre: lustre-MDT0000: Connection restored to f94f106d-ff06-f5f1-7838-f93f08a07601 (at 10.9.5.237@tcp) [13629.277394] Lustre: Skipped 1 previous similar message [13631.519669] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13631.522343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13633.643027] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13633.894115] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13642.057041] Lustre: 18071:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254771/real 1491254771] req@ffff880037fa8900 x1563693942833808/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254777 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13642.101946] Lustre: server umount lustre-MDT0000 complete [13642.234514] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13647.682103] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13648.208071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13648.226547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13648.309195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13648.323378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing \/usr\/sbin\/lustre_rmmod ldiskfs [13648.464088] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13648.584817] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13648.590327] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13648.597122] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing /usr/sbin/lustre_rmmod ldiskfs [13648.786280] Key type lgssc unregistered [13648.788604] Lustre: 18848:0:(gss_mech_switch.c:81:lgss_mech_unregister()) Unregister krb5 mechanism [13653.539058] LNet: Removed LNI 10.9.5.243@tcp [13653.790163] LNet: 18901:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [13655.148394] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [13655.153210] alg: No test for adler32 (adler32-zlib) [13655.155612] alg: No test for crc32 (crc32-table) [13656.104038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13656.157161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13656.165802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13656.219042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13656.365925] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_mem_leak [13656.464539] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_mem_leak [13656.477297] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13656.478760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_mem_leak [13656.887971] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13657.516779] Lustre: DEBUG MARKER: rc=0; [13657.516779] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13657.516779] if [[ $? -eq 0 && $val -ne 0 ]]; then [13657.516779] echo $(hostname -s): $val; [13657.516779] rc=$val; [13657.516779] fi; [13657.516779] exit $rc [13658.100687] Lustre: DEBUG MARKER: dmesg [13659.405104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 54a: test llverdev and partial verify of device ================================== 21:26:33 \(1491254793\) [13659.549225] Lustre: DEBUG MARKER: == conf-sanity test 54a: test llverdev and partial verify of device ================================== 21:26:33 (1491254793) [13660.180854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing run_llverdev \/dev\/lvm-Role_OSS\/P1 -p [13660.337725] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing run_llverdev /dev/lvm-Role_OSS/P1 -p [13661.421438] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13661.667452] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13665.442766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13665.442766] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13665.442766] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13665.760075] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13666.376091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13666.391145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13666.399799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13666.437209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13666.595233] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13666.683824] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13666.695698] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13666.695706] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13668.600355] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13668.843304] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13669.120656] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13669.282448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13676.018728] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13676.280812] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13676.544720] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13676.790359] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13676.945032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13676.990780] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [13677.007451] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [13677.009476] LNet: Accept all, port 7988 [13677.019038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13677.042755] Lustre: MGS: Connection restored to 1073a2fa-d1a3-ed46-58c8-26ed11314546 (at 0@lo) [13677.064823] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13677.087997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13677.103197] Lustre: lustre-MDT0000: new disk, initializing [13677.124405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13677.133400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13677.293831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13677.561324] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13678.069322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13678.069323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13678.230550] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13678.232524] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13678.372152] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13678.626285] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13678.887318] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13679.145080] Lustre: DEBUG MARKER: sync; sync; sync [13682.118394] Lustre: lustre-MDT0000: Connection restored to 1073a2fa-d1a3-ed46-58c8-26ed11314546 (at 0@lo) [13684.422474] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13684.677610] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13685.954476] Lustre: MGS: Connection restored to 208649c3-b078-4852-3e92-4f5f90f3fa6c (at 10.9.5.244@tcp) [13686.925465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13687.093687] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13693.317846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13693.317846] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13693.317846] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13693.587429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13693.587429] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13693.587429] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13693.873103] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13694.139027] Lustre: DEBUG MARKER: lctl get_param -n at_min [13694.403810] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13694.891200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13694.902597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13695.056828] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13695.057115] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13695.249444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13695.250841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13695.410690] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13695.412643] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13700.719797] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13700.722814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13702.367417] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13702.621283] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13708.784057] Lustre: 23388:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254838/real 1491254838] req@ffff8800651a1e00 x1563694005748736/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254844 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13708.820929] Lustre: server umount lustre-MDT0000 complete [13708.954036] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13709.262176] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13709.880403] Lustre: DEBUG MARKER: rc=0; [13709.880403] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13709.880403] if [[ $? -eq 0 && $val -ne 0 ]]; then [13709.880403] echo $(hostname -s): $val; [13709.880403] rc=$val; [13709.880403] fi; [13709.880403] exit $rc [13710.461149] Lustre: DEBUG MARKER: dmesg [13711.015244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 54b: test llverfs and partial verify of filesystem =============================== 21:27:26 \(1491254846\) [13711.186918] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem =============================== 21:27:26 (1491254846) [13711.338096] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13711.598601] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13711.850740] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13712.110341] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13712.255122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.274685] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [13712.277881] Lustre: Skipped 1 previous similar message [13712.323646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13712.473741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13712.738676] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13713.230094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13713.235199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13713.394098] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13713.394100] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13713.534958] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13713.786418] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13714.036206] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13714.291190] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13714.541324] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13716.711044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13716.865843] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13717.320207] Lustre: 21190:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254847/real 1491254847] req@ffff880055e6fc00 x1563694005748896/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13717.329035] Lustre: lustre-MDT0000: Connection restored to 10.9.5.243@tcp (at 0@lo) [13717.331381] Lustre: Skipped 2 previous similar messages [13728.623853] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [13728.626433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [13731.169701] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13731.421677] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13737.601191] Lustre: 25199:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254867/real 1491254867] req@ffff8800689a0300 x1563694005749408/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254873 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13737.653084] Lustre: server umount lustre-MDT0000 complete [13737.784419] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13743.245167] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13743.890897] Lustre: DEBUG MARKER: rc=0; [13743.890897] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13743.890897] if [[ $? -eq 0 && $val -ne 0 ]]; then [13743.890897] echo $(hostname -s): $val; [13743.890897] rc=$val; [13743.890897] fi; [13743.890897] exit $rc [13744.497124] Lustre: DEBUG MARKER: dmesg [13745.980777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 55: check lov_objid size ========================================================= 21:28:00 \(1491254880\) [13746.136538] Lustre: DEBUG MARKER: == conf-sanity test 55: check lov_objid size ========================================================= 21:28:00 (1491254880) [13746.342661] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13746.593551] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13746.876479] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13747.053816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13748.146671] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13750.096735] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13750.352947] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13750.619152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13750.779148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13750.820110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13750.837215] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [13750.840493] Lustre: Skipped 2 previous similar messages [13750.858689] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13750.861197] Lustre: Skipped 4 previous similar messages [13750.885061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13750.887365] Lustre: Skipped 1 previous similar message [13750.903179] Lustre: lustre-MDT0000: new disk, initializing [13750.926077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13750.936521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13751.104154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13751.399974] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13751.909667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13751.931611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13752.085753] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13752.086831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13752.231354] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13752.491702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13752.752617] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13753.010161] Lustre: DEBUG MARKER: sync; sync; sync [13758.317911] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13758.576303] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13760.877209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13761.065198] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13771.930224] Lustre: lustre-MDT0000: Connection restored to f962d99d-7e30-fad6-7e0d-7dd9726fb79e (at 10.9.5.237@tcp) [13771.932927] Lustre: Skipped 4 previous similar messages [13772.803074] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13773.062457] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13773.967094] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800689a1800 x1563694005750320/t0(0) o13->lustre-OST03ff-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13774.899348] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [13784.890112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13785.237033] Lustre: 27334:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254914/real 1491254914] req@ffff880055e6f900 x1563694005750416/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254920 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13785.285849] Lustre: server umount lustre-MDT0000 complete [13785.413391] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13795.676553] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13795.944781] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13796.205900] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13796.453950] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13796.610705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13796.674423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13796.825704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13797.087295] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13797.576246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13797.576254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13797.736596] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13797.737960] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13797.882494] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13798.144972] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13798.399366] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13798.658709] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13798.912172] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13801.155910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13801.311739] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13801.671047] Lustre: 21190:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254932/real 1491254932] req@ffff8800689a0300 x1563694005750576/t0(0) o8->lustre-OST03ff-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491254937 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13802.311364] Lustre: DEBUG MARKER: debugfs -R 'stat lov_objid' /dev/lvm-Role_MDS/P1 2>/dev/null [13803.264941] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13803.520195] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13805.255722] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [13806.703106] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800689a0c00 x1563694005750960/t0(0) o13->lustre-OST03ff-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13815.244128] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13815.692041] Lustre: 28924:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254945/real 1491254945] req@ffff880055e6fc00 x1563694005751024/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254951 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13815.740042] Lustre: server umount lustre-MDT0000 complete [13815.875276] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13826.337408] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13826.617693] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13826.906872] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13827.100089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13828.333846] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13828.612092] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13828.878370] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13829.148163] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13829.320794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13829.363355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13829.381442] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [13829.385297] Lustre: Skipped 6 previous similar messages [13829.400754] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13829.403381] Lustre: Skipped 4 previous similar messages [13829.424127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13829.426531] Lustre: Skipped 1 previous similar message [13829.442870] Lustre: lustre-MDT0000: new disk, initializing [13829.467671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13829.477665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13829.649906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13829.904806] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13830.398852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13830.411733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13830.564488] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13830.566046] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13830.704554] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13830.956217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13831.210624] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13831.487215] Lustre: DEBUG MARKER: sync; sync; sync [13836.804221] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13837.071695] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13839.414235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13839.659035] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13846.696262] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13846.947887] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13848.497175] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [13849.343068] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800689a1200 x1563694005751904/t0(0) o13->lustre-OST0800-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13858.488132] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13859.118043] Lustre: 30768:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491254988/real 1491254988] req@ffff8800689a1200 x1563694005751984/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491254994 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13859.191338] Lustre: server umount lustre-MDT0000 complete [13859.318529] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13869.725284] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13869.988499] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13870.238475] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13870.489968] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13870.648262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13870.726992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13870.879566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13871.136993] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13871.629315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13871.631910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13871.791843] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13871.791854] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13871.930230] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13872.189971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13872.466472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13872.740148] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13872.993219] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13875.276123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13875.425725] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13876.431937] Lustre: DEBUG MARKER: debugfs -R 'stat lov_objid' /dev/lvm-Role_MDS/P1 2>/dev/null [13877.392101] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13877.654547] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13879.323240] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [13880.959104] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880055e6d200 x1563694005752528/t0(0) o13->lustre-OST0800-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13889.315070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13889.846819] Lustre: server umount lustre-MDT0000 complete [13889.977245] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13900.791192] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13901.051081] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13905.028441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13905.028441] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13905.028441] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13905.341679] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13905.915675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13905.941983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13905.974835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13906.016798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13906.150624] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13906.172759] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13906.264757] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13906.266661] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13906.581192] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13906.827328] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13907.106527] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13907.302758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13913.773796] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13914.414721] Lustre: DEBUG MARKER: rc=0; [13914.414721] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13914.414721] if [[ $? -eq 0 && $val -ne 0 ]]; then [13914.414721] echo $(hostname -s): $val; [13914.414721] rc=$val; [13914.414721] fi; [13914.414721] exit $rc [13915.018195] Lustre: DEBUG MARKER: dmesg [13915.569927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 56a: check big OST indexes and out-of-index-order start ========================== 21:30:50 \(1491255050\) [13915.750282] Lustre: DEBUG MARKER: == conf-sanity test 56a: check big OST indexes and out-of-index-order start ========================== 21:30:50 (1491255050) [13915.883026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13915.883026] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13915.883026] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13916.217869] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13916.464165] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13916.737655] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-J size=16 -E la [13916.931099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13918.937241] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13919.208307] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13919.471064] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13919.729518] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13919.875319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13919.887558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13919.909880] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [13919.912009] Lustre: Skipped 11 previous similar messages [13919.930902] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13919.933091] Lustre: Skipped 4 previous similar messages [13919.956378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13919.958418] Lustre: Skipped 1 previous similar message [13919.974074] Lustre: lustre-MDT0000: new disk, initializing [13920.000916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13920.012311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13920.186224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13920.457130] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13921.041873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13921.043205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13921.282505] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13921.290955] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13921.437859] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13921.706282] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13921.975824] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13922.239803] Lustre: DEBUG MARKER: sync; sync; sync [13927.567671] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13927.843106] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13930.261567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13930.421386] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13938.636163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13938.870736] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13939.980360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13939.980360] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13939.980360] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13940.268106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13940.268106] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13940.268106] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13940.544092] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13940.804949] Lustre: DEBUG MARKER: lctl get_param -n at_min [13941.059658] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13941.581379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [13941.617228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [13941.787185] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [13941.794336] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [13941.978534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13941.983159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13942.142713] Lustre: DEBUG MARKER: osc.lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13942.143686] Lustre: DEBUG MARKER: osc.lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13942.283703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13942.283703] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13942.283703] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13942.564477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13942.564477] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13942.564477] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13942.840926] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [13943.094858] Lustre: DEBUG MARKER: lctl get_param -n at_min [13943.351426] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13943.887793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [13943.894330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [13944.189563] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [13944.194303] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [13944.374906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13944.375541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13944.533438] Lustre: DEBUG MARKER: osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13944.535662] Lustre: DEBUG MARKER: osc.lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13946.187965] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13946.445277] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13947.685297] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [13948.319082] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800651a0c00 x1563694005754048/t0(0) o13->lustre-OST2710-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13957.676195] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13958.615068] Lustre: 5201:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491255088/real 1491255088] req@ffff880055e6c900 x1563694005754160/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491255094 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [13958.623625] Lustre: 5201:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [13958.696659] Lustre: server umount lustre-MDT0000 complete [13958.828223] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13963.916060] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13964.172363] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13968.135353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13968.135353] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13968.135353] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13968.479972] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13969.081722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13969.109091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13969.130742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13969.152921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13969.385161] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13969.407464] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13969.431080] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13969.443457] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13969.756313] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13970.014765] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13970.298900] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [13970.500922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13977.402544] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13978.052307] Lustre: DEBUG MARKER: rc=0; [13978.052307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13978.052307] if [[ $? -eq 0 && $val -ne 0 ]]; then [13978.052307] echo $(hostname -s): $val; [13978.052307] rc=$val; [13978.052307] fi; [13978.052307] exit $rc [13978.676084] Lustre: DEBUG MARKER: dmesg [13979.245908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ======================== 21:31:54 \(1491255114\) [13979.399528] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ======================== 21:31:54 (1491255114) [13979.588520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_56b needs \>= 3 MDTs [13979.762569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [13979.973716] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13980.671828] Lustre: DEBUG MARKER: rc=0; [13980.671828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13980.671828] if [[ $? -eq 0 && $val -ne 0 ]]; then [13980.671828] echo $(hostname -s): $val; [13980.671828] rc=$val; [13980.671828] fi; [13980.671828] exit $rc [13981.326341] Lustre: DEBUG MARKER: dmesg [13982.039319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 57a: initial registration from failnode should fail \(should return errs\) ========= 21:31:57 \(1491255117\) [13982.217603] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========= 21:31:57 (1491255117) [13982.838663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13983.011578] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [13985.030779] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13985.280513] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13985.540024] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [13985.685046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13987.851389] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13988.112823] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13988.368608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13988.619244] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13988.781703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13988.821951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13988.858558] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13988.860852] Lustre: Skipped 4 previous similar messages [13988.880190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13988.882294] Lustre: Skipped 1 previous similar message [13988.898246] Lustre: lustre-MDT0000: new disk, initializing [13988.923469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13988.933599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13989.095139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13989.351559] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [13989.894272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13989.899938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13990.056045] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13990.062225] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [13990.201790] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13990.457199] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13990.711106] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13990.966407] Lustre: DEBUG MARKER: sync; sync; sync [13996.311230] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13996.569488] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13997.865709] Lustre: Denying initial registration attempt from nid 10.9.5.244@tcp, specified as failover [13998.666552] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13998.931937] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14005.130783] Lustre: server umount lustre-MDT0000 complete [14005.260099] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14009.320066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14009.320066] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14009.320066] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14009.644480] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14010.294120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14010.303703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14010.310641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14010.338346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14010.547417] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14010.664156] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14010.702265] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14010.757171] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14011.147320] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14011.395380] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14011.691567] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14011.891748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14018.853656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14019.479183] Lustre: DEBUG MARKER: rc=0; [14019.479183] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14019.479183] if [[ $? -eq 0 && $val -ne 0 ]]; then [14019.479183] echo $(hostname -s): $val; [14019.479183] rc=$val; [14019.479183] fi; [14019.479183] exit $rc [14020.082850] Lustre: DEBUG MARKER: dmesg [14020.722795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 57b: initial registration from servicenode should not fail ======================= 21:32:35 \(1491255155\) [14020.877845] Lustre: DEBUG MARKER: == conf-sanity test 57b: initial registration from servicenode should not fail ======================= 21:32:35 (1491255155) [14021.520428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [14021.671666] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing load_modules_local [14022.142630] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14022.391953] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14022.653932] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [14022.817813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14024.986328] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14025.242051] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14025.487938] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14025.735470] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14025.888617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14025.930793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14025.965520] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14025.967743] Lustre: Skipped 4 previous similar messages [14025.987115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14025.989165] Lustre: Skipped 1 previous similar message [14026.004853] Lustre: lustre-MDT0000: new disk, initializing [14026.035464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14026.207403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14026.470027] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14026.970213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14026.973318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14027.126514] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14027.127567] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14027.264615] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14027.516983] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14027.767065] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14028.020746] Lustre: DEBUG MARKER: sync; sync; sync [14033.361053] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14033.613580] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14035.756905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14035.915916] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14042.744969] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14043.011050] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14043.179687] LustreError: 11791:0:(ldlm_resource.c:1095:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff88006190d180) refcount nonzero (1) after lock cleanup; forcing cleanup. [14043.184823] LustreError: 11791:0:(ldlm_resource.c:1677:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x0:0x0].0x0 (ffff88006190d180) refcount = 1 [14044.875244] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14044.880194] LustreError: Skipped 1 previous similar message [14049.193059] Lustre: 11791:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491255178/real 1491255178] req@ffff880055e6db00 x1563694005755632/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491255184 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [14049.200660] Lustre: 11791:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14049.360112] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14059.733239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14059.733239] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14059.733239] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14060.079397] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14060.718109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14060.721947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14060.724864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14060.780083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14060.976098] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14060.988521] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14061.010878] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14061.104727] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14061.390276] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14061.637858] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14061.918271] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14062.071858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14068.676036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14069.299772] Lustre: DEBUG MARKER: rc=0; [14069.299772] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14069.299772] if [[ $? -eq 0 && $val -ne 0 ]]; then [14069.299772] echo $(hostname -s): $val; [14069.299772] rc=$val; [14069.299772] fi; [14069.299772] exit $rc [14069.884049] Lustre: DEBUG MARKER: dmesg [14070.415288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 58: missing llog files must not prevent MDT from mounting ======================== 21:33:25 \(1491255205\) [14070.573999] Lustre: DEBUG MARKER: == conf-sanity test 58: missing llog files must not prevent MDT from mounting ======================== 21:33:25 (1491255205) [14070.774920] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14071.034261] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14071.285625] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14071.539467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14071.695176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14071.734535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.753310] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [14071.755448] Lustre: Skipped 13 previous similar messages [14071.771131] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14071.773346] Lustre: Skipped 4 previous similar messages [14071.792820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14071.794834] Lustre: Skipped 1 previous similar message [14071.810680] Lustre: lustre-MDT0000: new disk, initializing [14071.831793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14071.833861] Lustre: Skipped 1 previous similar message [14071.842459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14072.004575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14072.260738] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14072.848129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14072.851135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14073.016329] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14073.021945] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14073.170531] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14073.427428] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14073.688547] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14073.939265] Lustre: DEBUG MARKER: sync; sync; sync [14079.249702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14079.513314] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14081.758961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14081.934165] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14088.455596] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x601 [14088.945911] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14089.196454] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14090.840092] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [14090.842449] Lustre: Skipped 1 previous similar message [14091.535082] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880055e6c000 x1563694005756528/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14091.540207] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [14098.100595] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14101.433340] Lustre: server umount lustre-MDT0000 complete [14101.435663] Lustre: Skipped 1 previous similar message [14101.569848] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14101.865112] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14102.130490] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14102.267198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14102.400191] Lustre: DEBUG MARKER: find /mnt/lustre-mds1/O/1/d* -type f -delete [14102.681224] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [14102.974145] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14103.255883] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14103.520652] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14103.789185] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14103.941090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14104.156733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14104.417273] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14104.941523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14104.972686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14105.157507] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14105.195410] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14105.340037] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14105.605815] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14105.871390] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14106.139198] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14106.412594] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14106.680252] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14107.662298] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14107.913160] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14109.007072] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064149500 x1563694005757072/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14112.841524] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [14120.233086] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14128.324822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14128.324822] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14128.324822] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14128.641709] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14129.232256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14129.243739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14129.254068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14129.260364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14129.475192] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14129.495044] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14129.529559] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14129.545303] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14129.836839] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14130.087279] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14130.368631] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14130.560400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14137.137397] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14137.770956] Lustre: DEBUG MARKER: rc=0; [14137.770956] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14137.770956] if [[ $? -eq 0 && $val -ne 0 ]]; then [14137.770956] echo $(hostname -s): $val; [14137.770956] rc=$val; [14137.770956] fi; [14137.770956] exit $rc [14138.354126] Lustre: DEBUG MARKER: dmesg [14138.929517] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 59: writeconf mount option ======================================================= 21:34:34 \(1491255274\) [14139.084173] Lustre: DEBUG MARKER: == conf-sanity test 59: writeconf mount option ======================================================= 21:34:34 (1491255274) [14139.241134] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14139.500886] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14139.748339] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14139.996703] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14140.150591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14140.187692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14140.223111] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14140.225389] Lustre: Skipped 4 previous similar messages [14140.246251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14140.248300] Lustre: Skipped 1 previous similar message [14140.263959] Lustre: lustre-MDT0000: new disk, initializing [14140.298084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14140.460268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14140.712289] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14141.201753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14141.211756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14141.365320] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14141.376051] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14141.508796] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14141.766503] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14142.030681] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14142.287530] Lustre: DEBUG MARKER: sync; sync; sync [14147.565634] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14147.836039] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14148.094631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.lustre [14150.366278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14150.523783] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14156.761664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.lustre [14157.677529] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14157.947116] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14159.465391] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14159.470190] LustreError: Skipped 1 previous similar message [14164.297133] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14174.537121] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14174.796880] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14175.064921] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14175.326283] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o writeconf /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14175.490658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14175.527821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14175.712466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14175.970675] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14176.460582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14176.465867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14176.626437] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14176.626474] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14176.768347] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14177.021284] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14177.284211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14177.547126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14177.804499] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14178.060326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.lustre [14179.295241] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [14180.503032] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14181.480917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14181.654436] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14182.576604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.lustre [14183.909832] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [14184.848025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14185.019451] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14191.600054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.lustre [14195.631953] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14195.634836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14201.485470] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14201.737037] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14207.900069] Lustre: 21210:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491255337/real 1491255337] req@ffff880064148900 x1563694005759568/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491255343 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [14207.907782] Lustre: 21210:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [14208.074438] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14213.738917] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14213.999374] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14214.256996] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [14214.416086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14216.431141] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14217.061174] Lustre: DEBUG MARKER: rc=0; [14217.061174] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14217.061174] if [[ $? -eq 0 && $val -ne 0 ]]; then [14217.061174] echo $(hostname -s): $val; [14217.061174] rc=$val; [14217.061174] fi; [14217.061174] exit $rc [14217.673118] Lustre: DEBUG MARKER: dmesg [14219.073335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ======================== 21:35:53 \(1491255353\) [14219.224943] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ======================== 21:35:53 (1491255353) [14219.414762] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14219.676650] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14219.960411] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14221.162061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14221.316301] Lustre: DEBUG MARKER: dumpe2fs /dev/lvm-Role_MDS/P1 [14222.160018] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14222.409615] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14226.506886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14226.506886] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14226.506886] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14226.885749] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14227.476956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14227.502855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14227.511796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14227.535165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14227.715397] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14227.771313] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14227.809388] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14227.819073] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14229.735806] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14229.996124] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14230.284104] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14230.489924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14237.171089] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14237.844414] Lustre: DEBUG MARKER: rc=0; [14237.844414] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14237.844414] if [[ $? -eq 0 && $val -ne 0 ]]; then [14237.844414] echo $(hostname -s): $val; [14237.844414] rc=$val; [14237.844414] fi; [14237.844414] exit $rc [14238.456078] Lustre: DEBUG MARKER: dmesg [14239.001176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 61: large xattr ================================================================== 21:36:14 \(1491255374\) [14239.165556] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ================================================================== 21:36:14 (1491255374) [14239.304982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14239.304982] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14239.304982] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14239.593850] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [14239.593850] grep -E -q '(ea_inode|large_xattr)' [14239.867102] Lustre: DEBUG MARKER: tune2fs -O large_xattr /dev/lvm-Role_MDS/P1 [14240.179254] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14240.452434] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14240.709152] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14240.959229] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14241.120114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14241.132501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14241.168110] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14241.170308] Lustre: Skipped 9 previous similar messages [14241.194133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14241.196219] Lustre: Skipped 1 previous similar message [14241.213028] Lustre: lustre-MDT0000: new disk, initializing [14241.238217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14241.240262] Lustre: Skipped 3 previous similar messages [14241.249654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14241.427384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14241.695212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14242.196969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14242.198755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14242.360983] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14242.370835] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14242.511241] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14242.773739] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14243.043263] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14243.309088] Lustre: DEBUG MARKER: sync; sync; sync [14248.591057] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14248.855675] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14251.216972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14251.376064] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14259.035232] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [14259.035232] grep -E -q '(ea_inode|large_xattr)' [14259.360505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save large xattr trusted.big on \/mnt\/lustre\/f61.conf-sanity [14259.520562] Lustre: DEBUG MARKER: save large xattr trusted.big on /mnt/lustre/f61.conf-sanity [14259.748632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark shrink value of trusted.big on \/mnt\/lustre\/f61.conf-sanity [14259.897995] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [14260.093019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow value of trusted.big on \/mnt\/lustre\/f61.conf-sanity [14260.244832] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [14260.464247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check value of trusted.big on \/mnt\/lustre\/f61.conf-sanity after remounting MDS [14260.619714] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [14260.758965] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14261.016628] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [14261.147273] Lustre: Failing over lustre-MDT0000 [14265.262094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14267.268084] Lustre: server umount lustre-MDT0000 complete [14267.270608] Lustre: Skipped 3 previous similar messages [14267.398981] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14277.723522] Lustre: DEBUG MARKER: hostname [14278.031866] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14278.277038] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14278.523225] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14278.666696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14278.885609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14279.139571] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14279.636507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14279.658447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14279.811866] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14279.824105] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14279.959524] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14280.211267] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14280.465654] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14280.634173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [14280.647248] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [14280.731281] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14281.482036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14281.517981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14281.717894] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14281.725045] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14281.948292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14282.106434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14283.440784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark remove large xattr trusted.big from \/mnt\/lustre\/f61.conf-sanity [14283.604181] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [14283.750944] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14284.005956] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [14284.135260] Lustre: Failing over lustre-MDT0000 [14290.351713] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14300.658617] Lustre: DEBUG MARKER: hostname [14300.979658] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14301.256702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14301.527492] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14301.684083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14301.909089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14302.174894] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14302.675229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14302.675272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14302.841241] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14302.843527] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14302.988128] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14303.252141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14303.505767] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14303.759094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14304.544942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14304.553417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14304.729399] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14304.810450] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14308.480921] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [14308.498231] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [14308.968245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [14309.119776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [14312.901232] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14313.152747] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14313.535110] LustreError: 21191:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880055e6d500 x1563694005761712/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14314.285812] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [14325.477695] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14335.600646] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [14335.994155] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14336.264267] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14336.516744] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14336.762072] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14336.915884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14336.939110] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [14336.941658] Lustre: Skipped 35 previous similar messages [14337.126433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14337.381231] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14337.871118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14337.872538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14338.029095] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14338.030408] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14338.167161] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14338.419344] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14338.671553] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14338.928476] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14339.181063] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14341.290813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14341.447745] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14348.111539] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14348.361817] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14362.723549] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14367.113719] Lustre: DEBUG MARKER: tune2fs -O ^large_xattr /dev/lvm-Role_MDS/P1 [14367.473087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14368.100030] Lustre: DEBUG MARKER: rc=0; [14368.100030] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14368.100030] if [[ $? -eq 0 && $val -ne 0 ]]; then [14368.100030] echo $(hostname -s): $val; [14368.100030] rc=$val; [14368.100030] fi; [14368.100030] exit $rc [14368.705874] Lustre: DEBUG MARKER: dmesg [14369.285811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 62: start with disabled journal ================================================== 21:38:24 \(1491255504\) [14369.444223] Lustre: DEBUG MARKER: == conf-sanity test 62: start with disabled journal ================================================== 21:38:24 (1491255504) [14369.591800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14369.591800] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14369.591800] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14369.860817] Lustre: DEBUG MARKER: tune2fs -O ^has_journal /dev/lvm-Role_MDS/P1 [14370.133395] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14370.400164] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14370.656968] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14370.904340] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14371.057865] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.060326] LustreError: 31291:0:(osd_handler.c:6894:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/lvm--Role_MDS-P1 is mounted w/o journal [14371.065373] LustreError: 31291:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000-osd failed (-22) [14371.067732] LustreError: 31291:0:(obd_mount.c:199:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14371.070074] LustreError: 31291:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -22 [14371.074157] LustreError: 31291:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-22) [14371.199687] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14373.396017] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14373.647107] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14380.338056] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14380.599593] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14384.798350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14384.798350] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14384.798350] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14385.129591] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14385.737916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14385.744867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14385.808580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14385.838838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14385.983134] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14386.016195] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14386.074766] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14386.128753] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14388.791083] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14389.059952] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14389.340999] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14389.537260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14396.405223] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14396.662758] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14396.913430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14397.158245] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14397.300920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14397.313875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14397.348538] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14397.350902] Lustre: Skipped 4 previous similar messages [14397.370309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14397.372463] Lustre: Skipped 1 previous similar message [14397.388076] Lustre: lustre-MDT0000: new disk, initializing [14397.419520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14397.581990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14397.836323] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14398.322253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14398.327158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14398.480165] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14398.480252] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14398.620043] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14398.871698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14399.127848] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14399.381254] Lustre: DEBUG MARKER: sync; sync; sync [14404.657762] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14404.907957] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14407.408901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14407.561379] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14413.769181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14413.769181] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14413.769181] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14414.051155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14414.051155] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14414.051155] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14414.325790] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [14414.622539] Lustre: DEBUG MARKER: lctl get_param -n at_min [14414.925789] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14415.437765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [14415.437842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [14415.598979] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [14415.613390] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [14415.804247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14415.805560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14415.973250] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14415.974844] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14419.119914] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14419.122821] LustreError: Skipped 1 previous similar message [14419.125332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14419.130630] Lustre: Skipped 1 previous similar message [14422.946034] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14423.208224] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14429.539755] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14429.847462] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14430.466067] Lustre: DEBUG MARKER: rc=0; [14430.466067] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14430.466067] if [[ $? -eq 0 && $val -ne 0 ]]; then [14430.466067] echo $(hostname -s): $val; [14430.466067] rc=$val; [14430.466067] fi; [14430.466067] exit $rc [14431.075478] Lustre: DEBUG MARKER: dmesg [14431.604437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 63: Verify each page can at least hold 3 ldiskfs inodes ========================== 21:39:26 \(1491255566\) [14431.771610] Lustre: DEBUG MARKER: == conf-sanity test 63: Verify each page can at least hold 3 ldiskfs inodes ========================== 21:39:26 (1491255566) [14431.931712] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14432.410914] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_module ldiskfs [14432.423796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing load_module ldiskfs [14432.569267] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_module ldiskfs [14432.580914] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing load_module ldiskfs [14432.716782] Lustre: DEBUG MARKER: cat /proc/slabinfo [14433.102573] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14433.742409] Lustre: DEBUG MARKER: rc=0; [14433.742409] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14433.742409] if [[ $? -eq 0 && $val -ne 0 ]]; then [14433.742409] echo $(hostname -s): $val; [14433.742409] rc=$val; [14433.742409] fi; [14433.742409] exit $rc [14434.337964] Lustre: DEBUG MARKER: dmesg [14434.899563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 64: check lfs df --lazy ========================================================== 21:39:30 \(1491255570\) [14435.051058] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ========================================================== 21:39:30 (1491255570) [14435.206389] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14435.476119] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14435.725436] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14435.971348] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14436.114760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14436.323219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14436.579843] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14437.074903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14437.081161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14437.235305] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14437.237695] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14437.375729] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14437.625037] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14437.877465] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14438.146439] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14438.411904] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14440.620230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14440.793450] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14443.777258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14443.939194] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14454.304301] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14454.307125] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14469.327800] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14469.330477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14474.422291] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14474.675444] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14486.839068] Lustre: 5332:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491255616/real 1491255616] req@ffff88005a994c00 x1563694005764864/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491255622 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [14486.846541] Lustre: 5332:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [14486.998430] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14491.986699] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14492.233674] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14492.489979] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [14492.644761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14494.547469] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14495.171330] Lustre: DEBUG MARKER: rc=0; [14495.171330] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14495.171330] if [[ $? -eq 0 && $val -ne 0 ]]; then [14495.171330] echo $(hostname -s): $val; [14495.171330] rc=$val; [14495.171330] fi; [14495.171330] exit $rc [14495.761287] Lustre: DEBUG MARKER: dmesg [14497.093909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================== 21:40:31 \(1491255631\) [14497.244484] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================== 21:40:31 (1491255631) [14497.384180] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14497.639518] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14497.884563] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14498.138344] Lustre: DEBUG MARKER: debugfs -c -R "stat last_rcvd" /dev/lvm-Role_MDS/P1 [14498.388702] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [14498.637125] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [14498.774943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14498.900092] Lustre: DEBUG MARKER: rm -f /mnt/lustre-brpt/last_rcvd [14499.145160] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [14499.418599] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14501.368416] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14501.639081] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14501.910447] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14502.072844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14502.109964] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14502.147954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14502.150321] Lustre: Skipped 5 previous similar messages [14502.311607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14502.588392] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14503.083376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14503.100636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14503.257152] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14503.267046] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14503.404726] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14503.657118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14503.908220] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14504.166766] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14504.422815] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14504.689363] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14504.940197] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14511.270813] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14511.545262] Lustre: DEBUG MARKER: debugfs -c -R "stat last_rcvd" /dev/lvm-Role_MDS/P1 [14511.883430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14512.532487] Lustre: DEBUG MARKER: rc=0; [14512.532487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14512.532487] if [[ $? -eq 0 && $val -ne 0 ]]; then [14512.532487] echo $(hostname -s): $val; [14512.532487] rc=$val; [14512.532487] fi; [14512.532487] exit $rc [14513.129922] Lustre: DEBUG MARKER: dmesg [14513.694484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 66: replace nids ================================================================= 21:40:48 \(1491255648\) [14513.851611] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ================================================================= 21:40:48 (1491255648) [14513.994183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14513.994183] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14513.994183] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14514.270561] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14514.542899] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14514.803658] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14515.050611] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14515.206926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14515.446214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14515.698446] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14516.181291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14516.183739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14516.339174] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14516.342598] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14516.478905] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14516.731503] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14516.989389] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14517.259234] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14517.519390] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14518.755917] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14519.733106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14519.984742] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14527.460175] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [14527.712022] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 10.9.5.244@tcp [14527.839822] LustreError: 9868:0:(mgs_llog.c:1460:mgs_replace_nids()) Only MGS is allowed to be started [14527.842190] LustreError: 9868:0:(mgs_handler.c:1061:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [14528.093414] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 10.9.5.244@tcp [14531.215810] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14531.218368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14535.182780] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 10.9.5.244@tcp [14535.307827] LustreError: 10014:0:(mgs_llog.c:1460:mgs_replace_nids()) Only MGS is allowed to be started [14535.310368] LustreError: 10014:0:(mgs_llog.c:1460:mgs_replace_nids()) Skipped 1 previous similar message [14535.312842] LustreError: 10014:0:(mgs_handler.c:1061:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [14535.315350] LustreError: 10014:0:(mgs_handler.c:1061:mgs_iocontrol()) Skipped 1 previous similar message [14535.453345] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14535.709235] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14541.915355] Lustre: server umount lustre-MDT0000 complete [14541.917602] Lustre: Skipped 6 previous similar messages [14542.045516] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14542.330084] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14542.586469] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14542.839672] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14543.090104] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o nosvc /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14543.233892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14543.379202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14543.633650] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14544.113127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14544.120672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14544.270802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14544.270837] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14544.409709] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14544.663027] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14544.911673] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14545.166380] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 [14545.417314] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-WRONG0000 10.9.5.244@tcp [14545.542843] LustreError: 11419:0:(mgs_handler.c:1061:mgs_iocontrol()) MGS: error replacing nids: rc = -22 [14545.668769] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 wrong nids list [14545.917230] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-OST0000 10.9.5.244@tcp [14546.169688] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-MDT0000 [14546.422948] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-MDT0000 wrong nids list [14546.670045] Lustre: DEBUG MARKER: /usr/sbin/lctl replace_nids lustre-MDT0000 10.9.5.243@tcp [14546.962789] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14547.210894] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14553.487853] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14553.787567] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14554.049225] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14554.296555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14554.542239] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14554.694032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14554.927670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14555.181453] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14555.677901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14555.678165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14555.836927] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14555.838499] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14555.978262] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14556.232384] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14556.482402] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14556.738846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14556.991712] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14559.076327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14559.237595] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14566.351738] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14566.354271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [14572.252917] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14572.517274] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14578.907049] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14586.180802] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14586.433826] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14590.447623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14590.447623] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14590.447623] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14590.782283] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14591.390252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14591.397773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14591.420330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14591.422916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14591.666111] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14591.690575] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14591.698675] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14591.710479] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14594.392842] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14594.638679] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14594.919411] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14595.077778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14601.707208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14602.391334] Lustre: DEBUG MARKER: rc=0; [14602.391334] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14602.391334] if [[ $? -eq 0 && $val -ne 0 ]]; then [14602.391334] echo $(hostname -s): $val; [14602.391334] rc=$val; [14602.391334] fi; [14602.391334] exit $rc [14603.004563] Lustre: DEBUG MARKER: dmesg [14603.638386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 67: test routes conversion and configuration ===================================== 21:42:18 \(1491255738\) [14603.792610] Lustre: DEBUG MARKER: == conf-sanity test 67: test routes conversion and configuration ===================================== 21:42:18 (1491255738) [14604.090720] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14604.707029] Lustre: DEBUG MARKER: rc=0; [14604.707029] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14604.707029] if [[ $? -eq 0 && $val -ne 0 ]]; then [14604.707029] echo $(hostname -s): $val; [14604.707029] rc=$val; [14604.707029] fi; [14604.707029] exit $rc [14605.308740] Lustre: DEBUG MARKER: dmesg [14605.912207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 68: be able to reserve specific sequences in FLDB ================================ 21:42:20 \(1491255740\) [14606.067183] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ================================ 21:42:20 (1491255740) [14606.211969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14606.211969] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14606.211969] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14606.506507] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14606.770127] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14607.018244] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14607.268856] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14607.414259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14607.427072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14607.686246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14607.945270] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14608.445374] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14608.448772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14608.603361] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14608.611057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14608.748797] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14609.010743] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14609.264711] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14609.517909] Lustre: DEBUG MARKER: sync; sync; sync [14614.799788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14615.060276] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14617.277223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14617.479173] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14623.680210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n seq.ctl*.space [14623.934619] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.ctl*.fldb=[0x240000400-10737419264\):0:mdt [14624.081481] Lustre: ctl-lustre-MDT0000: [0x0000000240000400-0x0000000280000400]:0:mdt sequences allocated: rc = 0 [14624.208136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.srv*MDT0000.space=clear [14624.607201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param seq.srv*MDT0000.space [14631.829485] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14632.081949] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14638.520757] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14644.951343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14645.577676] Lustre: DEBUG MARKER: rc=0; [14645.577676] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14645.577676] if [[ $? -eq 0 && $val -ne 0 ]]; then [14645.577676] echo $(hostname -s): $val; [14645.577676] rc=$val; [14645.577676] fi; [14645.577676] exit $rc [14646.189732] Lustre: DEBUG MARKER: dmesg [14647.520546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_69 skipping SLOW test 69 [14647.679336] Lustre: DEBUG MARKER: SKIP: conf-sanity test_69 skipping SLOW test 69 [14647.886697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 70a: start MDT0, then OST, then MDT1 ============================================= 21:43:03 \(1491255783\) [14648.053930] Lustre: DEBUG MARKER: == conf-sanity test 70a: start MDT0, then OST, then MDT1 ============================================= 21:43:03 (1491255783) [14648.236226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_70a needs \>= 2 MDTs [14648.387400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_70a needs >= 2 MDTs [14648.572257] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14649.184822] Lustre: DEBUG MARKER: rc=0; [14649.184822] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14649.184822] if [[ $? -eq 0 && $val -ne 0 ]]; then [14649.184822] echo $(hostname -s): $val; [14649.184822] rc=$val; [14649.184822] fi; [14649.184822] exit $rc [14649.770690] Lustre: DEBUG MARKER: dmesg [14650.356057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 70b: start OST, MDT1, MDT0 ======================================================= 21:43:05 \(1491255785\) [14650.501064] Lustre: DEBUG MARKER: == conf-sanity test 70b: start OST, MDT1, MDT0 ======================================================= 21:43:05 (1491255785) [14650.699232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_70b needs \>= 2 MDTs [14650.845398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_70b needs >= 2 MDTs [14651.031430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14651.639564] Lustre: DEBUG MARKER: rc=0; [14651.639564] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14651.639564] if [[ $? -eq 0 && $val -ne 0 ]]; then [14651.639564] echo $(hostname -s): $val; [14651.639564] rc=$val; [14651.639564] fi; [14651.639564] exit $rc [14652.214882] Lustre: DEBUG MARKER: dmesg [14652.738387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 70c: stop MDT0, mkdir fail, create remote dir fail =============================== 21:43:07 \(1491255787\) [14652.885828] Lustre: DEBUG MARKER: == conf-sanity test 70c: stop MDT0, mkdir fail, create remote dir fail =============================== 21:43:07 (1491255787) [14653.063816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_70c needs \>= 2 MDTs [14653.215070] Lustre: DEBUG MARKER: SKIP: conf-sanity test_70c needs >= 2 MDTs [14653.401235] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14654.019352] Lustre: DEBUG MARKER: rc=0; [14654.019352] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14654.019352] if [[ $? -eq 0 && $val -ne 0 ]]; then [14654.019352] echo $(hostname -s): $val; [14654.019352] rc=$val; [14654.019352] fi; [14654.019352] exit $rc [14654.639669] Lustre: DEBUG MARKER: dmesg [14655.220860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 70d: stop MDT1, mkdir succeed, create remote dir fail ============================ 21:43:10 \(1491255790\) [14655.371884] Lustre: DEBUG MARKER: == conf-sanity test 70d: stop MDT1, mkdir succeed, create remote dir fail ============================ 21:43:10 (1491255790) [14655.549504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_70d needs \>= 2 MDTs [14655.692917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_70d needs >= 2 MDTs [14655.886550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14656.495335] Lustre: DEBUG MARKER: rc=0; [14656.495335] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14656.495335] if [[ $? -eq 0 && $val -ne 0 ]]; then [14656.495335] echo $(hostname -s): $val; [14656.495335] rc=$val; [14656.495335] fi; [14656.495335] exit $rc [14657.075764] Lustre: DEBUG MARKER: dmesg [14657.618168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 70e: Sync-on-Cancel will be enabled by default on DNE ============================ 21:43:12 \(1491255792\) [14657.770806] Lustre: DEBUG MARKER: == conf-sanity test 70e: Sync-on-Cancel will be enabled by default on DNE ============================ 21:43:12 (1491255792) [14657.952678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_70e needs \>= 2 MDTs [14658.100230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_70e needs >= 2 MDTs [14658.292271] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14658.924512] Lustre: DEBUG MARKER: rc=0; [14658.924512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14658.924512] if [[ $? -eq 0 && $val -ne 0 ]]; then [14658.924512] echo $(hostname -s): $val; [14658.924512] rc=$val; [14658.924512] fi; [14658.924512] exit $rc [14659.506031] Lustre: DEBUG MARKER: dmesg [14660.077839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 71a: start MDT0 OST0, MDT1, OST1 ================================================= 21:43:15 \(1491255795\) [14660.230692] Lustre: DEBUG MARKER: == conf-sanity test 71a: start MDT0 OST0, MDT1, OST1 ================================================= 21:43:15 (1491255795) [14660.409406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_71a needs \>= 2 MDTs [14660.556370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_71a needs >= 2 MDTs [14660.738684] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14661.360365] Lustre: DEBUG MARKER: rc=0; [14661.360365] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14661.360365] if [[ $? -eq 0 && $val -ne 0 ]]; then [14661.360365] echo $(hostname -s): $val; [14661.360365] rc=$val; [14661.360365] fi; [14661.360365] exit $rc [14661.942209] Lustre: DEBUG MARKER: dmesg [14662.541051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 71b: start MDT1, OST0, MDT0, OST1 ================================================ 21:43:17 \(1491255797\) [14662.712709] Lustre: DEBUG MARKER: == conf-sanity test 71b: start MDT1, OST0, MDT0, OST1 ================================================ 21:43:17 (1491255797) [14662.892255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_71b needs \>= 2 MDTs [14663.048609] Lustre: DEBUG MARKER: SKIP: conf-sanity test_71b needs >= 2 MDTs [14663.233149] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14663.848768] Lustre: DEBUG MARKER: rc=0; [14663.848768] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14663.848768] if [[ $? -eq 0 && $val -ne 0 ]]; then [14663.848768] echo $(hostname -s): $val; [14663.848768] rc=$val; [14663.848768] fi; [14663.848768] exit $rc [14664.428584] Lustre: DEBUG MARKER: dmesg [14664.984115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 71c: start OST0, OST1, MDT1, MDT0 ================================================ 21:43:20 \(1491255800\) [14665.141462] Lustre: DEBUG MARKER: == conf-sanity test 71c: start OST0, OST1, MDT1, MDT0 ================================================ 21:43:20 (1491255800) [14665.324949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_71c needs \>= 2 MDTs [14665.468100] Lustre: DEBUG MARKER: SKIP: conf-sanity test_71c needs >= 2 MDTs [14665.650513] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14666.262074] Lustre: DEBUG MARKER: rc=0; [14666.262074] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14666.262074] if [[ $? -eq 0 && $val -ne 0 ]]; then [14666.262074] echo $(hostname -s): $val; [14666.262074] rc=$val; [14666.262074] fi; [14666.262074] exit $rc [14666.838202] Lustre: DEBUG MARKER: dmesg [14667.397974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 71d: start OST0, MDT1, MDT0, OST1 ================================================ 21:43:22 \(1491255802\) [14667.550289] Lustre: DEBUG MARKER: == conf-sanity test 71d: start OST0, MDT1, MDT0, OST1 ================================================ 21:43:22 (1491255802) [14667.735710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_71d needs \>= 2 MDTs [14667.885229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_71d needs >= 2 MDTs [14668.078303] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14668.707566] Lustre: DEBUG MARKER: rc=0; [14668.707566] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14668.707566] if [[ $? -eq 0 && $val -ne 0 ]]; then [14668.707566] echo $(hostname -s): $val; [14668.707566] rc=$val; [14668.707566] fi; [14668.707566] exit $rc [14669.304448] Lustre: DEBUG MARKER: dmesg [14669.856995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 71e: start OST0, MDT1, OST1, MDT0 ================================================ 21:43:24 \(1491255804\) [14670.010504] Lustre: DEBUG MARKER: == conf-sanity test 71e: start OST0, MDT1, OST1, MDT0 ================================================ 21:43:24 (1491255804) [14670.198493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_71e needs \>= 2 MDTs [14670.356715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_71e needs >= 2 MDTs [14670.542294] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14671.162264] Lustre: DEBUG MARKER: rc=0; [14671.162264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14671.162264] if [[ $? -eq 0 && $val -ne 0 ]]; then [14671.162264] echo $(hostname -s): $val; [14671.162264] rc=$val; [14671.162264] fi; [14671.162264] exit $rc [14671.734964] Lustre: DEBUG MARKER: dmesg [14672.269075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 72: test fast symlink with extents flag enabled ================================== 21:43:27 \(1491255807\) [14672.422793] Lustre: DEBUG MARKER: == conf-sanity test 72: test fast symlink with extents flag enabled ================================== 21:43:27 (1491255807) [14672.658395] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14672.915599] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14673.194371] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14673.385115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14673.531560] Lustre: DEBUG MARKER: tune2fs -O extents /dev/lvm-Role_MDS/P1 [14674.711159] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14676.640447] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14676.889333] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14677.137421] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14677.279523] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [14677.293397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14677.341891] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [14677.344898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14677.379596] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14677.381840] Lustre: Skipped 14 previous similar messages [14677.402719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14677.404790] Lustre: Skipped 3 previous similar messages [14677.420554] Lustre: lustre-MDT0000: new disk, initializing [14677.422560] Lustre: Skipped 2 previous similar messages [14677.452368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14677.454724] Lustre: Skipped 2 previous similar messages [14677.626188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14677.900842] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14678.427302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14678.436513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14678.589912] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14678.592112] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14678.733298] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14678.991416] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14679.276297] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14679.535686] Lustre: DEBUG MARKER: sync; sync; sync [14684.841099] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14685.105353] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14687.319843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14687.481251] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14693.980370] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14694.240251] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14696.413776] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [14696.416293] Lustre: Skipped 2 previous similar messages [14696.671083] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880055e6f600 x1563694005768816/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14696.676457] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [14706.401185] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14706.409593] LustreError: Skipped 4 previous similar messages [14706.579435] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14713.675256] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [14714.008648] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14714.645350] Lustre: DEBUG MARKER: rc=0; [14714.645350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14714.645350] if [[ $? -eq 0 && $val -ne 0 ]]; then [14714.645350] echo $(hostname -s): $val; [14714.645350] rc=$val; [14714.645350] fi; [14714.645350] exit $rc [14715.246134] Lustre: DEBUG MARKER: dmesg [14715.798149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 73: failnode to update from mountdata properly =================================== 21:44:10 \(1491255850\) [14715.948240] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly =================================== 21:44:10 (1491255850) [14716.416349] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14716.672538] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14716.923772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14717.171895] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14717.316045] LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 [14717.330594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14717.555563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14717.826116] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14718.384512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14718.396474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14718.550511] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14718.563371] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14718.698953] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14718.964749] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14719.232626] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14719.501684] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14719.770349] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14721.110335] Lustre: Found index 0 for lustre-OST0000, updating log [14722.049767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14722.199269] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14728.933269] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14729.191175] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14741.554086] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14752.502666] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14752.751615] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14756.730637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14756.730637] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14756.730637] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14757.075140] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14757.654133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14757.707678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14757.713444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14757.740404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14757.910039] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14757.977127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14757.999369] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14758.016043] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14758.339948] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14758.596817] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14758.883305] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14759.037620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14765.796390] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14766.433253] Lustre: DEBUG MARKER: rc=0; [14766.433253] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14766.433253] if [[ $? -eq 0 && $val -ne 0 ]]; then [14766.433253] echo $(hostname -s): $val; [14766.433253] rc=$val; [14766.433253] fi; [14766.433253] exit $rc [14767.034974] Lustre: DEBUG MARKER: dmesg [14767.656508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 75: The order of --index should be irrelevant ==================================== 21:45:02 \(1491255902\) [14767.832146] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ==================================== 21:45:02 (1491255902) [14767.979133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14767.979133] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14767.979133] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14768.340937] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14768.593610] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14768.883558] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14769.081628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14770.142577] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14770.397300] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14770.684625] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --index=0 --mdt --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [14770.853076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14771.908167] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14772.548345] Lustre: DEBUG MARKER: rc=0; [14772.548345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14772.548345] if [[ $? -eq 0 && $val -ne 0 ]]; then [14772.548345] echo $(hostname -s): $val; [14772.548345] rc=$val; [14772.548345] fi; [14772.548345] exit $rc [14773.149396] Lustre: DEBUG MARKER: dmesg [14773.746502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 76a: set permanent params set_param -P =========================================== 21:45:08 \(1491255908\) [14773.897812] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params set_param -P =========================================== 21:45:08 (1491255908) [14774.035240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14774.035240] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14774.035240] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14774.324615] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14774.599583] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14774.856031] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14775.106030] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14775.266865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14775.304673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14775.569519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14775.832361] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14776.336117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14776.337702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14776.498237] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14776.500165] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14776.644940] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14776.913161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14777.169525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14777.427343] Lustre: DEBUG MARKER: sync; sync; sync [14782.711544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14782.970943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14785.222313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14785.381252] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14792.957602] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P osc.*.max_dirty_mb=64 [14803.067060] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14803.330323] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14815.663461] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14827.915545] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14827.915545] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14827.915545] if [ $running -ne $mpts ]; then [14827.915545] echo $(hostname) env are INSANE!; [14827.915545] exit 1; [14827.915545] fi [14828.223175] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14828.223175] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14828.223175] if [ $running -ne $mpts ]; then [14828.223175] echo $(hostname) env are INSANE!; [14828.223175] exit 1; [14828.223175] fi [14829.211394] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14829.473210] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14829.727802] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14829.982586] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14830.185015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14830.431777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14830.703489] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14831.216349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14831.224503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14831.380906] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14831.385269] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14831.530463] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14831.803194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14832.073847] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14832.333564] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14832.608861] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14834.799076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14834.944435] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14837.751394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14837.901630] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14846.154250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14846.319786] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14853.571468] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [14853.574041] Lustre: Skipped 60 previous similar messages [14854.500735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14854.659247] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14862.847127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14863.004073] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14871.253152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14871.418207] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14879.888962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14880.047323] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14887.572661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14887.734378] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14889.179203] Lustre: DEBUG MARKER: lctl get_param -n timeout [14889.507811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14889.666231] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14889.802543] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14890.106227] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [14901.765138] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P obdfilter.*.client_cache_count=256 [14908.685644] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14908.946773] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14910.127175] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079445e00 x1563694005776848/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14910.132728] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [14913.937426] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [14913.940278] Lustre: Skipped 5 previous similar messages [14921.334190] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14929.449280] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14929.449280] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14929.449280] if [ $running -ne $mpts ]; then [14929.449280] echo $(hostname) env are INSANE!; [14929.449280] exit 1; [14929.449280] fi [14929.760875] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14929.760875] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14929.760875] if [ $running -ne $mpts ]; then [14929.760875] echo $(hostname) env are INSANE!; [14929.760875] exit 1; [14929.760875] fi [14930.732562] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14930.997194] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14931.255455] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14931.509021] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14931.665068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14931.889442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14932.150456] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14932.647077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14932.647088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14932.804963] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14932.807650] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14932.946334] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14933.202073] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14933.461413] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14933.726122] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14933.984486] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14936.159269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14936.313040] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14939.118048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14939.270278] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14942.194072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14942.367047] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14945.255609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14945.402612] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14948.311792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14948.476817] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14951.424706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14951.594948] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14954.513891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14954.677648] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14956.920848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14957.099622] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14958.551303] Lustre: DEBUG MARKER: lctl get_param -n timeout [14958.877088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14959.036952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14959.168739] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14965.999760] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14966.257661] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14973.550160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14973.550160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14973.550160] LustreError: Skipped 10 previous similar messages [14973.563597] LustreError: Skipped 1 previous similar message [14978.608700] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14984.675502] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14985.352239] Lustre: DEBUG MARKER: rc=0; [14985.352239] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14985.352239] if [[ $? -eq 0 && $val -ne 0 ]]; then [14985.352239] echo $(hostname -s): $val; [14985.352239] rc=$val; [14985.352239] fi; [14985.352239] exit $rc [14985.963083] Lustre: DEBUG MARKER: dmesg [14986.654284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 76b: verify params log setup correctly =========================================== 21:48:41 \(1491256121\) [14986.818162] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly =========================================== 21:48:41 (1491256121) [14986.958253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14986.958253] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14986.958253] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14987.876665] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14988.135386] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14994.047395] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14994.047395] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14994.047395] if [ $running -ne $mpts ]; then [14994.047395] echo $(hostname) env are INSANE!; [14994.047395] exit 1; [14994.047395] fi [14994.351432] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14994.351432] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14994.351432] if [ $running -ne $mpts ]; then [14994.351432] echo $(hostname) env are INSANE!; [14994.351432] exit 1; [14994.351432] fi [14995.329882] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14995.596801] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14995.859054] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14996.117049] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14996.280690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14996.526978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14996.810714] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [14997.300230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14997.322274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14997.475502] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14997.496101] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [14997.633877] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14997.889701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14998.171637] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14998.442044] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14998.737801] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15000.879197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15001.037107] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15001.358181] Lustre: 21190:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491256131/real 1491256131] req@ffff880037926400 x1563694005779536/t0(0) o8->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491256136 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [15001.365170] Lustre: 21190:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [15003.842117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15004.007205] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15006.854899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15007.025523] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15009.869243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15010.021543] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15012.834627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15012.991730] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15015.877872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15016.035176] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15018.909944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15019.070436] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15021.274165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15021.426486] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15022.890086] Lustre: DEBUG MARKER: lctl get_param -n timeout [15023.227588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [15023.394286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [15023.534078] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [15028.823244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mgs.MGS.live.params [15029.797361] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15030.055670] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15042.385329] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15048.440631] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15049.052181] Lustre: DEBUG MARKER: rc=0; [15049.052181] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15049.052181] if [[ $? -eq 0 && $val -ne 0 ]]; then [15049.052181] echo $(hostname -s): $val; [15049.052181] rc=$val; [15049.052181] fi; [15049.052181] exit $rc [15049.659949] Lustre: DEBUG MARKER: dmesg [15050.268819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 76c: verify changelog_mask is applied with set_param -P ========================== 21:49:45 \(1491256185\) [15050.433439] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with set_param -P ========================== 21:49:45 (1491256185) [15050.565260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15050.565260] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15050.565260] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15052.798920] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [15052.798920] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [15052.798920] if [ $running -ne $mpts ]; then [15052.798920] echo $(hostname) env are INSANE!; [15052.798920] exit 1; [15052.798920] fi [15053.117854] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [15053.117854] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [15053.117854] if [ $running -ne $mpts ]; then [15053.117854] echo $(hostname) env are INSANE!; [15053.117854] exit 1; [15053.117854] fi [15054.082765] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15054.347471] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15054.602542] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15054.854125] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15055.000069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15055.077556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15055.079573] Lustre: Skipped 9 previous similar messages [15055.226564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15055.482051] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15055.982892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15055.982899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15056.143569] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15056.143570] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15056.284427] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15056.545996] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15056.805340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15057.070512] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15057.333669] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15059.498027] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15059.653827] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15062.462650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15062.616411] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15065.387323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15065.548433] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15068.416684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15068.580520] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15071.378539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15071.527461] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15074.425029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15074.600097] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15077.466887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15077.627117] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15079.881986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15080.088372] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15081.516890] Lustre: DEBUG MARKER: lctl get_param -n timeout [15081.852993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [15082.005134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [15082.141499] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [15087.428482] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdd.*.changelog_mask=-CLOSE [15087.689665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*.changelog_mask | [15087.689665] grep 'CLOSE' [15087.947672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*.changelog_mask | [15087.947672] grep 'CLOSE' [15089.214022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*.changelog_mask | [15089.214022] grep 'CLOSE' [15090.480432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*.changelog_mask | [15090.480432] grep 'CLOSE' [15090.754498] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15091.012482] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15099.215178] Lustre: server umount lustre-MDT0000 complete [15099.217442] Lustre: Skipped 9 previous similar messages [15099.353416] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15099.634040] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15099.901974] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15100.151334] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15100.401989] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15100.552092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15100.800751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15101.065615] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15101.605052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15101.611636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15101.772594] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15101.774860] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15101.939229] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15102.202144] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15102.463674] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15102.724972] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15102.985072] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15103.243464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*.changelog_mask [15104.251883] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15104.514656] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15110.903894] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15120.222017] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15120.879509] Lustre: DEBUG MARKER: rc=0; [15120.879509] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15120.879509] if [[ $? -eq 0 && $val -ne 0 ]]; then [15120.879509] echo $(hostname -s): $val; [15120.879509] rc=$val; [15120.879509] fi; [15120.879509] exit $rc [15121.487418] Lustre: DEBUG MARKER: dmesg [15122.117496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ============================== 21:50:57 \(1491256257\) [15122.275421] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ============================== 21:50:57 (1491256257) [15122.417253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15122.417253] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15122.417253] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15122.741059] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [15122.989665] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15123.248751] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsopti [15123.440812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15123.585170] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds [15123.862658] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/S1 [15124.122740] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 [15124.373629] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-fs2mds; mount -t lustre /dev/lvm-Role_MDS/S1 /mnt/lustre-fs2mds [15124.538802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15124.574777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15124.823916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15125.086036] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15125.602189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15125.606563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15125.759432] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15125.760704] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15125.898855] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15126.162503] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15126.422181] Lustre: DEBUG MARKER: sync; sync; sync [15131.709806] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/S1 2>/dev/null [15131.979813] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids [15135.080875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15135.242163] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15141.540575] Lustre: DEBUG MARKER: grep -c /mnt/lustre-fs2mds' ' /proc/mounts [15141.794956] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-fs2mds [15154.103064] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15161.250635] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15161.899811] Lustre: DEBUG MARKER: rc=0; [15161.899811] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15161.899811] if [[ $? -eq 0 && $val -ne 0 ]]; then [15161.899811] echo $(hostname -s): $val; [15161.899811] rc=$val; [15161.899811] fi; [15161.899811] exit $rc [15162.496091] Lustre: DEBUG MARKER: dmesg [15163.116156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 78: run resize2fs on MDT and OST filesystems ===================================== 21:51:38 \(1491256298\) [15163.273286] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ===================================== 21:51:38 (1491256298) [15164.023464] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15164.270803] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15165.072878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15165.072878] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15165.072878] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15165.370111] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15165.908723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15165.927665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15165.975633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15165.989213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15166.360261] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15166.370182] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15166.420571] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15166.423144] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15166.742811] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15167.003117] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15167.301906] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=180000 --mkfsoptions="-E lazy_itable_i [15167.570833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15168.731430] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15169.003415] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15169.252517] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15169.496350] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15169.682725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15169.694966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15169.951529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15170.204570] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15170.689923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15170.690946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15170.849190] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15170.851168] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15170.988421] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15171.243934] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15171.501482] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15171.759676] Lustre: DEBUG MARKER: sync; sync; sync [15177.308864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15177.564782] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15179.853802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15180.011929] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15186.455330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create test files [15186.627156] Lustre: DEBUG MARKER: create test files [15199.561812] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15199.828035] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15212.196377] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15217.809109] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [15218.445375] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 [15218.966495] Lustre: DEBUG MARKER: resize2fs /dev/lvm-Role_MDS/P1 [15220.133448] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [15220.748452] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15223.468153] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15223.718600] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15223.970698] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15224.185470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15224.416595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15224.695573] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15225.204124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15225.209782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15225.369632] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15225.369697] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15225.509271] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15225.768084] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15226.044125] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15226.315871] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15226.588527] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15228.773539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15228.926439] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15236.451656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check files after expanding the MDT and OST filesystems [15236.615876] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [15237.058731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create more files after expanding the MDT and OST filesystems [15237.254936] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [15239.279710] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [15239.282244] LustreError: Skipped 1 previous similar message [15239.284310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [15239.288915] Lustre: Skipped 1 previous similar message [15245.475902] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15245.733454] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15252.072220] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15258.516709] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [15259.128767] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 [15259.657863] Lustre: DEBUG MARKER: resize2fs -P /dev/lvm-Role_MDS/P1 [15260.162063] Lustre: DEBUG MARKER: resize2fs /dev/lvm-Role_MDS/P1 302538 [15260.775773] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [15261.347761] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15264.085757] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15264.338315] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15264.592365] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15264.749430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15265.193062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15265.447819] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15265.935419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15265.935421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15266.097444] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15266.097591] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15266.234561] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15266.491680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15266.743984] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15267.008139] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15267.268537] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15269.563461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15269.722773] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15277.056623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check files after shrinking the MDT and OST filesystems [15277.215785] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [15284.813088] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15285.075282] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15291.418902] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15297.798946] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15298.047606] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15302.076762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15302.076762] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15302.076762] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15302.402305] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15303.027859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15303.047379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15303.058119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15303.064909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15303.237281] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15303.242822] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15303.344546] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15303.353935] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15306.028726] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15306.278227] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15306.572261] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15306.727674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15313.221223] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15313.841469] Lustre: DEBUG MARKER: rc=0; [15313.841469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15313.841469] if [[ $? -eq 0 && $val -ne 0 ]]; then [15313.841469] echo $(hostname -s): $val; [15313.841469] rc=$val; [15313.841469] fi; [15313.841469] exit $rc [15314.437713] Lustre: DEBUG MARKER: dmesg [15315.031803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 79: format MDT\/OST without mgs option \(should return errors\) ===================== 21:54:10 \(1491256450\) [15315.198924] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ===================== 21:54:10 (1491256450) [15315.328107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15315.328107] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15315.328107] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15315.728631] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15315.988056] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15316.268692] Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_init" - [15316.559752] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15316.805373] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15317.087077] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15317.273191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15317.416445] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts [15317.663452] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15317.921512] Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_init" - [15319.559981] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15319.806310] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15323.664879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15323.664879] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15323.664879] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15323.982372] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15324.571783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15324.579489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15324.591942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15324.596043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15324.784986] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15324.847735] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15324.851829] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15324.901166] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15325.207672] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15325.453977] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15325.738273] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15325.897428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15332.682285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15333.336854] Lustre: DEBUG MARKER: rc=0; [15333.336854] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15333.336854] if [[ $? -eq 0 && $val -ne 0 ]]; then [15333.336854] echo $(hostname -s): $val; [15333.336854] rc=$val; [15333.336854] fi; [15333.336854] exit $rc [15333.939339] Lustre: DEBUG MARKER: dmesg [15334.517673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 80: mgc import reconnect race ==================================================== 21:54:29 \(1491256469\) [15334.664499] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ==================================================== 21:54:29 (1491256469) [15334.803775] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15335.060606] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15335.311476] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15335.557653] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15335.711328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15335.752858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15335.787349] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [15335.789632] Lustre: Skipped 22 previous similar messages [15335.810695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [15335.812737] Lustre: Skipped 7 previous similar messages [15335.828561] Lustre: lustre-MDT0000: new disk, initializing [15335.830468] Lustre: Skipped 3 previous similar messages [15335.866253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15335.868564] Lustre: Skipped 3 previous similar messages [15336.047684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15336.311224] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15336.797732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15336.809210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15336.968635] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15336.969791] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15337.109680] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15337.358866] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15337.607768] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15337.860367] Lustre: DEBUG MARKER: sync; sync; sync [15343.250596] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15343.502552] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15345.696808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15345.852925] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15352.523416] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x906 [15352.774073] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mgs/MGS/evict_client bd0cc2ab-c21e-4d9c-f74c-3bb0e86fe7fb [15352.901710] Lustre: 23283:0:(genops.c:1532:obd_export_evict_by_uuid()) MGS: evicting bd0cc2ab-c21e-4d9c-f74c-3bb0e86fe7fb at adminstrative request [15354.804146] Lustre: *** cfs_fail_loc=906, val=2147483648*** [15364.796925] Lustre: MGS: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [15385.271381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15385.425128] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15392.551317] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15392.816968] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15399.134244] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15409.818876] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15410.476434] Lustre: DEBUG MARKER: rc=0; [15410.476434] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15410.476434] if [[ $? -eq 0 && $val -ne 0 ]]; then [15410.476434] echo $(hostname -s): $val; [15410.476434] rc=$val; [15410.476434] fi; [15410.476434] exit $rc [15411.122624] Lustre: DEBUG MARKER: dmesg [15411.708881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 81: sparse OST indexing ========================================================== 21:55:46 \(1491256546\) [15411.866259] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ========================================================== 21:55:46 (1491256546) [15412.008291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15412.008291] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15412.008291] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15412.898400] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15413.149689] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15419.545481] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15419.794215] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15421.608068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15421.608068] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15421.608068] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15421.916405] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15422.501177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15422.508359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15422.528634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15422.547600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15422.770915] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15422.778846] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15422.821231] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15422.822145] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15423.131302] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15423.380191] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15423.669093] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15423.830685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15426.733544] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15426.990157] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15427.238135] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15427.485130] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15427.637566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15427.676734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15427.924447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15428.179048] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15428.664817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15428.667687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15428.823205] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15428.824261] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15428.963174] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15429.212687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15429.464625] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15429.728756] Lustre: DEBUG MARKER: sync; sync; sync [15434.997418] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15435.253482] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15437.588500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15437.744187] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15445.964264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15446.111518] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15454.287125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15454.441505] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15455.249619] Lustre: MGS: Connection restored to f962d99d-7e30-fad6-7e0d-7dd9726fb79e (at 10.9.5.237@tcp) [15455.252327] Lustre: Skipped 95 previous similar messages [15456.194724] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15456.444907] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15456.751118] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005fb90300 x1563694005791904/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [15456.756356] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 20 previous similar messages [15458.343163] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [15458.345717] Lustre: Skipped 22 previous similar messages [15468.769203] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15479.492821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15479.492821] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15479.492821] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15479.821451] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15480.390125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15480.395552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15480.400822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15480.489960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15480.612151] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15480.699981] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15480.710983] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15480.773701] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15481.080446] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15481.328503] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15481.614708] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15481.804435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15488.435169] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15489.050287] Lustre: DEBUG MARKER: rc=0; [15489.050287] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15489.050287] if [[ $? -eq 0 && $val -ne 0 ]]; then [15489.050287] echo $(hostname -s): $val; [15489.050287] rc=$val; [15489.050287] fi; [15489.050287] exit $rc [15489.637861] Lustre: DEBUG MARKER: dmesg [15490.201703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 82a: specify OSTs for file \(succeed\) or directory \(fail\) ========================= 21:57:05 \(1491256625\) [15490.352922] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (fail) ========================= 21:57:05 (1491256625) [15490.481721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15490.481721] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15490.481721] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15491.341046] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15491.617437] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15496.207379] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15496.454754] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15498.264855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15498.264855] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15498.264855] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15498.576421] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15499.245735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15499.300882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15499.300882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15499.334137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15499.557643] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15499.571673] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15499.573611] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15499.629247] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15499.945121] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15500.199509] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15500.481456] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15500.639219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15503.595893] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15503.886228] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15504.157263] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15504.422381] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15504.595668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15504.666240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15504.914960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15505.172757] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15505.684152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15505.687839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15505.846533] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15505.846534] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15505.999381] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15506.254159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15506.516701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15506.777929] Lustre: DEBUG MARKER: sync; sync; sync [15512.112329] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15512.373089] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15514.625845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15514.772303] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15517.643578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15517.804048] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15525.993290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15526.156856] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15527.188147] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [15527.188147] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [15527.453548] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [15527.453548] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [15529.288808] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15529.546398] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15541.919872] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15552.764227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15552.764227] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15552.764227] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15553.090156] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15553.615933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15553.677845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15553.715852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15553.718513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15553.898810] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15553.913483] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15553.978987] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15554.020947] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15554.361842] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15554.613770] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15554.904915] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15555.064959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15561.777180] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15562.485838] Lustre: DEBUG MARKER: rc=0; [15562.485838] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15562.485838] if [[ $? -eq 0 && $val -ne 0 ]]; then [15562.485838] echo $(hostname -s): $val; [15562.485838] rc=$val; [15562.485838] fi; [15562.485838] exit $rc [15563.082678] Lustre: DEBUG MARKER: dmesg [15563.670172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ==================== 21:58:18 \(1491256698\) [15563.829441] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ==================== 21:58:18 (1491256698) [15563.963782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15563.963782] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15563.963782] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15564.823537] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15565.075299] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15569.568228] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15569.822719] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15572.214829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15572.214829] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15572.214829] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15572.537391] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15573.232571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15573.237152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15573.244244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15573.258455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15573.473290] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15573.508496] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15573.521812] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15573.547031] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15573.841705] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15574.094523] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15574.376382] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15574.578831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15578.602125] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15578.868424] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15579.133261] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15579.396255] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15579.560388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15579.604211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15579.863601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15580.137669] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15580.628776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15580.634488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15580.797670] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15580.797672] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15580.941553] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15581.195575] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15581.452554] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15581.703542] Lustre: DEBUG MARKER: sync; sync; sync [15587.014976] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15587.268078] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15589.523102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15589.669496] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15592.520354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15592.669337] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15601.205275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15601.442992] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15604.430597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15604.620225] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15610.959487] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [15610.959487] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [15611.220280] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [15611.220280] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [15611.736568] Lustre: DEBUG MARKER: lctl pool_new lustre.test_82b [15619.997990] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_82b 2>/dev/null || echo foo [15620.249683] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_82b 2>/dev/null || echo foo [15620.545146] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_82b lustre-OST[5caa,7348,8a10] [15627.864399] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.test_82b [15628.296508] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [15628.558039] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.test_82b [15628.821024] Lustre: DEBUG MARKER: lctl pool_remove lustre.test_82b lustre-OST5caa_UUID [15633.098610] Lustre: DEBUG MARKER: lctl pool_remove lustre.test_82b lustre-OST7348_UUID [15641.382267] Lustre: DEBUG MARKER: lctl pool_remove lustre.test_82b lustre-OST8a10_UUID [15651.646107] Lustre: DEBUG MARKER: lctl pool_destroy lustre.test_82b [15657.906095] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_82b 2>/dev/null || echo foo [15658.154991] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_82b 2>/dev/null || echo foo [15660.144297] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15660.398141] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15668.469335] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15668.474430] LustreError: Skipped 38 previous similar messages [15672.564049] Lustre: 4145:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491256802/real 1491256802] req@ffff8800580bb900 x1563694005799504/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491256808 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [15672.570994] Lustre: 4145:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [15672.736215] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15677.738558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15677.738558] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15677.738558] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15678.069053] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15678.592095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15678.693071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15678.733391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15678.773435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15678.888351] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15678.997363] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15678.998802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15679.047111] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15679.351508] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15679.613766] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15679.932055] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15680.148236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15686.688240] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15687.358892] Lustre: DEBUG MARKER: rc=0; [15687.358892] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15687.358892] if [[ $? -eq 0 && $val -ne 0 ]]; then [15687.358892] echo $(hostname -s): $val; [15687.358892] rc=$val; [15687.358892] fi; [15687.358892] exit $rc [15687.960961] Lustre: DEBUG MARKER: dmesg [15688.541157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 83: ENOSPACE on OST doesn\'t cause message VFS: Busy inodes after unmount ... ===== 22:00:23 \(1491256823\) [15688.691386] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ===== 22:00:23 (1491256823) [15689.162561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mount the OST \/dev\/lvm-Role_OSS\/P1 as a ldiskfs filesystem [15689.311275] Lustre: DEBUG MARKER: mount the OST /dev/lvm-Role_OSS/P1 as a ldiskfs filesystem [15690.837911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark run llverfs in partial mode on the OST ldiskfs \/mnt\/lustre-ost1 [15690.987870] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [15691.620292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing run_llverfs \/mnt\/lustre-ost1 -vpl no [15691.766722] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing run_llverfs /mnt/lustre-ost1 -vpl no [15692.466796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark unmount the OST \/dev\/lvm-Role_OSS\/P1 [15692.622417] Lustre: DEBUG MARKER: unmount the OST /dev/lvm-Role_OSS/P1 [15698.283459] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15698.535112] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15698.796015] Lustre: DEBUG MARKER: tunefs.lustre --quiet --writeconf /dev/lvm-Role_MDS/P1 [15698.949602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15707.261214] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15707.523282] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15711.525873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15711.525873] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15711.525873] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15711.870552] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15712.420455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15712.463141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15712.477553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15712.589413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15712.674422] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15712.717499] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15712.789433] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15712.857205] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15713.175796] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15713.448887] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15713.740350] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15713.933776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15720.893383] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15721.558819] Lustre: DEBUG MARKER: rc=0; [15721.558819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15721.558819] if [[ $? -eq 0 && $val -ne 0 ]]; then [15721.558819] echo $(hostname -s): $val; [15721.558819] rc=$val; [15721.558819] fi; [15721.558819] exit $rc [15722.188514] Lustre: DEBUG MARKER: dmesg [15722.774622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 84: check recovery_hard_time ===================================================== 22:00:57 \(1491256857\) [15722.935995] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ===================================================== 22:00:57 (1491256857) [15723.090306] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15723.352325] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15723.604836] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15723.855828] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o recovery_time_hard=60,recovery_time_soft=60 /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15724.070883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15724.145054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15724.232563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [15724.234802] Lustre: Skipped 9 previous similar messages [15724.418986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15724.698164] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15725.204432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15725.223122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15725.379809] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15725.392170] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15725.526711] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15725.781379] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15726.032540] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15726.284719] Lustre: DEBUG MARKER: sync; sync; sync [15731.646533] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15731.898284] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15734.201393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15734.352550] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15742.503873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15742.656234] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15749.106981] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [15754.749138] Lustre: DEBUG MARKER: sync; sync; sync [15755.712605] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [15755.971041] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [15756.123892] LustreError: 8999:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [15756.140952] Turning device dm-0 (0xfc00000) read-only [15756.275691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [15756.410136] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15758.655358] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x20000709 fail_val=5 [15758.921760] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15759.174631] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [15759.303575] Lustre: Failing over lustre-MDT0000 [15765.370076] Removing read-only on unknown block (0xfc00000) [15765.392941] Lustre: server umount lustre-MDT0000 complete [15765.395109] Lustre: Skipped 9 previous similar messages [15765.532693] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15765.811165] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [15776.151418] Lustre: DEBUG MARKER: hostname [15776.463720] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15776.714887] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15776.963634] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o recovery_time_hard=60,recovery_time_soft=60 /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15777.116318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15777.333508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15777.586218] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15778.074493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15778.085231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15778.241382] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15778.249535] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15778.383184] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15778.642277] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15778.908484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15779.178046] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15780.947280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15780.949819] Lustre: 9767:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [15785.947093] Lustre: 9767:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [15785.967427] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15786.270054] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15786.572620] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15786.575274] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [15786.877042] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15786.879424] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [15787.781598] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15787.784208] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [15788.086027] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15788.088407] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [15789.893201] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15789.895688] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [15790.197023] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15790.199268] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [15794.102229] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15794.104747] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 13 previous similar messages [15794.407027] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15794.409244] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 13 previous similar messages [15802.212653] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15802.215443] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 26 previous similar messages [15802.517041] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15802.519253] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 26 previous similar messages [15818.421764] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [15818.424326] LustreError: 9794:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 53 previous similar messages [15818.726050] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [15818.728259] LustreError: 9794:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 53 previous similar messages [15822.930212] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [15825.334196] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [15825.338279] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 3 previous similar messages [15829.540291] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [15829.544377] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 6 previous similar messages [15837.646616] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [15837.650768] Lustre: 9794:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 20 previous similar messages [15840.652035] Lustre: 9794:0:(ldlm_lib.c:2044:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [15840.654416] Lustre: 9794:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [15840.656810] Lustre: lustre-MDT0000: disconnecting 1 stale clients [15840.659214] Lustre: 9794:0:(ldlm_lib.c:1625:abort_req_replay_queue()) @@@ aborted: req@ffff880064540900 x1563695711790384/t0(4294967487) o101->c1e7c2a4-abde-d9da-4806-6c1a5b4381fc@10.9.5.237@tcp:-1/-1 lens 824/0 e 0 to 0 dl 1491256981 ref 1 fl Complete:/4/ffffffff rc 0/-1 [15840.671970] Lustre: lustre-MDT0000: Denying connection for new client c1e7c2a4-abde-d9da-4806-6c1a5b4381fc(at 10.9.5.237@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [15840.677797] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 1 recovered and 1 was evicted. [15846.268798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status [15846.524673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status [15846.777404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status [15847.031982] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [15850.799740] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [15850.802159] LustreError: Skipped 1 previous similar message [15850.804146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [15850.808596] Lustre: Skipped 1 previous similar message [15861.323136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15861.590870] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15867.938609] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15868.278521] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15868.929598] Lustre: DEBUG MARKER: rc=0; [15868.929598] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15868.929598] if [[ $? -eq 0 && $val -ne 0 ]]; then [15868.929598] echo $(hostname -s): $val; [15868.929598] rc=$val; [15868.929598] fi; [15868.929598] exit $rc [15869.545424] Lustre: DEBUG MARKER: dmesg [15870.129188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 85: osd_ost init: fail ea_fid_set ================================================ 22:03:25 \(1491257005\) [15870.280357] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ================================================ 22:03:25 (1491257005) [15929.991186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15930.146782] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15932.022743] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15932.659542] Lustre: DEBUG MARKER: rc=0; [15932.659542] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15932.659542] if [[ $? -eq 0 && $val -ne 0 ]]; then [15932.659542] echo $(hostname -s): $val; [15932.659542] rc=$val; [15932.659542] fi; [15932.659542] exit $rc [15933.255722] Lustre: DEBUG MARKER: dmesg [15933.970856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 86: Replacing mkfs.lustre -G option ============================================== 22:04:29 \(1491257069\) [15934.127749] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option ============================================== 22:04:29 (1491257069) [15934.262882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15934.262882] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15934.262882] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15934.856186] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15935.103909] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15945.418490] Lustre: DEBUG MARKER: hostname [15945.726550] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15945.981021] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15946.234323] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15946.388162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15946.602714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15946.860907] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15947.361868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15947.372572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15947.526220] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15947.526771] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15947.665091] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15947.922596] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15948.188291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15948.456788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15949.234897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15949.261776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15949.414364] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15949.538116] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15951.436231] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15951.689985] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [15958.015907] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15963.837542] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15964.091777] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15968.141782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15968.141782] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15968.141782] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15968.488422] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15969.115940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15969.120243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15969.124775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15969.139054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15969.329632] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [15969.382073] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [15969.416295] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15969.426107] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [15969.731928] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15969.984094] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15970.269395] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [15970.470305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15977.350396] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [15977.609108] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15977.857728] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15978.112922] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15978.257088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15978.269773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15978.305752] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [15978.308077] Lustre: Skipped 24 previous similar messages [15978.328447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [15978.330776] Lustre: Skipped 9 previous similar messages [15978.347626] Lustre: lustre-MDT0000: new disk, initializing [15978.349857] Lustre: Skipped 4 previous similar messages [15978.378854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15978.381303] Lustre: Skipped 4 previous similar messages [15978.543721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15978.799254] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15979.292465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15979.294231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15979.447975] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15979.449446] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15979.588129] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15979.846149] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15980.109934] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15980.373531] Lustre: DEBUG MARKER: sync; sync; sync [15985.656750] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15985.911201] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15988.201827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15988.370991] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [15994.675217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15994.675217] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15994.675217] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15994.944561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15994.944561] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15994.944561] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15995.212676] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [15995.481210] Lustre: DEBUG MARKER: lctl get_param -n at_min [15995.738050] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [15996.226038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [15996.228195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [15996.392751] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [15996.398685] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [15996.583068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15996.584231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15996.741936] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15996.746069] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16003.739950] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16003.992659] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16010.320457] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16010.636549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16011.271850] Lustre: DEBUG MARKER: rc=0; [16011.271850] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16011.271850] if [[ $? -eq 0 && $val -ne 0 ]]; then [16011.271850] echo $(hostname -s): $val; [16011.271850] rc=$val; [16011.271850] fi; [16011.271850] exit $rc [16011.873704] Lustre: DEBUG MARKER: dmesg [16012.479576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ====================== 22:05:47 \(1491257147\) [16012.627984] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ====================== 22:05:47 (1491257147) [16018.130715] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16018.376872] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16022.398657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16022.398657] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16022.398657] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16022.745810] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16023.346871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16023.353813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16023.361180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16023.404238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16023.558695] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16023.600159] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16023.643624] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16023.691849] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16026.393329] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16026.639623] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16026.923272] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16027.112003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16033.948847] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16034.197581] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16034.478542] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16034.738966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16034.887055] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16035.144620] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16035.391188] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16035.635226] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16035.778096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16035.788339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16036.044546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16036.297345] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16036.787293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16036.787773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16036.947370] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16036.948705] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16037.087994] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16037.339920] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16037.593564] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16037.846903] Lustre: DEBUG MARKER: sync; sync; sync [16043.158951] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16043.427240] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16045.751386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16045.917694] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16054.166277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16054.332575] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16061.677490] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16061.680128] Lustre: Skipped 46 previous similar messages [16062.621778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16062.778819] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16071.041425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16071.201658] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16079.444086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16079.593823] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16088.006030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16088.177889] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16096.495462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16096.654414] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16103.190933] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16103.445875] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16104.335125] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064541800 x1563694005809648/t0(0) o13->lustre-OST0004-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [16104.341970] LustreError: 21192:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [16105.558168] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [16105.560999] Lustre: Skipped 15 previous similar messages [16115.822806] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16116.088392] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16116.336692] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16116.472969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [16116.601069] Lustre: DEBUG MARKER: ls -sal /mnt/lustre-mds1/ROOT/f87.conf-sanity [16116.869046] Lustre: DEBUG MARKER: ls -s /mnt/lustre-mds1/ROOT/f87.conf-sanity [16117.719853] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16117.969087] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16118.227480] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16124.343454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16124.973729] Lustre: DEBUG MARKER: rc=0; [16124.973729] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16124.973729] if [[ $? -eq 0 && $val -ne 0 ]]; then [16124.973729] echo $(hostname -s): $val; [16124.973729] rc=$val; [16124.973729] fi; [16124.973729] exit $rc [16125.583643] Lustre: DEBUG MARKER: dmesg [16126.148875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 88: check the default mount options can be overridden ============================ 22:07:41 \(1491257261\) [16126.304648] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ============================ 22:07:41 (1491257261) [16126.604721] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16126.860830] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16127.167236] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16127.327231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16127.480979] Lustre: DEBUG MARKER: tunefs.lustre /dev/lvm-Role_MDS/P1 | [16127.480979] grep -e .*opts:.*errors=remount-ro.* [16127.624371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16127.820550] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16128.079064] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16128.370956] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16128.575784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16128.728132] Lustre: DEBUG MARKER: tunefs.lustre /dev/lvm-Role_MDS/P1 | [16128.728132] grep -e .*opts:.*errors=panic.* [16128.914911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16129.172920] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16129.823851] Lustre: DEBUG MARKER: rc=0; [16129.823851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16129.823851] if [[ $? -eq 0 && $val -ne 0 ]]; then [16129.823851] echo $(hostname -s): $val; [16129.823851] rc=$val; [16129.823851] fi; [16129.823851] exit $rc [16130.422476] Lustre: DEBUG MARKER: dmesg [16130.999569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 89: check tunefs --param and --erase-param{s} options ============================ 22:07:46 \(1491257266\) [16131.157705] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ============================ 22:07:46 (1491257266) [16131.294171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16131.294171] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16131.294171] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16132.168719] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16132.421929] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16136.535882] Lustre: DEBUG MARKER: tunefs.lustre --param failover.node=192.0.2.254@tcp0 /dev/lvm-Role_MDS/P1 >/dev/null [16136.688849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16136.852915] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /dev/lvm-Role_MDS/P1 [16137.123029] Lustre: DEBUG MARKER: tunefs.lustre --param failover.node=192.0.2.255@tcp0 /dev/lvm-Role_MDS/P1 >/dev/null [16137.275993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16137.443684] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /dev/lvm-Role_MDS/P1 [16137.720298] Lustre: DEBUG MARKER: tunefs.lustre --erase-param failover.node /dev/lvm-Role_MDS/P1 >/dev/null [16137.867149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16138.010261] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /dev/lvm-Role_MDS/P1 [16138.283251] Lustre: DEBUG MARKER: tunefs.lustre --erase-params /dev/lvm-Role_MDS/P1 >/dev/null [16138.436320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16138.577869] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /dev/lvm-Role_MDS/P1 [16138.844399] Lustre: DEBUG MARKER: tunefs.lustre --param failover.node=192.0.2.254@tcp0 --erase-params /dev/lvm-Role_MDS/P1 >/dev/null [16138.994228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16139.131672] Lustre: DEBUG MARKER: tunefs.lustre --dryrun /dev/lvm-Role_MDS/P1 [16140.021109] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16140.288209] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16144.382709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16144.382709] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16144.382709] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16144.796301] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16145.328353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16145.350315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16145.397405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16145.449504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16145.574921] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16145.596698] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16145.676501] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16145.724504] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16146.106790] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16146.357442] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16146.651292] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16146.856449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16153.701171] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16154.338485] Lustre: DEBUG MARKER: rc=0; [16154.338485] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16154.338485] if [[ $? -eq 0 && $val -ne 0 ]]; then [16154.338485] echo $(hostname -s): $val; [16154.338485] rc=$val; [16154.338485] fi; [16154.338485] exit $rc [16154.965322] Lustre: DEBUG MARKER: dmesg [16155.559095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ==================================== 22:08:10 \(1491257290\) [16155.724981] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ==================================== 22:08:10 (1491257290) [16156.444602] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16156.704362] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16160.803144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16160.803144] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16160.803144] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16161.136754] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16161.700485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16161.727004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16161.755372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16161.788608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16162.005746] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16162.007610] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16162.045044] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16162.065488] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16162.415545] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16162.678471] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16162.971366] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16163.176384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16169.785522] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16170.058911] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16170.327849] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16170.573905] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16170.742836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16170.755453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16171.062631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16171.321352] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16171.816537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16171.827349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16171.987453] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16171.988722] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16172.137504] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16172.396715] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16172.649924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16172.909542] Lustre: DEBUG MARKER: sync; sync; sync [16178.197766] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16178.453295] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16180.701163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16180.867103] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16188.532961] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16188.673900] Lustre: *** cfs_fail_loc=159, val=0*** [16189.805339] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16195.681902] Lustre: lustre-MDT0000: Client 5ea0b0cd-9f0d-ba64-78f9-186cd4ee462f (at 10.9.5.237@tcp) reconnecting [16195.821282] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16195.957704] Lustre: *** cfs_fail_loc=159, val=0*** [16195.959838] Lustre: Skipped 5 previous similar messages [16197.086970] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16202.963817] Lustre: lustre-MDT0000: Client 5ea0b0cd-9f0d-ba64-78f9-186cd4ee462f (at 10.9.5.237@tcp) reconnecting [16204.879808] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [16204.883975] LustreError: Skipped 2 previous similar messages [16204.886358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [16204.891102] Lustre: Skipped 2 previous similar messages [16210.122313] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16210.394182] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16216.750843] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16223.632720] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16224.258679] Lustre: DEBUG MARKER: rc=0; [16224.258679] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16224.258679] if [[ $? -eq 0 && $val -ne 0 ]]; then [16224.258679] echo $(hostname -s): $val; [16224.258679] rc=$val; [16224.258679] fi; [16224.258679] exit $rc [16224.833481] Lustre: DEBUG MARKER: dmesg [16226.157032] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ======================= 22:09:20 \(1491257360\) [16226.317929] Lustre: DEBUG MARKER: == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ======================= 22:09:20 (1491257360) [16226.464106] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16228.389464] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16228.635862] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16228.883288] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16229.037772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16229.251602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16229.508489] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16230.010358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16230.011955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16230.167239] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16230.167494] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16230.309655] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16230.559761] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16230.810316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16231.073909] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16231.341331] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16233.442333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16233.597663] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16240.868478] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16242.122614] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16243.378401] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16243.511205] Lustre: *** cfs_fail_loc=159, val=0*** [16243.513244] Lustre: Skipped 6 previous similar messages [16244.633547] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16250.524354] Lustre: lustre-MDT0000: Client fa6b0db3-0409-db14-ab24-3ad1c37e3b4d (at 10.9.5.237@tcp) reconnecting [16250.704167] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16250.839904] Lustre: *** cfs_fail_loc=159, val=0*** [16251.969341] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16257.856639] Lustre: lustre-MDT0000: Client fa6b0db3-0409-db14-ab24-3ad1c37e3b4d (at 10.9.5.237@tcp) reconnecting [16257.998161] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16258.134219] Lustre: *** cfs_fail_loc=159, val=0*** [16258.136323] Lustre: Skipped 4 previous similar messages [16259.264734] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16265.150405] Lustre: lustre-MDT0000: Client fa6b0db3-0409-db14-ab24-3ad1c37e3b4d (at 10.9.5.237@tcp) reconnecting [16265.303881] Lustre: DEBUG MARKER: cat /sys/module/mdt/parameters/max_mod_rpcs_per_client [16265.700817] Lustre: DEBUG MARKER: echo 16 > /sys/module/mdt/parameters/max_mod_rpcs_per_client [16266.145041] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16266.287156] Lustre: *** cfs_fail_loc=159, val=0*** [16266.289457] Lustre: Skipped 1 previous similar message [16267.409216] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16273.296414] Lustre: lustre-MDT0000: Client 9e43b1c9-aaed-11be-3235-2971cb108ac1 (at 10.9.5.237@tcp) reconnecting [16273.456871] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16274.741559] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16280.770285] Lustre: DEBUG MARKER: echo 8 > /sys/module/mdt/parameters/max_mod_rpcs_per_client [16288.083383] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16288.359950] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16294.557046] Lustre: 30623:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491257424/real 1491257424] req@ffff880063b7c300 x1563694005812224/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491257430 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [16294.564090] Lustre: 30623:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [16294.765464] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16301.232987] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16301.860879] Lustre: DEBUG MARKER: rc=0; [16301.860879] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16301.860879] if [[ $? -eq 0 && $val -ne 0 ]]; then [16301.860879] echo $(hostname -s): $val; [16301.860879] rc=$val; [16301.860879] fi; [16301.860879] exit $rc [16302.456615] Lustre: DEBUG MARKER: dmesg [16303.818877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ================================== 22:10:38 \(1491257438\) [16303.967645] Lustre: DEBUG MARKER: == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ================================== 22:10:38 (1491257438) [16304.109697] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16306.033018] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16306.286090] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16306.532316] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16306.686594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16306.895670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16307.149175] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16307.641478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16307.642147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16307.804021] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16307.804587] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16307.942722] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16308.194910] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16308.450068] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16308.715101] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16308.966718] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16311.066831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16311.219494] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16313.463991] Lustre: DEBUG MARKER: cat /sys/module/mdt/parameters/max_mod_rpcs_per_client [16313.846730] Lustre: DEBUG MARKER: echo 8 > /sys/module/mdt/parameters/max_mod_rpcs_per_client [16314.286432] Lustre: DEBUG MARKER: echo 8 > /sys/module/mdt/parameters/max_mod_rpcs_per_client [16321.557542] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16321.808757] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16328.259745] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16333.999208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16334.638339] Lustre: DEBUG MARKER: rc=0; [16334.638339] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16334.638339] if [[ $? -eq 0 && $val -ne 0 ]]; then [16334.638339] echo $(hostname -s): $val; [16334.638339] rc=$val; [16334.638339] fi; [16334.638339] exit $rc [16335.226153] Lustre: DEBUG MARKER: dmesg [16336.560552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ================= 22:11:10 \(1491257470\) [16336.708469] Lustre: DEBUG MARKER: == conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ================= 22:11:10 (1491257470) [16336.846466] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16338.766965] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16339.018657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16339.265128] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16339.421695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16339.492341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16339.494661] Lustre: Skipped 7 previous similar messages [16339.640289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16339.895241] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16340.377472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16340.379639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16340.532592] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16340.535638] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16340.671752] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16340.920949] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16341.173184] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16341.426998] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16341.681205] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16343.790744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16343.941644] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16351.274739] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [16351.427851] Lustre: *** cfs_fail_loc=159, val=0*** [16351.430659] Lustre: Skipped 31 previous similar messages [16351.559100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [16358.440231] Lustre: lustre-MDT0000: Client 5ceeb0df-3162-9c44-6022-ec948423a89d (at 10.9.5.237@tcp) reconnecting [16358.442782] Lustre: Skipped 1 previous similar message [16365.604870] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16365.868995] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16372.081900] Lustre: server umount lustre-MDT0000 complete [16372.084240] Lustre: Skipped 7 previous similar messages [16372.216940] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16379.000236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16379.634248] Lustre: DEBUG MARKER: rc=0; [16379.634248] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16379.634248] if [[ $? -eq 0 && $val -ne 0 ]]; then [16379.634248] echo $(hostname -s): $val; [16379.634248] rc=$val; [16379.634248] fi; [16379.634248] exit $rc [16380.240611] Lustre: DEBUG MARKER: dmesg [16381.625274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 91: evict-by-nid support ========================================================= 22:11:55 \(1491257515\) [16381.792309] Lustre: DEBUG MARKER: == conf-sanity test 91: evict-by-nid support ========================================================= 22:11:55 (1491257515) [16382.206606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16382.206606] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16382.206606] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16382.491914] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16384.430740] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16384.681513] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16384.947363] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16385.130166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16385.364514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16385.638562] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16386.134833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16386.136387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16386.287757] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16386.292127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16386.429077] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16386.684868] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16386.946534] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16387.206567] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16387.466503] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16389.644370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16389.792094] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16395.881683] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.lustre*.exports.* [16396.135150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.'10\.9\.5\.237@tcp'.uuid [16396.517475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.'10\.9\.5\.237@tcp'.uuid [16407.143545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.'10\.9\.5\.237@tcp'.uuid [16407.408349] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client nid:10\.9\.5\.237@tcp [16407.542333] Lustre: lustre-MDT0000: evicting 233736d6-46c8-aecb-68b0-1aac00d3f63b (at 10.9.5.237@tcp) by administrative request [16407.681403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.'10\.9\.5\.237@tcp'.uuid [16429.223480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre*.exports.'10\.9\.5\.237@tcp'.uuid [16436.657931] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16436.914459] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16443.260153] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16449.222828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16449.874754] Lustre: DEBUG MARKER: rc=0; [16449.874754] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16449.874754] if [[ $? -eq 0 && $val -ne 0 ]]; then [16449.874754] echo $(hostname -s): $val; [16449.874754] rc=$val; [16449.874754] fi; [16449.874754] exit $rc [16450.497725] Lustre: DEBUG MARKER: dmesg [16451.870867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 92: ldev returns MGS NID correctly in command substitution ======================= 22:13:06 \(1491257586\) [16452.026019] Lustre: DEBUG MARKER: == conf-sanity test 92: ldev returns MGS NID correctly in command substitution ======================= 22:13:06 (1491257586) [16452.571284] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16453.223223] Lustre: DEBUG MARKER: rc=0; [16453.223223] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16453.223223] if [[ $? -eq 0 && $val -ne 0 ]]; then [16453.223223] echo $(hostname -s): $val; [16453.223223] rc=$val; [16453.223223] fi; [16453.223223] exit $rc [16453.838678] Lustre: DEBUG MARKER: dmesg [16454.442308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 93: register mulitple MDT at the same time ======================================= 22:13:09 \(1491257589\) [16454.600713] Lustre: DEBUG MARKER: == conf-sanity test 93: register mulitple MDT at the same time ======================================= 22:13:09 (1491257589) [16454.807164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: conf-sanity test_93 needs \>= 3 MDTs [16454.985622] Lustre: DEBUG MARKER: SKIP: conf-sanity test_93 needs >= 3 MDTs [16455.200190] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16455.838096] Lustre: DEBUG MARKER: rc=0; [16455.838096] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16455.838096] if [[ $? -eq 0 && $val -ne 0 ]]; then [16455.838096] echo $(hostname -s): $val; [16455.838096] rc=$val; [16455.838096] fi; [16455.838096] exit $rc [16456.480482] Lustre: DEBUG MARKER: dmesg [16457.031376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 94: ldev outputs correct labels for file system name query ======================= 22:13:12 \(1491257592\) [16457.192315] Lustre: DEBUG MARKER: == conf-sanity test 94: ldev outputs correct labels for file system name query ======================= 22:13:12 (1491257592) [16457.521626] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16458.167297] Lustre: DEBUG MARKER: rc=0; [16458.167297] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16458.167297] if [[ $? -eq 0 && $val -ne 0 ]]; then [16458.167297] echo $(hostname -s): $val; [16458.167297] rc=$val; [16458.167297] fi; [16458.167297] exit $rc [16458.780032] Lustre: DEBUG MARKER: dmesg [16459.327944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 95: ldev should only allow one label filter ====================================== 22:13:14 \(1491257594\) [16459.492152] Lustre: DEBUG MARKER: == conf-sanity test 95: ldev should only allow one label filter ====================================== 22:13:14 (1491257594) [16459.982157] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16460.629624] Lustre: DEBUG MARKER: rc=0; [16460.629624] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16460.629624] if [[ $? -eq 0 && $val -ne 0 ]]; then [16460.629624] echo $(hostname -s): $val; [16460.629624] rc=$val; [16460.629624] fi; [16460.629624] exit $rc [16461.213990] Lustre: DEBUG MARKER: dmesg [16461.783329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ================ 22:13:16 \(1491257596\) [16461.952079] Lustre: DEBUG MARKER: == conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ================ 22:13:16 (1491257596) [16462.296239] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16462.940599] Lustre: DEBUG MARKER: rc=0; [16462.940599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16462.940599] if [[ $? -eq 0 && $val -ne 0 ]]; then [16462.940599] echo $(hostname -s): $val; [16462.940599] rc=$val; [16462.940599] fi; [16462.940599] exit $rc [16463.531054] Lustre: DEBUG MARKER: dmesg [16464.114519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 97: ldev returns correct ouput when querying based on role ======================= 22:13:19 \(1491257599\) [16464.271358] Lustre: DEBUG MARKER: == conf-sanity test 97: ldev returns correct ouput when querying based on role ======================= 22:13:19 (1491257599) [16464.676518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16465.333024] Lustre: DEBUG MARKER: rc=0; [16465.333024] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16465.333024] if [[ $? -eq 0 && $val -ne 0 ]]; then [16465.333024] echo $(hostname -s): $val; [16465.333024] rc=$val; [16465.333024] fi; [16465.333024] exit $rc [16465.936662] Lustre: DEBUG MARKER: dmesg [16466.531431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 98: Buffer-overflow check while parsing mount_opts =============================== 22:13:21 \(1491257601\) [16466.681791] Lustre: DEBUG MARKER: == conf-sanity test 98: Buffer-overflow check while parsing mount_opts =============================== 22:13:21 (1491257601) [16466.823367] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16468.771108] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16469.030765] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16469.282389] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16469.441846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16469.665160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16469.921151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16470.420873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16470.421255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16470.577651] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16470.579385] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16470.718207] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16470.971637] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16471.241484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16471.502043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16471.756723] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16473.888474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16474.052536] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16488.410486] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16488.676785] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16495.027071] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16501.428758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16502.062473] Lustre: DEBUG MARKER: rc=0; [16502.062473] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16502.062473] if [[ $? -eq 0 && $val -ne 0 ]]; then [16502.062473] echo $(hostname -s): $val; [16502.062473] rc=$val; [16502.062473] fi; [16502.062473] exit $rc [16502.700557] Lustre: DEBUG MARKER: dmesg [16504.032067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 99: Adding meta_bg option ======================================================== 22:13:58 \(1491257638\) [16504.189229] Lustre: DEBUG MARKER: == conf-sanity test 99: Adding meta_bg option ======================================================== 22:13:58 (1491257638) [16506.092716] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16506.730964] Lustre: DEBUG MARKER: rc=0; [16506.730964] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16506.730964] if [[ $? -eq 0 && $val -ne 0 ]]; then [16506.730964] echo $(hostname -s): $val; [16506.730964] rc=$val; [16506.730964] fi; [16506.730964] exit $rc [16507.340476] Lustre: DEBUG MARKER: dmesg [16507.927700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo =============================== 22:14:03 \(1491257643\) [16508.090959] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo =============================== 22:14:03 (1491257643) [16508.823507] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16509.075339] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16513.111239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [16513.111239] /usr/sbin/lctl lustre_build_version 2>/dev/null || [16513.111239] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [16513.440307] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16513.964634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16514.005472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16514.042506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16514.113723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16514.274424] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [16514.303731] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16514.340165] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [16514.407975] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [16516.330137] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16516.590123] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16516.879391] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --mkfsoptions="-E lazy_itable_i [16517.077256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16523.957292] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16524.229927] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16524.481765] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16524.733115] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16524.885653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [16524.896911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16525.173938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16525.433753] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16525.941719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16525.946790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16526.102031] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16526.102965] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16526.247564] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16526.501905] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16526.755812] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16527.018346] Lustre: DEBUG MARKER: sync; sync; sync [16532.339257] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16532.611051] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16534.901900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16535.070142] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16541.437912] Lustre: DEBUG MARKER: lshowmount -v [16541.802992] Lustre: DEBUG MARKER: lshowmount -v [16549.457065] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16549.726890] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16556.098204] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16561.763967] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16562.410886] Lustre: DEBUG MARKER: rc=0; [16562.410886] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16562.410886] if [[ $? -eq 0 && $val -ne 0 ]]; then [16562.410886] echo $(hostname -s): $val; [16562.410886] rc=$val; [16562.410886] fi; [16562.410886] exit $rc [16563.029743] Lustre: DEBUG MARKER: dmesg [16564.422732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 101: Race MDT-\>OST reconnection with create ====================================== 22:14:58 \(1491257698\) [16564.597527] Lustre: DEBUG MARKER: == conf-sanity test 101: Race MDT->OST reconnection with create ====================================== 22:14:58 (1491257698) [16564.736480] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16566.673751] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16566.932328] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16567.181246] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16567.329869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16567.550229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16567.803290] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16568.298047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16568.301636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16568.457415] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16568.459379] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16568.597752] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16568.853545] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16569.110208] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16569.367157] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16569.626918] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16571.892212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16572.115086] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16579.393848] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16579.554950] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16579.561844] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16579.696277] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16579.964748] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16580.094893] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16580.097378] Lustre: Skipped 1 previous similar message [16580.105856] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16580.110544] LustreError: Skipped 1 previous similar message [16580.240715] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16580.519381] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16580.855394] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16581.130723] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16581.269131] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16581.271872] Lustre: Skipped 3 previous similar messages [16581.283099] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16581.288407] LustreError: Skipped 3 previous similar messages [16581.425980] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16581.691960] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16581.988118] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16583.608175] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16583.764161] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16583.766968] Lustre: Skipped 3 previous similar messages [16583.776149] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16583.781396] LustreError: Skipped 3 previous similar messages [16583.911638] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16584.172960] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16584.438269] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16584.700304] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16584.973464] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16585.239891] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16585.500556] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16585.763121] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16586.033392] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16587.482502] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16587.785018] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16587.916685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16587.919749] Lustre: Skipped 10 previous similar messages [16587.929219] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16587.934977] LustreError: Skipped 10 previous similar messages [16588.065500] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16588.323547] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16588.586546] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16588.879399] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16589.137686] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16589.398469] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16589.661354] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16589.936096] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16591.861475] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16592.151821] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16592.418693] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16592.692401] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16592.960092] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16593.270019] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16593.537514] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16593.824764] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16594.091424] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16594.360350] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16597.575422] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16597.726979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16597.730636] Lustre: Skipped 18 previous similar messages [16597.742611] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16597.748479] LustreError: Skipped 18 previous similar messages [16597.879371] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16598.140916] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16598.417357] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16598.684385] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16598.977647] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16599.302266] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16599.591208] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16599.851437] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16600.117994] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16602.943913] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16603.253382] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16603.524020] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16603.798448] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16604.073667] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16604.365219] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16604.627153] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16604.890728] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16605.151246] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16605.413637] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16610.071433] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16610.365548] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16610.640112] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16610.927899] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16611.187674] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16611.453997] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16611.718203] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16612.014223] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16612.282059] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16612.611464] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16617.772303] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16617.923184] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16617.926519] Lustre: Skipped 29 previous similar messages [16617.936962] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16617.943015] LustreError: Skipped 29 previous similar messages [16618.073150] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16618.337547] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16618.600087] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16618.861711] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16619.132210] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16619.394111] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16619.661444] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16619.942420] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16620.262994] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16627.144221] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16627.419483] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16627.684226] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16627.944461] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16628.210159] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16628.482113] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16628.741366] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16628.999113] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16629.255967] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16629.548928] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16641.956488] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16642.239479] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16642.509920] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16642.777522] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16643.072210] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16643.342821] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16643.607113] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16643.881941] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16644.153806] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16644.506153] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16664.162765] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16664.310171] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16664.313557] Lustre: Skipped 29 previous similar messages [16664.325755] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16664.331784] LustreError: Skipped 29 previous similar messages [16664.335024] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16664.338472] Lustre: Skipped 168 previous similar messages [16664.462613] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16664.718388] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16664.991635] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16665.279747] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16665.555533] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16665.818433] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16666.086261] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16666.357639] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16666.638690] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16697.989553] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16698.278970] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16698.542783] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16698.808391] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16699.066380] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16699.373204] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16699.646115] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16699.912515] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16700.170754] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16700.430981] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16762.307157] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16762.463600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16762.466868] Lustre: Skipped 19 previous similar messages [16762.471052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [16762.477152] Lustre: Skipped 126 previous similar messages [16762.485547] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [16762.491469] LustreError: Skipped 19 previous similar messages [16762.624655] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16762.909214] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16763.176656] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16763.436914] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16763.707517] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16763.979186] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16764.248091] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16764.512123] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16764.783156] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16813.130634] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16813.403093] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16813.660876] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16813.931667] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16814.195278] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16814.491570] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16814.785772] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16815.054637] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16815.313269] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16815.574109] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 deactivate; /usr/sbin/lctl --device lustre-OST0000-osc-MDT0000 activate [16828.171327] LustreError: 12369:0:(osp_precreate.c:619:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [16870.816807] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [16870.820166] LustreError: Skipped 13 previous similar messages [16872.318652] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16872.583412] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16882.313119] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16887.598656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16888.242701] Lustre: DEBUG MARKER: rc=0; [16888.242701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16888.242701] if [[ $? -eq 0 && $val -ne 0 ]]; then [16888.242701] echo $(hostname -s): $val; [16888.242701] rc=$val; [16888.242701] fi; [16888.242701] exit $rc [16888.832592] Lustre: DEBUG MARKER: dmesg [16890.185821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 102: obdclass module cleanup upon error ========================================== 22:20:24 \(1491258024\) [16890.363659] Lustre: DEBUG MARKER: == conf-sanity test 102: obdclass module cleanup upon error ========================================== 22:20:24 (1491258024) [16891.060319] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16891.308767] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16891.818776] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16892.066883] Lustre: DEBUG MARKER: lustre_rmmod [16896.838069] LNet: Removed LNI 10.9.5.243@tcp [16897.155146] LNet: 24955:0:(workitem.c:319:cfs_wi_sched_destroy()) waiting for 2 threads of WI sched[cfs_rh] to terminate [16897.485127] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [16897.489597] alg: No test for adler32 (adler32-zlib) [16897.492132] alg: No test for crc32 (crc32-table) [16898.370603] Lustre: DEBUG MARKER: lsmod | grep libcfs [16898.632329] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000060a [16898.898638] Lustre: DEBUG MARKER: mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16899.055656] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [16899.058997] Lustre: *** cfs_fail_loc=60a, val=0*** [16899.204086] Lustre: DEBUG MARKER: lsmod | grep obdclass [16899.456267] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 [16899.708351] Lustre: DEBUG MARKER: mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16899.858037] Lustre: Lustre: Build Version: 2.9.54_76_ge2ce727 [16899.874808] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [16899.876950] LNet: Accept all, port 7988 [16899.901773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16899.922616] Lustre: MGS: Connection restored to 23599521-a413-f802-2824-696236992c93 (at 0@lo) [16899.974681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16900.681984] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16900.943840] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [16907.121046] Lustre: 25732:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258036/real 1491258036] req@ffff880064560000 x1563697385308592/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491258042 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [16907.156923] Lustre: server umount lustre-MDT0000 complete [16907.285815] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16907.771306] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16908.385848] Lustre: DEBUG MARKER: rc=0; [16908.385848] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16908.385848] if [[ $? -eq 0 && $val -ne 0 ]]; then [16908.385848] echo $(hostname -s): $val; [16908.385848] rc=$val; [16908.385848] fi; [16908.385848] exit $rc [16909.002775] Lustre: DEBUG MARKER: dmesg [16910.342714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test 103: rename filesystem name ====================================================== 22:20:44 \(1491258044\) [16910.517802] Lustre: DEBUG MARKER: == conf-sanity test 103: rename filesystem name ====================================================== 22:20:44 (1491258044) [16912.730363] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [16912.730363] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [16912.730363] if [ $running -ne $mpts ]; then [16912.730363] echo $(hostname) env are INSANE!; [16912.730363] exit 1; [16912.730363] fi [16913.056866] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [16913.056866] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [16913.056866] if [ $running -ne $mpts ]; then [16913.056866] echo $(hostname) env are INSANE!; [16913.056866] exit 1; [16913.056866] fi [16915.656691] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [16915.914479] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16916.180276] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16916.444656] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16916.599611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16916.618913] Lustre: MGS: Connection restored to 23599521-a413-f802-2824-696236992c93 (at 0@lo) [16916.660097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16916.825179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16917.094399] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [16917.607983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16917.618429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16917.776897] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16917.780211] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16917.916133] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16918.170316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16918.431864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16918.689181] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16918.943317] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16920.154054] Lustre: MGS: Connection restored to bb1fb615-6465-24d8-71e8-9b6a2f507085 (at 10.9.5.244@tcp) [16921.139601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16921.310665] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16921.657136] Lustre: 25521:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258052/real 1491258052] req@ffff880060ef4f00 x1563697385308752/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491258057 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [16923.419598] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16923.422383] Lustre: Skipped 2 previous similar messages [16924.362211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16924.522588] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16931.808181] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16932.776176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16932.947459] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16940.368566] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16941.326095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16941.510099] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16949.899263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16950.066220] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16957.440524] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16957.443242] Lustre: Skipped 1 previous similar message [16958.455262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16958.624126] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16966.985098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16967.144862] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16974.607530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16974.764792] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [16976.190297] Lustre: DEBUG MARKER: lctl get_param -n timeout [16976.506103] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [16976.659320] Lustre: DEBUG MARKER: Using TIMEOUT=20 [16976.797969] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [16977.091271] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [17101.859863] LustreError: 26604:0:(llog.c:616:llog_process_thread()) Local llog found corrupted [17174.224027] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new lustre.pool1 [17182.492441] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new lustre.lustre [17190.774529] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.lustre lustre-OST0000 [17201.631200] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17201.886031] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [17202.527068] LustreError: 25522:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079c0f000 x1563697386119072/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17203.567069] LustreError: 25522:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079c0f000 x1563697386119088/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17205.997833] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17206.003428] LustreError: Skipped 2 previous similar messages [17210.057040] Lustre: 28828:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258339/real 1491258339] req@ffff880055cafc00 x1563697386119184/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491258345 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [17210.110203] Lustre: server umount lustre-MDT0000 complete [17210.236487] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17216.386621] Lustre: DEBUG MARKER: tunefs.lustre --fsname=mylustre --rename=lustre -v /dev/lvm-Role_MDS/P1 [17216.541724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [17216.579686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17221.253937] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17221.253937] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17221.253937] if [ $running -ne $mpts ]; then [17221.253937] echo $(hostname) env are INSANE!; [17221.253937] exit 1; [17221.253937] fi [17221.583132] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17221.583132] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17221.583132] if [ $running -ne $mpts ]; then [17221.583132] echo $(hostname) env are INSANE!; [17221.583132] exit 1; [17221.583132] fi [17222.610133] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [17222.886176] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [17223.151786] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [17223.417353] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [17223.577582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17223.622965] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [17223.626226] Lustre: Skipped 5 previous similar messages [17223.682251] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17223.848027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17224.106678] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [17224.658084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17224.660896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17224.816818] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17224.829579] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17224.969920] Lustre: DEBUG MARKER: lctl set_param -n mdt.mylustre*.enable_remote_dir=1 [17225.227321] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17225.490472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17225.762946] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [17226.051849] Lustre: DEBUG MARKER: lctl set_param -n mdt.mylustre*.enable_remote_dir=1 [17228.375812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17228.527219] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17229.676048] Lustre: 25521:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258359/real 1491258359] req@ffff880066103c00 x1563697386119456/t0(0) o8->mylustre-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491258364 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17231.414692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17231.570354] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17234.560961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17234.725325] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17237.666316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17237.822039] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17240.813530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17240.976097] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17244.048243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17244.204428] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17247.112897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17247.273205] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17249.458225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17249.616410] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17251.063110] Lustre: DEBUG MARKER: lctl get_param -n timeout [17251.390673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [17251.547529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17251.684462] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [17256.970926] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list mylustre.pool1 [17257.223327] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list mylustre.lustre [17257.477173] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list mylustre.lustre [17257.739906] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add mylustre.lustre mylustre-OST0001 [17266.781276] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17267.049875] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [17268.463088] LustreError: 25523:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062d7b000 x1563697386121984/t0(0) o13->mylustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17268.468444] LustreError: 25523:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [17271.182085] Lustre: mylustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [17271.184830] Lustre: Skipped 6 previous similar messages [17271.407076] LustreError: 25523:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062d7b000 x1563697386122064/t0(0) o13->mylustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17271.412512] LustreError: 25523:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [17276.181663] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17276.187136] LustreError: Skipped 10 previous similar messages [17279.222032] Lustre: 31644:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258408/real 1491258408] req@ffff880062d7b000 x1563697386122144/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491258414 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [17279.229931] Lustre: 31644:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [17279.273125] Lustre: server umount mylustre-MDT0000 complete [17279.403511] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17285.363260] Lustre: DEBUG MARKER: tunefs.lustre --fsname=tfs --rename=mylustre -v /dev/lvm-Role_MDS/P1 [17285.518742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [17285.556639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17290.419494] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17290.419494] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17290.419494] if [ $running -ne $mpts ]; then [17290.419494] echo $(hostname) env are INSANE!; [17290.419494] exit 1; [17290.419494] fi [17290.733088] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17290.733088] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17290.733088] if [ $running -ne $mpts ]; then [17290.733088] echo $(hostname) env are INSANE!; [17290.733088] exit 1; [17290.733088] fi [17291.710093] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [17291.971598] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [17292.222027] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [17292.468105] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [17292.623259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17292.659834] Lustre: MGS: Connection restored to MGC10.9.5.243@tcp_0 (at 0@lo) [17292.662090] Lustre: Skipped 13 previous similar messages [17292.719045] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17292.869946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17293.123475] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [17293.613960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17293.613961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17293.770756] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17293.780386] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17293.915598] Lustre: DEBUG MARKER: lctl set_param -n mdt.tfs*.enable_remote_dir=1 [17294.183446] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17294.448032] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17294.712598] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [17294.973671] Lustre: DEBUG MARKER: lctl set_param -n mdt.tfs*.enable_remote_dir=1 [17297.079843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17297.241252] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17297.714124] Lustre: 25521:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258428/real 1491258428] req@ffff8800035e7c00 x1563697386122416/t0(0) o8->tfs-OST0006-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491258433 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17299.983596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17300.134292] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17302.967166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17303.111668] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17305.901733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17306.055946] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17308.897145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17309.144095] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17311.938056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17312.093332] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17314.936304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17315.100553] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17317.242787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17317.401825] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17318.820725] Lustre: DEBUG MARKER: lctl get_param -n timeout [17319.151953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [17319.303871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17319.434375] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [17324.723110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list tfs.pool1 [17324.975681] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list tfs.lustre [17325.230531] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list tfs.lustre [17326.203923] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17326.460833] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [17327.407138] LustreError: 25522:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062d7a100 x1563697386124080/t0(0) o13->tfs-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17329.043486] Lustre: tfs-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [17329.046088] Lustre: Skipped 2 previous similar messages [17334.042378] LustreError: 137-5: tfs-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17334.042379] LustreError: 137-5: tfs-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17334.052764] LustreError: Skipped 1 previous similar message [17338.633055] Lustre: 1997:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258468/real 1491258468] req@ffff8800628d2400 x1563697386124272/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491258474 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [17338.641047] Lustre: 1997:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [17338.683340] Lustre: server umount tfs-MDT0000 complete [17338.817116] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17345.127060] Lustre: DEBUG MARKER: tunefs.lustre --fsname=lustre --rename=tfs -v /dev/lvm-Role_MDS/P1 [17345.274401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [17345.285648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17350.349699] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17350.349699] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17350.349699] if [ $running -ne $mpts ]; then [17350.349699] echo $(hostname) env are INSANE!; [17350.349699] exit 1; [17350.349699] fi [17350.692964] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17350.692964] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17350.692964] if [ $running -ne $mpts ]; then [17350.692964] echo $(hostname) env are INSANE!; [17350.692964] exit 1; [17350.692964] fi [17351.716768] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [17351.987832] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [17352.240299] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [17352.496514] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [17352.642504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17352.734537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17352.883944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17353.140408] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [17353.641353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17353.666678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17353.829373] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17353.829374] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17353.976300] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17354.239926] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17354.507169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17354.780141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [17355.040420] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17357.247125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17357.397533] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17360.234442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17360.394197] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17363.268868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17363.430028] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17366.312479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17366.475060] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17369.304520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17369.462755] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17372.363040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17372.518664] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17375.424041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17375.574809] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17377.684630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17377.851247] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [17379.277056] Lustre: DEBUG MARKER: lctl get_param -n timeout [17379.605792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [17379.763646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17379.909525] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [17385.286329] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17385.981315] Lustre: DEBUG MARKER: rc=0; [17385.981315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17385.981315] if [[ $? -eq 0 && $val -ne 0 ]]; then [17385.981315] echo $(hostname -s): $val; [17385.981315] rc=$val; [17385.981315] fi; [17385.981315] exit $rc [17386.603283] Lustre: DEBUG MARKER: dmesg [17387.844887] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17388.099285] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [17388.511119] LustreError: 25522:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800628d0000 x1563697386126384/t0(0) o13->lustre-OST0004-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17388.516192] LustreError: 25522:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [17389.504037] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [17389.506826] Lustre: Skipped 4 previous similar messages [17394.502539] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17394.508475] LustreError: Skipped 6 previous similar messages [17400.276110] Lustre: 4744:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491258529/real 1491258529] req@ffff8800628d3000 x1563697386126576/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491258535 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [17400.284474] Lustre: 4744:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [17400.329489] Lustre: server umount lustre-MDT0000 complete [17400.481023] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17407.033669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [17407.033669] /usr/sbin/lctl lustre_build_version 2>/dev/null || [17407.033669] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [17407.381365] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [17407.899055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [17407.973949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [17407.974178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [17408.233900] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [17408.249712] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [17408.295541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [17408.306647] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [17408.447840] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [17408.769069] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17409.050193] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17409.395324] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-E lazy_itable_ [17409.889204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [17419.220967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == conf-sanity test complete, duration 10114 sec ===================================================== 22:29:14 \(1491258554\) [17419.400324] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10114 sec ===================================================== 22:29:14 (1491258554) [17420.280226] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17420.556278] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' Console [trevis-41vm7] departed by on pts/53 at 04-03 22:29. Console [trevis-41vm7] joined by on pts/56 at 04-04 14:36. Console [trevis-41vm7] disconnected from at 04-04 15:32. Console [trevis-41vm7] connected to at 04-04 15:32. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-63-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Tue Jun 23 16:02:31 UTC 2015 (4b89d0c) [ 0.000000] Command line: initrd=/images/SLES-11.4-x86_64/initrd ksdevice=bootif lang= console=tty0 console=ttyS0,115200 install=http://10.9.0.10/cblr/links/SLES-11.4-x86_64 text autoyast=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/SLES-11.4-x86_64/linux BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] No AGP bridge found [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 7d9c5000 - 7fea5000 [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1bee801, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007d600000 s71680 r8192 d22528 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7d610801, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/SLES-11.4-x86_64/initrd ksdevice=bootif lang= console=tty0 console=ttyS0,115200 install=http://10.9.0.10/cblr/links/SLES-11.4-x86_64 text autoyast=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/SLES-11.4-x86_64/linux BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Memory: 2013056k/2097140k available (4556k kernel code, 460k absent, 83624k reserved, 7590k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013487] Security Framework initialized [ 0.014362] AppArmor: AppArmor initialized [ 0.016176] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.019209] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020981] Mount-cache hash table entries: 256 [ 0.022180] Initializing cgroup subsys cpuacct [ 0.023086] Initializing cgroup subsys memory [ 0.024041] Initializing cgroup subsys devices [ 0.024945] Initializing cgroup subsys freezer [ 0.025852] Initializing cgroup subsys net_cls [ 0.026748] Initializing cgroup subsys blkio [ 0.028018] Initializing cgroup subsys perf_event [ 0.029107] Disabled fast string operations [ 0.029986] mce: CPU supports 10 MCE banks [ 0.033299] ACPI: Core revision 20110413 [ 0.035941] Enabling x2apic [ 0.036002] Enabled x2apic [ 0.036007] Switched APIC routing to physical x2apic. [ 0.038113] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.040005] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.064019] kvm-clock: cpu 1, msr 0:7d710801, secondary cpu clock [ 0.064039] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.064068] Brought up 2 CPUs [ 0.064069] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068414] devtmpfs: initialized [ 0.072123] print_constraints: dummy: [ 0.073028] Time: 15:32:59 Date: 04/04/17 [ 0.074089] NET: Registered protocol family 16 [ 0.075151] ACPI: bus type pci registered [ 0.076096] PCI: Using configuration type 1 for base access [ 0.077595] bio: create slab at 0 [ 0.081844] ACPI: Interpreter enabled [ 0.082671] ACPI: (supports S0 S5) [ 0.083598] ACPI: Using IOAPIC for interrupt routing [ 0.088136] ACPI: No dock devices found. [ 0.088988] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.090812] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.093339] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.094895] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.125501] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.131657] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.132845] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.134244] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.135633] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.136826] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.138189] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.140013] vgaarb: loaded [ 0.141000] PCI: Using ACPI for IRQ routing [ 0.141247] NetLabel: Initializing [ 0.142038] NetLabel: domain hash size = 128 [ 0.144011] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.145000] NetLabel: unlabeled traffic allowed by default [ 0.146049] Switching to clocksource kvm-clock [ 0.146961] AppArmor: AppArmor Filesystem Enabled [ 0.147314] pnp: PnP ACPI init [ 0.148092] ACPI: bus type pnp registered [ 0.149724] pnp: PnP ACPI: found 6 devices [ 0.150606] ACPI: ACPI bus type pnp unregistered [ 0.158025] NET: Registered protocol family 2 [ 0.162784] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.165170] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.170315] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.172341] TCP: Hash tables configured (established 262144 bind 65536) [ 0.173515] TCP reno registered [ 0.174293] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.175415] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.176817] NET: Registered protocol family 1 [ 0.177774] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.178876] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.179958] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.181187] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.182263] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.183985] pci 0000:00:01.2: PCI INT D disabled [ 0.185040] Unpacking initramfs... [ 0.836439] Freeing initrd memory: 37760k freed [ 0.857340] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.858557] Initialise module verification [ 0.859637] audit: initializing netlink socket (disabled) [ 0.860684] type=2000 audit(1491319980.860:1): initialized [ 0.877958] bounce pool size: 64 pages [ 0.878827] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.880838] VFS: Disk quotas dquot_6.5.2 [ 0.881800] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883104] msgmni has been set to 1001 [ 0.884418] alg: No test for stdrng (krng) [ 0.885325] Asymmetric key parser 'x509' registered [ 0.886360] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.888048] io scheduler noop registered [ 0.888910] io scheduler deadline registered [ 0.889908] io scheduler cfq registered (default) [ 0.891092] GHES: HEST is not enabled! [ 0.891987] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.893301] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.133231] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.169104] Non-volatile memory driver v1.3 [ 1.170020] Linux agpgart interface v0.103 [ 1.170973] Fixed MDIO Bus: probed [ 1.171807] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.174061] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.175049] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.176107] mousedev: PS/2 mouse device common for all mice [ 1.177353] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 1.179398] cpuidle: using governor ladder [ 1.180282] cpuidle: using governor menu [ 1.181123] EFI Variables Facility v0.08 2004-May-17 [ 1.182290] TCP cubic registered [ 1.183067] Registering the dns_resolver key type [ 1.184112] Loading module verification certificates [ 1.185094] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 1.187035] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 1.188992] registered taskstats version 1 [ 1.190037] Magic number: 13:355:537 [ 1.192905] Freeing unused kernel memory: 1360k freed [ 1.194558] Write protecting the kernel read-only data: 10240k [ 1.198688] Freeing unused kernel memory: 1568k freed [ 1.202416] Freeing unused kernel memory: 916k freed Moving into tmpfs... done. [ 1.387873] SCSI subsystem initialized [ 1.392552] alua: device handler registered [ 1.395164] emc: device handler registered [ 1.397749] hp_sw: device handler registered [ 1.400402] rdac: device handler registered >>> SUSE Linux Enterprise Server 11 installation program v3.3.108 (c) 1996-2010 SUSE Linux Products GmbH <<< Starting udev... [ 1.412200] udev: starting version 147 [ 1.427187] usbcore: registered new interface driver usbfs [ 1.428266] usbcore: registered new interface driver hub [ 1.434642] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.435972] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.437046] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 [ 1.440715] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000902000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.443175] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 1.444902] ACPI: Power Button [PWRF] [ 1.452561] usbcore: registered new device driver usb [ 1.461190] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.463391] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.464462] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 1.476552] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.480331] rtc_cmos 00:01: RTC can wake from S4 [ 1.481519] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.482714] rtc0: alarms up to one day, 114 bytes nvram [ 1.489801] [drm] Initialized drm 1.1.0 20060810 [ 1.499508] This driver is only used in secure boot mode as default [ 1.515188] FDC 0 is a S82078B [ 1.516762] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.521357] scsi0 : ata_piix [ 1.522412] uhci_hcd: USB Universal Host Controller Interface driver [ 1.523959] scsi1 : ata_piix [ 1.524743] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.525952] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.527189] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.529024] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.530039] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.531708] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.532852] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.534093] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.535703] usb usb1: Product: UHCI Host Controller [ 1.536696] usb usb1: Manufacturer: Linux 3.0.101-63-default uhci_hcd [ 1.537882] usb usb1: SerialNumber: 0000:00:01.2 [ 1.538912] hub 1-0:1.0: USB hub found [ 1.539769] hub 1-0:1.0: 2 ports detected [ 1.577962] vda: vda1 vda2 vda4 [ 1.836679] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.877037] Refined TSC clocksource calibration: 2593.643 MHz. ok [?1l%G78[9;0]Loading basic drivers... ok Starting hardware detection... ok (If a driver is not working for you, try booting with brokenmodules=driver_name.) Red Hat Qemu virtual machine drivers: ata_piix*, ata_generic, pata_acpi Red Hat Virtio block device drivers: virtio_pci* Virtio Storage 0 drivers: virtio_blk* Activating usb devices... ok Realtek RTL-8139/8139C/8139C+ drivers: 8139cp*, 8139too Sending DHCP request to eth0... Loading Installation System (1/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (2/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (3/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (4/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (5/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (6/6) -  - missing (optional) Starting haveged daemon haveged: haveged starting up haveged: arch: x86 vendor: intel generic: 0 i_cache: 32 d_cache: 32 loop_idx: 30 loop_idxmax: 40 loop_sz: 31836 loop_szmax: 124334 etime: 10753 havege_ndpt 0 7[?25l done8[?25h starting hald... ok starting syslogd (logging to /dev/tty4)... ok starting klogd... ok starting yast... > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.1.1: open serial > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > pci.1: sysfs drivers > pci.2: get sysfs pci data > pci.4: build list > pci.3: macio > pci.4: vio > pci.5: xen > pci.6: ps3 > pci.7: platform > pci.8: of_platform > pci.9: vm > pci.10: virtio > pci.11: ibmebus > pci.12: uisvirtpci > serial.1: read info > serial.2: build list > misc.5: misc data > usb.1: sysfs drivers > usb.2: usb > usb.3.1: joydev mod > usb.3.2: evdev mod > usb.3.3: input > usb.3.4: lp > usb.3.5: serial > braille.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent > hal.1: read hal data > cpu.1: cpuinfo > cpu.2: assign udi > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > pci.1: sysfs drivers > pci.2: get sysfs pci data > pci.4: build list > pci.3: macio > pci.4: vio > pci.5: xen > pci.6: ps3 > pci.7: platform > pci.8: of_platform > pci.9: vm > pci.10: virtio > pci.11: ibmebus > pci.12: uisvirtpci > misc.5: misc data > misc.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > cpu.1: cpuinfo > misc.5: misc data > usb.1: sysfs drivers > usb.2: usb > usb.3.1: joydev mod > usb.3.2: evdev mod > usb.3.3: input > usb.3.4: lp > usb.3.5: serial > input.1: joydev mod > input.1.1: evdev mod > input.2: input > kbd.2: uml > cpu.1: cpuinfo > kbd.3: serial console > kbd.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent [?25lProbing connected terminal... 78 Initializing virtual console...  H H H H H H H H H [?25h Found an unknown terminal on /dev/console (80 columns x 24 lines). 0 *** Starting YaST2 *** (B)0[?7h[?1h=[?1h=[?1h= YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com   Initializing ...                     [Help] [Back] [Abort] [Next]                                           [Next]   [Next] [Help]      F1 Help  F9 Abort [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkx Downloading installation system language extension... x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj     YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com                                                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com                                                              [Help] [Back] [Abort] [Next]    F1 Help  F9 AbortInitializing... Initializing the installation environment...  YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Initializing...                             Initializing the installation environment...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort  - Probe hardware - Retrieve & Read Control File - Process Profiles and Rules - Parse control file ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Preparing System for Automatic Installation  => Preprobing stage obing hardware...  Analyzing the Computer           - Probe USB devices - Probe FireWire devices - Probe floppy disks devices - Probe hard disk controllers - Load kernel modules for hard disk controllers - Probe hard disks - Search for Linux partitions - Search for system files - Evaluate update possibility - Initialize package manager±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  System Probing  Probing USB devices...  =>  1  x  FireWire devices...  =>  2  x  floppy disks devices... =>  3 x  hard disk controllers... =>  4 x  Loading kernel modules for hard disk controllers... =>  50 x  Probing hard disks...  => 60%  x  Searching for Linux partitions... => 7  x  system files...  => 8  x  Evaluating update possibility... => 9  x  Initializing package manager...  => [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxInitializing repositories...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqjartitions files ssibility manager    Adding the Repository... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading: http://10.9.0.10/cblr/.../SLES-11.4-x86_64/media.1/products  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± links/SLES-11.4-x86_64/media.1/media   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± content.asc   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± key  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±    100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../descr/32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Basis-Devel  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/Dom0-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Dom0-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± _KVM-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/Minimal-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Minimal-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/WBEM-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± WBEM-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± apparmor-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/base-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± base-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhcp_dns_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.p  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± irectory_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ocumentation-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± file_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gateway_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pa  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/gnome-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/jeos-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jeos-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/kde-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± kde-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vm_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lamp_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/laptop-11-38.47.37.x86_64.pat.gz   100%  1.82 kB/s - http://10.9.0.10/.../descr/laptop-11-38.47.37.x86_6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± http://10.9.0.10/.../laptop-32bit-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mail_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/ofed-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ofed-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± racle_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cblr/links/.../setup/descr/packages.DU.gz   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± en  8  13  22  31  9  ±±±44  53%  62%  71  80  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± use/setup/descr/packages.gz  5  10  7  23  8  34  9  ±±46  52% 7%  63%  8  74  9  85  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± tterns   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../print_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sap_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± le-manager_client-1-0.17.30  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-1-0.17.30.x86_64.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/x11-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± x11-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± en_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cblr/links/SLES-11.4-x86_64/license.tar.gz   22  ±±46  69%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../gpg-pubkey-307e3d54-53287cdc.asc   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 9db7c82-510a966b  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± d25d3d9-36e12d04  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 50a3dd1c-50f35137  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 9c800aca-53287d18  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± b37b98a992f  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Building repository 'SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109' cache  100%  7  9  100  Initializing package manager...    ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading: http://10.9.0.10/.../SLES-11.4-x86_64/add_on_products.xml cblr/links/SLES-11.4-x86_64/add_on_products x    Retrieving control file (/cblr/svc/op/ks/system/trevis-41vm7) from HTTP serv Processing Profiles and Rules  arsing control file  Initial Configuration    F10 Next 8 Back  F9 Abort  F10 Next [Help] [Help] [Help] [Next] MM- Configure General Settings - Execute pre-install user scripts - Set up language - Create partition plans   - Configure Software selections    - Configure Bootloader  - Configure runlevel         Preparing System for Automated Installation [Next] [Help] Configuring general settings...  =>  14  x  Executing pre-install user scripts...  =>  28  x  Setting up language  => Configuring language... [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkx Downloading installation system language extension... x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjigure runlevel     YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Preparing System for Automated Installation        x Configure General Settings    x Execute pre-install user scripts    => Set up language    - Create partition plans    - Configure Software selections    - Configure Bootloader    - Configure runlevel                      Configuring language...     ±±±±±±±±±±±±±±±28%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±         [Help] [Back] [Abort] [Next]    F1 Help  F8 Back  F9 Abort  F10 Next YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Preparing System for Automated Installation        x Configure General Settings    x Execute pre-install user scripts    => Set up language    - Create partition plans    - Configure Software selections    - Configure Bootloader    - Configure runlevel                      Configuring language...     ±±±±±±±±±±±±±±±28%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±         [Help] [Back] [Abort] [Next]    F1 Help  F8 Back  F9 Abort  F10 Next ±±±±42x reating partition plans...=> 57% x onfiguring Software selections...=>[?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Error x x These packages could not be found in the software repositories: x x compat-glibc-headersx x policycoreutils-pythonx x10x xx x[OK] [Stop]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj9 lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj8lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj7lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj6lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj5lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj4lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj3lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj2lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj1lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj0 Create partition plans  Configure Software selections  Configure Bootloader  Configure runlevel              71%  x Bootloader...  =>85  x runlevel...  =>100  x   9 Abort  F10 Next  8 Back  F9 Abort  F10 Next  ackage Installation           9 Abort   erform  lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx xx xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  F10 Next 8 Back  F9 Abort  F10 Next Preparing disks... Deleting logical volume /dev/lvm-Role_MDS/forcemnt  Deleting logical volume /dev/lvm-Role_MDS/forcemnt  1 S1  Deleting logical volume /dev/lvm-Role_MDS/S1 2 P Deleting logical volume /dev/lvm-Role_MDS/P1  3 Removing volume group lvm-Role_MDS  Removing volume group lvm-Role_MDS 4 Deleting partition /dev/vda4  Deleting partition /dev/vda4  5 2 Deleting partition /dev/vda2  6 1 Deleting partition /dev/vda1  7 Setting disk label of /dev/vda to MSDOS Setting disk label of /dev/vda to MSDOS 8 Creating volume /dev/vda1  Creating volume /dev/vda1  9 2 Creating volume /dev/vda2  10 Setting type of partition /dev/vda2 to 82 Setting type of partition /dev/vda2 to 82  1 Formatting partition /dev/vda1 (19.99 GB) with ext3  w   xFormatting partition /dev/vda1 (19.99 GB) with ext3 v  2  2  3 4  5  6  7 8  9  10  1 2  3  4  5 6  7  8  9 20  1  2  3 4  5  6 7  8  9  30 1  2  3  4 5  6  7  8 9  40  1  ±±±±42 3  ±±±44  ±±45  ±46 7  48 49 50 1 2% 3%  4 5%  6%  7%  8 9%  60  1  2 3  4  5  6 7  8  9  70 1  2  3  4 5  6  7 8  9  80  1 2  3  4  5 6  7  8  9 90  1  2  3 4  5  6  7 100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 2 (13.67swap  x wx   xFormatting partition /dev/vda2 (13.67 GB) with swap v   3 Mounting /dev/vda2 to swap   x wx   xMounting /dev/vda2 to swap v   4 Adding entry for mount point swap to /etc/fstab  x wx   xAdding entry for mount point swap to /etc/fstab v   5 Mounting /dev/vda1 to /   x wx   xMounting /dev/vda1 to / v  6 Adding entry for mount point / to /etc/fstab  x wx   xAdding entry for mount point / to /etc/fstab v   7  ackage                                9 Abort   erform  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xMediaxSizexPackagesxTimex xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Adding entry for mount point / to /etc/fstab  100% Preparing disks...   ±±±±±±±±±±±±±±±±±±±±±±±17%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26 Installing Packages...  xSizexPackagesxTq Total x 1.09 GBx 643x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 403.54 MBx 433x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 1.09 GBx 643x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 403.54 MBx 433x http-10.9.0.10-e8820372xxx tu Downloading master-boot-code (download size 5.00 kB) Downloading master-boot-code (download size 5.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing master-boot-code-1.14-70.11.i586.rpm (installed size 3.00 kB) (Remaining: 1.09 GB) Installing master-boot-code-1.14-70.11.i586.rpm (installed size 3.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  96%  100  2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading adaptec-firmware (download size 22.00 kB)  Downloading adaptec-firmware (download size 22.00 kB)  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing adaptec-firmware-1.35-2.15.4.noarch.rpm (installed size 27.00 k Installing adaptec-firmware-1.35-2.15.4.noarch.rpm (installed size 27.00  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  80%  99  100  1 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading branding-SLES (download size 85.00 kB)   w   x~ kB) x    xDownloading branding-SLES (download size 85.00 kB) v   18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing branding-SLES-11-3.20.30.noarch.rpm (installed size 2.00 kB)  x w   xInstalling branding-SLES-11-3.20.30.noarch.rpm (installed size 2.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading brocade-firmware (download size 999.00 kB)   x w   xDownloading brocade-firmware (download size 999.00 kB) v   5  10  5  21  6  31  7  ±±±±42  ±47  52% 8%  63%  8  74  9  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing brocade-firmware-3.2.3.0-0.11.35.noarch.rpm (installed size 3.0  x w   xInstalling brocade-firmware-3.2.3.0-0.11.35.noarch.rpm (installed size 3v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  9  24  30  6  ±±±±42  48 54%  9%  65  72  8  84  9  95  100  xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 1.09 GBx 639x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 400.44 MBx 429x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading desktop-translations (download size 2.42 MB)   x x   x~.06 MB) w    xDownloading desktop-translations (download size 2.42 MB) v   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing desktop-translations-11.1-10.50.1.noarch.rpm (installed size 17  x w   xInstalling desktop-translations-11.1-10.50.1.noarch.rpm (installed size v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  383 8 7  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2-branding-SLES (download size 6.00 kB)   x x   x~17.00 MB) w    xDownloading glib2-branding-SLES (download size 6.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-branding-SLES-2.16-47.3.noarch.rpm (installed size 4.00 k 7  x x   xInstalling glib2-branding-SLES-2.16-47.3.noarch.rpm (installed size 4.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  97%  100  7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gnome-icon-theme (download size 3.52 MB)   x   x~ kB) x    xDownloading gnome-icon-theme (download size 3.52 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gnome-icon-theme-2.28.0-1.2.11.noarch.rpm (installed size 10.43  x   xInstalling gnome-icon-theme-2.28.0-1.2.11.noarch.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  73.01 6 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hicolor-icon-theme (download size 35.00 kB)   x   x~43 MB) x    xDownloading hicolor-icon-theme (download size 35.00 kB) q   ±±45  92%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hicolor-icon-theme-0.10-102.22.noarch.rpm (installed size 48.00 6  x   xInstalling hicolor-icon-theme-0.10-102.22.noarch.rpm (installed size 48.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  23  8  33  8  ±±±±43  48 53%  8%  63%  8  95  100  2.96 5  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icom-firmware (download size 18.00 kB)   x   x~00 kB) x    xDownloading icom-firmware (download size 18.00 kB) q   88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icom-firmware-1.0-124.28.4.noarch.rpm (installed size 47.00 kB)  x   xInstalling icom-firmware-1.0-124.28.4.noarch.rpm (installed size 47.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  16  99%  100  1 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kernel-firmware (download size 15.15 MB)   x   x~B) x    xDownloading kernel-firmware (download size 15.15 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  - 12.02 MB/s (download size 15.15 MB) 5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-firmware-20110923-0.57.21.noarch.rpm (installed size 39.  x   xInstalling kernel-firmware-20110923-0.57.21.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  33.77 3 2  3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading latex2html-pngicons (download size 13.00 kB)   x   x~9.14 MB) x    xDownloading latex2html-pngicons (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing latex2html-pngicons-2002.2.1-604.31.noarch.rpm (installed size 2  x   xInstalling latex2html-pngicons-2002.2.1-604.31.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  33  9  ±46  54%  62%  9  77  82  8  95  100  6 2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading licenses (download size 340.00 kB)   x   x~e 8.00 kB) x    xDownloading licenses (download size 340.00 kB) q   5  10  7  22  8  33  8  ±47  52% 61%  6  71  80  5  90  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing licenses-20070810-2.15.noarch.rpm (installed size 3.29 MB)  x   xInstalling licenses-20070810-2.15.noarch.rpm (installed size 3.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  6  81  6  91  6  100  0.48 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man-pages (download size 2.20 MB)   x   xDownloading man-pages (download size 2.20 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-pages-3.15-2.29.1.noarch.rpm (installed size 2.03 MB)  x   xInstalling man-pages-3.15-2.29.1.noarch.rpm (installed size 2.03 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  28.45 0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man-pages-posix (download size 2.33 MB)   x   xDownloading man-pages-posix (download size 2.33 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-pages-posix-2003a-3.7.noarch.rpm (installed size 2.20 MB)  x   xInstalling man-pages-posix-2003a-3.7.noarch.rpm (installed size 2.20 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  6.2 19 1  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading providers (download size 24.00 kB)   x   xDownloading providers (download size 24.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing providers-2006.11.21-105.21.noarch.rpm (installed size 136.00 k 1  x   xInstalling providers-2006.11.21-105.21.noarch.rpm (installed size 136.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  25  31  79%  9  100  11 8  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-sam-data (download size 5.06 MB)   x   x~ kB) x    xDownloading suse-sam-data (download size 5.06 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-sam-data-0.8.5-0.4.9.noarch.rpm (installed size 5.01 MB)  x   xInstalling suse-sam-data-0.8.5-0.4.9.noarch.rpm (installed size 5.01 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  91  6  100  1.10 7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-trans-en_US (download size 129.00 kB)   x   xDownloading yast2-trans-en_US (download size 129.00 kB) q   1  23  35  48 60%  73  85  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-trans-en_US-2.17.35-0.7.37.noarch.rpm (installed size 584  x   xInstalling yast2-trans-en_US-2.17.35-0.7.37.noarch.rpm (installed size 5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  33  8  ±±±44  50 61%  8  73  80  5  96  100  0.53 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-trans-stats (download size 9.00 kB)   x   x~84.00 kB) x    xDownloading yast2-trans-stats (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-trans-stats-2.15.0-2.17.noarch.rpm (installed size 1.00 k  x   xInstalling yast2-trans-stats-2.15.0-2.17.noarch.rpm (installed size 1.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  5  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading boost-license (download size 23.00 kB)   x   x~ kB) x    xDownloading boost-license (download size 23.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing boost-license-1.36.0-12.6.49.x86_64.rpm (installed size 2.00 kB  x   xInstalling boost-license-1.36.0-12.6.49.x86_64.rpm (installed size 2.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  92%  100  4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cracklib-dict-full (download size 3.60 MB)   x   x~kB) x    xDownloading cracklib-dict-full (download size 3.60 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cracklib-dict-full-2.8.12-43.16.x86_64.rpm (installed size 10.4  x   xInstalling cracklib-dict-full-2.8.12-43.16.x86_64.rpm (installed size 10q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  10.07 3 1021.49 M  3  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading filesystem (download size 51.00 kB)   x   x~.46 MB) x    xDownloading filesystem (download size 51.00 kB) q   31  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing filesystem-11.1-3.5.3.x86_64.rpm (installed size 0.00 B) 021.49 MB)  x   xInstalling filesystem-11.1-3.5.3.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ofed-doc (download size 220.00 kB)   x   xDownloading ofed-doc (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-doc-1.5.4.1-20.26.x86_64.rpm (installed size 829.00 kB)  x   xInstalling ofed-doc-1.5.4.1-20.26.x86_64.rpm (installed size 829.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  8  34  ±±±±±41  ±46  51 6%  62%  7  7  84  90  7  100  09.26 1 0.68  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sash (download size 317.00 kB)   x   xDownloading sash (download size 317.00 kB) q   1  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sash-3.7-170.1.30.x86_64.rpm (installed size 769.00 kB) 0.68  x   xInstalling sash-3.7-170.1.30.x86_64.rpm (installed size 769.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  ±±±±±41  49 58%  66%  74  83  91  8  100  8.51 0 19.93  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-release-DVD (download size 4.00 kB)   x   xDownloading sles-release-DVD (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-release-DVD-11.4-1.109.x86_64.rpm (installed size 1.00 kB) 19.93  x   xInstalling sles-release-DVD-11.4-1.109.x86_64.rpm (installed size 1.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  75%  100  09  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading terminfo (download size 397.00 kB)   x   x~B) x    xDownloading terminfo (download size 397.00 kB) q   7  12  20  8  36  ±±±44  52% 60%  8  76  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing terminfo-5.6-90.55.x86_64.rpm (installed size 1.53 MB)  x   xInstalling terminfo-5.6-90.55.x86_64.rpm (installed size 1.53 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  6.97 8 8.39  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading terminfo-base (download size 114.00 kB)   x   xDownloading terminfo-base (download size 114.00 kB) q   13  28  ±±±±42  56%  70%  84  98  100   30 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing terminfo-base-5.6-90.55.x86_64.rpm (installed size 996.00 kB) 8.39  x   xInstalling terminfo-base-5.6-90.55.x86_64.rpm (installed size 996.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  9  25  31  8  ±±±44  50 7%  63%  9  76  82  8  94  9  100  00 7 7.42  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-country-data (download size 44.00 kB)   x   xDownloading yast2-country-data (download size 44.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-country-data-2.17.55-0.7.1.x86_64.rpm (installed size 166 7.42  x   xInstalling yast2-country-data-2.17.55-0.7.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  81%  99  100  5.84 6 26  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxq Total x 1017.26 MBx 616x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-e8820372xxx tu Downloading pam-doc (download size 496.00 kB)   x   x~66.00 kB) x    xDownloading pam-doc (download size 496.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  83  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-doc-1.1.5-0.17.2.x86_64.rpm (installed size 1.62 MB) 26  x   xInstalling pam-doc-1.1.5-0.17.2.x86_64.rpm (installed size 1.62 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  9  24  31  6  ±±±±±41  ±46  51 6%  61%  6  73  80  6  93  9  100  SizexPackagesxT1015.64 MBx 616x x x 305.84 MBx 406x x x qu  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading release-notes-sles (download size 517.00 kB)   x   xDownloading release-notes-sles (download size 517.00 kB) q   6  12  8  24  30  7  ±±±±43  49 55%  61%  8  74  80  6  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing release-notes-sles-11.4.26-0.35.1.x86_64.rpm (installed size 1. 5.64  x   xInstalling release-notes-sles-11.4.26-0.35.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  26  33  40  ±±45  52% 9%  65  71  8  85  92  7  100  3.86  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading snmp-mibs (download size 263.00 kB)   x   x~1.78 MB) x    xDownloading snmp-mibs (download size 263.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  9  85  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing snmp-mibs-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 1.54 MB) 3.86  x   xInstalling snmp-mibs-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 1.54 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  9  24  30  8  ±±45  50 5%  60%  8  73  8  84  90  7  100  2.32  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-i18ndata (download size 3.14 MB)   x   x~B) x    xDownloading glibc-i18ndata (download size 3.14 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-i18ndata-2.11.3-17.102.1.x86_64.rpm (installed size 10.45 2.32  x   xInstalling glibc-i18ndata-2.11.3-17.102.1.x86_64.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  01.86  2 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-schema (download size 72.00 kB)   x   x~45 MB) x    xDownloading yast2-schema (download size 72.00 kB) q   19  ±±±±±41  64%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-schema-2.17.7-8.1.x86_64.rpm (installed size 925.00 kB) 01.86  x   xInstalling yast2-schema-2.17.7-8.1.x86_64.rpm (installed size 925.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  36  ±±±±43  50 7%  64%  71  7  84  91  8  100  0.9  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-admin_en-pdf (download size 3.07 MB)   x   xDownloading sles-admin_en-pdf (download size 3.07 MB) q   5  10 - 232.09 kB/s (download size 3.07 MB)  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-admin_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4.09 M 0.9  x   xInstalling sles-admin_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4.09q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  9  35  ±±±±±41  ±47  53%  8%  64%  70  6  81  7  93  9  100   996.87  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-manuals_en (download size 14.49 MB)   x   x~ MB) x    xDownloading sles-manuals_en (download size 14.49 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-manuals_en-11.4-0.33.1.noarch.rpm (installed size 22.03 MB 996.87 MB)   x   xInstalling sles-manuals_en-11.4-0.33.1.noarch.rpm (installed size 22.03 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  74.84  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-storage_en-pdf (download size 2.90 MB)   x   x~MB) x    xDownloading sles-storage_en-pdf (download size 2.90 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-storage_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.84 74.84  x   xInstalling sles-storage_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  ±47  52% 8%  63%  8  73  8  84  9  94  9  100  1.00  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-installquick_en-pdf (download size 4.22 MB)   x   x~84 MB) x    xDownloading sles-installquick_en-pdf (download size 4.22 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-installquick_en-pdf-11.4-0.33.1.noarch.rpm (installed size 1.00  x   xInstalling sles-installquick_en-pdf-11.4-0.33.1.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  31  6  ±±±±±41  ±46  52% 8%  63%  9  74  9  84  9  94  9  100  66.58  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-deployment_en-pdf (download size 3.65 MB)   x   x~ze 4.42 MB) x    xDownloading sles-deployment_en-pdf (download size 3.65 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-deployment_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4 66.58  x   xInstalling sles-deployment_en-pdf-11.4-0.33.1.noarch.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  6  ±±±±±41  ±46  51 6%  62%  7  72  7  82  8  93  8  100  2.26  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-tuning_en-pdf (download size 706.00 kB)   x   x~ 4.33 MB) x    xDownloading sles-tuning_en-pdf (download size 706.00 kB) q   6  11  6  23  30  6  ±±±±±41  48 54%  9%  64  70  5  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-tuning_en-pdf-11.4-0.33.1.noarch.rpm (installed size 1.04 2.26  x   xInstalling sles-tuning_en-pdf-11.4-0.33.1.noarch.rpm (installed size 1.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  1.21  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-security_en-pdf (download size 2.19 MB)   x   x~4 MB) x    xDownloading sles-security_en-pdf (download size 2.19 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-security_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.0 1.21  x   xInstalling sles-security_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  7  ±±±±43  49 55%  61%  7  72  8  84  90  6  100  58.17  4  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfs-doc (download size 108.00 kB)   x   x~.04 MB) x    xDownloading nfs-doc (download size 108.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-doc-1.2.3-18.47.1.x86_64.rpm (installed size 363.00 kB) 58.17  x   xInstalling nfs-doc-1.2.3-18.47.1.x86_64.rpm (installed size 363.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  29  ±47  61%  6  72  80  7  92  8  100  xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 957.82 MBx 603x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc (download size 1.86 MB)   x   xDownloading glibc (download size 1.86 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-2.11.3-17.102.1.x86_64.rpm (installed size 5.30 MB) 7.82  x   xInstalling glibc-2.11.3-17.102.1.x86_64.rpm (installed size 5.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  71  6  81  6  91  6  100  2.5  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zlib (download size 45.00 kB)   x   xDownloading zlib (download size 45.00 kB) q   35  70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zlib-1.2.7-0.14.1.x86_64.rpm (installed size 87.00 kB) 2.5  x   xInstalling zlib-1.2.7-0.14.1.x86_64.rpm (installed size 87.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  73%  99  100  44  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-base (download size 1.10 MB)   x   xDownloading perl-base (download size 1.10 MB) q   5  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  72  8  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-base-5.10.0-64.80.1.x86_64.rpm (installed size 3.30 MB) 44  x   xInstalling perl-base-5.10.0-64.80.1.x86_64.rpm (installed size 3.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  32  7  ±±±±42  ±47  52% 7%  62%  8  73  9  85  90  6  100  49.13  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dosfstools (download size 85.00 kB)   x   xDownloading dosfstools (download size 85.00 kB) q   18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dosfstools-3.0.26-3.1.x86_64.rpm (installed size 217.00 kB) 49.13  x   xInstalling dosfstools-3.0.26-3.1.x86_64.rpm (installed size 217.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±47  60%  85  91  6  100  8.92 599 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ethtool (download size 94.00 kB)   x   xDownloading ethtool (download size 94.00 kB) q   16  33  50 67%  84  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ethtool-6.2.6.39-0.29.3.x86_64.rpm (installed size 262.00 kB) 8.92  x   xInstalling ethtool-6.2.6.39-0.29.3.x86_64.rpm (installed size 262.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 73%  8  90  7  100  67  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgomp1 (download size 66.00 kB)   x   xDownloading libgomp1 (download size 66.00 kB) q   24  48 72%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgomp1-5.3.1+r233831-10.1.x86_64.rpm (installed size 140.00 k 67  x   xInstalling libgomp1-5.3.1+r233831-10.1.x86_64.rpm (installed size 140.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  91%  9  100  53  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsqlite3-0 (download size 288.00 kB)   x   x~ kB) x    xDownloading libsqlite3-0 (download size 288.00 kB) q   5  11  6  22  7  33  8  ±±±44  50 5%  61%  6  72  7  83  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsqlite3-0-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 766.00 53  x   xInstalling libsqlite3-0-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 766.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  25  33  ±±±±±41  50 8%  66%  75  83  91  9  100  7.78  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libbz2-1 (download size 44.00 kB)   x   x~0 kB) x    xDownloading libbz2-1 (download size 44.00 kB) q   39  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libbz2-1-1.0.5-34.255.1.x86_64.rpm (installed size 82.00 kB) 7.78  x   xInstalling libbz2-1-1.0.5-34.255.1.x86_64.rpm (installed size 82.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  88  99  100  0  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libquadmath0 (download size 142.00 kB)   x   xDownloading libquadmath0 (download size 142.00 kB) q   11  22  33  ±±45  56%  67%  78  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libquadmath0-5.3.1+r233831-10.1.x86_64.rpm (installed size 252. 0  x   xInstalling libquadmath0-5.3.1+r233831-10.1.x86_64.rpm (installed size 25q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  50 76%  99  100  46  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libusb-1_0-0 (download size 91.00 kB)   x   x~2.00 kB) x    xDownloading libusb-1_0-0 (download size 91.00 kB) q   17  34  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libusb-1_0-0-1.0.9-8.4.4.1.x86_64.rpm (installed size 239.00 kB 46  x   xInstalling libusb-1_0-0-1.0.9-8.4.4.1.x86_64.rpm (installed size 239.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  37  64%  90  8  100  22  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc_s1 (download size 49.00 kB)   x   x~kB) x    xDownloading libgcc_s1 (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc_s1-5.3.1+r233831-10.1.x86_64.rpm (installed size 91.00 k 22  x   xInstalling libgcc_s1-5.3.1+r233831-10.1.x86_64.rpm (installed size 91.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  70%  99  100  13  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpng12-0 (download size 72.00 kB)   x   x~ kB) x    xDownloading libpng12-0 (download size 72.00 kB) q   22  ±±±44  67%  89  100   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpng12-0-1.2.31-5.43.1.x86_64.rpm (installed size 160.00 kB) 13  x   xInstalling libpng12-0-1.2.31-5.43.1.x86_64.rpm (installed size 160.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  80%  99  100  6.98  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgfortran3 (download size 273.00 kB)   x   x~) x    xDownloading libgfortran3 (download size 273.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgfortran3-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.14 6.98  x   xInstalling libgfortran3-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  5.84  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++6 (download size 383.00 kB)   x   x~14 MB) x    xDownloading libstdc++6 (download size 383.00 kB) q   8  16  25  33  ±±±±±41  50 8%  66%  71  9  87  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++6-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.50 M 5.84  x   xInstalling libstdc++6-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.50q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  25  33  ±±±±±41  50 8%  66%  75  83  91  9  100  4.3  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libicu (download size 4.24 MB)   x   x~ MB) x    xDownloading libicu (download size 4.24 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libicu-4.0-46.1.x86_64.rpm (installed size 16.52 MB) 4.3  x   xInstalling libicu-4.0-46.1.x86_64.rpm (installed size 16.52 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  27.83  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 927.83 MBx 588x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-e8820372xxx tu Downloading xorg-x11-libXdmcp (download size 11.00 kB)   x   xDownloading xorg-x11-libXdmcp (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXdmcp-7.4-1.15.x86_64.rpm (installed size 23.00 kB) 27.83  x   xInstalling xorg-x11-libXdmcp-7.4-1.15.x86_64.rpm (installed size 23.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 5 0  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXau (download size 8.00 kB)   x   x~B) x    xDownloading xorg-x11-libXau (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXau-7.4-1.15.x86_64.rpm (installed size 15.00 kB) 0  x   xInstalling xorg-x11-libXau-7.4-1.15.x86_64.rpm (installed size 15.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 4 79  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libICE (download size 41.00 kB)   x   xDownloading xorg-x11-libICE (download size 41.00 kB) q   38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libICE-7.4-1.15.x86_64.rpm (installed size 101.00 kB) 79  x   xInstalling xorg-x11-libICE-7.4-1.15.x86_64.rpm (installed size 101.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  63%  99  100  7 3 6  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vlan (download size 34.00 kB)   x   x~) x    xDownloading vlan (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vlan-1.9-62.24.1.x86_64.rpm (installed size 98.00 kB) 6  x   xInstalling vlan-1.9-62.24.1.x86_64.rpm (installed size 98.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  30  72%  90  6  100  61 2 5  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading utempter (download size 19.00 kB)   x   xDownloading utempter (download size 19.00 kB) q   86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing utempter-0.5.5-106.20.1.x86_64.rpm (installed size 42.00 kB) 5  x   xInstalling utempter-0.5.5-106.20.1.x86_64.rpm (installed size 42.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  40  98%  100  56 1 5  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tunctl (download size 7.00 kB)   x   xDownloading tunctl (download size 7.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tunctl-20070815-10.22.x86_64.rpm (installed size 11.00 kB) 5  x   xInstalling tunctl-20070815-10.22.x86_64.rpm (installed size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  5 0 4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcpd (download size 78.00 kB)   x   xDownloading tcpd (download size 78.00 kB) q   2  ±±±±±41  61%  82  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcpd-7.6-856.13.x86_64.rpm (installed size 210.00 kB) 4  x   xInstalling tcpd-7.6-856.13.x86_64.rpm (installed size 210.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  24  9  40  50 5%  65%  88  95  100  3 399 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sdpnetstat (download size 47.00 kB)   x   xDownloading sdpnetstat (download size 47.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sdpnetstat-1.60-32.26.x86_64.rpm (installed size 119.00 kB) 3  x   xInstalling sdpnetstat-1.60-32.26.x86_64.rpm (installed size 119.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  84%  99  100  23 8 22  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rds-tools (download size 34.00 kB)   x   xDownloading rds-tools (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rds-tools-2.0.4-6.18.x86_64.rpm (installed size 83.00 kB) 22  x   xInstalling rds-tools-2.0.4-6.18.x86_64.rpm (installed size 83.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  40  92%  9  100  15 7 14  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading prctl (download size 8.00 kB)   x   xDownloading prctl (download size 8.00 kB) q   100%  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing prctl-1.3-488.22.x86_64.rpm (installed size 12.00 kB) 14  x   xInstalling prctl-1.3-488.22.x86_64.rpm (installed size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  99  100  4 6 3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading popt (download size 46.00 kB)   x   xDownloading popt (download size 46.00 kB) q   34  69%  100  - 45.38 kB/s (download size 46.00 kB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing popt-1.7-37.63.64.1.x86_64.rpm (installed size 69.00 kB) 3  x   xInstalling popt-1.7-37.63.64.1.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  48 53%  60%  6  71  6  81  6  92  7  100  07 5 06  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pax (download size 58.00 kB)   x   xDownloading pax (download size 58.00 kB) q   27  54%  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pax-3.4-135.21.x86_64.rpm (installed size 118.00 kB) 06  x   xInstalling pax-3.4-135.21.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  91%  9  100  4.96 4 6.95  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading patch (download size 63.00 kB)   x   xDownloading patch (download size 63.00 kB) q   25  50 76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing patch-2.5.9-252.22.2.x86_64.rpm (installed size 118.00 kB) 6.95  x   xInstalling patch-2.5.9-252.22.2.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  80%  8  99  100  84 3 83  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netdate (download size 13.00 kB)   x   xDownloading netdate (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netdate-1.2-713.22.x86_64.rpm (installed size 19.00 kB) 83  x   xInstalling netdate-1.2-713.22.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  78%  87  99  100  2 2 1  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netcat (download size 59.00 kB)   x   xDownloading netcat (download size 59.00 kB) q   26  54%  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netcat-1.10-1005.22.x86_64.rpm (installed size 129.00 kB) 1  x   xInstalling netcat-1.10-1005.22.x86_64.rpm (installed size 129.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  64%  70  5  80  6  94  9  100  70 1 69  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading net-tools (download size 241.00 kB)   x   xDownloading net-tools (download size 241.00 kB) q   6  13  9  26  33  9  ±46  53%  9%  66  73  9  86  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing net-tools-1.60-725.35.1.x86_64.rpm (installed size 758.00 kB) 69  x   xInstalling net-tools-1.60-725.35.1.x86_64.rpm (installed size 758.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  24  32  7  ±±±44  51 9%  65  71  7  83  9  94  9  100  3.96 0 5.95  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mingetty (download size 15.00 kB)   x   xDownloading mingetty (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mingetty-1.0.7s-92.22.x86_64.rpm (installed size 25.00 kB) 5.95  x   xInstalling mingetty-1.0.7s-92.22.x86_64.rpm (installed size 25.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  90%  9  100  3 89 2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lsscsi (download size 30.00 kB)   x   xDownloading lsscsi (download size 30.00 kB) q   53%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsscsi-0.23-0.12.1.x86_64.rpm (installed size 63.00 kB) 2  x   xInstalling lsscsi-0.23-0.12.1.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  92  9  100  87 8 86  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxtables9 (download size 33.00 kB)   x   xDownloading libxtables9 (download size 33.00 kB) q   ±47  96%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxtables9-1.4.16.3-1.37.x86_64.rpm (installed size 47.00 kB) 86  x   xInstalling libxtables9-1.4.16.3-1.37.x86_64.rpm (installed size 47.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 7 1  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxcrypt (download size 38.00 kB)   x   x~) x    xDownloading libxcrypt (download size 38.00 kB) q   ±±±±42  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxcrypt-3.0.3-0.6.1.x86_64.rpm (installed size 132.00 kB) 1  x   xInstalling libxcrypt-3.0.3-0.6.1.x86_64.rpm (installed size 132.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  27  ±±±±±41  55%  68%  79  90  6  100  70 6 68  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libx86 (download size 214.00 kB)   x   xDownloading libx86 (download size 214.00 kB) q   7  14  22  9  37  ±±±44  52% 9%  67  74  82  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libx86-1.1-13.17.x86_64.rpm (installed size 1.24 MB) 68  x   xInstalling libx86-1.1-13.17.x86_64.rpm (installed size 1.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  6  91  6  100  2.46 5 4.45  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libusb-0_1-4 (download size 27.00 kB)   x   xDownloading libusb-0_1-4 (download size 27.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libusb-0_1-4-0.1.12-139.1.1.x86_64.rpm (installed size 67.00 kB 4.45  x   xInstalling libusb-0_1-4-0.1.12-139.1.1.x86_64.rpm (installed size 67.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  92%  9  100  39 4 38  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsepol1 (download size 96.00 kB)   x   x~kB) x    xDownloading libsepol1 (download size 96.00 kB) q   16  33  50 67%  83  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsepol1-2.0.41-2.2.9.x86_64.rpm (installed size 240.00 kB) 38  x   xInstalling libsepol1-2.0.41-2.2.9.x86_64.rpm (installed size 240.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  80%  99  100  16 3 15  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libselinux1 (download size 52.00 kB)   x   xDownloading libselinux1 (download size 52.00 kB) q   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libselinux1-2.0.91-4.2.1.x86_64.rpm (installed size 116.00 kB) 15  x   xInstalling libselinux1-2.0.91-4.2.1.x86_64.rpm (installed size 116.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  99%  100  04 2 03  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpixman-1-0 (download size 181.00 kB)   x   x~) x    xDownloading libpixman-1-0 (download size 181.00 kB) q   8  17  26  35  ±±±44  53%  62%  70  9  88  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpixman-1-0-0.24.4-0.15.1.x86_64.rpm (installed size 541.00 k 03  x   xInstalling libpixman-1-0-0.24.4-0.15.1.x86_64.rpm (installed size 541.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  35  ±47  59%  70  82  94  9  100  1.52 1 3.50  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpcap0 (download size 99.00 kB)   x   x~ kB) x    xDownloading libpcap0 (download size 99.00 kB) q   15  32  48 64%  80  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpcap0-0.9.8-50.10.1.x86_64.rpm (installed size 245.00 kB) 3.50  x   xInstalling libpcap0-0.9.8-50.10.1.x86_64.rpm (installed size 245.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 78%  86  93  9  100  28 0 27  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenct1 (download size 21.00 kB)   x   xDownloading libopenct1 (download size 21.00 kB) q   78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenct1-0.6.15-5.20.x86_64.rpm (installed size 39.00 kB) 27  x   xInstalling libopenct1-0.6.15-5.20.x86_64.rpm (installed size 39.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 79 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnuma1 (download size 27.00 kB)   x   xDownloading libnuma1 (download size 27.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnuma1-2.0.10-0.9.31.x86_64.rpm (installed size 44.00 kB) 3  x   xInstalling libnuma1-2.0.10-0.9.31.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 8 18  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnscd (download size 16.00 kB)   x   xDownloading libnscd (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnscd-2.0.2-73.18.x86_64.rpm (installed size 33.00 kB) 18  x   xInstalling libnscd-2.0.2-73.18.x86_64.rpm (installed size 33.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  96%  100  16 7 5  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnl (download size 121.00 kB)   x   xDownloading libnl (download size 121.00 kB) q   13  26  39  52% 66%  79  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnl-1.1-49.22.6.1.x86_64.rpm (installed size 362.00 kB) 5  x   xInstalling libnl-1.1-49.22.6.1.x86_64.rpm (installed size 362.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  70%  88  96  100  0.81 6 2.80  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnfnetlink0 (download size 22.00 kB)   x   xDownloading libnfnetlink0 (download size 22.00 kB) q   73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnfnetlink0-1.0.0+git1-9.5.56.x86_64.rpm (installed size 45.0 2.80  x   xInstalling libnfnetlink0-1.0.0+git1-9.5.56.x86_64.rpm (installed size 45q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  57%  96%  100  77 5 75  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses6 (download size 241.00 kB)   x   x~.00 kB) x    xDownloading libncurses6 (download size 241.00 kB) q   6  13  9  26  33  9  ±46  53%  9%  66  73  9  86  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses6-5.6-90.55.x86_64.rpm (installed size 856.00 kB) 75  x   xInstalling libncurses6-5.6-90.55.x86_64.rpm (installed size 856.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  37  ±±±44  52% 9%  65  72  80  7  98  100  299.93 4 1.92  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses5 (download size 242.00 kB)   x   xDownloading libncurses5 (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% 9%  66  72  9  86  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses5-5.6-90.55.x86_64.rpm (installed size 915.00 kB) 1.92  x   xInstalling libncurses5-5.6-90.55.x86_64.rpm (installed size 915.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  20  7  38  ±±45  52% 9%  66  74  81  8  9  100  04 3 03  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading liblzo2-2 (download size 45.00 kB)   x   xDownloading liblzo2-2 (download size 45.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblzo2-2-2.03-12.3.1.x86_64.rpm (installed size 135.00 kB) 03  x   xInstalling liblzo2-2-2.03-12.3.1.x86_64.rpm (installed size 135.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±47  95%  100  8.91 2 0.89  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading liblzma5 (download size 80.00 kB)   x   xDownloading liblzma5 (download size 80.00 kB) q   19  40  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblzma5-5.0.3-0.12.1.x86_64.rpm (installed size 139.00 kB) 0.89  x   xInstalling liblzma5-5.0.3-0.12.1.x86_64.rpm (installed size 139.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  92%  9  100  77 1 76  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libltdl7 (download size 21.00 kB)   x   xDownloading libltdl7 (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libltdl7-2.2.6-2.131.1.x86_64.rpm (installed size 39.00 kB) 76  x   xInstalling libltdl7-2.2.6-2.131.1.x86_64.rpm (installed size 39.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 0 2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libjpeg (download size 67.00 kB)   x   xDownloading libjpeg (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libjpeg-6.2.0-879.12.1.x86_64.rpm (installed size 143.00 kB) 2  x   xInstalling libjpeg-6.2.0-879.12.1.x86_64.rpm (installed size 143.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  89%  9  100  59 69 58  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libiptc0 (download size 33.00 kB)   x   xDownloading libiptc0 (download size 33.00 kB) q   48 98%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libiptc0-1.4.16.3-1.37.x86_64.rpm (installed size 65.00 kB) 58  x   xInstalling libiptc0-1.4.16.3-1.37.x86_64.rpm (installed size 65.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  87%  99  100  3 8 2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libibverbs (download size 69.00 kB)   x   xDownloading libibverbs (download size 69.00 kB) q   22  ±46  69%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibverbs-1.1.4-0.16.37.x86_64.rpm (installed size 234.00 kB) 2  x   xInstalling libibverbs-1.1.4-0.16.37.x86_64.rpm (installed size 234.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  2  3  ±±±±42  52% 77%  86  95  100  30 7 29  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libibumad3 (download size 43.00 kB)   x   x~) x    xDownloading libibumad3 (download size 43.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibumad3-1.3.7-10.26.x86_64.rpm (installed size 62.00 kB) 29  x   xInstalling libibumad3-1.3.7-10.26.x86_64.rpm (installed size 62.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  69%  74  9  84  9  94  9  100  24 6 3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgssglue1 (download size 21.00 kB)   x   xDownloading libgssglue1 (download size 21.00 kB) q   76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgssglue1-0.1-20.2.1.x86_64.rpm (installed size 40.00 kB) 3  x   xInstalling libgssglue1-0.1-20.2.1.x86_64.rpm (installed size 40.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 5 19  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libevent-1_4-2 (download size 64.00 kB)   x   xDownloading libevent-1_4-2 (download size 64.00 kB) q   24  49 74%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libevent-1_4-2-1.4.5-24.24.1.x86_64.rpm (installed size 226.00 19  x   xInstalling libevent-1_4-2-1.4.5-24.24.1.x86_64.rpm (installed size 226.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  ±±45  63%  91  9  100  7.98 4 19.97  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libelf1 (download size 44.00 kB)   x   x~0 kB) x    xDownloading libelf1 (download size 44.00 kB) q   36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libelf1-0.152-4.9.17.x86_64.rpm (installed size 83.00 kB) 19.97  x   xInstalling libelf1-0.152-4.9.17.x86_64.rpm (installed size 83.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  77%  99  100  0 3 89  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libelf0 (download size 36.00 kB)   x   xDownloading libelf0 (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libelf0-0.8.12-1.1.52.x86_64.rpm (installed size 101.00 kB) 89  x   xInstalling libelf0-0.8.12-1.1.52.x86_64.rpm (installed size 101.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  63%  91  9  100  8 2 7  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libdb-4_5 (download size 470.00 kB)   x   xDownloading libdb-4_5 (download size 470.00 kB) q   6  13  20  7  34  40  ±47  54%  61%  8  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libdb-4_5-4.5.20-95.39.x86_64.rpm (installed size 1.22 MB) 7  x   xInstalling libdb-4_5-4.5.20-95.39.x86_64.rpm (installed size 1.22 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  6.58 1 8.57  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libattr (download size 11.00 kB)   x   xDownloading libattr (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libattr-2.4.43-1.18.x86_64.rpm (installed size 20.00 kB) 8.57  x   xInstalling libattr-2.4.43-1.18.x86_64.rpm (installed size 20.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  6 0 5  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libaio (download size 15.00 kB)   x   xDownloading libaio (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libaio-0.3.109-0.1.46.x86_64.rpm (installed size 32.00 kB) 5  x   xInstalling libaio-0.3.109-0.1.46.x86_64.rpm (installed size 32.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  98%  100  3 59 2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libHX13 (download size 24.00 kB)   x   xDownloading libHX13 (download size 24.00 kB) q   68%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libHX13-1.23-4.13.1.x86_64.rpm (installed size 47.00 kB) 2  x   xInstalling libHX13-1.23-4.13.1.x86_64.rpm (installed size 47.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  48 8 47  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading keyutils-libs (download size 16.00 kB)   x   xDownloading keyutils-libs (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing keyutils-libs-1.2-107.29.15.x86_64.rpm (installed size 36.00 kB 47  x   xInstalling keyutils-libs-1.2-107.29.15.x86_64.rpm (installed size 36.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  99%  100  5 7 4  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading initviocons (download size 25.00 kB)   x   x~kB) x    xDownloading initviocons (download size 25.00 kB) q   63%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing initviocons-0.5-55.21.x86_64.rpm (installed size 56.00 kB) 4  x   xInstalling initviocons-0.5-55.21.x86_64.rpm (installed size 56.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  34  68%  80  96  100  39 6 38  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading infinipath-psm (download size 160.00 kB)   x   xDownloading infinipath-psm (download size 160.00 kB) q   9  1  2  40  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing infinipath-psm-3.1-10.24.x86_64.rpm (installed size 381.00 kB) 38  x   xInstalling infinipath-psm-3.1-10.24.x86_64.rpm (installed size 381.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  31  48 64%  81  98  100  02 5 01  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpart (download size 40.00 kB)   x   x~) x    xDownloading gpart (download size 40.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpart-0.1h-565.22.x86_64.rpm (installed size 78.00 kB) 01  x   xInstalling gpart-0.1h-565.22.x86_64.rpm (installed size 78.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  77%  92  9  100  5.95 4 7.93  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gdbm (download size 26.00 kB)   x   xDownloading gdbm (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gdbm-1.8.3-374.25.x86_64.rpm (installed size 55.00 kB) 7.93  x   xInstalling gdbm-1.8.3-374.25.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  58%  90%  6  100  89 3 88  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fping (download size 25.00 kB)   x   xDownloading fping (download size 25.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fping-2.4b2-97.5.x86_64.rpm (installed size 44.00 kB) 88  x   xInstalling fping-2.4b2-97.5.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  86  92  9  100  5 2 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fillup (download size 20.00 kB)   x   xDownloading fillup (download size 20.00 kB) q   80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fillup-1.42-242.21.x86_64.rpm (installed size 37.00 kB) 4  x   xInstalling fillup-1.42-242.21.x86_64.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  1 1 0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dos2unix (download size 11.00 kB)   x   xDownloading dos2unix (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dos2unix-3.1-438.22.1.x86_64.rpm (installed size 22.00 kB) 0  x   xInstalling dos2unix-3.1-438.22.1.x86_64.rpm (installed size 22.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  84%  92  8  100  79 0 78  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dmapi (download size 27.00 kB)   x   xDownloading dmapi (download size 27.00 kB) q   59%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dmapi-2.2.10-8.7.2.x86_64.rpm (installed size 80.00 kB) 78  x   xInstalling dmapi-2.2.10-8.7.2.x86_64.rpm (installed size 80.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  95%  100  1 49 0  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpufrequtils (download size 50.00 kB)   x   xDownloading cpufrequtils (download size 50.00 kB) q   13  ±46  78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpufrequtils-004-35.20.x86_64.rpm (installed size 195.00 kB) 0  x   xInstalling cpufrequtils-004-35.20.x86_64.rpm (installed size 195.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  52% 63%  75  84  9  94  9  100  52 8 51  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bridge-utils (download size 51.00 kB)   x   xDownloading bridge-utils (download size 51.00 kB) q   31 - 15.74 kB/s (download size 51.00 kB)  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bridge-utils-1.4-23.18.1.x86_64.rpm (installed size 161.00 kB) 51  x   xInstalling bridge-utils-1.4-23.18.1.x86_64.rpm (installed size 161.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  57%  76%  87  93  8  100  37 7 35  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading audit-libs (download size 72.00 kB)   x   x~) x    xDownloading audit-libs (download size 72.00 kB) q   21  ±±±44  66%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing audit-libs-1.8-0.34.26.x86_64.rpm (installed size 158.00 kB) 35  x   xInstalling audit-libs-1.8-0.34.26.x86_64.rpm (installed size 158.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  62%  99  100  21 6 20  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libfontenc (download size 14.00 kB)   x   xDownloading xorg-x11-libfontenc (download size 14.00 kB) q   100%   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libfontenc-7.4-1.15.x86_64.rpm (installed size 30.00 k 20  x   xInstalling xorg-x11-libfontenc-7.4-1.15.x86_64.rpm (installed size 30.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  18 5 17  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading file (download size 266.00 kB)   x   x~ kB) x    xDownloading file (download size 266.00 kB) q   5  11  7  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing file-4.24-43.27.1.x86_64.rpm (installed size 2.17 MB) 17  x   xInstalling file-4.24-43.27.1.x86_64.rpm (installed size 2.17 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  33  9  ±±45  51 6%  62%  8  74  9  85  91  7  100  3.01 4 5.00  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading strace (download size 98.00 kB)   x   xDownloading strace (download size 98.00 kB) q   16  32  48 65%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing strace-4.5.18-10.26.19.x86_64.rpm (installed size 313.00 kB) 5.00  x   xInstalling strace-4.5.18-10.26.19.x86_64.rpm (installed size 313.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  4  61%  8  9  6  100  2.70 3 4.69  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pptp (download size 61.00 kB)   x   xDownloading pptp (download size 61.00 kB) q   25  52% 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pptp-1.7.2-1.22.x86_64.rpm (installed size 134.00 kB) 4.69  x   xInstalling pptp-1.7.2-1.22.x86_64.rpm (installed size 134.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  60%  5  79  84  90  5  100  57 2 56  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading checkmedia (download size 37.00 kB)   x   xDownloading checkmedia (download size 37.00 kB) q   ±±±±43  87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing checkmedia-3.0-0.9.1.x86_64.rpm (installed size 94.00 kB) 56  x   xInstalling checkmedia-3.0-0.9.1.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  9  100  48 1 47  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading update-alternatives (download size 15.00 kB)   x   xDownloading update-alternatives (download size 15.00 kB) q   97%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing update-alternatives-1.14.19-1.22.noarch.rpm (installed size 28. 47  x   xInstalling update-alternatives-1.14.19-1.22.noarch.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  83%  99  100  5 0 4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading convmv (download size 29.00 kB)   x   x~8.00 kB) x    xDownloading convmv (download size 29.00 kB) q   55%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing convmv-1.12-1.22.noarch.rpm (installed size 51.00 kB) 4  x   xInstalling convmv-1.12-1.22.noarch.rpm (installed size 51.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  52% 86%  99  100  0 39 39  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atmel-firmware (download size 166.00 kB)   x   xDownloading atmel-firmware (download size 166.00 kB) q   9  1  28  3  4 57%  6  7  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atmel-firmware-1.3-139.15.4.noarch.rpm (installed size 752.00 k 39  x   xInstalling atmel-firmware-1.3-139.15.4.noarch.rpm (installed size 752.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  21  6  34  ±±±±42  ±47  56%  61%  70  5  80  5  94  9  100  1.67 8 3.66  0 9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zip (download size 253.00 kB)   x   x~ kB) x    xDownloading zip (download size 253.00 kB) q   5  11  8  24  30  7  ±±±±43  49 56%  62%  8  75  81  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zip-3.0-4.9.1.x86_64.rpm (installed size 795.00 kB) 3.66  x   xInstalling zip-3.0-4.9.1.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  8  ±46  57%  70%  8  86  91  6  100  0.89 7 2.88  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc43 (download size 4.00 kB)   x   xDownloading libgcc43 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) 2.88  x   xInstalling libgcc43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  74%  100  6  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pcre (download size 381.00 kB)   x   xDownloading pcre (download size 381.00 kB) q   8  16  25  33  ±±±±42  50 8%  63%  71  9  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pcre-7.8-2.18.x86_64.rpm (installed size 1.30 MB)  x   xInstalling pcre-7.8-2.18.x86_64.rpm (installed size 1.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  8  33  8  ±±±±43  50 5%  63%  8  76  81  6  94  9  100  89.5 5 1.5  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++43 (download size 4.00 kB)   x   xDownloading libstdc++43 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) 1.5  x   xInstalling libstdc++43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  74%  100  4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsmbios2 (download size 155.00 kB)   x   x~) x    xDownloading libsmbios2 (download size 155.00 kB) q   1  2  3  ±±±±±41  51 61%  72  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsmbios2-2.0.2-11.8.4.x86_64.rpm (installed size 646.00 kB)  x   xInstalling libsmbios2-2.0.2-11.8.4.x86_64.rpm (installed size 646.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  2  3  4 59%  6  7  8  9  100  8.96 3 0.95  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libboost_regex1_36_0 (download size 237.00 kB)   x   xDownloading libboost_regex1_36_0 (download size 237.00 kB) q   6  13  20  7  33  40  ±47  54%  60%  7  74  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libboost_regex1_36_0-1.36.0-12.6.49.x86_64.rpm (installed size 0.95  x   xInstalling libboost_regex1_36_0-1.36.0-12.6.49.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  07 2 06  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libxcb (download size 97.00 kB)   x   x~e 910.00 kB) x    xDownloading xorg-x11-libxcb (download size 97.00 kB) q   16  32  49 65%  82  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libxcb-7.4-1.29.1.x86_64.rpm (installed size 528.00 kB 06  x   xInstalling xorg-x11-libxcb-7.4-1.29.1.x86_64.rpm (installed size 528.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  3  9  ±46  51 6%  61%  9  76  91  9  100  7.55 1 09.54  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading delayacct-utils (download size 18.00 kB)   x   x~kB) x    xDownloading delayacct-utils (download size 18.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing delayacct-utils-2.6.32.12-3.3.38.x86_64.rpm (installed size 35. 09.54  x   xInstalling delayacct-utils-2.6.32.12-3.3.38.x86_64.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±±41  91%  9  100  2 0 1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procinfo (download size 37.00 kB)   x   x~5.00 kB) x    xDownloading procinfo (download size 37.00 kB) q   ±±±±43  88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procinfo-18-198.16.1.x86_64.rpm (installed size 72.00 kB) 1  x   xInstalling procinfo-18-198.16.1.x86_64.rpm (installed size 72.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  66%  71  9  9  100  45 29 44  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ncurses-utils (download size 73.00 kB)   x   xDownloading ncurses-utils (download size 73.00 kB) q   21  ±±±±43  66%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ncurses-utils-5.6-90.55.x86_64.rpm (installed size 103.00 kB) 44  x   xInstalling ncurses-utils-5.6-90.55.x86_64.rpm (installed size 103.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  20  34  52% 60%  9  9  100  3 8 3  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading joe (download size 320.00 kB)   x   xDownloading joe (download size 320.00 kB) q   9  14  9  24  9  35  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing joe-3.5-92.22.x86_64.rpm (installed size 1.20 MB) 3  x   xInstalling joe-3.5-92.22.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  24  9  34  9  4 55%  61%  7  73  8  84  91  6  100  6.1 7 8.1  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi-libs (download size 1.50 MB)   x   xDownloading openmpi-libs (download size 1.50 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-libs-1.8.4-1.26.x86_64.rpm (installed size 5.33 MB) 8.1  x   xInstalling openmpi-libs-1.8.4-1.26.x86_64.rpm (installed size 5.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  0.83 6 2.81  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librdmacm (download size 58.00 kB)   x   xDownloading librdmacm (download size 58.00 kB) q   27  55%  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librdmacm-1.0.15-12.29.x86_64.rpm (installed size 175.00 kB) 2.81  x   xInstalling librdmacm-1.0.15-12.29.x86_64.rpm (installed size 175.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  20  6  32  ±±±44  55%  63%  8  94  9  100  66 5 64  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmthca-rdmav2 (download size 18.00 kB)   x   xDownloading libmthca-rdmav2 (download size 18.00 kB) q   87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmthca-rdmav2-1.0.5-5.27.1.x86_64.rpm (installed size 35.00 k 64  x   xInstalling libmthca-rdmav2-1.0.5-5.27.1.x86_64.rpm (installed size 35.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 4 1  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmlx4 (download size 28.00 kB)   x   x~ kB) x    xDownloading libmlx4-rdmav2 (download size 28.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmlx4-rdmav2-1.0.1-15.30.x86_64.rpm (installed size 53.00 kB) 1  x   xInstalling libmlx4-rdmav2-1.0.1-15.30.x86_64.rpm (installed size 53.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  95%  100  57 3 56  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcxgb3-rdmav2 (download size 13.00 kB)   x   x~B) x    xDownloading libcxgb3-rdmav2 (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcxgb3-rdmav2-1.2.5-0.8.1.x86_64.rpm (installed size 23.00 kB 56  x   xInstalling libcxgb3-rdmav2-1.2.5-0.8.1.x86_64.rpm (installed size 23.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  5 2 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ibvexdmtools (download size 14.00 kB)   x   x~kB) x    xDownloading ibvexdmtools (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ibvexdmtools-0.0.1-75.23.23.x86_64.rpm (installed size 23.00 kB 4  x   xInstalling ibvexdmtools-0.0.1-75.23.23.x86_64.rpm (installed size 23.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  97%  100  2 1 1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librpcsecgss (download size 130.00 kB)   x   x~kB) x    xDownloading librpcsecgss (download size 130.00 kB) q   12  24  36  49 61%  73  86  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librpcsecgss-0.18-1.15.x86_64.rpm (installed size 586.00 kB) 1  x   xInstalling librpcsecgss-0.18-1.15.x86_64.rpm (installed size 586.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  27  38  49 60%  71  82  8  99  100  79.95 0 1.94  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libdw1 (download size 98.00 kB)   x   xDownloading libdw1 (download size 98.00 kB) q   16  32  49 65%  82  98  100   40 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libdw1-0.152-4.9.17.x86_64.rpm (installed size 212.00 kB) 1.94  x   xInstalling libdw1-0.152-4.9.17.x86_64.rpm (installed size 212.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  60%  9  9  100  7 19 73  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libasm1 (download size 24.00 kB)   x   xDownloading libasm1 (download size 24.00 kB) q   66%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libasm1-0.152-4.9.17.x86_64.rpm (installed size 31.00 kB) 73  x   xInstalling libasm1-0.152-4.9.17.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 8 0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading db-utils (download size 74.00 kB)   x   xDownloading db-utils (download size 74.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing db-utils-4.5.20-95.39.x86_64.rpm (installed size 280.00 kB) 0  x   xInstalling db-utils-4.5.20-95.39.x86_64.rpm (installed size 280.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  31  7  ±±±±42  49 58%  81%  6  92  9  100  44 7 43 499 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcap2 (download size 12.00 kB)   x   xDownloading libcap2 (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcap2-2.11-2.17.1.x86_64.rpm (installed size 19.00 kB) 43  x   xInstalling libcap2-2.11-2.17.1.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 6 1  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libacl (download size 18.00 kB)   x   xDownloading libacl (download size 18.00 kB) q   88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libacl-2.2.47-30.36.1.x86_64.rpm (installed size 31.00 kB) 1  x   xInstalling libacl-2.2.47-30.36.1.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  39 5 38  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading attr (download size 52.00 kB)   x   xDownloading attr (download size 52.00 kB) q   3 - 15.74 kB/s (download size 52.00 kB)  62%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing attr-2.4.43-1.18.x86_64.rpm (installed size 149.00 kB) 38  x   xInstalling attr-2.4.43-1.18.x86_64.rpm (installed size 149.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  2  61%  8  76  84  91  6  100  25 4 24  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading genisoimage (download size 348.00 kB)   x   xDownloading genisoimage (download size 348.00 kB) q   9  18  27  32  ±±±±±41  50 5%  64%  9  78  87  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing genisoimage-1.1.8-3.32.3.x86_64.rpm (installed size 1.61 MB) 24  x   xInstalling genisoimage-1.1.8-3.32.3.x86_64.rpm (installed size 1.61 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  37  ±±45  53%  9%  64  72  80  8  93  9  100  7.64 3 899.62  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc46 (download size 4.00 kB)   x   xDownloading libgcc46 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB) 899.62  x   xInstalling libgcc46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  74%  100  3 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++46 (download size 4.00 kB)   x   xDownloading libstdc++46 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB)  x   xInstalling libstdc++46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  74%  100  1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading makedumpfile (download size 160.00 kB)   x   xDownloading makedumpfile (download size 160.00 kB) q   9  1  2  40  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing makedumpfile-1.5.6-2.29.x86_64.rpm (installed size 372.00 kB)  x   xInstalling makedumpfile-1.5.6-2.29.x86_64.rpm (installed size 372.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  34  51 68%  81  7  93  8  100  27 0 26  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libebl1 (download size 71.00 kB)   x   xDownloading libebl1 (download size 71.00 kB) q   22  ±±45  68%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libebl1-0.152-4.9.17.x86_64.rpm (installed size 251.00 kB) 26  x   xInstalling libebl1-0.152-4.9.17.x86_64.rpm (installed size 251.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  36  ±±±±43  52% 9%  6  78  99  100  03 09 01  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading acl (download size 68.00 kB)   x   xDownloading acl (download size 68.00 kB) q   23  ±47  71%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing acl-2.2.47-30.36.1.x86_64.rpm (installed size 192.00 kB) 01  x   xInstalling acl-2.2.47-30.36.1.x86_64.rpm (installed size 192.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  19  37  64%  70  5  80  9  94  9  100  6.84 8 8.83  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 898.83 MBx 490x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-e8820372xxx tu Downloading fsx (download size 14.00 kB)   x   xDownloading fsx (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fsx-1.0-1.wc1.x86_64.rpm (installed size 37.00 kB) 8.83  x   xInstalling fsx-1.0-1.wc1.x86_64.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  79  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading connectathon (download size 107.00 kB)   x   xDownloading connectathon (download size 107.00 kB) q   14  29  ±±±44  59%  74  89  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing connectathon-1.19-1.intel.x86_64.rpm (installed size 975.00 kB) 79  x   xInstalling connectathon-1.19-1.intel.x86_64.rpm (installed size 975.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  24  31  7  ±±±±42  50 6%  61%  7  72  7  84  9  95  100  7.84  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-rsh (download size 12.00 kB)   x   x~B) x    xDownloading pdsh-rcmd-rsh (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-rsh-2.29-1.wc1.x86_64.rpm (installed size 25.00 kB) 7.84  x   xInstalling pdsh-rcmd-rsh-2.29-1.wc1.x86_64.rpm (installed size 25.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading byacc (download size 89.00 kB)   x   xDownloading byacc (download size 89.00 kB) q   17  35  53%  71%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing byacc-1.9.20070509-7.wc1.x86_64.rpm (installed size 255.00 kB) 1  x   xInstalling byacc-1.9.20070509-7.wc1.x86_64.rpm (installed size 255.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  50 75%  93  8  100  warning: group jenkins does not exist - using root warning: user jenkins does not exist - using root warning: group jenkins does not exist - using root warning: user jenkins does not exist - using root warning: group jenkins does not exist - using root  56  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-ssh (download size 13.00 kB)  user jenkins does not exist - using root group jenkins does not exist - using root   Downloading pdsh-rcmd-ssh (download size 13.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-ssh-2.29-1.wc1.x86_64.rpm (installed size 27.00 kB) 56  x   xInstalling pdsh-rcmd-ssh-2.29-1.wc1.x86_64.rpm (installed size 27.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbench (download size 1.05 MB)   x   xDownloading dbench (download size 1.05 MB) q   5  10  5  20  6  32  8  ±±±44  50 6%  62%  8  73  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbench-4.00-1.8b5143b.wc1.x86_64.rpm (installed size 25.32 MB) 4  x   xInstalling dbench-4.00-1.8b5143b.wc1.x86_64.rpm (installed size 25.32 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  72.22  4  ±±±±42 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 872.22 MBx 484x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-e8820372xxx tu Downloading lsof (download size 314.00 kB)   x   x~) x    xDownloading lsof (download size 314.00 kB) q   5 - 15.74 kB/s (download size 314.00 kB)  10  5  20  5  30  5  40  ±±45  50 6%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsof-4.80-1.33.1.x86_64.rpm (installed size 925.00 kB) 72.22  x   xInstalling lsof-4.80-1.33.1.x86_64.rpm (installed size 925.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  24  31  ±±±±±4  ±47  54%  61%  6  73  8  9  94  9  100  1.3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcpdump (download size 290.00 kB)   x   xDownloading tcpdump (download size 290.00 kB) q   5  10  6  21  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcpdump-3.9.8-1.29.1.x86_64.rpm (installed size 702.00 kB) 1.3  x   xInstalling tcpdump-3.9.8-1.29.1.x86_64.rpm (installed size 702.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  27  36  ±±45  54%  63%  72  81  8  93  9  100  0.63  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libreadline5 (download size 106.00 kB)   x   xDownloading libreadline5 (download size 106.00 kB) q   14  30  ±±45  60%  75  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libreadline5-5.2-147.29.1.x86_64.rpm (installed size 292.00 kB) 0.63  x   xInstalling libreadline5-5.2-147.29.1.x86_64.rpm (installed size 292.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  33  55%  77%  99  100  34  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtiff3 (download size 117.00 kB)   x   x~B) x    xDownloading libtiff3 (download size 117.00 kB) q   13  27  40  54%  68%  82  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtiff3-3.8.2-141.168.1.x86_64.rpm (installed size 386.00 kB) 34  x   xInstalling libtiff3-3.8.2-141.168.1.x86_64.rpm (installed size 386.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  33  49 66%  82  96  100  69.97  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtirpc1 (download size 71.00 kB)   x   x~) x    xDownloading libtirpc1 (download size 71.00 kB) q   22  ±±45  67%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtirpc1-0.2.1-1.9.1.x86_64.rpm (installed size 159.00 kB) 69.97  x   xInstalling libtirpc1-0.2.1-1.9.1.x86_64.rpm (installed size 159.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  81%  99  100  81  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl (download size 6.87 MB)   x   xDownloading perl (download size 6.87 MB) q   5  10  5  20  5  30  5 - 2.08 MB/s (download size 6.87 MB)  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-5.10.0-64.80.1.x86_64.rpm (installed size 41.58 MB) 81  x   xInstalling perl-5.10.0-64.80.1.x86_64.rpm (installed size 41.58 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  28.23  8 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading star (download size 342.00 kB)   x   xDownloading star (download size 342.00 kB) q   5  10  5  24  9  38  ±±±±43  52% 7%  66%  71  6  85  90  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing star-1.5final-28.23.27.1.x86_64.rpm (installed size 1.03 MB) 28.23  x   xInstalling star-1.5final-28.23.27.1.x86_64.rpm (installed size 1.03 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  9  25  31  7  ±±±±42  48 54%  9%  64  70  6  82  8  93  9  100  7.20  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sqlite3 (download size 49.00 kB)   x   xDownloading sqlite3 (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sqlite3-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 61.00 kB) 7.20  x   xInstalling sqlite3-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 61.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  94%  9  100  14  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bash (download size 296.00 kB)   x   xDownloading bash (download size 296.00 kB) q   1  5  20  6  31  7  ±±±±42  ±47  53%  8%  64%  9  74  80  5  91  6  100   ±±±44 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bash-3.2-147.29.1.x86_64.rpm (installed size 864.00 kB) 14  x   xInstalling bash-3.2-147.29.1.x86_64.rpm (installed size 864.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  9  36  ±±±±43  51 8%  65%  70  9  85  93  8  100  6.30  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxml2 (download size 607.00 kB)   x   xDownloading libxml2 (download size 607.00 kB) q   5  10  6  21  6  32  7  ±±±±42  ±47  53%  8%  63%  8  74  9  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxml2-2.7.6-0.64.1.x86_64.rpm (installed size 1.55 MB) 6.30  x   xInstalling libxml2-2.7.6-0.64.1.x86_64.rpm (installed size 1.55 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  25  33  ±±±±±41  49 57%  65%  73  81  9  96  100  4.75  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Module-Build (download size 194.00 kB)   x   xDownloading perl-Module-Build (download size 194.00 kB) q   8  16  24  32  ±±±±±41  49 57%  65%  74  82  90  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Module-Build-0.2808.01-0.80.1.x86_64.rpm (installed size 4 4.75  x   xInstalling perl-Module-Build-0.2808.01-0.80.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  20  38  51 6%  62%  7  72  9  84  90  5  100  3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-32bit (download size 1.06 MB)   x   x~ 415.00 kB) x    xDownloading glibc-32bit (download size 1.06 MB) q   5  10  5  20  6  32  8  ±±±44  50 5%  60%  6  72  8  84  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-32bit-2.11.3-17.102.1.x86_64.rpm (installed size 2.88 MB) 3  x   xInstalling glibc-32bit-2.11.3-17.102.1.x86_64.rpm (installed size 2.88 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  32  8  ±±45  51 8%  64%  70  5  80  5  92  8  100  1.46  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading freetype2 (download size 360.00 kB)   x   x~B) x    xDownloading freetype2 (download size 360.00 kB) q   8  13  22  31  9  ±±±44  53%  62%  71  80  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing freetype2-2.3.7-25.44.1.x86_64.rpm (installed size 891.00 kB) 1.46  x   xInstalling freetype2-2.3.7-25.44.1.x86_64.rpm (installed size 891.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  50 7%  6  73  81  7  92  9  100  0.59  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenssl0_9_8 (download size 772.00 kB)   x   xDownloading libopenssl0_9_8 (download size 772.00 kB) q   5  11  6  22  8  34  40  ±46  53%  9%  65  71  7  82  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenssl0_9_8-0.9.8j-0.105.1.x86_64.rpm (installed size 2.12 0.59  x   xInstalling libopenssl0_9_8-0.9.8j-0.105.1.x86_64.rpm (installed size 2.1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  8  34  40  ±46  52% 8%  63%  9  75  81  6  92  8  100  18.48  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pmtools (download size 515.00 kB)   x   x~2 MB) x    xDownloading pmtools (download size 515.00 kB) q   6  12  8  24  31  7  ±±±±43  49 55%  62%  8  74  80  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pmtools-20071116-44.35.1.x86_64.rpm (installed size 1.63 MB) 18.48  x   xInstalling pmtools-20071116-44.35.1.x86_64.rpm (installed size 1.63 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  21  9  34  ±±±±±41  49 56%  64%  71  9  87  92  7  100  6.84  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libaugeas0 (download size 164.00 kB)   x   xDownloading libaugeas0 (download size 164.00 kB) q   9  1  2  38  4 58%  6  77  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libaugeas0-0.9.0-3.17.2.x86_64.rpm (installed size 334.00 kB) 6.84  x   xInstalling libaugeas0-0.9.0-3.17.2.x86_64.rpm (installed size 334.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  38  57%  76%  98  100  52  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpython2_6-1_0 (download size 563.00 kB)   x   xDownloading libpython2_6-1_0 (download size 563.00 kB) q   5  11  7  22  8  34  9  ±±45  51 6%  62%  8  73  9  85  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpython2_6-1_0-2.6.9-39.1.x86_64.rpm (installed size 1.54 MB) 52  x   xInstalling libpython2_6-1_0-2.6.9-39.1.x86_64.rpm (installed size 1.54 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  61%  9  77  85  93  9  100  4.98  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cracklib (download size 56.00 kB)   x   x~B) x    xDownloading cracklib (download size 56.00 kB) q   28  57%  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cracklib-2.8.12-56.13.1.x86_64.rpm (installed size 183.00 kB) 4.98  x   xInstalling cracklib-2.8.12-56.13.1.x86_64.rpm (installed size 183.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  9  ±±±±43  49 54%  9%  64  9  75  80  5  90  5  100  xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 814.80 MBx 466x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libexpat1 (download size 62.00 kB)   x   xDownloading libexpat1 (download size 62.00 kB) q   25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libexpat1-2.0.1-88.41.1.x86_64.rpm (installed size 167.00 kB) 80 MB / 04:15)  x   xInstalling libexpat1-2.0.1-88.41.1.x86_64.rpm (installed size 167.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  76%  99  100  64  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sg3_utils (download size 652.00 kB)   x   xDownloading sg3_utils (download size 652.00 kB) q   6  11  7  22  8  33  8  ±±±±43  48 53%  8%  63%  8  75  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) 64  x   xInstalling sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  71  7  83  8  93  8  100  sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) Additional rpm output: /var/tmp/rpm-tmp.45892: line 2: ln: command not found /var/tmp/rpm-tmp.45892: line 3: ln: command not found /var/tmp/rpm-tmp.45892: line 4: ln: command not found  2.83  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vim-base (download size 170.00 kB)  /var/tmp/rpm-tmp.45892: line 3: ln: command not found /var/tmp/rpm-tmp.45892: line 4: ln: command not found   Downloading vim-base (download size 170.00 kB)   9  18  2  37  ±4  56%  66%  75  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vim-base-7.2-8.17.1.x86_64.rpm (installed size 322.00 kB) 2.83  x   xInstalling vim-base-7.2-8.17.1.x86_64.rpm (installed size 322.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  15  20  36  ±±±±±41  ±46  51 7%  62%  79  96  100  51  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bzip2 (download size 47.00 kB)   x   xDownloading bzip2 (download size 47.00 kB) q   33  68%  100   ±±45 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bzip2-1.0.5-34.255.1.x86_64.rpm (installed size 90.00 kB) 51  x   xInstalling bzip2-1.0.5-34.255.1.x86_64.rpm (installed size 90.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  ±±±±42  58%  65%  77  8  96  100  42  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading unzip (download size 260.00 kB)   x   xDownloading unzip (download size 260.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  86  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing unzip-6.00-11.17.1.x86_64.rpm (installed size 721.00 kB) 42  x   xInstalling unzip-6.00-11.17.1.x86_64.rpm (installed size 721.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  20  5  34  9  ±47  54%  60%  5  74  9  8  95  100  1.7  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Test-Simple (download size 90.00 kB)   x   xDownloading perl-Test-Simple (download size 90.00 kB) q   17  35  53%  70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Test-Simple-0.72-0.80.1.x86_64.rpm (installed size 167.00 1.74  x   xInstalling perl-Test-Simple-0.72-0.80.1.x86_64.rpm (installed size 167.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  36  59%  74  80  5  92  9  100  56  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libssh2-1 (download size 68.00 kB)   x   x~0 kB) x    xDownloading libssh2-1 (download size 68.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libssh2-1-1.2.9-4.2.11.1.x86_64.rpm (installed size 164.00 kB) 56  x   xInstalling libssh2-1-1.2.9-4.2.11.1.x86_64.rpm (installed size 164.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  78%  99  100  40  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssl (download size 536.00 kB)   x   x~) x    xDownloading openssl (download size 536.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssl-0.9.8j-0.105.1.x86_64.rpm (installed size 1.13 MB) 40  x   xInstalling openssl-0.9.8j-0.105.1.x86_64.rpm (installed size 1.13 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  ±47  53%  8%  64%  9  76  81  6  91  6  100  0.27  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libesmtp (download size 62.00 kB)   x   xDownloading libesmtp (download size 62.00 kB) q   25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libesmtp-1.0.4-157.17.2.x86_64.rpm (installed size 180.00 kB) 0.27  x   xInstalling libesmtp-1.0.4-157.17.2.x86_64.rpm (installed size 180.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  56%  62%  7  97  100  09  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtalloc2 (download size 220.00 kB)   x   xDownloading libtalloc2 (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  50 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtalloc2-3.6.3-87.1.x86_64.rpm (installed size 53.00 kB) 09  x   xInstalling libtalloc2-3.6.3-87.1.x86_64.rpm (installed size 53.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  99%  100  4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-base (download size 4.29 MB)   x   xDownloading python-base (download size 4.29 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-base-2.6.9-39.1.x86_64.rpm (installed size 19.59 MB) 4  x   xInstalling python-base-2.6.9-39.1.x86_64.rpm (installed size 19.59 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  790.45  5  ±46 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam (download size 450.00 kB)   x   xDownloading pam (download size 450.00 kB) q   7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-1.1.5-0.17.2.x86_64.rpm (installed size 1.50 MB) 790.4508  x   xInstalling pam-1.1.5-0.17.2.x86_64.rpm (installed size 1.50 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  88.9  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading expat (download size 54.00 kB)   x   xDownloading expat (download size 54.00 kB) q   29  59%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing expat-2.0.1-88.41.1.x86_64.rpm (installed size 149.00 kB) 88.9  x   xInstalling expat-2.0.1-88.41.1.x86_64.rpm (installed size 149.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  24  9  34  77%  94  9  100  81  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fontconfig (download size 274.00 kB)   x   xDownloading fontconfig (download size 274.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fontconfig-2.6.0-10.19.1.x86_64.rpm (installed size 889.00 kB) 81  x   xInstalling fontconfig-2.6.0-10.19.1.x86_64.rpm (installed size 889.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  23  30  7  ±±45  53%  60%  7  74  81  8  95  100  7.94  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading elilo (download size 122.00 kB)   x   x~) x    xDownloading elilo (download size 122.00 kB) q   11  24  37  50 63%  77  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing elilo-3.14-0.38.1.x86_64.rpm (installed size 404.00 kB) 7.94  x   xInstalling elilo-3.14-0.38.1.x86_64.rpm (installed size 404.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  27  ±±±±43  59%  75  80  6  91  6  100  55  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssl-certs (download size 177.00 kB)   x   xDownloading openssl-certs (download size 177.00 kB) q   8  17  26  3  ±±45  54%  63%  72  81  90  9  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssl-certs-2.7-0.6.1.noarch.rpm (installed size 261.00 kB) 55  x   xInstalling openssl-certs-2.7-0.6.1.noarch.rpm (installed size 261.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  29  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-xml (download size 190.00 kB)   x   xDownloading python-xml (download size 190.00 kB) q   8  16  25  33  ±±±±±41  50 8%  67%  75  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-xml-2.6.9-39.1.x86_64.rpm (installed size 838.00 kB) 297  x   xInstalling python-xml-2.6.9-39.1.x86_64.rpm (installed size 838.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  9  37  ±46  54%  61%  6  74  80  5  91  6  100  6.47  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gdb (download size 3.28 MB)   x   xDownloading gdb (download size 3.28 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gdb-7.9.1-8.2.x86_64.rpm (installed size 9.14 MB) 6.47  x   xInstalling gdb-7.9.1-8.2.x86_64.rpm (installed size 9.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  77.33  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python (download size 269.00 kB)   x   xDownloading python (download size 269.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-2.6.9-39.1.x86_64.rpm (installed size 1.14 MB) 77.335  x   xInstalling python-2.6.9-39.1.x86_64.rpm (installed size 1.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  22  31  6  ±±±±43  49 55%  60%  5  71  6  81  6  92  7  100  6.19  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ppp (download size 324.00 kB)   x   xDownloading ppp (download size 324.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ppp-2.4.5.git-2.31.7.x86_64.rpm (installed size 853.00 kB) 6.194  x   xInstalling ppp-2.4.5.git-2.31.7.x86_64.rpm (installed size 853.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  34  ±±±±±41  49 56%  63%  7  80  5  90  5  100  5.36  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxml2-python (download size 162.00 kB)   x   xDownloading libxml2-python (download size 162.00 kB) q   9  1  2  3  4 59%  6  7  8  - 143.74 kB/s (download size 162.00 kB) 9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxml2-python-2.7.6-0.64.4.x86_64.rpm (installed size 832.00 k 5.36  x   xInstalling libxml2-python-2.7.6-0.64.4.x86_64.rpm (installed size 832.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  24  32  9  48 56%  64%  71  9  85  90  5  100  4.55  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 774.55 MBx 445x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-e8820372xxx tu Downloading libibumad-devel (download size 34.00 kB)   x   x~ kB) x    xDownloading libibumad-devel (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibumad-devel-1.3.7-10.26.x86_64.rpm (installed size 112.00 k 4.552  x   xInstalling libibumad-devel-1.3.7-10.26.x86_64.rpm (installed size 112.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  62%  98  100  44  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 774.44 MBx 444x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-e8820372xxx tu Downloading liblcms1 (download size 93.00 kB)   x   x~ kB) x    xDownloading liblcms1 (download size 93.00 kB) q   17  34  51 68%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblcms1-1.17-77.16.1.x86_64.rpm (installed size 218.00 kB) 44  x   xInstalling liblcms1-1.17-77.16.1.x86_64.rpm (installed size 218.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  58%  8  99  100  62 7 22  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yp-tools (download size 65.00 kB)   x   xDownloading yp-tools (download size 65.00 kB) q   24  49 73%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yp-tools-2.12-0.2.5.x86_64.rpm (installed size 206.00 kB) 22  x   xInstalling yp-tools-2.12-0.2.5.x86_64.rpm (installed size 206.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  1  24  32  7  ±46  55%  61%  70  7  82  94  9  100  4 6 0  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-libyui (download size 192.00 kB)   x   xDownloading yast2-libyui (download size 192.00 kB) q   8  16  24  33  ±±±±±41  49 58%  66%  75  83  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-libyui-2.21.1-2.5.93.x86_64.rpm (installed size 679.00 kB 0  x   xInstalling yast2-libyui-2.21.1-2.5.93.x86_64.rpm (installed size 679.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  2  37  ±4  56%  66%  75  84  9  9  100  5.76 5 3.36  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xz (download size 107.00 kB)   x   x~kB) x    xDownloading xz (download size 107.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xz-5.0.3-0.12.1.x86_64.rpm (installed size 228.00 kB) 3.361  x   xInstalling xz-5.0.3-0.12.1.x86_64.rpm (installed size 228.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  33  8  49 57%  65%  71  89  97  100  54 4 14  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysfsutils (download size 75.00 kB)   x   xDownloading sysfsutils (download size 75.00 kB) q   21  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysfsutils-2.1.0-102.25.1.x86_64.rpm (installed size 337.00 kB) 14  x   xInstalling sysfsutils-2.1.0-102.25.1.x86_64.rpm (installed size 337.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  20  5  ±±±44  63%  82  93  9  100  21 3 2.81  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-release (download size 16.00 kB)   x   x~B) x    xDownloading sles-release (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB) 2.81  x   xInstalling sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  15 2 75  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading qlvnictools (download size 31.00 kB)   x   xDownloading qlvnictools (download size 31.00 kB) q   52% 100%   48 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 kB) 75  x   xInstalling qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  2  64%  96  100  06 1 6  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pth (download size 216.00 kB)  Installing sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB)qlvnictools (download size 31.00 kB) qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 k ~B) Downloading pth (download size 216.00 kB)   7  14  22  9  36  ±±±44  51 9%  66  74  81  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pth-2.0.7-102.22.x86_64.rpm (installed size 856.00 kB) 6  x   xInstalling pth-2.0.7-102.22.x86_64.rpm (installed size 856.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  25  32  40  ±47  55%  62%  70  8  84  9  94  9  100  4.22 0 1.82  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mtools (download size 250.00 kB)   x   xDownloading mtools (download size 250.00 kB) q   6  12  9  25  31  8  ±±±44  51 7%  64%  70  6  83  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mtools-3.9.11-165.31.x86_64.rpm (installed size 542.00 kB) 1.82  x   xInstalling mtools-3.9.11-165.31.x86_64.rpm (installed size 542.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  34  ±±±4  58%  70%  81  90  5  100  3.69 299 29  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mstflint (download size 254.00 kB)   x   xDownloading mstflint (download size 254.00 kB) q   6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mstflint-3.6.0-1.8.x86_64.rpm (installed size 1.38 MB) 29  x   xInstalling mstflint-3.6.0-1.8.x86_64.rpm (installed size 1.38 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  21  30  5  ±±±±±41  ±47  53%  9%  64  9  74  82  91  9  100  2.31 8 69.91  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libzio (download size 10.00 kB)   x   xDownloading libzio (download size 10.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libzio-0.9-5.77.x86_64.rpm (installed size 19.00 kB) 69.910  x   xInstalling libzio-0.9-5.77.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  29 7 89  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libuuid1 (download size 37.00 kB)   x   xDownloading libuuid1 (download size 37.00 kB) q   ±±±±43  87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libuuid1-2.19.1-6.72.1.x86_64.rpm (installed size 19.00 kB) 89  x   xInstalling libuuid1-2.19.1-6.72.1.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7 6 7  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++33 (download size 198.00 kB)   x   xDownloading libstdc++33 (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++33-3.3.3-11.9.x86_64.rpm (installed size 833.00 kB) 7  x   xInstalling libstdc++33-3.3.3-11.9.x86_64.rpm (installed size 833.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  23  30  8  ±46  53%  61%  9  76  84  92  9  100  1.46 5 05  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnet (download size 218.00 kB)   x   xDownloading libnet (download size 218.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  7  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnet-1.1.2.1-140.24.1.x86_64.rpm (installed size 1.61 MB) 05  x   xInstalling libnet-1.1.2.1-140.24.1.x86_64.rpm (installed size 1.61 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  33  ±±±±±41  49 54%  62%  8  73  8  83  8  93  8  100  69.85 4 7.4  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgpg-error0 (download size 42.00 kB)   x   xDownloading libgpg-error0 (download size 42.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgpg-error0-1.10-0.7.29.x86_64.rpm (installed size 162.00 kB) 7.4  x   xInstalling libgpg-error0-1.10-0.7.29.x86_64.rpm (installed size 162.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  19  2  38  4 63%  71  8  92  9  100  69 3 29  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading less (download size 121.00 kB)   x   x~B) x    xDownloading less (download size 121.00 kB) q   13  26  39  53%  66%  79  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing less-424b-10.24.1.x86_64.rpm (installed size 266.00 kB) 29  x   xInstalling less-424b-10.24.1.x86_64.rpm (installed size 266.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 56%  66%  81  90  8  100  43 2 03  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading keyutils (download size 54.00 kB)   x   xDownloading keyutils (download size 54.00 kB) q   29  59%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing keyutils-1.2-107.29.15.x86_64.rpm (installed size 87.00 kB) 03  x   xInstalling keyutils-1.2-107.29.15.x86_64.rpm (installed size 87.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  ±47  66%  74  80  5  90  5  100  34 1 6.94  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iptables (download size 263.00 kB)   x   xDownloading iptables (download size 263.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  73  9  85  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iptables-1.4.16.3-1.37.x86_64.rpm (installed size 1.25 MB) 6.94  x   xInstalling iptables-1.4.16.3-1.37.x86_64.rpm (installed size 1.25 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  7  82  7  93  8  100  8.10 0 5.70  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading insserv (download size 39.00 kB)   x   xDownloading insserv (download size 39.00 kB) q   4  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing insserv-1.12.0-25.11.47.x86_64.rpm (installed size 69.00 kB) 5.703:59  x   xInstalling insserv-1.12.0-25.11.47.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  03 89 63  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ifplugd (download size 42.00 kB)   x   xDownloading ifplugd (download size 42.00 kB) q   37  76%  100  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ifplugd-0.28-210.20.x86_64.rpm (installed size 100.00 kB) 63  x   xInstalling ifplugd-0.28-210.20.x86_64.rpm (installed size 100.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  56%  74%  90  5  100  7.9 8 5  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icedax (download size 174.00 kB)   x   xDownloading icedax (download size 174.00 kB) q   9  18  27  36  ±±45  55%  64%  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icedax-1.1.8-3.32.3.x86_64.rpm (installed size 387.00 kB) 5  x   xInstalling icedax-1.1.8-3.32.3.x86_64.rpm (installed size 387.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  34  50 67%  81  7  93  9  100  56 7 15  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gmp (download size 137.00 kB)   x   xDownloading gmp (download size 137.00 kB) q   11  23  34  ±46  58%  70%  81  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gmp-4.2.3-10.99.x86_64.rpm (installed size 310.00 kB) 15  x   xInstalling gmp-4.2.3-10.99.x86_64.rpm (installed size 310.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  ±±±±±41  61%  82  7  93  8  100  25 6 4.8  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading elfutils (download size 260.00 kB)   x   xDownloading elfutils (download size 260.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  86  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing elfutils-0.152-4.9.17.x86_64.rpm (installed size 795.00 kB) 4.8  x   xInstalling elfutils-0.152-4.9.17.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  18  23  3  ±±±±±41  ±46  52% 7%  65%  73  8  86  91  8  100  6.48 5 08  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xdg-utils (download size 49.00 kB)   x   xDownloading xdg-utils (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xdg-utils-1.0.2-36.18.noarch.rpm (installed size 183.00 kB) 08  x   xInstalling xdg-utils-1.0.2-36.18.noarch.rpm (installed size 183.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  28  37  49 64%  70  8  5  90  5  100  30 4 3.90  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lsb-release (download size 9.00 kB)   x   xDownloading lsb-release (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsb-release-2.0-1.2.18.noarch.rpm (installed size 15.00 kB) 3.90  x   xInstalling lsb-release-2.0-1.2.18.noarch.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  99  100  28 3 88  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses5-32bit (download size 242.00 kB)   x   xDownloading libncurses5-32bit (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% - 127.74 kB/s (download size 242.00 kB) 9%  66  72  9  86  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses5-32bit-5.6-90.55.x86_64.rpm (installed size 810.00 k 88  x   xInstalling libncurses5-32bit-5.6-90.55.x86_64.rpm (installed size 810.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  23  9  37  ±±45  53%  60%  5  72  80  7  98  100  5.49 2 09  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wireless-tools (download size 168.00 kB)   x   x~ kB) x    xDownloading wireless-tools (download size 168.00 kB) q   9  18  2  3  ±47  57%  66%  7  85  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wireless-tools-30.pre7-3.13.1.x86_64.rpm (installed size 408.00 09  x   xInstalling wireless-tools-30.pre7-3.13.1.x86_64.rpm (installed size 408.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  23  30  7  ±±45  53%  60%  71  6  81  7  93  8  100  0 1 2.6  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ruby (download size 1.76 MB)   x   x~00 kB) x    xDownloading ruby (download size 1.76 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ruby-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 6.43 MB) 2.68  x   xInstalling ruby-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 6.43 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  58.66 0 56.26  6 50 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openslp (download size 61.00 kB)   x   xDownloading openslp (download size 61.00 kB) q   26  52% 79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openslp-1.2.0-172.24.1.x86_64.rpm (installed size 137.00 kB) 56.267  x   xInstalling openslp-1.2.0-172.24.1.x86_64.rpm (installed size 137.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  19  66%  92  8  100  53 79 13  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libblocxx6 (download size 349.00 kB)   x   xDownloading libblocxx6 (download size 349.00 kB) q   5  12  20  5  32  8  ±±±±43  50 5%  61%  6  71  6  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libblocxx6-2.1.0.342-122.6.x86_64.rpm (installed size 1.15 MB) 13  x   xInstalling libblocxx6-2.1.0.342-122.6.x86_64.rpm (installed size 1.15 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  32  8  ±±±±43  48 54%  9%  65  70  6  81  6  92  7  100  7.38 8 4.98  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-gettext (download size 21.00 kB)   x   x~) x    xDownloading perl-gettext (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-gettext-1.05-91.95.x86_64.rpm (installed size 44.00 kB) 4.986  x   xInstalling perl-gettext-1.05-91.95.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  76%  91  8  100  4 7 4  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Writer (download size 24.00 kB)   x   xDownloading perl-XML-Writer (download size 24.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Writer-0.604-2.10.x86_64.rpm (installed size 52.00 kB) 4  x   xInstalling perl-XML-Writer-0.604-2.10.x86_64.rpm (installed size 52.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  72%  84  99  100  29 6 88  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Parser (download size 231.00 kB)   x   x~B) x    xDownloading perl-XML-Parser (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Parser-2.36-1.18.x86_64.rpm (installed size 688.00 kB) 88  x   xInstalling perl-XML-Parser-2.36-1.18.x86_64.rpm (installed size 688.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  23  9  34  9  ±±±44  56%  65%  71  80  9  94  9  100  6.61 5 21  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-NamespaceSupport (download size 16.00 kB)   x   x~B) x    xDownloading perl-XML-NamespaceSupport (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-NamespaceSupport-1.09-1.22.x86_64.rpm (installed size 21  x   xInstalling perl-XML-NamespaceSupport-1.09-1.22.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  80  98  100  59 4 18  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-LibXML-Common (download size 17.00 kB)   x   x~e 29.00 kB) x    xDownloading perl-XML-LibXML-Common (download size 17.00 kB) q   95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-LibXML-Common-0.13-70.7.x86_64.rpm (installed size 36. 18  x   xInstalling perl-XML-LibXML-Common-0.13-70.7.x86_64.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  84%  90  8  100  5 3 5  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-URI (download size 111.00 kB)   x   x~6.00 kB) x    xDownloading perl-URI (download size 111.00 kB) q   14  28  ±±±±43  57%  72%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-URI-1.37-1.22.x86_64.rpm (installed size 252.00 kB) 5  x   xInstalling perl-URI-1.37-1.22.x86_64.rpm (installed size 252.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  6  21  6  33  8  ±±±±43  52% 77%  84  90  5  100  30 2 3.90  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-TimeDate (download size 34.00 kB)   x   xDownloading perl-TimeDate (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-TimeDate-1.16-71.22.x86_64.rpm (installed size 76.00 kB) 3.90  x   xInstalling perl-TimeDate-1.16-71.22.x86_64.rpm (installed size 76.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  7  23  9  37  ±±±±42  ±47  52% 63%  7  87  92  9  100  23 1 83  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Socket6 (download size 24.00 kB)   x   xDownloading perl-Socket6 (download size 24.00 kB) q   67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Socket6-0.22-1.24.1.x86_64.rpm (installed size 61.00 kB) 83  x   xInstalling perl-Socket6-0.22-1.24.1.x86_64.rpm (installed size 61.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  78%  94  9  100  17 0 77  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Parse-RecDescent (download size 91.00 kB)   x   xDownloading perl-Parse-RecDescent (download size 91.00 kB) q   17  35  52% 70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Parse-RecDescent-1.96.0-1.22.x86_64.rpm (installed size 23 77  x   xInstalling perl-Parse-RecDescent-1.96.0-1.22.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  83%  99  100  5.95 69 54  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-HTML-Tagset (download size 15.00 kB)   x   x~230.00 kB) x    xDownloading perl-HTML-Tagset (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-HTML-Tagset-3.20-1.22.x86_64.rpm (installed size 20.00 kB) 54  x   xInstalling perl-HTML-Tagset-3.20-1.22.x86_64.rpm (installed size 20.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  9  76  97  100  3 8 3  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-SHA1 (download size 48.00 kB)   x   x~B) x    xDownloading perl-Digest-SHA1 (download size 48.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-SHA1-2.11-1.22.x86_64.rpm (installed size 112.00 kB 3  x   xInstalling perl-Digest-SHA1-2.11-1.22.x86_64.rpm (installed size 112.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  33  ±46  54%  95%  100  82 7 42  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-MD4 (download size 23.00 kB)   x   x~kB) x    xDownloading perl-Digest-MD4 (download size 23.00 kB) q   69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-MD4-1.5-1.22.x86_64.rpm (installed size 44.00 kB) 42  x   xInstalling perl-Digest-MD4-1.5-1.22.x86_64.rpm (installed size 44.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  83%  98  100  77 6 37  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Config-IniFiles (download size 38.00 kB)   x   xDownloading perl-Config-IniFiles (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Config-IniFiles-2.39-1.24.1.x86_64.rpm (installed size 84. 37  x   xInstalling perl-Config-IniFiles-2.39-1.24.1.x86_64.rpm (installed size 8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  75%  82  99  100  69 5 29  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Config-Crontab (download size 50.00 kB)   x   x~4.00 kB) x    xDownloading perl-Config-Crontab (download size 50.00 kB) q   31  63%  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Config-Crontab-1.21-73.21.x86_64.rpm (installed size 115.0 29  x   xInstalling perl-Config-Crontab-1.21-73.21.x86_64.rpm (installed size 115q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  76%  82  99  100  58 4 18  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Carp-Clan (download size 24.00 kB)   x   x~.00 kB) x    xDownloading perl-Carp-Clan (download size 24.00 kB) q   59%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Carp-Clan-6.00-1.16.x86_64.rpm (installed size 46.00 kB) 18  x   xInstalling perl-Carp-Clan-6.00-1.16.x86_64.rpm (installed size 46.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  24  39  ±±45  82%  91  8  100  4 3 3 399 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading numactl (download size 64.00 kB)   x   xDownloading numactl (download size 64.00 kB) q   24  49 75%  100  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing numactl-2.0.10-0.9.31.x86_64.rpm (installed size 154.00 kB) 3  x   xInstalling numactl-2.0.10-0.9.31.x86_64.rpm (installed size 154.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  2  ±±±±±41  66%  84  9  9  100  38 2 2.98  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading login (download size 74.00 kB)   x   xDownloading login (download size 74.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing login-3.41-0.4.2.x86_64.rpm (installed size 291.00 kB) 2.98  x   xInstalling login-3.41-0.4.2.x86_64.rpm (installed size 291.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  9  25  31  6  ±±±±±41  ±46  53%  8%  64%  9  75  80  5  91  6  100  10 1 70  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lcms (download size 49.00 kB)   x   xDownloading lcms (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lcms-1.17-77.16.1.x86_64.rpm (installed size 156.00 kB) 70  x   xInstalling lcms-1.17-77.16.1.x86_64.rpm (installed size 156.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  18  3  52% 67%  86  93  8  100  4.95 0 55  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xz-lang (download size 36.00 kB)   x   xDownloading xz-lang (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xz-lang-5.0.3-0.12.1.x86_64.rpm (installed size 108.00 kB) 55  x   xInstalling xz-lang-5.0.3-0.12.1.x86_64.rpm (installed size 108.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  3  60%  8  99  100  84 59 44  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsensors3 (download size 73.00 kB)   x   xDownloading libsensors3 (download size 73.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsensors3-2.10.6-10.15.x86_64.rpm (installed size 436.00 kB) 44  x   xInstalling libsensors3-2.10.6-10.15.x86_64.rpm (installed size 436.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  9  33  48 63%  77  92  9  100  42 8 02  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iputils (download size 80.00 kB)   x   x~) x    xDownloading iputils (download size 80.00 kB) q   19  40  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iputils-ss021109-292.28.1.x86_64.rpm (installed size 219.00 kB) 025  x   xInstalling iputils-ss021109-292.28.1.x86_64.rpm (installed size 219.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  33  ±±±±42  50 9%  71  8  85  90  5  100  20 7 1.80  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading info (download size 159.00 kB)   x   x~B) x    xDownloading info (download size 159.00 kB) q   9  1  30  4  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing info-4.12-1.86.x86_64.rpm (installed size 278.00 kB) 1.80  x   xInstalling info-4.12-1.86.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  37  60%  72  81  90  8  100  3.93 6 53  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libSM (download size 16.00 kB)   x   xDownloading xorg-x11-libSM (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libSM-7.4-1.18.x86_64.rpm (installed size 35.00 kB) 53  x   xInstalling xorg-x11-libSM-7.4-1.18.x86_64.rpm (installed size 35.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 5 0  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading syslinux (download size 403.00 kB)   x   xDownloading syslinux (download size 403.00 kB) q   7  15  23  31  9  ±47  55%  63%  71  9  87  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing syslinux-3.82-8.17.6.x86_64.rpm (installed size 1.07 MB) 0  x   xInstalling syslinux-3.82-8.17.6.x86_64.rpm (installed size 1.07 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  7  22  7  32  9  4 56%  61%  7  75  81  6  92  7  100  2.83 4 0.43  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading reiserfs (download size 228.00 kB)   x   xDownloading reiserfs (download size 228.00 kB) q   6  13  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing reiserfs-3.6.21-1.15.1.x86_64.rpm (installed size 1016.00 kB) 0.43  x   xInstalling reiserfs-3.6.21-1.15.1.x86_64.rpm (installed size 1016.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  27  33  ±±±±4  49 54%  61%  7  76  83  9  95  100  1.84 3 49.44  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libblkid1 (download size 90.00 kB)   x   xDownloading libblkid1 (download size 90.00 kB) q   17  35  53%  71%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libblkid1-2.19.1-6.72.1.x86_64.rpm (installed size 153.00 kB) 49.44  x   xInstalling libblkid1-2.19.1-6.72.1.x86_64.rpm (installed size 153.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  84%  99  100  69 2 29  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmpfr1 (download size 125.00 kB)   x   xDownloading libmpfr1 (download size 125.00 kB) q   11  23  36  49 62%  75  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmpfr1-2.3.2-3.118.2.x86_64.rpm (installed size 285.00 kB) 29  x   xInstalling libmpfr1-2.3.2-3.118.2.x86_64.rpm (installed size 285.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  ±±45  67%  90  9  100  41 1 01  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading facter (download size 52.00 kB)   x   xDownloading facter (download size 52.00 kB) q   3  61%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing facter-1.6.18-0.3.1.x86_64.rpm (installed size 146.00 kB) 01  x   xInstalling facter-1.6.18-0.3.1.x86_64.rpm (installed size 146.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±42  ±47  52% 60%  6  74  82  8  93  8  100  27 0 8.87  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal (download size 137.00 kB)   x   xDownloading limal (download size 137.00 kB) q   11  23  34  ±46  58%  70%  81  93  100  2% ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-1.5.3-0.2.11.x86_64.rpm (installed size 538.00 kB) 8.87  x   xInstalling limal-1.5.3-0.2.11.x86_64.rpm (installed size 538.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  35  ±47  56%  61%  6  72  7  83  8  94  9  100  0.74 49 34  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-XPath (download size 76.00 kB)   x   xDownloading perl-XML-XPath (download size 76.00 kB) q   2  ±±±±±41  63%  84  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-XPath-1.13-74.15.x86_64.rpm (installed size 191.00 kB) 344  x   xInstalling perl-XML-XPath-1.13-74.15.x86_64.rpm (installed size 191.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  21  7  35  ±±±±43  48 60%  9  75  80  5  90  5  100  56 8 15  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Simple (download size 71.00 kB)   x   x~B) x    xDownloading perl-XML-Simple (download size 71.00 kB) q   22  ±±45  68%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Simple-2.18-1.15.x86_64.rpm (installed size 156.00 kB) 15  x   xInstalling perl-XML-Simple-2.18-1.15.x86_64.rpm (installed size 156.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  60%  72  80  93  9  100  40 7 00  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-SAX (download size 75.00 kB)   x   x~B) x    xDownloading perl-XML-SAX (download size 75.00 kB) q   21  ±±±±42  64%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-SAX-0.96-2.7.x86_64.rpm (installed size 260.00 kB) 00  x   xInstalling perl-XML-SAX-0.96-2.7.x86_64.rpm (installed size 260.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  48 56%  66%  72  7  82  7  92  8  100  15 6 7.75  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-X500-DN (download size 14.00 kB)   x   xDownloading perl-X500-DN (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-X500-DN-0.29-78.22.x86_64.rpm (installed size 16.00 kB) 7.75  x   xInstalling perl-X500-DN-0.29-78.22.x86_64.rpm (installed size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  29  3  48 53%  60%  7  83  97  100  3 5 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-HTML-Parser (download size 109.00 kB)   x   xDownloading perl-HTML-Parser (download size 109.00 kB) q   14  29  ±±±±43  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-HTML-Parser-3.56-1.18.1.x86_64.rpm (installed size 248.00 3  x   xInstalling perl-HTML-Parser-3.56-1.18.1.x86_64.rpm (installed size 248.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  26  33  59%  65  79  84  9  96  100  49.89 4 49  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-HMAC (download size 21.00 kB)   x   x~0 kB) x    xDownloading perl-Digest-HMAC (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-HMAC-1.01-436.17.x86_64.rpm (installed size 34.00 k 49  x   xInstalling perl-Digest-HMAC-1.01-436.17.x86_64.rpm (installed size 34.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  20  80%  6  92  8  100  6 3 6  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Crypt-SmbHash (download size 15.00 kB)   x   x~ kB) x    xDownloading perl-Crypt-SmbHash (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Crypt-SmbHash-0.12-132.17.x86_64.rpm (installed size 25.00 6  x   xInstalling perl-Crypt-SmbHash-0.12-132.17.x86_64.rpm (installed size 25.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  80  5  98  100  3 2 3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Bit-Vector (download size 198.00 kB)   x   x~00 kB) x    xDownloading perl-Bit-Vector (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Bit-Vector-6.4-1.14.x86_64.rpm (installed size 1.29 MB) 3  x   xInstalling perl-Bit-Vector-6.4-1.14.x86_64.rpm (installed size 1.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  9  24  9  38  ±±±±43  48 53%  8%  67%  72  7  82  7  92  8  100  8.54 1 6.14  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmng (download size 183.00 kB)   x   xDownloading libmng (download size 183.00 kB) q   8  17  26  34  ±±±±43  52% 61%  70  8  87  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmng-1.0.10-46.18.x86_64.rpm (installed size 566.00 kB) 6.14  x   xInstalling libmng-1.0.10-46.18.x86_64.rpm (installed size 566.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±45  56%  67%  79  90  9  100  7.99 0 5.59  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zsh (download size 2.58 MB)   x   xDownloading zsh (download size 2.58 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zsh-4.3.6-67.9.1.x86_64.rpm (installed size 7.97 MB) 5.59  x   xInstalling zsh-4.3.6-67.9.1.x86_64.rpm (installed size 7.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  0.02 39 37.62  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading screen (download size 453.00 kB)   x   xDownloading screen (download size 453.00 kB) q   7  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing screen-4.0.2-162.19.2.1.x86_64.rpm (installed size 725.00 kB) 37.621  x   xInstalling screen-4.0.2-162.19.2.1.x86_64.rpm (installed size 725.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  26  35  ±±±±43  48 57%  63%  71  9  86  95  100  39.31 8 6.91  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading recode (download size 744.00 kB)   x   xDownloading recode (download size 744.00 kB) q   6  11  6  22  7  33  9  ±±±44  50 5%  61%  7  72  8  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing recode-3.6-635.18.x86_64.rpm (installed size 2.23 MB) 6.91  x   xInstalling recode-3.6-635.18.x86_64.rpm (installed size 2.23 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  34  40  ±46  51 7%  63%  8  74  9  84  9  94  9  100  7.08 7 4.68  3 3%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pinentry (download size 60.00 kB)   x   xDownloading pinentry (download size 60.00 kB) q   26  53%  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pinentry-0.7.5-61.33.1.x86_64.rpm (installed size 136.00 kB) 4.680  x   xInstalling pinentry-0.7.5-61.33.1.x86_64.rpm (installed size 136.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  35  ±47  86%  92  9  100  6.95 6 55  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading make (download size 356.00 kB)   x   xDownloading make (download size 356.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing make-3.81-128.20.x86_64.rpm (installed size 985.00 kB) 55  x   xInstalling make-3.81-128.20.x86_64.rpm (installed size 985.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  23  30  5  ±±±±42  50 7%  63%  9  75  82  7  93  9  100  5.99 5 3.59  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading m4 (download size 234.00 kB)   x   xDownloading m4 (download size 234.00 kB) q   6  13  20  7  34  ±±±±±41  ±47  54%  61%  8  75  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing m4-1.4.12-1.19.x86_64.rpm (installed size 441.00 kB) 3.59  x   xInstalling m4-1.4.12-1.19.x86_64.rpm (installed size 441.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  33  48 62%  9  75  92  9  100  56 4 16  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtool (download size 514.00 kB)   x   xDownloading libtool (download size 514.00 kB) q   6  12  8  24  31  7  ±±±±43  49 56%  62%  8  74  80  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtool-2.2.6-2.131.1.x86_64.rpm (installed size 2.33 MB) 16  x   xInstalling libtool-2.2.6-2.131.1.x86_64.rpm (installed size 2.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  9  34  40  ±46  51 7%  62%  8  73  9  85  90  5  100  3.23 3 0.83  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libreiserfs (download size 43.00 kB)   x   xDownloading libreiserfs (download size 43.00 kB) q   36  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libreiserfs-0.3.0.5-143.3.x86_64.rpm (installed size 111.00 kB) 0.8349  x   xInstalling libreiserfs-0.3.0.5-143.3.x86_64.rpm (installed size 111.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  67%  99  100  12 2 72  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gzip (download size 74.00 kB)   x   x~B) x    xDownloading gzip (download size 74.00 kB) q   21  ±±±±42  64%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gzip-1.3.12-69.23.1.x86_64.rpm (installed size 116.00 kB) 72  x   xInstalling gzip-1.3.12-69.23.1.x86_64.rpm (installed size 116.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  61%  7  74  89  95  100  01 1 61  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading grep (download size 232.00 kB)   x   xDownloading grep (download size 232.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing grep-2.7-5.7.1.x86_64.rpm (installed size 835.00 kB) 61  x   xInstalling grep-2.7-5.7.1.x86_64.rpm (installed size 835.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  3  40  ±47  55%  60%  5  70  5  80  5  90  5  100  2.19 0 29.79  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gettext-runtime (download size 699.00 kB)   x   xDownloading gettext-runtime (download size 699.00 kB) q   6  11  6  22  7  32  8  ±±±±43  48 54%  9%  64  70  5  80  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gettext-runtime-0.17-61.40.x86_64.rpm (installed size 2.18 MB) 29.79  x   xInstalling gettext-runtime-0.17-61.40.x86_64.rpm (installed size 2.18 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  33  9  ±±±44  50 6%  62%  8  74  81  7  92  7  100  0.02 29 7.62  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gawk (download size 773.00 kB)   x   x~) x    xDownloading gawk (download size 773.00 kB) q   6  11  6  22  8  34  40  ±±45  51 7%  63%  9  76  82  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gawk-3.1.8-2.6.1.x86_64.rpm (installed size 1.92 MB) 7.628  x   xInstalling gawk-3.1.8-2.6.1.x86_64.rpm (installed size 1.92 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  9  25  31  8  ±±±±43  49 55%  61%  8  74  80  5  90  5  100  28.10 8 5.70  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading findutils (download size 321.00 kB)   x   xDownloading findutils (download size 321.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing findutils-4.4.0-38.26.1.x86_64.rpm (installed size 896.00 kB) 5.70  x   xInstalling findutils-4.4.0-38.26.1.x86_64.rpm (installed size 896.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  38  ±±±44  49 54%  9%  64  9  75  80  5  91  6  100  7.23 7 4.82  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fastjar (download size 78.00 kB)   x   xDownloading fastjar (download size 78.00 kB) q   2  4  61%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fastjar-0.95-1.24.1.x86_64.rpm (installed size 175.00 kB) 4.82  x   xInstalling fastjar-0.95-1.24.1.x86_64.rpm (installed size 175.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  60%  73  9  8  100  05 6 65  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ed (download size 45.00 kB)   x   xDownloading ed (download size 45.00 kB) q   35  71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ed-0.2-1001.30.1.x86_64.rpm (installed size 71.00 kB) 65  x   xInstalling ed-0.2-1001.30.1.x86_64.rpm (installed size 71.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  76%  89  9  100  6.99 5 58  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading diffutils (download size 289.00 kB)   x   xDownloading diffutils (download size 289.00 kB) q   5  11  6  22  7  33  8  ±±±44  49 55%  60%  6  72  7  83  8  94  9  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing diffutils-2.8.7-143.23.1.x86_64.rpm (installed size 982.00 kB) 58  x   xInstalling diffutils-2.8.7-143.23.1.x86_64.rpm (installed size 982.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  21  7  34  9  ±46  52% 7%  63%  9  74  80  5  92  7  100  03 4 3.62  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading coreutils (download size 1.19 MB)   x   x~) x    xDownloading coreutils (download size 1.19 MB) q   5  11  6  21  7 - 281.79 kB/s (download size 1.19 MB)  32  7  ±±±±42  48 53%  8%  63%  9  74  9  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing coreutils-8.12-6.25.32.33.1.x86_64.rpm (installed size 4.60 MB) 3.627  x   xInstalling coreutils-8.12-6.25.32.33.1.x86_64.rpm (installed size 4.60 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  31  6  ±±±±±41  ±46  52% 7%  62%  7  73  8  83  8  94  9  100  1.42 3 19.0  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bc (download size 108.00 kB)   x   x~B) x    xDownloading bc (download size 108.00 kB) q   14  29  ±±±44  59%  74  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bc-1.06-838.15.x86_64.rpm (installed size 239.00 kB) 19.06  x   xInstalling bc-1.06-838.15.x86_64.rpm (installed size 239.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  32  50 8%  69%  83  91  7  100  19 2 8.79  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading FirmwareUpdateKit (download size 117.00 kB)   x   xDownloading FirmwareUpdateKit (download size 117.00 kB) q   11  25  39  52% 66%  80  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing FirmwareUpdateKit-1.1-2.17.x86_64.rpm (installed size 178.00 kB 8.79  x   xInstalling FirmwareUpdateKit-1.1-2.17.x86_64.rpm (installed size 178.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  ±±±44  70%  9  9  100  02 1 61  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpp43 (download size 2.80 MB)   x   x~kB) x    xDownloading cpp43 (download size 2.80 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpp43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.83 MB 61  x   xInstalling cpp43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.83 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  13.18 0 0.78  6 5%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-perl (download size 105.00 kB)   x   x~MB) x    xDownloading limal-perl (download size 105.00 kB) q   15  30  ±±45  61%  76  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-perl-1.5.3-0.2.11.x86_64.rpm (installed size 374.00 kB) 0.784  x   xInstalling limal-perl-1.5.3-0.2.11.x86_64.rpm (installed size 374.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  37  54%  71%  88  99  100  2.82 19 42  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-nfs-server (download size 115.00 kB)   x   xDownloading limal-nfs-server (download size 115.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-nfs-server-1.5.3-0.2.11.x86_64.rpm (installed size 487.00 42  x   xInstalling limal-nfs-server-1.5.3-0.2.11.x86_64.rpm (installed size 487.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  26  39  52% 7%  63%  8  73  9  84  9  95  100  34 8 09.94  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-ca-mgm (download size 660.00 kB)   x   x~00 kB) x    xDownloading limal-ca-mgm (download size 660.00 kB) q   7  12  9  24  9  36  ±±±±±41  ±46  53%  8%  65%  70  5  82  7  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-ca-mgm-1.5.24-0.3.2.x86_64.rpm (installed size 2.65 MB) 09.94  x   xInstalling limal-ca-mgm-1.5.24-0.3.2.x86_64.rpm (installed size 2.65 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  40  ±47  54%  61%  8  73  8  83  8  93  8  100  09.70 7 7.29  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-LibXML (download size 358.00 kB)   x   xDownloading perl-XML-LibXML (download size 358.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-LibXML-1.66-1.14.x86_64.rpm (installed size 1.11 MB) 7.293  x   xInstalling perl-XML-LibXML-1.66-1.14.x86_64.rpm (installed size 1.11 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  33  8  ±±±44  49 55%  61%  6  72  7  83  8  93  8  100  8.58 6 6.18  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-NetxAP (download size 54.00 kB)   x   xDownloading perl-NetxAP (download size 54.00 kB) q   29  59%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-NetxAP-0.02-604.22.x86_64.rpm (installed size 135.00 kB) 6.182  x   xInstalling perl-NetxAP-0.02-604.22.x86_64.rpm (installed size 135.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  57%  69%  75  84  96  100  45 5 05  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Date-Calc (download size 258.00 kB)   x   xDownloading perl-Date-Calc (download size 258.00 kB) q   6  12  8  24  31  7  ±±±±43  49 55%  62%  8  74  80  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Date-Calc-5.4-137.14.x86_64.rpm (installed size 1.03 MB) 05  x   xInstalling perl-Date-Calc-5.4-137.14.x86_64.rpm (installed size 1.03 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  8  25  31  6  ±±±44  50 6%  62%  8  73  8  83  8  93  8  100  7.42 4 5.02  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcsh (download size 357.00 kB)   x   xDownloading tcsh (download size 357.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcsh-6.18.01-0.9.1.x86_64.rpm (installed size 1.01 MB) 5.02  x   xInstalling tcsh-6.18.01-0.9.1.x86_64.rpm (installed size 1.01 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  5  ±±±±±41  ±47  53%  62%  71  6  84  9  9  100  6.40 3 4.00  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcl (download size 1.69 MB)   x   xDownloading tcl (download size 1.69 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcl-8.5.5-2.81.x86_64.rpm (installed size 5.05 MB) 4.00  x   xInstalling tcl-8.5.5-2.81.x86_64.rpm (installed size 5.05 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 7%  62%  7  72  7  83  8  93  8  100  1.35 2 698.95  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysvinit (download size 204.00 kB)   x   xDownloading sysvinit (download size 204.00 kB) q   7  15  23  31  9  ±47  54%  62%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysvinit-2.86-219.37.x86_64.rpm (installed size 642.00 kB) 698.950  x   xInstalling sysvinit-2.86-219.37.x86_64.rpm (installed size 642.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  23  9  3  ±±45  50 6%  61%  7  72  80  6  91  6  100  0.72 1 32  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading srptools (download size 42.00 kB)   x   xDownloading srptools (download size 42.00 kB) q   37  76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing srptools-0.0.4-11.26.x86_64.rpm (installed size 94.00 kB) 32  x   xInstalling srptools-0.0.4-11.26.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  72%  96  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  63 0 23  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sharutils (download size 163.00 kB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading sharutils (download size 163.00 kB)  8  1  2  3  ±47  57%  6  7  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sharutils-4.6.3-3.19.x86_64.rpm (installed size 577.00 kB) 23  x   xInstalling sharutils-4.6.3-3.19.x86_64.rpm (installed size 577.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  24  31  8  ±±±44  50 5%  61%  9  75  82  9  96  100  xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 697.67 MBx 345x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 200.07 MBx 209x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opensm (download size 502.00 kB)   x   xDownloading opensm (download size 502.00 kB) q   6  12  9  25  31  8  ±±±44  50 7%  63%  70  6  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opensm-3.3.13-12.26.x86_64.rpm (installed size 1.34 MB) 7.674:0  x   xInstalling opensm-3.3.13-12.26.x86_64.rpm (installed size 1.34 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  31  6  ±±45  54%  9%  64  73  80  5  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/opensm... q  198.73 8 6.33  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lukemftp (download size 129.00 kB)  Additional rpm output: Updating etc/sysconfig/opensm...   Downloading lukemftp (download size 129.00 kB)  1  23  35  48 60%  73  85  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lukemftp-1.5-725.17.x86_64.rpm (installed size 278.00 kB) 6.33  x   xInstalling lukemftp-1.5-725.17.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±±45  68%  7  90  9  100  46 7 05  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ib-bonding (download size 11.00 kB)   x   xDownloading ib-bonding (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ib-bonding-0.9.0-138.35.x86_64.rpm (installed size 18.00 kB) 05  x   xInstalling ib-bonding-0.9.0-138.35.x86_64.rpm (installed size 18.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  55%  93%  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  4 6 4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading groff (download size 1.92 MB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading groff (download size 1.92 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing groff-1.18.1.1-167.18.61.x86_64.rpm (installed size 7.24 MB) 4  x   xInstalling groff-1.18.1.1-167.18.61.x86_64.rpm (installed size 7.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/suseconfig... q  1.20 5 88.79  1 6%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading coreutils-lang (download size 1.38 MB)  Additional rpm output: Updating etc/sysconfig/suseconfig...   Downloading coreutils-lang (download size 1.38 MB)  5  10  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing coreutils-lang-8.12-6.25.32.33.1.x86_64.rpm (installed size 7.9 88.793:58  x   xInstalling coreutils-lang-8.12-6.25.32.33.1.x86_64.rpm (installed size 7q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  83.27 4 0.87  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading blktrace (download size 236.00 kB)   x   x~.93 MB) x    xDownloading blktrace (download size 236.00 kB) q   5  12  9  26  33  9  ±46  53%  60%  7  73  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing blktrace-1.0.3-0.7.1.x86_64.rpm (installed size 414.00 kB) 0.875  x   xInstalling blktrace-1.0.3-0.7.1.x86_64.rpm (installed size 414.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  9  25  30  9  55%  60%  75  90  5  100  2.8 3 46  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-theme-SLE (download size 2.72 MB)   x   xDownloading yast2-theme-SLE (download size 2.72 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-theme-SLE-2.17.27-0.18.9.noarch.rpm (installed size 2.87 46  x   xInstalling yast2-theme-SLE-2.17.27-0.18.9.noarch.rpm (installed size 2.8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  8  33  8  ±±±±43  48 54%  9%  64  9  74  9  84  9  94  9  100  0.00 2 77.60  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ulimit (download size 9.00 kB)   x   x~7 MB) x    xDownloading ulimit (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ulimit-1.2-2.4.11.noarch.rpm (installed size 13.00 kB) 77.604  x   xInstalling ulimit-1.2-2.4.11.noarch.rpm (installed size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  99%  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/ulimit... q  79.99 1 59  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading linux-kernel-headers (download size 813.00 kB)  Additional rpm output: Updating etc/sysconfig/ulimit...   Downloading linux-kernel-headers (download size 813.00 kB)  5  10  5  21  7  33  9  ±±45  51 7%  63%  9  75  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.95 59  x   xInstalling linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.9q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  Downloading linux-kernel-headers (download size 813.00 kB) Installing linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.9 ~5 MB) Additional rpm output: post linux-kernel-headers-3.0-19.18 /var/tmp/rpm-tmp.64369 1  7.04 0 4.63  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpp (download size 5.00 kB)  Additional rpm output: post linux-kernel-headers-3.0-19.18 /var/tmp/rpm-tmp.64369 1   Downloading cpp (download size 5.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpp-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB) 4.633  x   xInstalling cpp-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  33  54%  79%  100  3 199  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-nfs-server-perl (download size 135.00 kB)   x   xDownloading limal-nfs-server-perl (download size 135.00 kB) q   11  23  35  ±47  59%  71  83  95  100  7%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-nfs-server-perl-1.5.3-0.2.11.x86_64.rpm (installed size 6  x   xInstalling limal-nfs-server-perl-1.5.3-0.2.11.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  25  3  ±46  56%  66%  77  8  98  100  6.44 8 04  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-ca-mgm-perl (download size 351.00 kB)   x   x~ 610.00 kB) x    xDownloading limal-ca-mgm-perl (download size 351.00 kB) q   9  18  27  36  ±±±±±41  50 9%  68  77  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-ca-mgm-perl-1.5.24-0.3.2.x86_64.rpm (installed size 1.83 04  x   xInstalling limal-ca-mgm-perl-1.5.24-0.3.2.x86_64.rpm (installed size 1.8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  21  8  35  ±±±±42  49 56%  62%  9  76  83  90  7  100  4.61 7 2.21  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mail-plugins (download size 23.00 kB)   x   x~3 MB) x    xDownloading yast2-mail-plugins (download size 23.00 kB) q   70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mail-plugins-2.17.7-0.15.1.noarch.rpm (installed size 65. 2.212  x   xInstalling yast2-mail-plugins-2.17.7-0.15.1.noarch.rpm (installed size 6q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  53%  99%  100  55 6 15  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mpi-selector (download size 24.00 kB)   x   x~5.00 kB) x    xDownloading mpi-selector (download size 24.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mpi-selector-1.0.3-0.3.1.x86_64.rpm (installed size 49.00 kB) 15  x   xInstalling mpi-selector-1.0.3-0.3.1.x86_64.rpm (installed size 49.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  71%  6  81  6  99  100  0 5 0  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cvs (download size 387.00 kB)   x   xDownloading cvs (download size 387.00 kB) q   8  16  24  33  ±±±±±41  49 57%  62%  70  8  86  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cvs-1.12.12-144.23.5.1.x86_64.rpm (installed size 899.00 kB) 0  x   xInstalling cvs-1.12.12-144.23.5.1.x86_64.rpm (installed size 899.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  5  80  7  92  8  100  3.62 4 1.22  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading expect (download size 277.00 kB)   x   xDownloading expect (download size 277.00 kB) q   5  11  7  23  8  34  40  ±46  51 7%  63%  9  75  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing expect-5.44.1.11-1.241.1.x86_64.rpm (installed size 668.00 kB) 1.22  x   xInstalling expect-5.44.1.11-1.241.1.x86_64.rpm (installed size 668.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  26  35  ±±4  52% 62%  7  8  90  7  100  2.97 3 0.57  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading metamail (download size 153.00 kB)   x   x~) x    xDownloading metamail (download size 153.00 kB) q   1  2  31  ±±±±±4  52% 62%  73  84  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing metamail-2.7.19-1244.20.x86_64.rpm (installed size 422.00 kB) 0.57  x   xInstalling metamail-2.7.19-1244.20.x86_64.rpm (installed size 422.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  30  7  ±±±44  50 5%  60%  78  89  94  9  100  56 2 16  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man (download size 569.00 kB)   x   xDownloading man (download size 569.00 kB) q   5  11  6  22  8  33  9  ±±45  50 6%  61%  7  73  8  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-2.5.2-17.16.x86_64.rpm (installed size 1.93 MB) 16  x   xInstalling man-2.5.2-17.16.x86_64.rpm (installed size 1.93 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  35  ±±±±±41  ±47  52% 7%  63%  9  74  9  84  9  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/cron... q  0.63 1 68.23  7 8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-branding-SLES (download size 10.00 kB)  Additional rpm output: Updating etc/sysconfig/cron...   Downloading yast2-branding-SLES (download size 10.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-branding-SLES-2.17.0-1.11.noarch.rpm (installed size 18.0 68.231  x   xInstalling yast2-branding-SLES-2.17.0-1.11.noarch.rpm (installed size 18q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 0 1  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-common (download size 8.00 kB)   x   x~.00 kB) x    xDownloading yast2-nfs-common (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-common-2.17.8-0.2.1.noarch.rpm (installed size 4.00 k 1  x   xInstalling yast2-nfs-common-2.17.8-0.2.1.noarch.rpm (installed size 4.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  64%  73  96  100  89  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi (download size 1.13 MB)   x   x~ kB) x    xDownloading openmpi (download size 1.13 MB) q   5  10  6  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-1.8.4-1.26.x86_64.rpm (installed size 3.93 MB)  x   xInstalling openmpi-1.8.4-1.26.x86_64.rpm (installed size 3.93 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 6%  61%  6  71  6  81  6  91  7  100  66.67 8 4.27  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 664.27 MBx 324x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu Downloading powerman (download size 254.00 kB)   x   xDownloading powerman (download size 254.00 kB) q   6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing powerman-2.3.24-1.02.wc1.x86_64.rpm (installed size 1.16 MB) 4.2746  x   xInstalling powerman-2.3.24-1.02.wc1.x86_64.rpm (installed size 1.16 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  48 54%  9%  65  70  5  81  6  95  100  3.11  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fdtree (download size 12.00 kB)   x   xDownloading fdtree (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fdtree-1.0.2-1.wc1.x86_64.rpm (installed size 32.00 kB) 3.1139  x   xInstalling fdtree-1.0.2-1.wc1.x86_64.rpm (installed size 32.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  87%  99  100  08  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iozone (download size 690.00 kB)   x   xDownloading iozone (download size 690.00 kB) q   6  11  6  23  30  7  ±±±44  50 5%  60%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iozone-3-373.wc1.x86_64.rpm (installed size 1.00 MB) 08  x   xInstalling iozone-3-373.wc1.x86_64.rpm (installed size 1.00 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  23  9  38  ±±45  51 9%  65  70  6  83  9  95  100  2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading posix (download size 3.54 MB)   x   xDownloading posix (download size 3.54 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing posix-1.1-intel.x86_64.rpm (installed size 4.08 MB) 2  x   xInstalling posix-1.1-intel.x86_64.rpm (installed size 4.08 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±42  48 54%  60%  5  71  6  82  7  93  9  100  58.00  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading munge-libs (download size 42.00 kB)   x   xDownloading munge-libs (download size 42.00 kB) q   38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing munge-libs-0.5.10-1.wc1.x86_64.rpm (installed size 120.00 kB) 58.007  x   xInstalling munge-libs-0.5.10-1.wc1.x86_64.rpm (installed size 120.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  99%  100  7.88  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bonnie++ (download size 34.00 kB)   x   xDownloading bonnie++ (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bonnie++-1.03e-1.wc1.x86_64.rpm (installed size 77.00 kB) 7.88  x   xInstalling bonnie++-1.03e-1.wc1.x86_64.rpm (installed size 77.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  61%  80  92  7  100  0  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading compilebench (download size 248.00 kB)   x   xDownloading compilebench (download size 248.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing compilebench-0.6-1.wc1.x86_64.rpm (installed size 2.98 MB) 0  x   xInstalling compilebench-0.6-1.wc1.x86_64.rpm (installed size 2.98 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  23  8  34  40  ±±45  50 7%  63%  9  75  80  6  92  8  100  4.82  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading simul (download size 23.00 kB)   x   xDownloading simul (download size 23.00 kB) q   61%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing simul-1.14-1.wc1.x86_64.rpm (installed size 62.00 kB) 4.826  x   xInstalling simul-1.14-1.wc1.x86_64.rpm (installed size 62.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  69%  96  100  76  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading IOR (download size 78.00 kB)   x   xDownloading IOR (download size 78.00 kB) q   2  4  61%  82  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing IOR-2.10.3-wc1.x86_64.rpm (installed size 278.00 kB) 76  x   xInstalling IOR-2.10.3-wc1.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  ±±45  68%  7  83  99  100  49  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mdtest (download size 43.00 kB)   x   xDownloading mdtest (download size 43.00 kB) q   37  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mdtest-1.8.3-1.wc1.x86_64.rpm (installed size 127.00 kB) 49  x   xInstalling mdtest-1.8.3-1.wc1.x86_64.rpm (installed size 127.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 54%  65%  70  5  94  9  100  37  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading metabench (download size 29.00 kB)   x   xDownloading metabench (download size 29.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing metabench-1.0-1.wc2.x86_64.rpm (installed size 69.00 kB) 37  x   xInstalling metabench-1.0-1.wc2.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  90%  9  100  0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading munge (download size 221.00 kB)   x   xDownloading munge (download size 221.00 kB) q   7  14  21  8  36  ±±±±43  50 7%  65%  72  9  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing munge-0.5.10-1.wc1.x86_64.rpm (installed size 795.00 kB) 0  x   xInstalling munge-0.5.10-1.wc1.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  29  3  ±47  55%  63%  71  9  87  92  7  100  3.52  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-mrsh (download size 16.00 kB)   x   xDownloading pdsh-rcmd-mrsh (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-mrsh-2.29-1.wc1.x86_64.rpm (installed size 35.00 kB) 3.52  x   xInstalling pdsh-rcmd-mrsh-2.29-1.wc1.x86_64.rpm (installed size 35.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  49  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh (download size 208.00 kB)   x   x~) x    xDownloading pdsh (download size 208.00 kB) q   6  14  22  9  37  ±±45  53%  60%  8  76  83  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-2.29-1.wc1.x86_64.rpm (installed size 1.59 MB) 49  x   xInstalling pdsh-2.29-1.wc1.x86_64.rpm (installed size 1.59 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  31  6  ±±±44  51 9%  67  75  83  91  6  100  1.90  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 651.90 MBx 310x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu Downloading libcom_err2 (download size 60.00 kB)   x   xDownloading libcom_err2 (download size 60.00 kB) q   26  53%  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcom_err2-1.42.13.wc5-7suse.x86_64.rpm (installed size 126.00 1.904  x   xInstalling libcom_err2-1.42.13.wc5-7suse.x86_64.rpm (installed size 126.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  80%  99  100  78  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libext2fs2 (download size 389.00 kB)   x   x~00 kB) x    xDownloading libext2fs2 (download size 389.00 kB) q   8  16  24  32  7  ±±45  53%  61%  9  74  82  90  8  100  9%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libext2fs2-1.42.13.wc5-7suse.x86_64.rpm (installed size 1.35 MB 7825  x   xInstalling libext2fs2-1.42.13.wc5-7suse.x86_64.rpm (installed size 1.35 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  26  31  40  49 54%  63%  72  7  86  91  9  100  0.42  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading e2fsprogs (download size 1.29 MB)   x   x~MB) x    xDownloading e2fsprogs (download size 1.29 MB) q   5  10  6  22  7  33  9  ±±±44  50 5%  61%  7  - 819.69 kB/s (download size 1.29 MB) 72  8  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing e2fsprogs-1.42.13.wc5-7suse.x86_64.rpm (installed size 5.13 MB) 0.42  x   xInstalling e2fsprogs-1.42.13.wc5-7suse.x86_64.rpm (installed size 5.13 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  33  8  ±±±±43  48 53%  8%  63%  9  74  9  85  91  6  100  45.29  7 60  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx w Total x 645.29 MBx 307x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu Downloading ruby-devel (download size 1021.00 kB)   x   x~B) x    xDownloading ruby-devel (download size 1021.00 kB) q   6  11  6  21  7  32  8  ±±±±43  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ruby-devel-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 4.46 45.292  x   xInstalling ruby-devel-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 4.4q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  32  7  ±±±±42  ±47  53%  8%  63%  8  74  9  84  9  95  100  0.83  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 640.83 MBx 306x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu Downloading yast2-core (download size 1009.00 kB)   x   x~6 MB) x    xDownloading yast2-core (download size 1009.00 kB) q   6  12  8  23  8  34  9  ±±45  50 5%  61%  6  72  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-core-2.17.47-7.1.x86_64.rpm (installed size 3.28 MB) 0.83  x   xInstalling yast2-core-2.17.47-7.1.x86_64.rpm (installed size 3.28 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  32  7  ±±±±43  48 53%  9%  65  71  6  82  8  94  9  100  37.55  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-storage-lib (download size 718.00 kB)   x   xDownloading yast2-storage-lib (download size 718.00 kB) q   6  11  7  24  30  5  ±±±±42  48 53%  9%  64  71  7  82  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-storage-lib-2.17.161-5.1.x86_64.rpm (installed size 1.99 37.551  x   xInstalling yast2-storage-lib-2.17.161-5.1.x86_64.rpm (installed size 1.9q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  9  100  5.57  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libidn (download size 223.00 kB)   x   x~9 MB) x    xDownloading libidn (download size 223.00 kB) q   7  14  21  8  35  ±±±±42  50 7%  64%  71  8  86  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libidn-1.10-6.1.x86_64.rpm (installed size 788.00 kB) 5.570  x   xInstalling libidn-1.10-6.1.x86_64.rpm (installed size 788.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  20  8  36  ±±±44  52% 60%  8  75  8  90  5  100  4.80  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading binutils (download size 5.13 MB)   x   xDownloading binutils (download size 5.13 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing binutils-2.25.0-5.5.x86_64.rpm (installed size 33.97 MB) 4.80  x   xInstalling binutils-2.25.0-5.5.x86_64.rpm (installed size 33.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  00.82  2 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tar (download size 753.00 kB)   x   xDownloading tar (download size 753.00 kB) q   5  10  7  23  9  34  40  ±46  51 7%  63%  8  74  80  5  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tar-1.26-1.2.13.1.x86_64.rpm (installed size 2.48 MB) 00.821  x   xInstalling tar-1.26-1.2.13.1.x86_64.rpm (installed size 2.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  7  ±±±±42  49 54%  9%  64  9  74  9  84  90  5  100  598.34  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sed (download size 160.00 kB)   x   xDownloading sed (download size 160.00 kB) q   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sed-4.1.5-85.32.1.x86_64.rpm (installed size 491.00 kB) 598.3409  x   xInstalling sed-4.1.5-85.32.1.x86_64.rpm (installed size 491.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  7  23  40  52% 9%  64  9  74  9  85  90  5  100  7.86  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libksba (download size 104.00 kB)   x   xDownloading libksba (download size 104.00 kB) q   15  30  ±46  61%  77  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libksba-1.0.4-1.25.1.x86_64.rpm (installed size 271.00 kB) 7.86  x   xInstalling libksba-1.0.4-1.25.1.x86_64.rpm (installed size 271.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±47  70%  7  90  8  100  60 299 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpio-lang (download size 56.00 kB)   x   xDownloading cpio-lang (download size 56.00 kB) q   28  57%  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpio-lang-2.9-75.80.1.x86_64.rpm (installed size 225.00 kB) 60  x   xInstalling cpio-lang-2.9-75.80.1.x86_64.rpm (installed size 225.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  26  33  ±±±±±41  50 60%  7  76  88  9  100  38  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wget (download size 414.00 kB)   x   xDownloading wget (download size 414.00 kB) q   7  15  23  30  8  ±46  54%  61%  9  77  85  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wget-1.11.4-1.40.1.x86_64.rpm (installed size 1.46 MB) 38  x   xInstalling wget-1.11.4-1.40.1.x86_64.rpm (installed size 1.46 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  8  25  30  6  ±±±±42  48 55%  60%  5  71  6  82  8  95  100  5.91  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xfsprogs (download size 730.00 kB)   x   xDownloading xfsprogs (download size 730.00 kB) q   6  12  8  23  9  34  40  ±±45  51 6%  62%  7  73  8  84  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xfsprogs-3.1.8-0.10.2.x86_64.rpm (installed size 3.30 MB) 5.918  x   xInstalling xfsprogs-3.1.8-0.10.2.x86_64.rpm (installed size 3.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  35  40  ±±45  50 6%  61%  6  71  6  82  8  93  8  100  2.6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pciutils-ids (download size 206.00 kB)   x   xDownloading pciutils-ids (download size 206.00 kB) q   7  15  23  31  8  ±46  54%  62%  70  7  85  93  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pciutils-ids-2015.10.07-3.1.noarch.rpm (installed size 974.00 k 2.67  x   xInstalling pciutils-ids-2015.10.07-3.1.noarch.rpm (installed size 974.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  33  9  ±46  52% 9%  66  72  9  85  92  8  100  1.66  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-locale (download size 11.36 MB)   x   x~ kB) x    xDownloading glibc-locale (download size 11.36 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-locale-2.11.3-17.102.1.x86_64.rpm (installed size 106.13 1.66  x   xInstalling glibc-locale-2.11.3-17.102.1.x86_64.rpm (installed size 106.1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  485.53  4 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vim (download size 835.00 kB)   x   x~3 MB) x    xDownloading vim (download size 835.00 kB) q   5  10  6  21  6  32  8  ±±±44  50 6%  62%  7  72  8  83  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vim-7.2-8.17.1.x86_64.rpm (installed size 1.77 MB) 485.532:35  x   xInstalling vim-7.2-8.17.1.x86_64.rpm (installed size 1.77 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  3.76  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading timezone (download size 358.00 kB)   x   xDownloading timezone (download size 358.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing timezone-2017b-0.51.1.x86_64.rpm (installed size 1.21 MB) 3.76  x   xInstalling timezone-2017b-0.51.1.x86_64.rpm (installed size 1.21 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  2.55  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading usbutils (download size 229.00 kB)   x   xDownloading usbutils (download size 229.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing usbutils-004-1.18.1.x86_64.rpm (installed size 696.00 kB) 2.554  x   xInstalling usbutils-004-1.18.1.x86_64.rpm (installed size 696.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  15  21  32  ±±±±±41  50 9%  68  7  87  96  100  1.87  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libX11 (download size 726.00 kB)   x   xDownloading xorg-x11-libX11 (download size 726.00 kB) q   6  13  9  24  9  36  ±±±±42  ±47  53%  8%  64%  9  74  9  85  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libX11-7.4-5.11.68.1.x86_64.rpm (installed size 3.00 M 1.87  x   xInstalling xorg-x11-libX11-7.4-5.11.68.1.x86_64.rpm (installed size 3.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  9  35  ±±±±±41  ±46  51 7%  63%  8  73  9  85  91  6  100  78  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-devel (download size 4.87 MB)   x   x~ MB) x    xDownloading glibc-devel (download size 4.87 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-devel-2.11.3-17.102.1.x86_64.rpm (installed size 31.28 MB 783  x   xInstalling glibc-devel-2.11.3-17.102.1.x86_64.rpm (installed size 31.28 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  47.58  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Bootloader (download size 161.00 kB)   x   x~MB) x    xDownloading perl-Bootloader (download size 161.00 kB) q   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Bootloader-0.4.89.72-3.1.x86_64.rpm (installed size 369.00 47.5824  x   xInstalling perl-Bootloader-0.4.89.72-3.1.x86_64.rpm (installed size 369.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  25  ±±±±42  51 8%  66%  81  6  91  6  100  22  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading crash (download size 2.50 MB)   x   x~00 kB) x    xDownloading crash (download size 2.50 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  - 2.50 MB/s (download size 2.50 MB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing crash-7.0.9-26.1.x86_64.rpm (installed size 6.95 MB) 22  x   xInstalling crash-7.0.9-26.1.x86_64.rpm (installed size 6.95 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  0.28  7 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading supportutils (download size 126.00 kB)   x   xDownloading supportutils (download size 126.00 kB) q   12  25  38  50 63%  76  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing supportutils-1.20-121.1.noarch.rpm (installed size 292.00 kB) 0.281  x   xInstalling supportutils-1.20-121.1.noarch.rpm (installed size 292.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  18  40  62%  84  92  7  100  39.99  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading krb5 (download size 521.00 kB)   x   xDownloading krb5 (download size 521.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  85  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing krb5-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.49 MB) 39.99  x   xInstalling krb5-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.49 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  20  8  34  9  ±±±44  49 56%  65%  73  81  90  7  100  8.50  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpcbind (download size 48.00 kB)   x   xDownloading rpcbind (download size 48.00 kB) q   32  66%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpcbind-0.1.6+git20080930-6.24.1.x86_64.rpm (installed size 101 8.50  x   xInstalling rpcbind-0.1.6+git20080930-6.24.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  62%  9  9  100  Updating etc/sysconfig/rpcbind... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl (download size 314.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading cyrus-sasl (download size 314.00 kB)  5  10  5  20  5  30  5  40  ±±45  50 5%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-2.1.22-182.23.1.x86_64.rpm (installed size 1.21 MB) 4  x   xInstalling cyrus-sasl-2.1.22-182.23.1.x86_64.rpm (installed size 1.21 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  9  35  40  ±±45  52% 7%  65%  71  7  83  92  8  100  7.19  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2-lang (download size 499.00 kB)   x   x~) x    xDownloading glib2-lang (download size 499.00 kB) q   6  12  9  25  32  8  ±±±44  51 7%  64%  70  6  83  9  96  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-lang-2.22.5-0.8.26.1.x86_64.rpm (installed size 3.20 MB) 7.19  x   xInstalling glib2-lang-2.22.5-0.8.26.1.x86_64.rpm (installed size 3.20 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  6  81  6  92  7  100  3.9  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading open-iscsi (download size 354.00 kB)   x   x~) x    xDownloading open-iscsi (download size 354.00 kB) q   8  13  8  27  36  ±±45  54%  63%  72  81  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing open-iscsi-2.0.873-0.41.3.x86_64.rpm (installed size 1.67 MB) 3.90  x   xInstalling open-iscsi-2.0.873-0.41.3.x86_64.rpm (installed size 1.67 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  9  37  ±±±44  51 9%  66  74  81  6  94  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  2.31  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsync (download size 362.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading rsync (download size 362.00 kB)  8  13  22  30  5  ±±±44  52% 7%  66%  75  83  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsync-3.0.4-2.52.1.x86_64.rpm (installed size 713.00 kB) 2.3119  x   xInstalling rsync-3.0.4-2.52.1.x86_64.rpm (installed size 713.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  27  36  ±±45  54%  63%  7  82  9  98  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 431.62 MBx 280x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nscd (download size 99.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading nscd (download size 99.00 kB)  15  32  48 64%  80  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nscd-2.11.3-17.102.1.x86_64.rpm (installed size 138.00 kB) 1.621:4  x   xInstalling nscd-2.11.3-17.102.1.x86_64.rpm (installed size 138.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 96%  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  48  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading microcode_ctl (download size 1.33 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading microcode_ctl (download size 1.33 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing microcode_ctl-1.17-102.82.1.x86_64.rpm (installed size 2.73 MB) 48  x   xInstalling microcode_ctl-1.17-102.82.1.x86_64.rpm (installed size 2.73 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  8  25  31  6  ±±±±43  49 54%  9%  65  70  5  80  5  90  5  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  28.75  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading irqbalance (download size 29.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading irqbalance (download size 29.00 kB)  48 53%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing irqbalance-1.0.4-0.23.2.x86_64.rpm (installed size 46.00 kB) 28.758  x   xInstalling irqbalance-1.0.4-0.23.2.x86_64.rpm (installed size 46.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  92%  9  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/irqbalance...  1  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpio (download size 80.00 kB)  ~ script `munge' Updating etc/sysconfig/irqbalance...   Downloading cpio (download size 80.00 kB)  17  3  56%  76%  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpio-2.9-75.80.1.x86_64.rpm (installed size 177.00 kB) 1  x   xInstalling cpio-2.9-75.80.1.x86_64.rpm (installed size 177.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  71%  94  9  100  53  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pciutils (download size 77.00 kB)   x   xDownloading pciutils (download size 77.00 kB) q   2  ±±±±±41  62%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pciutils-3.1.7-11.13.1.x86_64.rpm (installed size 155.00 kB) 53  x   xInstalling pciutils-3.1.7-11.13.1.x86_64.rpm (installed size 155.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  71%  89  94  9  100  38  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXrender (download size 19.00 kB)   x   xDownloading xorg-x11-libXrender (download size 19.00 kB) q   74%  82  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXrender-7.4-1.23.1.x86_64.rpm (installed size 39.00 38  x   xInstalling xorg-x11-libXrender-7.4-1.23.1.x86_64.rpm (installed size 39.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXfixes (download size 11.00 kB)   x   x~00 kB) x    xDownloading xorg-x11-libXfixes (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXfixes-7.4-1.20.1.x86_64.rpm (installed size 23.00 4  x   xInstalling xorg-x11-libXfixes-7.4-1.20.1.x86_64.rpm (installed size 23.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kexec-tools (download size 140.00 kB)   x   x~0 kB) x    xDownloading kexec-tools (download size 140.00 kB) q   9  24  30  6  ±47  54%  66%  78  84  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kexec-tools-2.0.3-0.20.2.x86_64.rpm (installed size 388.00 kB) 2  x   xInstalling kexec-tools-2.0.3-0.20.2.x86_64.rpm (installed size 388.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  36  52% 69%  75  80  5  93  8  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  7.94  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading crash-eppic (download size 167.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading crash-eppic (download size 167.00 kB)  11  20  3  9  4 59%  68  7  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing crash-eppic-7.0.9-26.1.x86_64.rpm (installed size 358.00 kB) 7.94  x   xInstalling crash-eppic-7.0.9-26.1.x86_64.rpm (installed size 358.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  71%  89  97  100  59  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mailx (download size 323.00 kB)   x   xDownloading mailx (download size 323.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mailx-12.5-1.11.2.x86_64.rpm (installed size 527.00 kB) 59  x   xInstalling mailx-12.5-1.11.2.x86_64.rpm (installed size 527.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  24  36  48 60%  6  72  85  91  9  100  08  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bind-libs (download size 1.12 MB)   x   xDownloading bind-libs (download size 1.12 MB) q   5  10  6  21  6  31  6  ±±±±42  48 53%  8%  63%  8  74  9  84  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bind-libs-9.9.6P1-0.39.1.x86_64.rpm (installed size 3.66 MB) 08  x   xInstalling bind-libs-9.9.6P1-0.39.1.x86_64.rpm (installed size 3.66 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  31  6  ±±±±±41  ±46  51 6%  62%  7  73  8  83  8  93  9  100  3.42  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-digestmd5 (download size 27.00 kB)   x   xDownloading cyrus-sasl-digestmd5 (download size 27.00 kB) q   59%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-digestmd5-2.1.22-182.23.1.x86_64.rpm (installed size 3.427  x   xInstalling cyrus-sasl-digestmd5-2.1.22-182.23.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  37  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-gssapi (download size 18.00 kB)   x   x~ze 52.00 kB) x    xDownloading cyrus-sasl-gssapi (download size 18.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-gssapi-2.1.22-182.23.1.x86_64.rpm (installed size 31 37  x   xInstalling cyrus-sasl-gssapi-2.1.22-182.23.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libldap-2_4-2 (download size 221.00 kB)   x   x~31.00 kB) x    xDownloading libldap-2_4-2 (download size 221.00 kB) q   7  14  21  8  36  ±±±±43  50 7%  65%  72  9  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libldap-2_4-2-2.4.26-0.65.2.x86_64.rpm (installed size 667.00 k 4  x   xInstalling libldap-2_4-2-2.4.26-0.65.2.x86_64.rpm (installed size 667.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  1  2  37  ±4  52% 62%  71  8  9  9  100  2.69  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-plain (download size 13.00 kB)   x   x~ kB) x    xDownloading cyrus-sasl-plain (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-plain-2.1.22-182.23.1.x86_64.rpm (installed size 19. 2.69  x   xInstalling cyrus-sasl-plain-2.1.22-182.23.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2 (download size 160.00 kB)   x   x~9.00 kB) x    xDownloading glib2 (download size 160.00 kB) q   8  1  2  3  4 58%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-2.22.5-0.8.26.1.x86_64.rpm (installed size 497.00 kB) 7  x   xInstalling glib2-2.22.5-0.8.26.1.x86_64.rpm (installed size 497.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  24  36  49 62%  74  87  98  100  18  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bind-utils (download size 264.00 kB)   x   xDownloading bind-utils (download size 264.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  - 191.74 kB/s (download size 264.00 kB) 8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bind-utils-9.9.6P1-0.39.1.x86_64.rpm (installed size 1.05 MB) 18  x   xInstalling bind-utils-9.9.6P1-0.39.1.x86_64.rpm (installed size 1.05 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  24  31  7  ±±±±42  51 7%  63%  70  5  84  90  5  100  1.13  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sudo (download size 439.00 kB)   x   xDownloading sudo (download size 439.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sudo-1.7.6p2-0.29.1.x86_64.rpm (installed size 1.23 MB) 1.136  x   xInstalling sudo-1.7.6p2-0.29.1.x86_64.rpm (installed size 1.23 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  22  9  37  ±±±±42  ±47  53%  8%  63%  8  73  8  83  8  93  8  100  19.90  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcurl4 (download size 153.00 kB)   x   xDownloading libcurl4 (download size 153.00 kB) q   1  2  31  ±±±±±4  52% 62%  73  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcurl4-7.19.7-1.64.1.x86_64.rpm (installed size 457.00 kB) 19.90  x   xInstalling libcurl4-7.19.7-1.64.1.x86_64.rpm (installed size 457.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  28  ±±±±42  56%  70%  84  98  100  46  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-saslauthd (download size 49.00 kB)   x   xDownloading cyrus-sasl-saslauthd (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed size 46  x   xInstalling cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  68%  84  96  100  Downloading cyrus-sasl-saslauthd (download size 49.00 kB) Installing cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed si ~ze 121.00 kB) Additional rpm output: Updating etc/sysconfig/saslauthd...  34  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openldap2-client (download size 164.00 kB)  Additional rpm output: Updating etc/sysconfig/saslauthd...   Downloading openldap2-client (download size 164.00 kB)  9  1  2  38  4 58%  6  7  87  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openldap2-client-2.4.26-0.65.2.x86_64.rpm (installed size 560.0 34  x   xInstalling openldap2-client-2.4.26-0.65.2.x86_64.rpm (installed size 560q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±43  55%  65%  77  83  9  8  100  8.79  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libglib-2_0-0 (download size 299.00 kB)   x   x~.00 kB) x    xDownloading libglib-2_0-0 (download size 299.00 kB) q   5  10  5  21  6  32  7  ±±±±42  48 53%  8%  64%  9  74  80  5  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libglib-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 789.00 8.79  x   xInstalling libglib-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 789.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  64%  73  81  9  97  100  02  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading curl (download size 168.00 kB)   x   x~00 kB) x    xDownloading curl (download size 168.00 kB) q   9  18  2  3  ±47  57%  66%  7  85  9  100  70 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing curl-7.19.7-1.64.1.x86_64.rpm (installed size 309.00 kB) 02  x   xInstalling curl-7.19.7-1.64.1.x86_64.rpm (installed size 309.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  ±±±±±41  ±46  63%  76  85  92  9  100  7.7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgthread-2_0-0 (download size 41.00 kB)   x   xDownloading libgthread-2_0-0 (download size 41.00 kB) q   39  78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgthread-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 19. 7.7  x   xInstalling libgthread-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgmodule-2_0-0 (download size 40.00 kB)   x   x~9.00 kB) x    xDownloading libgmodule-2_0-0 (download size 40.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgmodule-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 15. 0  x   xInstalling libgmodule-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  68  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgobject-2_0-0 (download size 128.00 kB)   x   x~5.00 kB) x    xDownloading libgobject-2_0-0 (download size 128.00 kB) q   12  24  37  49 62%  74  87  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgobject-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 264 68  x   xInstalling libgobject-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 72%  97  100  43  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgio-2_0-0 (download size 244.00 kB)   x   x~64.00 kB) x    xDownloading libgio-2_0-0 (download size 244.00 kB) q   6  13  9  26  32  9  ±±45  52% 9%  65  72  8  85  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgio-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 671.00 435  x   xInstalling libgio-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 671.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  3  ±47  57%  66%  7  85  9  100  6.77  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 416.77 MBx 253x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e8820372xxx tu Downloading yast2-ycp-ui-bindings (download size 218.00 kB)   x   x~0 kB) x    xDownloading yast2-ycp-ui-bindings (download size 218.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ycp-ui-bindings-2.21.1-2.5.107.x86_64.rpm (installed size 6.777  x   xInstalling yast2-ycp-ui-bindings-2.21.1-2.5.107.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  37  ±4  56%  66%  75  8  94  9  100  01 7 11  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-slp (download size 59.00 kB)   x   x~ze 676.00 kB) x    xDownloading yast2-slp (download size 59.00 kB) q   26  54%  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-slp-2.17.0-0.7.55.x86_64.rpm (installed size 284.00 kB) 11  x   xInstalling yast2-slp-2.17.0-0.7.55.x86_64.rpm (installed size 284.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  9  38  ±±±±43  53%  8%  6  73  9  84  90  5  100  5.73 6 5.83  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ksymoops (download size 739.00 kB)   x   xDownloading ksymoops (download size 739.00 kB) q   6  12  7  22  8  35  ±±±±±41  48 54%  61%  7  74  80  7  93  100  - 738.72 kB/s (download size 739.00 kB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ksymoops-2.4.11-117.22.x86_64.rpm (installed size 2.19 MB) 5.83  x   xInstalling ksymoops-2.4.11-117.22.x86_64.rpm (installed size 2.19 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  34  9  ±±45  51 7%  62%  8  74  9  85  91  7  100  3.55 5 3.65  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading smartmontools (download size 451.00 kB)   x   xDownloading smartmontools (download size 451.00 kB) q   7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing smartmontools-6.3-0.8.18.x86_64.rpm (installed size 1.66 MB) 3.656  x   xInstalling smartmontools-6.3-0.8.18.x86_64.rpm (installed size 1.66 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  20  8  35  ±±±±42  49 57%  64%  70  8  85  91  6  100  Updating etc/sysconfig/smartmontools... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  1.89 4 1.99  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading setserial (download size 23.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading setserial (download size 23.00 kB)  70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing setserial-2.17-716.22.x86_64.rpm (installed size 44.00 kB) 1.99  x   xInstalling setserial-2.17-716.22.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  74%  83  99  100  4 3 4  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading scpm (download size 757.00 kB)   x   xDownloading scpm (download size 757.00 kB) q   6  11  7  22  8  34  ±±±±±41  ±47  53%  60%  6  72  7  83  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing scpm-1.1.7-1.19.x86_64.rpm (installed size 2.83 MB) 4  x   xInstalling scpm-1.1.7-1.19.x86_64.rpm (installed size 2.83 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  8  24  9  35  ±±±±42  48 53%  9%  64  70  5  81  6  91  7  100  Updating etc/sysconfig/scpm... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  59.02 2 09.12  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading psmisc (download size 75.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading psmisc (download size 75.00 kB)  21  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing psmisc-22.7-7.3.1.x86_64.rpm (installed size 257.00 kB) 09.125  x   xInstalling psmisc-22.7-7.3.1.x86_64.rpm (installed size 257.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  22  35  40  ±±45  53%  61%  8  73  80  5  93  8  100  8.77 1 8.87  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procps (download size 203.00 kB)   x   xDownloading procps (download size 203.00 kB) q   7  15  23  31  9  ±47  55%  63%  71  9  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procps-3.2.7-151.28.1.x86_64.rpm (installed size 601.00 kB) 8.87  x   xInstalling procps-3.2.7-151.28.1.x86_64.rpm (installed size 601.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  2  7  32  ±±±±±41  51 8%  64%  72  8  83  8  93  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  18 0 28  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procmail (download size 134.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading procmail (download size 134.00 kB)  11  23  35  ±47  59%  71  83  95  100  1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procmail-3.22-240.8.1.x86_64.rpm (installed size 271.00 kB) 28  x   xInstalling procmail-3.22-240.8.1.x86_64.rpm (installed size 271.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  20  ±46  54%  63%  8  73  80  6  92  8  100  7.91 79 01  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openslp-server (download size 260.00 kB)   x   xDownloading openslp-server (download size 260.00 kB) q   6  12  8  24  30  6  ±±±±43  49 55%  61%  7  73  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openslp-server-1.2.0-172.24.1.x86_64.rpm (installed size 897.00 01  x   xInstalling openslp-server-1.2.0-172.24.1.x86_64.rpm (installed size 897.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  24  33  40  ±±45  53%  62%  9  76  85  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  04 8 7.14  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openct (download size 206.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading openct (download size 206.00 kB)  7  15  23  31  8  ±46  54%  62%  9  77  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openct-0.6.15-5.20.x86_64.rpm (installed size 874.00 kB) 7.14  x   xInstalling openct-0.6.15-5.20.x86_64.rpm (installed size 874.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  16  23  30  7  ±±±44  51 8%  64%  71  8  83  8  93  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  6.18 7 6.28  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kbd (download size 1.09 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading kbd (download size 1.09 MB)  5  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kbd-1.14.1-16.33.1.x86_64.rpm (installed size 2.41 MB) 6.28  x   xInstalling kbd-1.14.1-16.33.1.x86_64.rpm (installed size 2.41 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/keyboard...  3.77 6 3.87  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iproute2 (download size 952.00 kB)  ~ script `munge' Updating etc/sysconfig/keyboard...   Downloading iproute2 (download size 952.00 kB)  6  11  6  21  7  32  7  ±±±±42  ±47  52% 7%  62%  7  72  7  82  7  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iproute2-3.0-4.30.x86_64.rpm (installed size 1.69 MB) 3.874  x   xInstalling iproute2-3.0-4.30.x86_64.rpm (installed size 1.69 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  25  31  8  ±46  51 6%  63%  70  8  84  9  94  9  100  arpd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  2.09 5 2.19  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hdparm (download size 91.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading hdparm (download size 91.00 kB)  17  34  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hdparm-9.27-1.6.39.x86_64.rpm (installed size 161.00 kB) 2.19  x   xInstalling hdparm-9.27-1.6.39.x86_64.rpm (installed size 161.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  60%  74  87  99  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/ide... q  1.93 4 03  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpm (download size 197.00 kB)  Additional rpm output: Updating etc/sysconfig/ide...   Downloading gpm (download size 197.00 kB)  8  16  24  32  40  48 56%  65%  73  81  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpm-1.20.1-452.83.x86_64.rpm (installed size 546.00 kB) 03  x   xInstalling gpm-1.20.1-452.83.x86_64.rpm (installed size 546.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  5  27  33  ±±±44  53%  8%  69%  76  81  6  91  7  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/mouse...  40 3 1.50  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fbset (download size 41.00 kB)  ~ script `munge' Updating etc/sysconfig/mouse...   Downloading fbset (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fbset-2.1-919.22.x86_64.rpm (installed size 127.00 kB) 1.503  x   xInstalling fbset-2.1-919.22.x86_64.rpm (installed size 127.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  ±±45  65%  83  91  6  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/console...  27 2 37  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading acpid (download size 38.00 kB)  ~ script `munge' Updating etc/sysconfig/console...   Downloading acpid (download size 38.00 kB)  ±±±±±41  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing acpid-1.0.6-91.25.20.x86_64.rpm (installed size 77.00 kB) 37  x   xInstalling acpid-1.0.6-91.25.20.x86_64.rpm (installed size 77.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  34  53%  67%  72  7  82  94  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  0 1 0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pm-profiler (download size 19.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading pm-profiler (download size 19.00 kB)  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pm-profiler-0.1-28.7.1.noarch.rpm (installed size 37.00 kB) 0  x   xInstalling pm-profiler-0.1-28.7.1.noarch.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  25  37  ±±45  52% 60%  7  74  82  7  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/pm-profiler... q  16 0 26  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ncurses (download size 312.00 kB)  Additional rpm output: Updating etc/sysconfig/pm-profiler...   Downloading yast2-ncurses (download size 312.00 kB)  7  12  7  23  8  33  8  ±±±±43  48 53%  8%  64%  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ncurses-2.17.23-1.14.x86_64.rpm (installed size 1.03 MB) 26  x   xInstalling yast2-ncurses-2.17.23-1.14.x86_64.rpm (installed size 1.03 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  0.13 69 0.23  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libxkbfile (download size 63.00 kB)   x   x~) x    xDownloading xorg-x11-libxkbfile (download size 63.00 kB) q   25  50 75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libxkbfile-7.4-1.14.x86_64.rpm (installed size 149.00 0.23  x   xInstalling xorg-x11-libxkbfile-7.4-1.14.x86_64.rpm (installed size 149.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  86%  99  100  49.98 8 08  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXt (download size 155.00 kB)   x   x~0 kB) x    xDownloading xorg-x11-libXt (download size 155.00 kB) q   1  2  3  ±±±±±41  51 61%  72  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXt-7.4-1.19.8.x86_64.rpm (installed size 404.00 kB) 08  x   xInstalling xorg-x11-libXt-7.4-1.19.8.x86_64.rpm (installed size 404.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  31  ±47  63%  79  95  100  59 7 399.69  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXext (download size 27.00 kB)   x   x~B) x    xDownloading xorg-x11-libXext (download size 27.00 kB) q   60%  10  2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXext-7.4-1.18.16.x86_64.rpm (installed size 72.00 k 399.69  x   xInstalling xorg-x11-libXext-7.4-1.18.16.x86_64.rpm (installed size 72.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  88%  99  100  2 6 2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++43-devel (download size 2.25 MB)   x   x~ kB) x    xDownloading libstdc++43-devel (download size 2.25 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++43-devel-4.3.4_20091019-0.37.30.x86_64.rpm (installed 2  x   xInstalling libstdc++43-devel-4.3.4_20091019-0.37.30.x86_64.rpm (installeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  31.45 5 81.55  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc43 (download size 2.08 MB)   x   x~d size 18.06 MB) x    xDownloading gcc43 (download size 2.08 MB) q   5  10  5  20  5  30  5  40 - 772.73 kB/s (download size 2.08 MB)  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.21 MB 81.5539  x   xInstalling gcc43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.21 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  24.24 4 74.34  29 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cups-libs (download size 408.00 kB)   x   x~MB) x    xDownloading cups-libs (download size 408.00 kB) q   7  15  23  31  9  ±47  54%  62%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cups-libs-1.3.9-8.46.56.1.x86_64.rpm (installed size 2.02 MB) 74.347  x   xInstalling cups-libs-1.3.9-8.46.56.1.x86_64.rpm (installed size 2.02 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  23  8  35  40  ±47  53%  9%  66  71  9  84  90  7  100  2.22 3 2.32  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gfxboot (download size 29.00 kB)   x   xDownloading gfxboot (download size 29.00 kB) q   55%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gfxboot-4.1.34-0.5.44.x86_64.rpm (installed size 75.00 kB) 2.32  x   xInstalling gfxboot-4.1.34-0.5.44.x86_64.rpm (installed size 75.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  85%  97  100  15 2 25  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading x86info (download size 62.00 kB)   x   xDownloading x86info (download size 62.00 kB) q   25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing x86info-1.29-0.5.34.x86_64.rpm (installed size 227.00 kB) 25  x   xInstalling x86info-1.29-0.5.34.x86_64.rpm (installed size 227.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  ±±±±43  71%  97  100  1.93 1 03  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading efibootmgr (download size 38.00 kB)   x   xDownloading efibootmgr (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing efibootmgr-0.6.0-2.23.x86_64.rpm (installed size 74.00 kB) 03  x   xInstalling efibootmgr-0.6.0-2.23.x86_64.rpm (installed size 74.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  64%  88  95  100  85 0 1.95  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpupower (download size 82.00 kB)   x   xDownloading cpupower (download size 82.00 kB) q   19  38  58%  7  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpupower-3.19-2.23.3.x86_64.rpm (installed size 206.00 kB) 1.95  x   xInstalling cpupower-3.19-2.23.3.x86_64.rpm (installed size 206.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  55%  68%  73  80  8  93  8  100  6 59 7  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfsidmap (download size 32.00 kB)   x   xDownloading nfsidmap (download size 32.00 kB) q   50 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfsidmap-0.25-0.13.1.x86_64.rpm (installed size 82.00 kB) 7  x   xInstalling nfsidmap-0.25-0.13.1.x86_64.rpm (installed size 82.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  37  49 57%  89%  96  100  57 8 67  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libldapcpp1 (download size 87.00 kB)   x   xDownloading libldapcpp1 (download size 87.00 kB) q   18  36  55%  73%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libldapcpp1-0.3.0-0.9.29.x86_64.rpm (installed size 274.00 kB) 67  x   xInstalling libldapcpp1-0.3.0-0.9.29.x86_64.rpm (installed size 274.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±46  70%  93  9  100  31 7 41  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-WWW-Curl (download size 53.00 kB)   x   x~) x    xDownloading perl-WWW-Curl (download size 53.00 kB) q   29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-WWW-Curl-4.09-2.1.72.x86_64.rpm (installed size 142.00 kB) 416  x   xInstalling perl-WWW-Curl-4.09-2.1.72.x86_64.rpm (installed size 142.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  59%  80  9  5  100  17 6 27  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcroco-0_6-3 (download size 79.00 kB)   x   x~B) x    xDownloading libcroco-0_6-3 (download size 79.00 kB) q   17  34  54%  63%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcroco-0_6-3-0.6.1-120.16.x86_64.rpm (installed size 241.00 k 27  x   xInstalling libcroco-0_6-3-0.6.1-120.16.x86_64.rpm (installed size 241.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  79%  9  100  0.93 5 03  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-transfer (download size 60.00 kB)   x   x~ kB) x    xDownloading yast2-transfer (download size 60.00 kB) q   26  53%  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-transfer-2.16.3-0.5.133.x86_64.rpm (installed size 322.00 03  x   xInstalling yast2-transfer-2.16.3-0.5.133.x86_64.rpm (installed size 322.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  29  3  ±±±44  49 58%  63%  9  76  81  7  92  8  100  62 4 0.72  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atk (download size 47.00 kB)   x   x~00 kB) x    xDownloading atk (download size 47.00 kB) q   33  67%  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atk-1.28.0-1.4.1.x86_64.rpm (installed size 152.00 kB) 0.72  x   xInstalling atk-1.28.0-1.4.1.x86_64.rpm (installed size 152.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  58%  85%  98  100  47 3 57  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgsf-1-114 (download size 89.00 kB)   x   xDownloading libgsf-1-114 (download size 89.00 kB) q   17  35  54%  72%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgsf-1-114-1.14.9-2.27.x86_64.rpm (installed size 243.00 kB) 57  x   xInstalling libgsf-1-114-1.14.9-2.27.x86_64.rpm (installed size 243.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 79%  9  100  23 2 33  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-python-bindings (download size 93.00 kB)   x   x~) x    xDownloading yast2-python-bindings (download size 93.00 kB) q   17  34  51 68%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-python-bindings-2.17.4-1.32.115.x86_64.rpm (installed siz 33  x   xInstalling yast2-python-bindings-2.17.4-1.32.115.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  3  63%  82  93  9  100  19.91 1 01  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-perl-bindings (download size 99.00 kB)   x   x~ize 326.00 kB) x    xDownloading yast2-perl-bindings (download size 99.00 kB) q   16  32  48 64%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-perl-bindings-2.17.2-1.32.121.x86_64.rpm (installed size 01  x   xInstalling yast2-perl-bindings-2.17.2-1.32.121.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  35  53%  71%  6  82  7  93  8  100  58 0 69.68  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ipoibtools (download size 19.00 kB)   x   x~e 344.00 kB) x    xDownloading ipoibtools (download size 19.00 kB) q   84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipoibtools-1.1-80.22.x86_64.rpm (installed size 55.00 kB) 69.68  x   xInstalling ipoibtools-1.1-80.22.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  59%  92  8  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  2 49 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXp (download size 17.00 kB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading xorg-x11-libXp (download size 17.00 kB)  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXp-7.4-1.18.7.x86_64.rpm (installed size 39.00 kB) 2  x   xInstalling xorg-x11-libXp-7.4-1.18.7.x86_64.rpm (installed size 39.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  49 8 58  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXmu (download size 48.00 kB)   x   x~) x    xDownloading xorg-x11-libXmu (download size 48.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXmu-7.4-1.17.x86_64.rpm (installed size 119.00 kB) 58  x   xInstalling xorg-x11-libXmu-7.4-1.17.x86_64.rpm (installed size 119.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  87%  99  100  37 7 47  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading DirectFB (download size 1.05 MB)   x   x~) x    xDownloading DirectFB (download size 1.05 MB) q   6  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  70  5  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing DirectFB-1.2.3-6.32.x86_64.rpm (installed size 3.58 MB) 47  x   xInstalling DirectFB-1.2.3-6.32.x86_64.rpm (installed size 3.58 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  7  72  7  83  8  93  8  100  5.78 6 5.88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++-devel (download size 5.00 kB)   x   xDownloading libstdc++-devel (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++-devel-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB 5.885  x   xInstalling libstdc++-devel-4.3-62.200.2.x86_64.rpm (installed size 1.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  75%  100  5  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc43-c++ (download size 3.22 MB)   x   x~kB) x    xDownloading gcc43-c++ (download size 3.22 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc43-c++-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 8.6  x   xInstalling gcc43-c++-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  07.1 4 57.2  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc (download size 6.00 kB)   x   x~.61 MB) x    xDownloading gcc (download size 6.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) 57.23  x   xInstalling gcc-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  26  40  55%  71%  87  100  3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cups-client (download size 118.00 kB)   x   xDownloading cups-client (download size 118.00 kB) q   13  26  40  54%  67%  81  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cups-client-1.3.9-8.46.56.1.x86_64.rpm (installed size 230.00 k  x   xInstalling cups-client-1.3.9-8.46.56.1.x86_64.rpm (installed size 230.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  20  6  33  9  ±±45  50 63%  8  79  84  93  8  100  6.95 2 05  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gfxboot-branding-SLES (download size 1.84 MB)   x   x~ kB) x    xDownloading gfxboot-branding-SLES (download size 1.84 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gfxboot-branding-SLES-4.1.34-0.5.44.x86_64.rpm (installed size 05  x   xInstalling gfxboot-branding-SLES-4.1.34-0.5.44.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  48 53%  9%  64  70  5  80  5  90  5  100  3.52 1 3.62  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libatk-1_0-0 (download size 41.00 kB)   x   x~e 3.43 MB) x    xDownloading libatk-1_0-0 (download size 41.00 kB) q   39  78%  100  5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libatk-1_0-0-1.28.0-1.4.1.x86_64.rpm (installed size 128.00 kB) 3.622  x   xInstalling libatk-1_0-0-1.28.0-1.4.1.x86_64.rpm (installed size 128.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 99%  100  40 0 50  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXprintUtil (download size 16.00 kB)   x   x~B) x    xDownloading xorg-x11-libXprintUtil (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXprintUtil-7.4-1.17.x86_64.rpm (installed size 35.0 50  x   xInstalling xorg-x11-libXprintUtil-7.4-1.17.x86_64.rpm (installed size 35q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  37 39 46  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc-c++ (download size 5.00 kB)   x   x~.00 kB) x    xDownloading gcc-c++ (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc-c++-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) 46  x   xInstalling gcc-c++-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  3  60%  82  100  8  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atk-lang (download size 136.00 kB)   x   xDownloading atk-lang (download size 136.00 kB) q   11  23  35  ±46  58%  70%  82  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atk-lang-1.28.0-1.4.1.x86_64.rpm (installed size 766.00 kB)  x   xInstalling atk-lang-1.28.0-1.4.1.x86_64.rpm (installed size 766.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 8%  63%  8  73  8  83  8  94  9  100  2.62 7 2.72  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 352.72 MBx 202x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e8820372xxx tu Downloading libibverbs-devel (download size 67.00 kB)   x   xDownloading libibverbs-devel (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibverbs-devel-1.1.4-0.16.37.x86_64.rpm (installed size 122.0 2.720  x   xInstalling libibverbs-devel-1.1.4-0.16.37.x86_64.rpm (installed size 122q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  23  30  63%  8  73  9  84  9  94  9  100  60  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi-devel (download size 4.78 MB)   x   x~.00 kB) x    xDownloading openmpi-devel (download size 4.78 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-devel-1.8.4-1.26.x86_64.rpm (installed size 11.69 MB) 6022  x   xInstalling openmpi-devel-1.8.4-1.26.x86_64.rpm (installed size 11.69 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  40.91  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 340.91 MBx 200x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e8820372xxx tu Downloading zlib-devel (download size 184.00 kB)   x   xDownloading zlib-devel (download size 184.00 kB) q   9  18  26  35  ±±±44  52% 61%  70  8  87  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zlib-devel-1.2.7-0.14.1.x86_64.rpm (installed size 656.00 kB) 40.9118  x   xInstalling zlib-devel-1.2.7-0.14.1.x86_64.rpm (installed size 656.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  3  ±±4  55%  64%  70  8  5  9  100  27 199 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenssl-devel (download size 2.87 MB)   x   xDownloading libopenssl-devel (download size 2.87 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenssl-devel-0.9.8j-0.105.1.x86_64.rpm (installed size 20.3 272  x   xInstalling libopenssl-devel-0.9.8j-0.105.1.x86_64.rpm (installed size 20q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  19.95  8 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 319.95 MBx 198x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e8820372xxx tu Downloading aaa_base (download size 173.00 kB)   x   x~.32 MB) x    xDownloading aaa_base (download size 173.00 kB) q   9  18  27  3  ±46  55%  64%  7  83  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing aaa_base-11-6.115.1.x86_64.rpm (installed size 355.00 kB) 19.9507  x   xInstalling aaa_base-11-6.115.1.x86_64.rpm (installed size 355.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  23  8  35  ±±±±43  50 6%  61%  6  72  7  83  90  5  100  Updating etc/sysconfig/shutdown... Updating etc/sysconfig/readonlyroot... Updating etc/passwd...unchanged Updating etc/group...unchanged Updating etc/shadow...unchanged  61  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading w3m (download size 942.00 kB)  group...unchanged hadow...unchanged   Downloading w3m (download size 942.00 kB)  6  12  7  22  7  32  7  ±±±±42  48 - 420.14 kB/s (download size 942.00 kB) 53%  8%  63%  8  73  8  83  8  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing w3m-0.5.3.git20161120-4.1.x86_64.rpm (installed size 2.17 MB) 61  x   xInstalling w3m-0.5.3.git20161120-4.1.x86_64.rpm (installed size 2.17 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  9  34  40  ±46  52% 7%  63%  8  73  80  5  90  5  100  7.43  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXpm (download size 46.00 kB)   x   xDownloading xorg-x11-libXpm (download size 46.00 kB) q   34  69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXpm-7.4-3.1.x86_64.rpm (installed size 118.00 kB) 7.436  x   xInstalling xorg-x11-libXpm-7.4-3.1.x86_64.rpm (installed size 118.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  39  93%  9  100  32  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXv (download size 12.00 kB)   x   xDownloading xorg-x11-libXv (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXv-7.4-1.20.1.x86_64.rpm (installed size 23.00 kB) 32  x   xInstalling xorg-x11-libXv-7.4-1.20.1.x86_64.rpm (installed size 23.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  29  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libs (download size 709.00 kB)   x   x~) x    xDownloading xorg-x11-libs (download size 709.00 kB) q   6  11  8  24  9  36  ±±±±42  ±47  54%  60%  5  72  8  83  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libs-7.4-8.26.49.1.x86_64.rpm (installed size 2.76 MB) 29  x   xInstalling xorg-x11-libs-7.4-8.26.49.1.x86_64.rpm (installed size 2.76 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  34  40  ±±45  52% 7%  62%  9  74  9  84  91  6  100  4.53  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 314.53 MBx 193x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e8820372xxx tu Downloading permissions (download size 25.00 kB)   x   x~B) x    xDownloading permissions (download size 25.00 kB) q   63%  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 kB 4.53  x   xInstalling permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  30  53%  76%  97  100  permissions (download size 25.00 kB) permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 kB) Additional rpm output: Updating etc/sysconfig/security...  5 6 44  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tk (download size 1.52 MB)  Additional rpm output: Updating etc/sysconfig/security...   Downloading tk (download size 1.52 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tk-8.5.5-3.14.1.x86_64.rpm (installed size 3.96 MB) 44  x   xInstalling tk-8.5.5-3.14.1.x86_64.rpm (installed size 3.96 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  8  33  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100   98.56 5 0.48  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cairo (download size 458.00 kB)   x   xDownloading cairo (download size 458.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cairo-1.8.8-2.1.48.x86_64.rpm (installed size 1.39 MB) 0.48  x   xInstalling cairo-1.8.8-2.1.48.x86_64.rpm (installed size 1.39 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  22  31  6  ±±±44  53%  62%  71  80  8  97  100  7.17 4 09.0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wodim (download size 388.00 kB)   x   xDownloading wodim (download size 388.00 kB) q   8  16  24  32  7  ±±45  53%  61%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wodim-1.1.8-3.32.3.x86_64.rpm (installed size 1.20 MB) 09.05  x   xInstalling wodim-1.1.8-3.32.3.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  ±47  53%  61%  6  74  80  5  90  5  100  5.9 3 7.89  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vlock (download size 20.00 kB)   x   xDownloading vlock (download size 20.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vlock-1.4-52.16.x86_64.rpm (installed size 41.00 kB) 7.89  x   xInstalling vlock-1.4-52.16.x86_64.rpm (installed size 41.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  79%  95  100  3 2 5  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading util-linux-lang (download size 905.00 kB)   x   xDownloading util-linux-lang (download size 905.00 kB) q   5  11  6  22  7  32  7  ±±±±43  48 53%  9%  64  9  75  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing util-linux-lang-2.19.1-6.72.1.x86_64.rpm (installed size 4.29 M 5  x   xInstalling util-linux-lang-2.19.1-6.72.1.x86_64.rpm (installed size 4.29q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  32  7  ±±±±43  48 53%  8%  64%  70  5  80  5  90  6  100  1.64 1 3.5  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpm (download size 1.60 MB)   x   x~ MB) x    xDownloading rpm (download size 1.60 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpm-4.4.2.3-37.63.64.1.x86_64.rpm (installed size 4.27 MB) 3.54  x   xInstalling rpm-4.4.2.3-37.63.64.1.x86_64.rpm (installed size 4.27 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  32  7  ±±±±42  ±47  53%  9%  64  9  74  9  84  9  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/services... q  87.36 0 299.28  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam-modules (download size 109.00 kB)  Additional rpm output: Updating etc/sysconfig/services...   Downloading pam-modules (download size 109.00 kB)  14  29  ±±±±43  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-modules-11-1.25.26.x86_64.rpm (installed size 457.00 kB) 299.283  x   xInstalling pam-modules-11-1.25.26.x86_64.rpm (installed size 457.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  20  5  30  5  ±±±±±41  ±46  51 6%  62%  7  72  7  82  7  92  7  100  6.92 29 8.83  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opie (download size 148.00 kB)   x   xDownloading opie (download size 148.00 kB) q   1  21  32  ±±±±43  54%  65%  7  86  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opie-2.4-662.18.1.x86_64.rpm (installed size 661.00 kB) 8.83  x   xInstalling opie-2.4-662.18.1.x86_64.rpm (installed size 661.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  9  26  33  40  ±46  57%  6  76  8  92  7  100  27 8 19  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading eject (download size 38.00 kB)   x   xDownloading eject (download size 38.00 kB) q   ±±±±±41  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing eject-2.1.0-115.19.2.1.x86_64.rpm (installed size 103.00 kB) 19  x   xInstalling eject-2.1.0-115.19.2.1.x86_64.rpm (installed size 103.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  8  ±±45  53%  60%  71  9  87  95  100  1 7 0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ibutils (download size 1.27 MB)   x   xDownloading ibutils (download size 1.27 MB) q   5  10  5  21 - 219.36 kB/s (download size 1.27 MB)  6  32  7  ±±±±42  48 53%  8%  63%  9  74  9  85  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ibutils-1.5.7-0.15.22.x86_64.rpm (installed size 10.49 MB) 0  x   xInstalling ibutils-1.5.7-0.15.22.x86_64.rpm (installed size 10.49 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  75.68 6 87.60  2 8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cdrkit-cdrtools-compat (download size 6.00 kB)   x   xDownloading cdrkit-cdrtools-compat (download size 6.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cdrkit-cdrtools-compat-1.1.8-3.32.3.x86_64.rpm (installed size 87.601  x   xInstalling cdrkit-cdrtools-compat-1.1.8-3.32.3.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±42  54%  66%  78  90  10  5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading util-linux (download size 761.00 kB)   x   x~e 0.00 B) x    xDownloading util-linux (download size 761.00 kB) q   5  10  5  21  6  32  8  ±±45  51 7%  63%  8  74  80  5  91  7  100  9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing util-linux-2.19.1-6.72.1.x86_64.rpm (installed size 2.16 MB)  x   xInstalling util-linux-2.19.1-6.72.1.x86_64.rpm (installed size 2.16 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  8  23  30  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  3.53 4 5.44  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading satsolver-tools (download size 351.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading satsolver-tools (download size 351.00 kB)  9  18  27  36  ±±±±±41  50 9%  68  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing satsolver-tools-0.17.9-0.5.2.x86_64.rpm (installed size 3.25 MB 5.440  x   xInstalling satsolver-tools-0.17.9-0.5.2.x86_64.rpm (installed size 3.25 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  23  8  35  40  ±46  52% 8%  64%  70  7  82  7  92  7  100  0.27 3 2.19  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpm-python (download size 43.00 kB)   x   x~MB) x    xDownloading rpm-python (download size 43.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpm-python-4.4.2.3-37.63.64.2.x86_64.rpm (installed size 137.00 2.19  x   xInstalling rpm-python-4.4.2.3-37.63.64.2.x86_64.rpm (installed size 137.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  48 94%  9  100  14 2 06  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-satsolver (download size 251.00 kB)   x   x~00 kB) x    xDownloading python-satsolver (download size 251.00 kB) q   6  12  9  25  31  8  ±±±44  51 7%  63%  70  6  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 749. 060:59  x   xInstalling python-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 74q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  34  ±±±±42  51 9%  68  76  85  93  9  100  69.41 1 1.33  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-satsolver (download size 303.00 kB)   x   x~9.00 kB) x    xDownloading perl-satsolver (download size 303.00 kB) q   5  10  5  21  6  31  6  ±±±±42  ±47  52% 8%  63%  8  74  9  84  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 944.00 1.33  x   xInstalling perl-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 944.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  6  23  30  7  ±±±44  50 7%  64%  71  7  84  91  8  100  8.49 0 0.40  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading module-init-tools (download size 146.00 kB)   x   x~00 kB) x    xDownloading module-init-tools (download size 146.00 kB) q   1  21  32  ±±±±43  54%  65%  76  87  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing module-init-tools-3.11.1-31.25.x86_64.rpm (installed size 488.0 0.40  x   xInstalling module-init-tools-3.11.1-31.25.x86_64.rpm (installed size 488q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  29  ±46  53%  66%  77  83  9  96  100  01 19 79.93  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pwdutils (download size 352.00 kB)   x   x~.00 kB) x    xDownloading pwdutils (download size 352.00 kB) q   9  18  27  36  ±±45  54%  63%  8  77  86  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pwdutils-3.2.15-0.17.28.x86_64.rpm (installed size 1.95 MB) 79.93  x   xInstalling pwdutils-3.2.15-0.17.28.x86_64.rpm (installed size 1.95 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  23  9  34  ±±±±±41  48 53%  8%  63%  8  73  8  84  90  6  100  6.07 8 7.98  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam-config (download size 108.00 kB)   x   xDownloading pam-config (download size 108.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-config-0.79-2.8.8.x86_64.rpm (installed size 499.00 kB) 7.98  x   xInstalling pam-config-0.79-2.8.8.x86_64.rpm (installed size 499.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  34  40  ±±45  51 6%  61%  6  71  7  83  8  95  100  **** write config for pam_ssh.so (session, disabled) **** write config for pam_selinux.so (session, disabled) **** write config for pam_gnome_keyring.so (session, disabled) **** write config for pam_exec.so (session, disabled) **** write config for pam_env.so (session, disabled)  5.58 7 49  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zisofs-tools (download size 25.00 kB)  exec.so (session, disabled)env.so (session, disabled)   Downloading zisofs-tools (download size 25.00 kB)   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zisofs-tools-1.0.6-138.22.x86_64.rpm (installed size 53.00 kB) 49  x   xInstalling zisofs-tools-1.0.6-138.22.x86_64.rpm (installed size 53.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  78%  92  9  100  3 6 4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam_mount (download size 102.00 kB)   x   x~) x    xDownloading pam_mount (download size 102.00 kB) q   13  29  ±±45  60%  76  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam_mount-0.47-13.16.1.x86_64.rpm (installed size 280.00 kB) 48  x   xInstalling pam_mount-0.47-13.16.1.x86_64.rpm (installed size 280.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  31  8  4 54%  66%  7  83  90  5  100  25 5 17  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading haveged (download size 45.00 kB)   x   xDownloading haveged (download size 45.00 kB) q   31  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing haveged-1.3-0.6.1.x86_64.rpm (installed size 153.00 kB) 17  x   xInstalling haveged-1.3-0.6.1.x86_64.rpm (installed size 153.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  85%  98  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  10 4 02  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading grub (download size 349.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading grub (download size 349.00 kB)  9  18  27  32  ±±±±±41  50 5%  64%  73  82  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing grub-0.97-162.172.1.x86_64.rpm (installed size 1.38 MB) 02  x   xInstalling grub-0.97-162.172.1.x86_64.rpm (installed size 1.38 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  21  9  34  ±±±±±41  49 56%  62%  7  73  8  87  92  8  100  3.72 3 5.64  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading scout (download size 48.00 kB)   x   xDownloading scout (download size 48.00 kB) q   33  66%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing scout-0.1.0-29.9.29.x86_64.rpm (installed size 133.00 kB) 5.64  x   xInstalling scout-0.1.0-29.9.29.x86_64.rpm (installed size 133.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  8%  63%  9  74  80  8  93  8  100  59 2 51  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-sam (download size 63.00 kB)   x   xDownloading suse-sam (download size 63.00 kB) q   25  50 76%  100  80  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-sam-0.8.5-0.4.9.noarch.rpm (installed size 174.00 kB) 51  x   xInstalling suse-sam-0.8.5-0.4.9.noarch.rpm (installed size 174.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  73%  86  9  100  42 1 34  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mpt-firmware (download size 5.00 kB)   x   xDownloading mpt-firmware (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mpt-firmware-1.0-236.22.noarch.rpm (installed size 1.00 kB) 34  x   xInstalling mpt-firmware-1.0-236.22.noarch.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  36  82%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ipw-firmware (download size 344.00 kB)   x   xDownloading ipw-firmware (download size 344.00 kB) q   9  18  23  32  7  ±46  51 60%  5  74  9  88  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipw-firmware-9-1.6.4.noarch.rpm (installed size 2.55 MB)  x   xInstalling ipw-firmware-9-1.6.4.noarch.rpm (installed size 2.55 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  24  31  8  ±±45  51 6%  61%  6  71  8  83  8  95  100  0.87 09 2.79  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootcycle (download size 8.00 kB)   x   xDownloading bootcycle (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootcycle-0.3-225.18.x86_64.rpm (installed size 5.00 kB) 2.797  x   xInstalling bootcycle-0.3-225.18.x86_64.rpm (installed size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  84%  97  100  Updating etc/sysconfig/bootloader... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  8 8  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading command-not-found (download size 12.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading command-not-found (download size 12.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing command-not-found-0.1.0-29.9.29.x86_64.rpm (installed size 13.0 8  x   xInstalling command-not-found-0.1.0-29.9.29.x86_64.rpm (installed size 13q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  32  ±±±±4  56%  69%  82  99  100  5 7 7  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 272.77 MBx 163x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 60.85 MBx 107x http-10.9.0.10-e8820372xxx tu Downloading nfs-client (download size 198.00 kB)   x   x~.00 kB) x    xDownloading nfs-client (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-client-1.2.3-18.47.1.x86_64.rpm (installed size 540.00 kB) 78  x   xInstalling nfs-client-1.2.3-18.47.1.x86_64.rpm (installed size 540.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  22  9  3  51 61%  72  8  91  6  100  Updating etc/sysconfig/nfs... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  24  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pango (download size 397.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading pango (download size 397.00 kB)  8  15  23  31  9  ±47  55%  63%  71  9  87  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pango-1.26.2-1.7.1.x86_64.rpm (installed size 1.48 MB) 24  x   xInstalling pango-1.26.2-1.7.1.x86_64.rpm (installed size 1.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  8  35  ±±±±42  ±47  53%  9%  64  9  74  82  90  8  100  0.76  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading deltarpm (download size 432.00 kB)   x   xDownloading deltarpm (download size 432.00 kB) q   7  14  22  9  36  ±±±44  51 9%  66  74  81  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing deltarpm-3.6.1-4.11.1.x86_64.rpm (installed size 1.48 MB) 0.76  x   xInstalling deltarpm-3.6.1-4.11.1.x86_64.rpm (installed size 1.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  8  26  34  ±±±±42  49 56%  65%  71  9  86  94  9  100  69.28  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsnmp15 (download size 1.12 MB)   x   xDownloading libsnmp15 (download size 1.12 MB) q   5  10  6  21  7  32  7  ±±±±42  48 53%  8%  63%  9  74  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsnmp15-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 3.45 MB) 69.287  x   xInstalling libsnmp15-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 3.45 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  32  7  ±±±±43  48 53%  9%  64  70  5  81  6  91  7  100  5.83  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading udev (download size 236.00 kB)   x   x~B) x    xDownloading udev (download size 236.00 kB) q   6  13  20  7  33  40  ±47  54%  61%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing udev-147-0.110.1.x86_64.rpm (installed size 943.00 kB) 5.836  x   xInstalling udev-147-0.110.1.x86_64.rpm (installed size 943.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  9  24  9  34  9  ±±±44  49 58%  63%  72  9  86  93  8  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.91  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcrypt11 (download size 339.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading libgcrypt11 (download size 339.00 kB)  8  13  8  23  32  7  ±46  51 6%  65%  70  5  84  9  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcrypt11-1.5.0-0.22.1.x86_64.rpm (installed size 719.00 kB) 4.91  x   xInstalling libgcrypt11-1.5.0-0.22.1.x86_64.rpm (installed size 719.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  9  28  37  ±46  55%  64%  71  8  83  8  9  100  2  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfs-kernel-server (download size 117.00 kB)   x   x~) x    xDownloading nfs-kernel-server (download size 117.00 kB) q   13  27  ±±±±±41  54%  68%  82  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-kernel-server-1.2.3-18.47.1.x86_64.rpm (installed size 248. 2  x   xInstalling nfs-kernel-server-1.2.3-18.47.1.x86_64.rpm (installed size 24q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  53%  71%  8  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  3.96  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gtk2-lang (download size 3.16 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading gtk2-lang (download size 3.16 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gtk2-lang-2.18.9-0.44.1.x86_64.rpm (installed size 19.71 MB) 3.96  x   xInstalling gtk2-lang-2.18.9-0.44.1.x86_64.rpm (installed size 19.71 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  44.25  5 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ksh (download size 1.13 MB)   x   xDownloading ksh (download size 1.13 MB) q   5  10  6  21  7  32  7 [19;31H ±±±±42  48 54%  9%  64  9  75  80  6  91  7  100  2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ksh-93u-0.37.1.x86_64.rpm (installed size 2.57 MB) 44.252  x   xInstalling ksh-93u-0.37.1.x86_64.rpm (installed size 2.57 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  23  8  33  40  ±±45  50 6%  61%  6  71  8  83  9  94  9  100  1.69  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libudev0 (download size 50.00 kB)   x   xDownloading libudev0 (download size 50.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libudev0-147-0.110.1.x86_64.rpm (installed size 55.00 kB) 1.691  x   xInstalling libudev0-147-0.110.1.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autofs (download size 368.00 kB)   x   xDownloading autofs (download size 368.00 kB) q   8  17  26  34  9  ±47  52% 60%  5  73  8  87  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autofs-5.0.6-3.10.33.1.x86_64.rpm (installed size 1.65 MB) 3  x   xInstalling autofs-5.0.6-3.10.33.1.x86_64.rpm (installed size 1.65 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  20  6  33  9  ±±45  51 6%  63%  8  76  81  8  95  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/autofs... q  39.98  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1 (download size 248.00 kB)  Additional rpm output: Updating etc/sysconfig/autofs...   Downloading dbus-1 (download size 248.00 kB)  6  12  9  25  32  8  ±±45  51 7%  64%  70  7  83  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-1.2.10-3.33.1.x86_64.rpm (installed size 827.00 kB) 39.98  x   xInstalling dbus-1-1.2.10-3.33.1.x86_64.rpm (installed size 827.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  8  27  36  ±±±44  50 8%  66%  75  84  91  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading at (download size 53.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading at (download size 53.00 kB)  29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing at-3.1.8-1069.22.30.1.x86_64.rpm (installed size 109.00 kB) 1  x   xInstalling at-3.1.8-1069.22.30.1.x86_64.rpm (installed size 109.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  70%  87  93  8  100  Updating etc/sysconfig/atd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  07  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssh (download size 1.09 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading openssh (download size 1.09 MB)  5  10  6  21  6  31  6  ±±±±±41  ±46  52% 7%  62%  7  72  7  82  7  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssh-6.6p1-35.1.x86_64.rpm (installed size 5.01 MB) 07  x   xInstalling openssh-6.6p1-35.1.x86_64.rpm (installed size 5.01 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  7  ±±±±43  48 53%  9%  64  9  74  9  84  9  94  9  100  Updating etc/sysconfig/ssh... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.06  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gtk2 (download size 2.23 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading gtk2 (download size 2.23 MB)  5  10  5  20  5  30  5  40  ±±45  50 - 1.07 MB/s (download size 2.23 MB) 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gtk2-2.18.9-0.44.1.x86_64.rpm (installed size 7.00 MB) 4.060  x   xInstalling gtk2-2.18.9-0.44.1.x86_64.rpm (installed size 7.00 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  27  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading device-mapper (download size 323.00 kB)   x   xDownloading device-mapper (download size 323.00 kB) q   9  14  9  24 - 79.74 kB/s (download size 323.00 kB)  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing device-mapper-1.02.77-0.15.14.x86_64.rpm (installed size 1.65 M 2748  x   xInstalling device-mapper-1.02.77-0.15.14.x86_64.rpm (installed size 1.65q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  20  8  36  ±±±44  51 9%  66  74  81  9  96  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  5.41  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading quota (download size 233.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading quota (download size 233.00 kB)  6  13  20  7  34  ±±±±±41  48 54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing quota-3.16-51.1.0.x86_64.rpm (installed size 1.15 MB) 5.41  x   xInstalling quota-3.16-51.1.0.x86_64.rpm (installed size 1.15 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  15  21  7  33  9  ±±45  52% 8%  65%  7  80  5  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.26  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librsvg (download size 149.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading librsvg (download size 149.00 kB)  1  21  32  ±±±±4  53%  64%  75  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librsvg-2.26.0-2.5.1.x86_64.rpm (installed size 420.00 kB) 4.267  x   xInstalling librsvg-2.26.0-2.5.1.x86_64.rpm (installed size 420.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  27  ±±±±43  58%  65%  84  98  100  3.85  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lvm2 (download size 936.00 kB)   x   xDownloading lvm2 (download size 936.00 kB) q   5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lvm2-2.02.98-0.42.3.x86_64.rpm (installed size 2.97 MB) 3.85  x   xInstalling lvm2-2.02.98-0.42.3.x86_64.rpm (installed size 2.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  8  34  40  ±46  51 6%  62%  9  75  81  8  93  8  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  0.88  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kpartx (download size 53.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading kpartx (download size 53.00 kB)  29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kpartx-0.4.9-115.1.x86_64.rpm (installed size 46.00 kB) 0.88  x   xInstalling kpartx-0.4.9-115.1.x86_64.rpm (installed size 46.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  97%  100  3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading parted (download size 589.00 kB)   x   xDownloading parted (download size 589.00 kB) q   5  10  5  21  6  32  7  ±±±±43  48 53%  9%  64  70  5  81  6  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing parted-2.3-10.51.1.x86_64.rpm (installed size 2.20 MB) 3  x   xInstalling parted-2.3-10.51.1.x86_64.rpm (installed size 2.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  24  30  6  ±±±±±41  ±46  53%  8%  65%  70  5  82  8  93  9  100  18.6  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libMagickCore1 (download size 1.26 MB)   x   xDownloading libMagickCore1 (download size 1.26 MB) q   6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libMagickCore1-6.4.3.6-7.70.1.x86_64.rpm (installed size 5.33 M 18.66  x   xInstalling libMagickCore1-6.4.3.6-7.70.1.x86_64.rpm (installed size 5.33q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  62%  7  72  7  82  8  94  9  100  3.30  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading multipath-tools (download size 233.00 kB)   x   x~ MB) x    xDownloading multipath-tools (download size 233.00 kB) q   6  13  20  7  34  ±±±±±41  ±47  54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing multipath-tools-0.4.9-115.1.x86_64.rpm (installed size 690.00 k 3.305  x   xInstalling multipath-tools-0.4.9-115.1.x86_64.rpm (installed size 690.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  25  34  ±±±±43  48 53%  8%  63%  8  75  87  92  7  100  2.63  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 212.63 MBx 140x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 60.85 MBx 107x http-10.9.0.10-e8820372xxx tu Downloading OpenIPMI (download size 874.00 kB)   x   x~ kB) x    xDownloading OpenIPMI (download size 874.00 kB) q   5  10  5  21  6  32  7  ±±±±43  48 54%  9%  65  70  6  81  7  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing OpenIPMI-2.0.16-0.15.1.x86_64.rpm (installed size 3.32 MB) 2.634  x   xInstalling OpenIPMI-2.0.16-0.15.1.x86_64.rpm (installed size 3.32 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  8  33  8  ±±±±43  49 54%  60%  6  72  7  83  8  93  8  100  Updating etc/sysconfig/ipmi... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  57.53 6 09.31  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading uuid-runtime (download size 41.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading uuid-runtime (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing uuid-runtime-2.19.1-6.72.1.x86_64.rpm (installed size 22.00 kB) 09.313  x   xInstalling uuid-runtime-2.19.1-6.72.1.x86_64.rpm (installed size 22.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  93%  8  100  1 5 29  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading puppet (download size 1.16 MB)   x   x~B) x    xDownloading puppet (download size 1.16 MB) q   5  10  5  20  6  31  7  ±±±±42  ±47  53%  8%  64%  9  74  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing puppet-2.7.26-0.5.1.x86_64.rpm (installed size 3.40 MB) 29  x   xInstalling puppet-2.7.26-0.5.1.x86_64.rpm (installed size 3.40 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/puppet... q  4.1 4 5.8  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netcfg (download size 246.00 kB)  Additional rpm output: Updating etc/sysconfig/puppet...   Downloading netcfg (download size 246.00 kB)  6  12  9  25  32  9  ±±45  52% 8%  65%  71  8  84  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netcfg-11.2-0.8.1.noarch.rpm (installed size 1.24 MB) 5.8  x   xInstalling netcfg-11.2-0.8.1.noarch.rpm (installed size 1.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  7  32  7  ±±±±43  48 53%  8%  63%  8  73  8  83  8  93  8  100  2.87 3 4.65  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxslt (download size 149.00 kB)   x   xDownloading libxslt (download size 149.00 kB) q   1  21  32  ±±±±43  53%  64%  75  86  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxslt-1.1.24-19.23.1.x86_64.rpm (installed size 386.00 kB) 4.652  x   xInstalling libxslt-1.1.24-19.23.1.x86_64.rpm (installed size 386.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  22  ±±±±4  59%  75  87  9  100  50 2 27  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dirmngr (download size 219.00 kB)   x   xDownloading dirmngr (download size 219.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  7  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dirmngr-1.0.2-1.19.x86_64.rpm (installed size 564.00 kB) 27  x   xInstalling dirmngr-1.0.2-1.19.x86_64.rpm (installed size 564.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±±±42  53%  65%  74  9  84  91  8  100  1.95 1 3.72  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dbus-client (download size 28.00 kB)   x   xDownloading yast2-dbus-client (download size 28.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dbus-client-2.17.0-0.1.65.x86_64.rpm (installed size 71.0 3.72  x   xInstalling yast2-dbus-client-2.17.0-0.1.65.x86_64.rpm (installed size 71q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  71%  96  100  88 0 65  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1-glib (download size 51.00 kB)   x   x~.00 kB) x    xDownloading dbus-1-glib (download size 51.00 kB) q   3  62%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-glib-0.76-34.22.1.x86_64.rpm (installed size 137.00 kB) 65  x   xInstalling dbus-1-glib-0.76-34.22.1.x86_64.rpm (installed size 137.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  93%  9  100  74  99 52  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcryptsetup1 (download size 48.00 kB)   x   x~) x    xDownloading libcryptsetup1 (download size 48.00 kB) q   32  66%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcryptsetup1-1.1.3-2.10.x86_64.rpm (installed size 88.00 kB) 52  x   xInstalling libcryptsetup1-1.1.3-2.10.x86_64.rpm (installed size 88.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  73%  99  100  66 8 43  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dmraid (download size 131.00 kB)   x   x~) x    xDownloading dmraid (download size 131.00 kB) q   12  24  36  48 60%  73  85  - 111.74 kB/s (download size 131.00 kB) 97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331.00 kB) 43  x   xInstalling dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  26  ±4  65%  70  8  6  93  8  100  dmraid (download size 131.00 kB) dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331 Additional rpm output: Updating etc/sysconfig/dmraid...  33 7 11  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libfprint0 (download size 85.00 kB)  Additional rpm output: Updating etc/sysconfig/dmraid...   Downloading libfprint0 (download size 85.00 kB)  18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libfprint0-0.0.6-18.22.136.x86_64.rpm (installed size 209.00 kB 11  x   xInstalling libfprint0-0.0.6-18.22.136.x86_64.rpm (installed size 209.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  61%  9  9  100  1 6 2.9  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading telnet (download size 54.00 kB)   x   x~kB) x    xDownloading telnet (download size 54.00 kB) q   29  59%  8  100  4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing telnet-1.2-134.24.1.x86_64.rpm (installed size 118.00 kB) 2.9  x   xInstalling telnet-1.2-134.24.1.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  85%  91  9  100  01 5 79  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icmpinfo (download size 14.00 kB)   x   xDownloading icmpinfo (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icmpinfo-1.11-691.22.x86_64.rpm (installed size 29.00 kB) 79  x   xInstalling icmpinfo-1.11-691.22.x86_64.rpm (installed size 29.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 7%  75%  80  95  100  0.99 4 6  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading finger (download size 19.00 kB)   x   xDownloading finger (download size 19.00 kB) q   83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing finger-1.3-104.22.x86_64.rpm (installed size 36.00 kB) 6  x   xInstalling finger-1.3-104.22.x86_64.rpm (installed size 36.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  84%  90  9  100  5 3 3  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ypbind (download size 38.00 kB)   x   xDownloading ypbind (download size 38.00 kB) q   ±±±±±41  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ypbind-1.22-1.17.x86_64.rpm (installed size 76.00 kB) 3  x   xInstalling ypbind-1.22-1.17.x86_64.rpm (installed size 76.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  68%  73  8  92  7  100  Updating etc/sysconfig/ypbind... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  88 2 65  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1-python (download size 126.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading dbus-1-python (download size 126.00 kB)  12  25  38  50 63%  76  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-python-0.83.0-27.1.43.x86_64.rpm (installed size 617.00 65  x   xInstalling dbus-1-python-0.83.0-27.1.43.x86_64.rpm (installed size 617.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  2  6  31  7  ±±±±42  49 56%  61%  9  76  83  91  9  100  27 1 0  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cryptsetup (download size 89.00 kB)   x   x~0 kB) x    xDownloading cryptsetup (download size 89.00 kB) q   17  36  54%  72%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cryptsetup-1.1.3-2.10.x86_64.rpm (installed size 250.00 kB) 0  x   xInstalling cryptsetup-1.1.3-2.10.x86_64.rpm (installed size 250.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  26  3  ±±±±43  51 8%  65%  72  80  7  95  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  03 0 1.81  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cryptconfig (download size 83.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading cryptconfig (download size 83.00 kB)  19  38  57%  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cryptconfig-0.3-68.8.17.x86_64.rpm (installed size 369.00 kB) 1.81  x   xInstalling cryptconfig-0.3-68.8.17.x86_64.rpm (installed size 369.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  29  34  ±±±±43  52% 60%  5  70  8  8  96  100  49.67 89 45  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 201.45 MBx 122x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 49.67 MBx89x http-10.9.0.10-e8820372xxx tu Downloading ipmitool (download size 351.00 kB)   x   xDownloading ipmitool (download size 351.00 kB) q   9  18  27  36  ±±±±±41  50 9%  68  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipmitool-1.8.15-0.29.2.x86_64.rpm (installed size 1.20 MB) 45  x   xInstalling ipmitool-1.8.15-0.29.2.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  6  31  6  ±±±±±41  ±47  52% 7%  6  72  7  82  7  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  0.2  1 5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsh (download size 40.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading rsh (download size 40.00 kB)  39  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsh-0.17-706.22.1.x86_64.rpm (installed size 91.00 kB) 0.20  x   xInstalling rsh-0.17-706.22.1.x86_64.rpm (installed size 91.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  ±±45  65%  81  7  93  9  100  16  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading postfix (download size 1017.00 kB)   x   xDownloading postfix (download size 1017.00 kB) q   6  11  6  21  7  32  8  ±±±±43  49 54%  60%  5  71  6  82  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing postfix-2.9.4-0.28.2.x86_64.rpm (installed size 2.71 MB) 16  x   xInstalling postfix-2.9.4-0.28.2.x86_64.rpm (installed size 2.71 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  6  ±±±±±41  ±46  51 8%  64%  71  6  81  6  91  8  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/mail...  197.45  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpg2-lang (download size 472.00 kB)  ~ script `munge' Updating etc/sysconfig/mail...   Downloading gpg2-lang (download size 472.00 kB)  6  13 - 47.74 kB/s (download size 472.00 kB)  20  7  33  40  ±47  54%  61%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpg2-lang-2.0.9-25.33.41.2.x86_64.rpm (installed size 2.31 MB) 197.45  x   xInstalling gpg2-lang-2.0.9-25.33.41.2.x86_64.rpm (installed size 2.31 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  22  8  33  8  ±±±±43  49 55%  60%  5  70  5  80  7  93  8  100  5.14  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ConsoleKit (download size 80.00 kB)   x   x~) x    xDownloading ConsoleKit (download size 80.00 kB) q   19  3  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ConsoleKit-0.2.10-64.69.1.x86_64.rpm (installed size 295.00 kB) 5.1439  x   xInstalling ConsoleKit-0.2.10-64.69.1.x86_64.rpm (installed size 295.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  32  8  ±±±44  65%  87  93  8  100  4.86  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpg2 (download size 1.16 MB)   x   x~B) x    xDownloading gpg2 (download size 1.16 MB) q   5  10  5  20  5  30  6  ±±±±±41  ±47  52% 7%  62%  7  72  8  83  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpg2-2.0.9-25.33.41.2.x86_64.rpm (installed size 3.77 MB) 4.86  x   xInstalling gpg2-2.0.9-25.33.41.2.x86_64.rpm (installed size 3.77 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  ±47  52% 7%  63%  9  75  80  5  90  5  100  1.09  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-build-key (download size 15.00 kB)   x   xDownloading suse-build-key (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-build-key-1.0-907.46.1.noarch.rpm (installed size 9.00 kB) 1.098  x   xInstalling suse-build-key-1.0-907.46.1.noarch.rpm (installed size 9.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  98%  100   imported gpg: key 50A3DD1C: public key "SuSE Package Signing Key (reserve key) " imported gpg: Total number processed: 4 gpg: imported: 4 (RSA: 3)  8  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading inst-source-utils (download size 44.00 kB)  gpg: Total number processed: 4 imported: 4 (RSA: 3)   Downloading inst-source-utils (download size 44.00 kB)  36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing inst-source-utils-2013.1.25-0.20.10.noarch.rpm (installed size 8  x   xInstalling inst-source-utils-2013.1.25-0.20.10.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  8  34  69%  76  83  8  99  100  0.94  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 190.94 MBx 114x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 49.67 MBx89x http-10.9.0.10-e8820372xxx tu Downloading cron (download size 71.00 kB)   x   x~e 139.00 kB) x    xDownloading cron (download size 71.00 kB) q   22  ±±±44  67%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cron-4.1-194.211.213.1.x86_64.rpm (installed size 152.00 kB) 0.9442  x   xInstalling cron-4.1-194.211.213.1.x86_64.rpm (installed size 152.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  30  6  67%  72  8  91  8  100  Updating etc/sysconfig/cron... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  52 8 79  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading PolicyKit (download size 108.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading PolicyKit (download size 108.00 kB)  14  29  ±±±44  59%  74  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing PolicyKit-0.9-14.43.1.x86_64.rpm (installed size 417.00 kB) 79  x   xInstalling PolicyKit-0.9-14.43.1.x86_64.rpm (installed size 417.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  8  27  33  ±±±±±41  ±46  52% 64%  7  89  9  100  11 7 3  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpgme (download size 109.00 kB)   x   xDownloading gpgme (download size 109.00 kB) q   12  27  ±±±±42  56%  71%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpgme-1.1.6-25.32.1.x86_64.rpm (installed size 344.00 kB) 3  x   xInstalling gpgme-1.1.6-25.32.1.x86_64.rpm (installed size 344.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  35  ±±4  67%  73  99  100  8.78 6 05  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading logrotate (download size 39.00 kB)   x   xDownloading logrotate (download size 39.00 kB) q   4  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing logrotate-3.7.7-10.30.1.x86_64.rpm (installed size 68.00 kB) 05  x   xInstalling logrotate-3.7.7-10.30.1.x86_64.rpm (installed size 68.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  83%  91  9  100  1 5 89.98  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dbus-server (download size 116.00 kB)   x   xDownloading yast2-dbus-server (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339.0 89.98  x   xInstalling yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  32  55%  73%  89  96  100  yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339 ~.00 kB) Additional rpm output: Failed to open connection to system message bus: Failed to connect to so ~cket /var/run/dbus/system_bus_socket: No such file or directory  38 4 65  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pm-utils (download size 43.00 kB)  Failed to open connection to system message bus: Failed to connect to so cket /var/run/dbus/system_bus_socket: No such file or directory   Downloading pm-utils (download size 43.00 kB)   36  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pm-utils-0.99.4.20071229-12.14.1.x86_64.rpm (installed size 100 651  x   xInstalling pm-utils-0.99.4.20071229-12.14.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  2  3  ±±±±42  ±47  55%  60%  6  85  90  5  100  2 3 56  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading PolicyKit-doc (download size 336.00 kB)   x   x~00.00 kB) x    xDownloading PolicyKit-doc (download size 336.00 kB) q   9  14  9  28  33  8  ±47  52% 7%  62%  71  6  81  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing PolicyKit-doc-0.9-14.43.1.x86_64.rpm (installed size 1.29 MB) 56  x   xInstalling PolicyKit-doc-0.9-14.43.1.x86_64.rpm (installed size 1.29 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  9  25  30  5  40  ±±45  50 5%  60%  6  72  7  83  8  94  9  100  6.99 2 8.27  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading polkit-default-privs (download size 12.00 kB)   x   xDownloading polkit-default-privs (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing polkit-default-privs-0.1_200812111413-3.6.noarch.rpm (installed 8.27  x   xInstalling polkit-default-privs-0.1_200812111413-3.6.noarch.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  67%  79  8  95  100  keep_always:auth_admin_keep_always:yes setting org.freedesktop.network-manager-settings.system.modify to auth_a ~dmin_keep_always:auth_admin_keep_always:auth_admin_keep_always setting org.freedesktop.hal.device-access.removable-block to auth_admin_ ~keep_always:auth_admin_keep_always:auth_admin_keep_always  7 1 4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgpgme11 (download size 105.00 kB)  setting org.freedesktop.hal.device-access.removable-block to auth_admin_ ~keep_always:auth_admin_keep_always:auth_admin_keep_always   Downloading libgpgme11 (download size 105.00 kB)   15  30  ±±45  61%  76  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgpgme11-1.1.6-25.32.1.x86_64.rpm (installed size 574.00 kB) 4  x   xInstalling libgpgme11-1.1.6-25.32.1.x86_64.rpm (installed size 574.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±±44  55%  66%  77  88  99  100  41 0 7.68  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xinetd (download size 125.00 kB)   x   x~) x    xDownloading xinetd (download size 125.00 kB) q   12  25  38  51 63%  76  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xinetd-2.3.14-130.133.1.x86_64.rpm (installed size 285.00 kB) 7.68  x   xInstalling xinetd-2.3.14-130.133.1.x86_64.rpm (installed size 285.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  36  59%  72  87  93  8  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  13 79 40  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-xml (download size 50.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading yast2-xml (download size 50.00 kB)  31  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-xml-2.16.1-1.23.x86_64.rpm (installed size 128.00 kB) 40  x   xInstalling yast2-xml-2.16.1-1.23.x86_64.rpm (installed size 128.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 79%  94  9  100  00 8 28  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hal (download size 464.00 kB)   x   xDownloading hal (download size 464.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  75  82  9  96  100  6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hal-0.5.12-23.76.1.x86_64.rpm (installed size 1.78 MB) 28  x   xInstalling hal-0.5.12-23.76.1.x86_64.rpm (installed size 1.78 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 7%  64%  71  8  83  8  93  8  100  Updating etc/sysconfig/hal... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.23 7 5.50  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mutt (download size 1.24 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading mutt (download size 1.24 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mutt-1.5.17-42.39.1.x86_64.rpm (installed size 4.33 MB) 5.50  x   xInstalling mutt-1.5.17-42.39.1.x86_64.rpm (installed size 4.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  33  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  39.90 6 1.17  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pcsc-lite (download size 158.00 kB)   x   xDownloading pcsc-lite (download size 158.00 kB) q   9  20  3  4  50 60%  7  8  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pcsc-lite-1.4.102-1.37.3.x86_64.rpm (installed size 520.00 kB) 1.170  x   xInstalling pcsc-lite-1.4.102-1.37.3.x86_64.rpm (installed size 520.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  20  7  40  52% 64%  76  81  9  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  39 5 0.66  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ivman (download size 62.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading ivman (download size 62.00 kB)  25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ivman-0.6.14-201.31.1.x86_64.rpm (installed size 150.00 kB) 0.66  x   xInstalling ivman-0.6.14-201.31.1.x86_64.rpm (installed size 150.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  58%  80%  9  6  100  24 4 51  99 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopensc2 (download size 325.00 kB)   x   xDownloading libopensc2 (download size 325.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  73  8  83  8  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopensc2-0.11.6-5.27.1.x86_64.rpm (installed size 838.00 kB) 51  x   xInstalling libopensc2-0.11.6-5.27.1.x86_64.rpm (installed size 838.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  30  8  ±±45  53%  61%  8  76  84  92  7  100  8.42 3 79.70  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opensc (download size 493.00 kB)   x   x~) x    xDownloading opensc (download size 493.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opensc-0.11.6-5.27.1.x86_64.rpm (installed size 1.83 MB) 79.7039  x   xInstalling opensc-0.11.6-5.27.1.x86_64.rpm (installed size 1.83 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  9  24  9  34  40  ±46  53%  60%  7  74  9  84  9  94  9  100  6.59 2 7.87  7 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 177.87 MBx 97x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 36.59 MBx72x http-10.9.0.10-e8820372xxx tu Downloading ntp (download size 763.00 kB)   x   xDownloading ntp (download size 763.00 kB) q   6  11  7  23  8  34  40  ±47  53%  9%  65  70  6  82  - 598.52 kB/s (download size 763.00 kB) 8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) 7.8742  x   xInstalling ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  9  26  33  8  ±±45  51 8%  65%  70  5  80  6  92  7  100  Installing ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) Additional rpm output: tr: write error: Broken pipe tr: write error Updating etc/sysconfig/ntp...  5.20  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mcelog (download size 91.00 kB)  tr: write error Updating etc/sysconfig/ntp...   Downloading mcelog (download size 91.00 kB)  17  35  52% 70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mcelog-1.0.2014.12.20-0.23.1.x86_64.rpm (installed size 217.00 5.2038  x   xInstalling mcelog-1.0.2014.12.20-0.23.1.x86_64.rpm (installed size 217.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  31  61%  89  94  9  100  mcelog... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.99  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysconfig (download size 350.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading sysconfig (download size 350.00 kB)  9  18  27  32  ±±±±±41  50 9%  64  73  82  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysconfig-0.71.61-0.15.1.x86_64.rpm (installed size 781.00 kB) 4.99  x   xInstalling sysconfig-0.71.61-0.15.1.x86_64.rpm (installed size 781.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  8  33  8  ±±±±43  48 55%  61%  6  71  6  82  90  5  100  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx  Updating etc/sysconfig/network/dhcp... Updating etc/sysconfig/network/config... Removing old autogenerated device configuration files:q 22  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libzypp (download size 2.14 MB)  Updating etc/sysconfig/network/config... Removing old autogenerated device configuration files:   Downloading libzypp (download size 2.14 MB)   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libzypp-9.40.2-14.7.x86_64.rpm (installed size 7.27 MB) 22  x   xInstalling libzypp-9.40.2-14.7.x86_64.rpm (installed size 7.27 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  66.96  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hwinfo (download size 609.00 kB)   x   xDownloading hwinfo (download size 609.00 kB) q   5  10  5  20  6  31  6  ±±±±42  ±47  52% 7%  63%  8  73  8  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hwinfo-15.57-0.12.2.x86_64.rpm (installed size 2.20 MB) 66.966  x   xInstalling hwinfo-15.57-0.12.2.x86_64.rpm (installed size 2.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  34  9  ±±45  51 6%  62%  8  73  9  85  90  5  100  4.7  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading SuSEfirewall2 (download size 67.00 kB)   x   xDownloading SuSEfirewall2 (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing SuSEfirewall2-3.6_SVNr208-2.11.1.noarch.rpm (installed size 234 4.7  x   xInstalling SuSEfirewall2-3.6_SVNr208-2.11.1.noarch.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  38  ±47  56%  63%  8  82  99  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  53  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dhcpcd (download size 51.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading dhcpcd (download size 51.00 kB)  31  63%  95  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dhcpcd-3.2.3-45.5.3.x86_64.rpm (installed size 94.00 kB) 53  x   xInstalling dhcpcd-3.2.3-45.5.3.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  80  94  9  100  44  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zypper (download size 1.02 MB)   x   xDownloading zypper (download size 1.02 MB) q   6  12  7  23  9  34  40  ±46 - 482.72 kB/s (download size 1.02 MB)  52% 7%  63%  9  75  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zypper-1.6.334-26.3.x86_64.rpm (installed size 4.86 MB) 44  x   xInstalling zypper-1.6.334-26.3.x86_64.rpm (installed size 4.86 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 6%  61%  6  71  6  81  6  91  6  100  59.57  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mkinitrd (download size 99.00 kB)   x   xDownloading mkinitrd (download size 99.00 kB) q   16  32  48 64%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mkinitrd-2.4.2-100.2.x86_64.rpm (installed size 177.00 kB) 59.575  x   xInstalling mkinitrd-2.4.2-100.2.x86_64.rpm (installed size 177.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  20  5  30  5  40  ±46  53%  9%  65  73  84  90  5  100  ~ script `munge' Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  40  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mdadm (download size 451.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading mdadm (download size 451.00 kB)  7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mdadm-3.3.1-9.1.x86_64.rpm (installed size 1.21 MB) 40  x   xInstalling mdadm-3.3.1-9.1.x86_64.rpm (installed size 1.21 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  32  7  ±±±±42  ±47  52% 7%  63%  8  74  9  84  91  6  100  Updating etc/sysconfig/mdadm... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  8.19  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kdump (download size 261.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading kdump (download size 261.00 kB)  6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  85  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kdump-0.8.4-0.55.1.x86_64.rpm (installed size 635.00 kB) 8.194  x   xInstalling kdump-0.8.4-0.55.1.x86_64.rpm (installed size 635.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  2  3  ±4  56%  66%  7  8  92  7  100  Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ... symlinks in /lib/mkinitrd/boot ...  7.57  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 157.57 MBx 86x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 36.59 MBx72x http-10.9.0.10-e8820372xxx tu Downloading klogd (download size 40.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading klogd (download size 40.00 kB)  4  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing klogd-1.4.1-708.44.32.x86_64.rpm (installed size 59.00 kB) 7.5742  x   xInstalling klogd-1.4.1-708.44.32.x86_64.rpm (installed size 59.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  68%  79  92  8  100  4 1 1  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-pkg-bindings (download size 530.00 kB)   x   xDownloading yast2-pkg-bindings (download size 530.00 kB) q   5  12  8  24  30  6  ±±±±±41  ±47  53%  60%  6  72  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-pkg-bindings-2.17.61-1.40.x86_64.rpm (installed size 1.86 139  x   xInstalling yast2-pkg-bindings-2.17.61-1.40.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  33  40  ±47  53%  60%  7  74  80  7  94  9  100  4.68 0 5.65  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ncurses-pkg (download size 233.00 kB)   x   x~86 MB) x    xDownloading yast2-ncurses-pkg (download size 233.00 kB) q   6  13  20  7  34  ±±±±±41  48 54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ncurses-pkg-2.17.20-0.5.1.x86_64.rpm (installed size 771. 5.65  x   xInstalling yast2-ncurses-pkg-2.17.20-0.5.1.x86_64.rpm (installed size 77q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  ±±±±±41  49 58%  66%  74  83  91  7  100  3.93 69 4.90  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-hardware-detection (download size 97.00 kB)   x   x~1.00 kB) x    xDownloading yast2-hardware-detection (download size 97.00 kB) q   16  32  49 65%  82  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-hardware-detection-2.17.2-2.7.1.x86_64.rpm (installed siz 4.90  x   xInstalling yast2-hardware-detection-2.17.2-2.7.1.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  30  ±±45  54%  9%  65  71  83  8  93  9  100  52 8 49  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading subscription-tools (download size 47.00 kB)   x   x~ize 417.00 kB) x    xDownloading subscription-tools (download size 47.00 kB) q   33  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing subscription-tools-1.0-0.21.1.noarch.rpm (installed size 163.00 49  x   xInstalling subscription-tools-1.0-0.21.1.noarch.rpm (installed size 163.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  32  ±46  53%  9%  66  72  8  84  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  36 7 33  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suseRegister (download size 54.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading suseRegister (download size 54.00 kB)  29  58%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suseRegister-1.4-1.38.24.noarch.rpm (installed size 213.00 kB) 338  x   xInstalling suseRegister-1.4-1.38.24.noarch.rpm (installed size 213.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  48 53%  78%  90  8  100  15 6 12  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading splashy (download size 1.11 MB)   x   x~) x    xDownloading splashy (download size 1.11 MB) q   5  11  6  22  7  32  7  ±±±±43  48 53%  8%  63%  8  74  9  84  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing splashy-0.3.8-106.21.x86_64.rpm (installed size 1.47 MB) 12  x   xInstalling splashy-0.3.8-106.21.x86_64.rpm (installed size 1.47 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  30  9  ±±±44  53%  61%  9  78  86  91  9  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ... symlinks in /lib/mkinitrd/boot ...  1.68 5 2.65  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cifs-utils (download size 166.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading cifs-utils (download size 166.00 kB)  9  1  28  3  4 57%  6  7  86  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cifs-utils-5.1-0.14.46.x86_64.rpm (installed size 85.00 kB) 2.65  x   xInstalling cifs-utils-5.1-0.14.46.x86_64.rpm (installed size 85.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  85%  99  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  59 4 57  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootsplash (download size 205.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading bootsplash (download size 205.00 kB)  6  14  22  30  8  ±±45  53%  61%  9  77  85  92  100  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootsplash-3.3-146.24.12.x86_64.rpm (installed size 555.00 kB) 57  x   xInstalling bootsplash-3.3-146.24.12.x86_64.rpm (installed size 555.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  37  48 60%  71  6  81  92  7  100  Updating etc/sysconfig/bootsplash... Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  05 3 02  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsyslog (download size 399.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading rsyslog (download size 399.00 kB)  7  15  20  8  36  ±±±44  52% 60%  8  76  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsyslog-5.10.1-0.11.1.x86_64.rpm (installed size 1.50 MB) 02  x   xInstalling rsyslog-5.10.1-0.11.1.x86_64.rpm (installed size 1.50 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  9  24  9  34  9  ±±±44  51 60%  8  74  9  87  92  8  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/syslog...  29.56 2 0.53  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2 (download size 682.00 kB)  ~ script `munge' Updating etc/sysconfig/syslog...   Downloading yast2 (download size 682.00 kB)  7  13  8  23  30  5  ±±±±42  49 56%  63%  70  7  84  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-2.17.140-1.1.x86_64.rpm (installed size 3.44 MB) 0.53  x   xInstalling yast2-2.17.140-1.1.x86_64.rpm (installed size 3.44 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±42  ±47  52% 7%  62%  7  72  7  83  8  93  8  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/yast2... q  6.12 1 47.09  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suspend (download size 91.00 kB)  Additional rpm output: Updating etc/sysconfig/yast2...   Downloading suspend (download size 91.00 kB)  17  35  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suspend-0.80.20081103-1.59.35.x86_64.rpm (installed size 350.00 47.097  x   xInstalling suspend-0.80.20081103-1.59.35.x86_64.rpm (installed size 350.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  30  ±±45  60%  78  85  9  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  5.77 0 6.75  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootsplash-branding-SLES (download size 1.19 MB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading bootsplash-branding-SLES (download size 1.19 MB)  5  11  6  21  6  32  7  ±±±±42  ±47  53%  8%  63%  8  74  9  84  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed size 1 6.75  x   xInstalling bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  24  31  8  ±±45  50 6%  62%  8  75  82  9  95  100  Downloading bootsplash-branding-SLES (download size 1.19 MB) Installing bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed size ~ 1.34 MB) Additional rpm output: Updating etc/sysconfig/bootsplash...  4.44 59 5.41  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-vm (download size 41.00 kB)  Additional rpm output: Updating etc/sysconfig/bootsplash...   Downloading yast2-vm (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-vm-2.17.17-0.7.66.x86_64.rpm (installed size 163.00 kB) 5.41  x   xInstalling yast2-vm-2.17.17-0.7.66.x86_64.rpm (installed size 163.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  31  9  52% 61%  73  80  5  91  8  100  28 8 25  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-squid (download size 109.00 kB)   x   xDownloading yast2-squid (download size 109.00 kB) q   14  29  ±±±44  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-squid-2.17.12-0.7.24.x86_64.rpm (installed size 547.00 kB 25  x   xInstalling yast2-squid-2.17.12-0.7.24.x86_64.rpm (installed size 547.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  9  35  40  56%  65%  74  9  86  91  6  100  3.74 7 4.71  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-packager (download size 362.00 kB)   x   x~kB) x    xDownloading yast2-packager (download size 362.00 kB) q   8  13  21  6  35  ±±±44  52% 7%  66%  75  83  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-packager-2.17.113-1.47.x86_64.rpm (installed size 1.72 MB 4.716  x   xInstalling yast2-packager-2.17.113-1.47.x86_64.rpm (installed size 1.72 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  9  ±46  51 6%  63%  8  74  9  84  9  94  9  100  2.02 6 2.99  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mouse (download size 26.00 kB)   x   x~MB) x    xDownloading yast2-mouse (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mouse-2.16.1-19.38.x86_64.rpm (installed size 87.00 kB) 2.99  x   xInstalling yast2-mouse-2.16.1-19.38.x86_64.rpm (installed size 87.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  ±±45  60%  83  8  93  9  100  1.93 5 0  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap (download size 112.00 kB)   x   xDownloading yast2-ldap (download size 112.00 kB) q   14  28  ±±±±42  57%  71%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-2.17.8-0.7.61.x86_64.rpm (installed size 475.00 kB) 0  x   xInstalling yast2-ldap-2.17.8-0.7.61.x86_64.rpm (installed size 475.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  27  40  ±±45  50 5%  61%  74  8  9  96  100  47 4 44  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-country (download size 104.00 kB)   x   x~) x    xDownloading yast2-country (download size 104.00 kB) q   15  30  ±46  61%  77  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-country-2.17.55-0.7.1.x86_64.rpm (installed size 478.00 k 44  x   xInstalling yast2-country-2.17.55-0.7.1.x86_64.rpm (installed size 478.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  ±±±±42  50 9%  64  70  6  83  8  93  8  100  00 3 1.97  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-tftp-server (download size 25.00 kB)   x   x~ kB) x    xDownloading yast2-tftp-server (download size 25.00 kB) q   64%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-tftp-server-2.17.2-0.7.38.noarch.rpm (installed size 93.0 1.97  x   xInstalling yast2-tftp-server-2.17.2-0.7.38.noarch.rpm (installed size 93q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  29  38  59%  72  9  88  96  100  0.91 2 88  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sysconfig (download size 56.00 kB)   x   x~.00 kB) x    xDownloading yast2-sysconfig (download size 56.00 kB) q   28  57%  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sysconfig-2.17.0-1.38.noarch.rpm (installed size 284.00 k 88  x   xInstalling yast2-sysconfig-2.17.0-1.38.noarch.rpm (installed size 284.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  18  ±±±44  59%  66  74  82  90  5  100  63 1 61  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-support (download size 36.00 kB)   x   x~ kB) x    xDownloading yast2-support (download size 36.00 kB) q   ±±±44  89%  100  90  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-support-2.17.21-1.35.noarch.rpm (installed size 150.00 kB 61  x   xInstalling yast2-support-2.17.21-1.35.noarch.rpm (installed size 150.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  25  30  7  48 55%  69%  77  82  9  94  9  100  49 0 46  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sshd (download size 28.00 kB)   x   x~kB) x    xDownloading yast2-sshd (download size 28.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sshd-2.17.2-1.21.noarch.rpm (installed size 114.00 kB) 46  x   xInstalling yast2-sshd-2.17.2-1.21.noarch.rpm (installed size 114.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  8  2  35  ±±±±43  59%  6  78  84  90  6  100  38 49 35  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-runlevel (download size 56.00 kB)   x   xDownloading yast2-runlevel (download size 56.00 kB) q   28  57%  86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-runlevel-2.17.9-0.7.3.noarch.rpm (installed size 209.00 k 35  x   xInstalling yast2-runlevel-2.17.9-0.7.3.noarch.rpm (installed size 209.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  ±±±44  74%  88  9  100  17 8 14  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-power-management (download size 34.00 kB)   x   x~ kB) x    xDownloading yast2-power-management (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-power-management-2.17.7-0.1.84.noarch.rpm (installed size 14  x   xInstalling yast2-power-management-2.17.7-0.1.84.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  34  ±±±±42  53%  62%  71  82  7  9  100  00 7 0.97  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-pam (download size 30.00 kB)   x   x~ze 175.00 kB) x    xDownloading yast2-pam (download size 30.00 kB) q   ±47 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-pam-2.17.3-0.5.211.noarch.rpm (installed size 129.00 kB) 0.97  x   xInstalling yast2-pam-2.17.3-0.5.211.noarch.rpm (installed size 129.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  20  6  32  ±±±±±41  ±46  63%  9  74  9  90  7  100  19.88 6 85  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-server (download size 129.00 kB)   x   x~) x    xDownloading yast2-nfs-server (download size 129.00 kB) q   12  24  37  49 61%  74  86  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-server-2.17.8-0.2.1.noarch.rpm (installed size 295.00 85  x   xInstalling yast2-nfs-server-2.17.8-0.2.1.noarch.rpm (installed size 295.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  23  3  8  ±±±±43  54%  69%  81  9  9  100  59 5 56  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-client (download size 115.00 kB)   x   x~00 kB) x    xDownloading yast2-nfs-client (download size 115.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-client-2.17.19-1.32.noarch.rpm (installed size 313.00 56  x   xInstalling yast2-nfs-client-2.17.19-1.32.noarch.rpm (installed size 313.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  31  8  ±±±±43  48 55%  62%  7  72  7  88  94  9  100  28 4 25  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-isns (download size 32.00 kB)   x   x~00 kB) x    xDownloading yast2-isns (download size 32.00 kB) q   49 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-isns-2.17.2-0.2.2.noarch.rpm (installed size 152.00 kB) 25  x   xInstalling yast2-isns-2.17.2-0.2.2.noarch.rpm (installed size 152.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  25  49 59%  72  7  83  8  94  9  100  13 3 10  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-iscsi-server (download size 40.00 kB)   x   xDownloading yast2-iscsi-server (download size 40.00 kB) q   4  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-iscsi-server-2.17.11-1.58.noarch.rpm (installed size 172. 10  x   xInstalling yast2-iscsi-server-2.17.11-1.58.noarch.rpm (installed size 17q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  27  ±46  55%  62%  74  9  84  90  6  100  8.96 2 39.94  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-irda (download size 19.00 kB)   x   x~2.00 kB) x    xDownloading yast2-irda (download size 19.00 kB) q   83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-irda-2.17.3-0.1.85.noarch.rpm (installed size 58.00 kB) 39.945  x   xInstalling yast2-irda-2.17.3-0.1.85.noarch.rpm (installed size 58.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  6  31  61%  7  8  92  9  100  1 1 88  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-instserver (download size 72.00 kB)   x   xDownloading yast2-instserver (download size 72.00 kB) q   22  ±±±44  67%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-instserver-2.17.7-0.2.1.noarch.rpm (installed size 289.00 88  x   xInstalling yast2-instserver-2.17.7-0.2.1.noarch.rpm (installed size 289.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  ±±±±43  52% 64%  70  6  81  6  99  100  63 0 60  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-firewall (download size 83.00 kB)   x   x~00 kB) x    xDownloading yast2-firewall (download size 83.00 kB) q   19  38  57%  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-firewall-2.17.13-0.11.11.noarch.rpm (installed size 498.0 60  x   xInstalling yast2-firewall-2.17.13-0.11.11.noarch.rpm (installed size 498q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  25  38  51 64%  76  81  6  93  8  100  14 39 11  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ca-management (download size 122.00 kB)   x   x~.00 kB) x    xDownloading yast2-ca-management (download size 122.00 kB) q   12  26  39  52% 65%  78  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ca-management-2.17.27-1.1.noarch.rpm (installed size 940. 11  x   xInstalling yast2-ca-management-2.17.27-1.1.noarch.rpm (installed size 94q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  8  23  30  5  40  49 56%  63%  72  8  7  92  7  100  7.22 8 8.19  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-boot-server (download size 16.00 kB)   x   x~0.00 kB) x    xDownloading yast2-boot-server (download size 16.00 kB) q   98%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-boot-server-2.14.0-276.38.noarch.rpm (installed size 45.0 8.19  x   xInstalling yast2-boot-server-2.14.0-276.38.noarch.rpm (installed size 45q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  30  7  77%  82  7  94  9  100  18 7 5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-autofs (download size 26.00 kB)   x   x~.00 kB) x    xDownloading yast2-autofs (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-autofs-2.17.1-0.1.61.noarch.rpm (installed size 98.00 kB) 5  x   xInstalling yast2-autofs-2.17.1-0.1.61.noarch.rpm (installed size 98.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  1  22  ±±±±43  49 69%  75  80  6  94  9  100  0 6 0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-add-on-creator (download size 100.00 kB)   x   x~B) x    xDownloading yast2-add-on-creator (download size 100.00 kB) q   15  32  48 64%  80  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-add-on-creator-2.17.15-0.5.93.noarch.rpm (installed size 0  x   xInstalling yast2-add-on-creator-2.17.15-0.5.93.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  21  36  48 59%  73  9  90  5  100  dd-on-creator (download size 100.00 kB)dd-on-creator-2.17.15-0.5.93.noarch.rpm (installed siz e 547.00 kB) Additional rpm output: Updating etc/sysconfig/add-on-creator...  6.55 5 7.52  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-registration-branding-SLE (download size 12.00 kB)  Additional rpm output: Updating etc/sysconfig/add-on-creator...   Downloading yast2-registration-branding-SLE (download size 12.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-registration-branding-SLE-2.17.38-0.8.43.noarch.rpm (inst 7.52  x   xInstalling yast2-registration-branding-SLE-2.17.38-0.8.43.noarch.rpm (inq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  100  4  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-online-update (download size 49.00 kB)   x   x~stalled size 1.00 kB) x    xDownloading yast2-online-update (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-online-update-2.17.24-1.82.noarch.rpm (installed size 187  x   xInstalling yast2-online-update-2.17.24-1.82.noarch.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  33  ±±±±42  52% 9%  6  7  84  9  95  100  36 3 34  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-metapackage-handler (download size 44.00 kB)   x   x~87.00 kB) x    xDownloading yast2-metapackage-handler (download size 44.00 kB) q   36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-metapackage-handler-0.8.7-1.34.noarch.rpm (installed size 34  x   xInstalling yast2-metapackage-handler-0.8.7-1.34.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  35  ±47  58%  64%  73  81  6  98  100  18 2 15  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-iscsi-client (download size 58.00 kB)   x   x~ze 191.00 kB) x    xDownloading yast2-iscsi-client (download size 58.00 kB) q   27  55%  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-iscsi-client-2.17.42-1.4.noarch.rpm (installed size 262.0 15  x   xInstalling yast2-iscsi-client-2.17.42-1.4.noarch.rpm (installed size 262q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  24  51 60%  71  9  84  9  94  9  100  5.92 1 6.89  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nis-client (download size 232.00 kB)   x   x~.00 kB) x    xDownloading yast2-nis-client (download size 232.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nis-client-2.17.8-0.1.1.x86_64.rpm (installed size 543.00 6.89  x   xInstalling yast2-nis-client-2.17.8-0.1.1.x86_64.rpm (installed size 543.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  28  34  ±±±4  49 60%  7  74  82  9  9  100  39 0 36  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-fingerprint-reader (download size 62.00 kB)   x   x~00 kB) x    xDownloading yast2-fingerprint-reader (download size 62.00 kB) q   25  52% 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-fingerprint-reader-2.17.7-0.1.201.x86_64.rpm (installed s 36  x   xInstalling yast2-fingerprint-reader-2.17.7-0.1.201.x86_64.rpm (installedq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  25  30  6  ±±±±42  49 54%  62%  75  80  5  90  5  100  13 29 10  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-security (download size 51.00 kB)   x   x~ size 268.00 kB) x    xDownloading yast2-security (download size 51.00 kB) q   31  62%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-security-2.17.16-0.7.25.noarch.rpm (installed size 232.00 10  x   xInstalling yast2-security-2.17.16-0.7.25.noarch.rpm (installed size 232.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  22  7  32  49 57%  6  73  8  83  8  93  8  100  4.90 8 5.88  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kerberos-client (download size 45.00 kB)   x   x~00 kB) x    xDownloading yast2-kerberos-client (download size 45.00 kB) q   35  71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kerberos-client-2.17.16-0.7.79.noarch.rpm (installed size 5.88  x   xInstalling yast2-kerberos-client-2.17.16-0.7.79.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  20  ±46  59%  70  9  85  91  7  100  71 7 6  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-registration (download size 173.00 kB)   x   x~ze 201.00 kB) x    xDownloading yast2-registration (download size 173.00 kB) q   9  18  27  36  ±4  55%  64%  73  8  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size 44 6  x   xInstalling yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  9  28  ±47  52% 66%  72  87  92  9  100  Installing yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size ~449.00 kB) Additional rpm output: Updating etc/sysconfig/suse_register... Updating etc/sysconfig/automatic_online_update...  27 6 24  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-online-update-frontend (download size 11.00 kB)  Updating etc/sysconfig/suse_register... Updating etc/sysconfig/automatic_online_update...   Downloading yast2-online-update-frontend (download size 11.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-online-update-frontend-2.17.24-1.82.noarch.rpm (installed 244  x   xInstalling yast2-online-update-frontend-2.17.24-1.82.noarch.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  100  5  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-samba-client (download size 71.00 kB)   x   x~ed size 1.00 kB) x    xDownloading yast2-samba-client (download size 71.00 kB) q   22  ±±45  67%  90  10  1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-samba-client-2.17.30-0.7.1.noarch.rpm (installed size 321  x   xInstalling yast2-samba-client-2.17.30-0.7.1.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  34  ±±±±43  58%  64%  73  82  90  5  100  3.95 4 4.93  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 134.93 MBx 38x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 13.95 MBx24x http-10.9.0.10-e8820372xxx tu Downloading kernel-default-base (download size 10.79 MB)   x   x~21.00 kB) x    xDownloading kernel-default-base (download size 10.79 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-default-base-3.0.101-97_lustre.x86_64.rpm (installed siz 4.9343  x   xInstalling kernel-default-base-3.0.101-97_lustre.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  WARNING: no dependencies for kernel module 'virtio:d00000002v00001AF4' f ~ound. Kernel Modules: mbcache jbd ext3 Features: acpi block resume.userspace resume.kernel 36120 blocks  SizexPackagesx 111.79 MBx 38x x x 13.95 MBx 24x x x qu  7 2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kernel-default (download size 19.16 MB)  Features: acpi block resume.userspace resume.kernel 36120 blocks    Downloading kernel-default (download size 19.16 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-default-3.0.101-97_lustre.x86_64.rpm (installed size 85. 11.7938  x   xInstalling kernel-default-3.0.101-97_lustre.x86_64.rpm (installed size 8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ~ms_helper ttm cirrus virtio_ring virtio virtio_blk usb-common usbcore o ~hci-hcd uhci-hcd ehci-hcd xhci-hcd hid usbhid crc-t10dif sd_mod virtio_ ~pci Features: acpi kms block usb resume.userspace resume.kernel 42005 blocks   26.72  6 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesx w Total x 26.72 MBx 36x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 13.95 MBx24x v http-10.9.0.10-6075eb91xxx tu Downloading syslog-ng (download size 247.00 kB)  Features: acpi kms block usb resume.userspace resume.kernel 42005 blocks    Downloading syslog-ng (download size 247.00 kB)  6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing syslog-ng-2.0.9-27.34.39.2.x86_64.rpm (installed size 787.00 kB 26.72 MB / 00:20)   x   xInstalling syslog-ng-2.0.9-27.34.39.2.x86_64.rpm (installed size 787.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  8  27  35  ±±±±43  50 60%  6  7  81  9  97  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/syslog...  5.96  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-slp-server (download size 31.00 kB)  ~ script `munge' Updating etc/sysconfig/syslog...   Downloading yast2-slp-server (download size 31.00 kB)  51 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-slp-server-2.17.8-7.1.noarch.rpm (installed size 128.00 k 5.9612  x   xInstalling yast2-slp-server-2.17.8-7.1.noarch.rpm (installed size 128.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  ±±±±43  50 67%  74  80  8  94  9  100  83  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-audit-laf (download size 37.00 kB)   x   x~ kB) x    xDownloading yast2-audit-laf (download size 37.00 kB) q   ±±±±43  87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-audit-laf-2.17.11-4.3.noarch.rpm (installed size 157.00 k 83  x   xInstalling yast2-audit-laf-2.17.11-4.3.noarch.rpm (installed size 157.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  22  7  ±±45  55%  68%  79  84  9  96  100  68  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ntp-client (download size 84.00 kB)   x   x~ kB) x    xDownloading yast2-ntp-client (download size 84.00 kB) q   18  3  57%  76%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ntp-client-2.17.20-10.34.noarch.rpm (installed size 452.0 68  x   xInstalling yast2-ntp-client-2.17.20-10.34.noarch.rpm (installed size 452q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  22  7  39  53%  60%  7  6  83  8  97  100  24  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-printer (download size 142.00 kB)   x   x~.00 kB) x    xDownloading yast2-printer (download size 142.00 kB) q   9  21  32  ±±±±43  54%  66%  77  88  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-printer-2.17.64-4.4.x86_64.rpm (installed size 721.00 kB) 24  x   xInstalling yast2-printer-2.17.64-4.4.x86_64.rpm (installed size 721.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  20  7  34  40  49 58%  65%  7  84  90  6  100  4.53  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-storage (download size 646.00 kB)   x   x~B) x    xDownloading yast2-storage (download size 646.00 kB) q   7  12  7  22  7  32  7  ±±±±42  ±47  52% 7%  64%  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-storage-2.17.161-5.1.x86_64.rpm (installed size 3.53 MB) 4.53  x   xInstalling yast2-storage-2.17.161-5.1.x86_64.rpm (installed size 3.53 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  6  31  6  ±±±±±41  ±47  53%  8%  64%  9  74  9  85  90  6  100  storage646storage161-5.1.x86_64.rpm (installed size 3.53 MB ) Additional rpm output: Updating etc/sysconfig/storage...  1.00  0 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-network (download size 380.00 kB)  Additional rpm output: Updating etc/sysconfig/storage...   Downloading yast2-network (download size 380.00 kB)  8  16  21  9  37  ±±±±42  50 8%  63%  71  80  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-network-2.17.210-3.3.x86_64.rpm (installed size 2.26 MB) 1.001  x   xInstalling yast2-network-2.17.210-3.3.x86_64.rpm (installed size 2.26 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±43  48 55%  61%  6  71  7  83  9  96  100  18.74 29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-bootloader (download size 353.00 kB)   x   x~) x    xDownloading yast2-bootloader (download size 353.00 kB) q   8  13  8  27  36  ±±45  54%  63%  72  7  86  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 MB 18.740  x   xInstalling yast2-bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  23  8  34  9  ±±45  50 5%  61%  7  72  7  82  7  92  7  100  bootloader (download size 353.00 kB)bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 MB) Additional rpm output: Updating etc/sysconfig/bootloader...  6.36  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap-client (download size 109.00 kB)  Additional rpm output: Updating etc/sysconfig/bootloader...   Downloading yast2-ldap-client (download size 109.00 kB)  12  31  8  ±46  61%  70  83  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581.0 6.36  x   xInstalling yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  27  38  ±±±44  55%  68%  73  87  93  9  100  Downloading yast2-ldap-client (download size 109.00 kB) Installing yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581 ~.00 kB) Additional rpm output: Updating etc/sysconfig/ldap...  5.79  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kdump (download size 58.00 kB)  Additional rpm output: Updating etc/sysconfig/ldap...   Downloading yast2-kdump (download size 58.00 kB)  24  51 79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kdump-2.17.30-8.1.x86_64.rpm (installed size 331.00 kB) 5.79  x   xInstalling yast2-kdump-2.17.30-8.1.x86_64.rpm (installed size 331.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  30  48 59%  65  74  8  98  100  47  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-users (download size 214.00 kB)   x   xDownloading yast2-users (download size 214.00 kB) q   6  13  21  8  36  ±±±±43  51 8%  66%  73  81  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-users-2.17.56-3.6.x86_64.rpm (installed size 1.06 MB) 47  x   xInstalling yast2-users-2.17.56-3.6.x86_64.rpm (installed size 1.06 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  8  35  40  ±46  53%  8%  65%  73  8  83  8  93  8  100  4.41  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 14.41 MBx 25x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 13.95 MBx24x v http-10.9.0.10-6075eb91xxx tu Downloading ofed-kmp-default (download size 66.00 kB)   x   xDownloading ofed-kmp-default (download size 66.00 kB) q   21  ±±45  69%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-kmp-default-1.5.4.1_3.0.101_63-20.26.x86_64.rpm (installed 4.412  x   xInstalling ofed-kmp-default-1.5.4.1_3.0.101_63-20.26.x86_64.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  ±±±±±41  73%  99  100  76 3 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nis-server (download size 42.00 kB)   x   x~ed size 201.00 kB) x    xDownloading yast2-nis-server (download size 42.00 kB) q   38  76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nis-server-2.17.3-0.7.130.noarch.rpm (installed size 159. 212  x   xInstalling yast2-nis-server-2.17.3-0.7.130.noarch.rpm (installed size 15q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  9  35  67%  85  99  100  60 2 06  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-http-server (download size 111.00 kB)   x   x~9.00 kB) x    xDownloading yast2-http-server (download size 111.00 kB) q   14  28  ±±±±43  57%  71%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-http-server-2.17.17-1.99.noarch.rpm (installed size 672.0 06  x   xInstalling yast2-http-server-2.17.17-1.99.noarch.rpm (installed size 672q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  29  38  ±±±44  53%  8%  65%  72  7  89  97  100  2.95 1 3.40  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-tune (download size 79.00 kB)   x   x~.00 kB) x    xDownloading yast2-tune (download size 79.00 kB) q   2  4  60%  81  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-tune-2.17.7-1.28.x86_64.rpm (installed size 449.00 kB) 3.40  x   xInstalling yast2-tune-2.17.7-1.28.x86_64.rpm (installed size 449.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  20  8  ±±±44  59%  65  70  7  82  7  95  100  51 0 2.96  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-restore (download size 65.00 kB)   x   xDownloading yast2-restore (download size 65.00 kB) q   24  49 73%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-restore-2.17.7-0.5.1.noarch.rpm (installed size 371.00 kB 2.96  x   xInstalling yast2-restore-2.17.7-0.5.1.noarch.rpm (installed size 371.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  37  50 65%  70  8  8  99  100  15 19 60  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autoyast2-installation (download size 231.00 kB)   x   x~kB) x    xDownloading autoyast2-installation (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autoyast2-installation-2.17.76-1.20.noarch.rpm (installed size 60  x   xInstalling autoyast2-installation-2.17.76-1.20.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  21  6  32  7  ±±±44  49 54%  61%  7  73  81  7  93  8  100  0.71 8 1.17  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kerberos-server (download size 46.00 kB)   x   x~e 1.43 MB) x    xDownloading yast2-kerberos-server (download size 46.00 kB) q   34  69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kerberos-server-2.17.8-0.6.2.noarch.rpm (installed size 2 1.17  x   xInstalling yast2-kerberos-server-2.17.8-0.6.2.noarch.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  34  ±±±±42  ±47  76%  81  9  94  9  100  48 7 0.94  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dns-server (download size 134.00 kB)   x   x~ 240.00 kB) x    xDownloading yast2-dns-server (download size 134.00 kB) q   11  23  35  ±47  59%  71  83  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dns-server-2.17.24-1.2.noarch.rpm (installed size 757.00 0.94  x   xInstalling yast2-dns-server-2.17.24-1.2.noarch.rpm (installed size 757.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  22  7  36  ±47  56%  64%  72  8  83  90  7  100   9.74 6 20  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-profile-manager (download size 116.00 kB)   x   x~0 kB) x    xDownloading yast2-profile-manager (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-profile-manager-2.17.2-0.7.5.x86_64.rpm (installed size 4 201  x   xInstalling yast2-profile-manager-2.17.2-0.7.5.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  8  24  3  40  ±46  52% 7%  70%  5  80  5  90  6  100  28 5  9.73  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap-server (download size 295.00 kB)   x   x~ 476.00 kB) x    xDownloading yast2-ldap-server (download size 295.00 kB) q   5  10  6  21  7  32  7  ±±±±43  48 54%  9%  65  70  6  81  6  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-server-2.17.44-0.5.1.x86_64.rpm (installed size 1.15 9.73 MB / 00:11)   x   xInstalling yast2-ldap-server-2.17.44-0.5.1.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  33  8  ±±±44  50 6%  63%  9  75  82  9  7  100  8.12 4 8.58  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sudo (download size 38.00 kB)   x   x~15 MB) x    xDownloading yast2-sudo (download size 38.00 kB) q   ±±±±42  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sudo-2.17.3-1.53.noarch.rpm (installed size 164.00 kB) 8.58  x   xInstalling yast2-sudo-2.17.3-1.53.noarch.rpm (installed size 164.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  2  36  58%  65%  77  82  7  92  8  100  7.96 3 42  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-samba-server (download size 75.00 kB)   x   xDownloading yast2-samba-server (download size 75.00 kB) q   21  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-samba-server-2.18.0-1.87.noarch.rpm (installed size 333.0 42  x   xInstalling yast2-samba-server-2.18.0-1.87.noarch.rpm (installed size 333q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  25  33  8  ±±±±43  50 5%  61%  6  74  80  6  91  6  100  64 2 10  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mail (download size 134.00 kB)   x   x~.00 kB) x    xDownloading yast2-mail (download size 134.00 kB) q   11  23  35  ±47  59%  71  83  95  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kB) 10  x   xInstalling yast2-mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  21  8  39  ±±±44  49 61%  8  73  9  85  90  7  100  mail (download size 134.00 kB) mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kB ) Additional rpm output: Updating etc/sysconfig/mail...  6.91 1 7.37  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-inetd (download size 44.00 kB)  Additional rpm output: Updating etc/sysconfig/mail...   Downloading yast2-inetd (download size 44.00 kB)  35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-inetd-2.17.3-1.53.noarch.rpm (installed size 208.00 kB) 7.37  x   xInstalling yast2-inetd-2.17.3-1.53.noarch.rpm (installed size 208.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  38  ±46  78%  8  99  100  7 0 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ofed (download size 28.00 kB)   x   xDownloading ofed (download size 28.00 kB) q   57%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-1.5.4.1-20.26.x86_64.rpm (installed size 52.00 kB) 1  x   xInstalling ofed-1.5.4.1-20.26.x86_64.rpm (installed size 52.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  89%  9  100  Updating etc/sysconfig/infiniband... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  66  9 2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-installation (download size 258.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading yast2-installation (download size 258.00 kB)  6  12  8  24  30  7  ±±±±43  49 55%  61%  8  74  80  6  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-installation-2.17.114-1.14.noarch.rpm (installed size 775 2  x   xInstalling yast2-installation-2.17.114-1.14.noarch.rpm (installed size 7q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  23  31  9  ±47  52% 9%  65  70  6  84  9  94  9  100  Downloading yast2-installation (download size 258.00 kB) Installing yast2-installation-2.17.114-1.14.noarch.rpm (installed size 7 ~75.00 kB) Additional rpm output: Updating etc/sysconfig/security...  5.90 8 6.36   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dhcp-server (download size 139.00 kB)  Additional rpm output: Updating etc/sysconfig/security...   Downloading yast2-dhcp-server (download size 139.00 kB)  11  22 - 31.74 kB/s (download size 139.00 kB)  34  ±46  57%  69%  80  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dhcp-server-2.17.6-0.4.2.21.noarch.rpm (installed size 92 6.36  x   xInstalling yast2-dhcp-server-2.17.6-0.4.2.21.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  5  20  7  35  ±±±±42  49 56%  63%  70  7  82  7  95  100  0 7 5.4  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ft (download size 70.00 kB)   x   x~920.00 kB) x    xDownloading yast2-ftp-server (download size 70.00 kB) q   22  ±±45  68%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ftp-server-2.17.9-0.5.2.noarch.rpm (installed size 534.00 5.40  x   xInstalling yast2-ftp-server-2.17.9-0.5.2.noarch.rpm (installed size 534.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  26  33  ±±45  53%  9%  6  76  8  98  100  4.48 6 4.94  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 4.94 MBx 7x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.48 MBx6x v http-10.9.0.10-6075eb91xxx tu Downloading yast2-update (download size 95.00 kB)   x   x~00 kB) x    xDownloading yast2-update (download size 95.00 kB) q   16  33  50 67%  84  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-update-2.17.27-4.12.x86_64.rpm (installed size 467.00 kB) 4.942  x   xInstalling yast2-update-2.17.27-4.12.x86_64.rpm (installed size 467.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  26  40  52% 64%  72  7  8  92  7  100  SizexPackagesx 4.94 MBx 7x x x 4.48 MBx 6x x x qqu 48 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx Total x 4.48 MBx 6x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.48 MBx6x tu Downloading yast2-repair (download size 263.00 kB)   x   x~B) x    xDownloading yast2-repair (download size 263.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  9  85  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-repair-2.17.13-1.98.noarch.rpm (installed size 1.78 MB) 481  x   xInstalling yast2-repair-2.17.13-1.98.noarch.rpm (installed size 1.78 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  5  30  5  ±±±±42  ±47  52% 8%  63%  8  73  9  84  9  96  100  2.71 5 SizexPackagesx Ti2.71 MBx 6x 00:x x 2.71 MBx 5x 00:qqqu 5 Time 00:00 00:00 qq ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dirinstall (download size 24.00 kB)   x   xDownloading yast2-dirinstall (download size 24.00 kB) q   68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dirinstall-2.17.5-0.7.83.noarch.rpm (installed size 73.00 2.71 MB)   x   xInstalling yast2-dirinstall-2.17.5-0.7.83.noarch.rpm (installed size 73.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  22  9  40  ±±45  69%  76  82  8  94  9  100  64 4 64 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-backup (download size 166.00 kB)   x   x~00 kB) x    xDownloading yast2-backup (download size 166.00 kB) q   9  1  28  3  4 57%  6  7  86  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-backup-2.17.14-0.5.1.noarch.rpm (installed size 689.00 kB 64  x   xInstalling yast2-backup-2.17.14-0.5.1.noarch.rpm (installed size 689.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  2  9  38  ±47  55%  65%  72  8  83  8  99  100  1.96 3 1.96 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-add-on (download size 50.00 kB)   x   x~kB) x    xDownloading yast2-add-on (download size 50.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-add-on-2.17.31-0.7.49.noarch.rpm (installed size 154.00 k 1.96  x   xInstalling yast2-add-on-2.17.31-0.7.49.noarch.rpm (installed size 154.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  54%  69%  76  81  8  95  100  81 2 81 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autoyast2 (download size 481.00 kB)   x   x~ kB) x    xDownloading autoyast2 (download size 481.00 kB) q   6  13  9  26  33  9  ±46  53%  9%  66  73  9  86  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autoyast2-2.17.76-1.20.noarch.rpm (installed size 1.56 MB) 81  x   xInstalling autoyast2-2.17.76-1.20.noarch.rpm (installed size 1.56 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±46  52% 9%  64  9  74  81  6  91  6  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/autoinstall... q   SizexPackagesx Tim 1.81 MBx 2x00:0 x x260.00 kBx 2x00:0 tu 1 260.00 k 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-wagon (download size 63.00 kB)  Additional rpm output: Updating etc/sysconfig/autoinstall...   Downloading yast2-wagon (download size 63.00 kB)  25  50 76%  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 kB 260.00 kB)  x   xInstalling yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  51 66%  73  9  84  9  94  9  100   Done.   SizexPackagesx Time  x 1x00:00 x x Done.x x00:00 qq   9   Perform Installation  lq xMedia x SizexPackagesx Time xTotal x x x00:00 xSUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109x x x xMedium 1 xDone.x x00:00 x mq lActions performed:q xUpdating etc/sysconfig/autoinstall... x x xDownloading yast2-wagon (download size 63.00 kB) xInstalling yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 mq Installing yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 kB)  Installing Packages... (Remaining: 260.00 kB)   F10 Next 8 Back  F9 Abort  F10 Next  Finishing Basic Installation   Finishing Basic Installation                Preparing ation finish-scripts...            [Help] [Back] [Abort] [Next]9 Abort   - Copy files to installed system - Save configuration - Install boot manager - Save installation settings - Prepare system for initial boot ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  => Executing autoinstall scripts in the installation environment...  2  4 Copying files to installed system...   7 system files to the installed system...  9 Moving to installed system...   12  x   => Setting up linker cache   4  7 aving default runlevel...  9 Initializing default window manager...  21 fonts...   4 Saving file system configuration...  6 iSCSI configuration...   9 fcoe configuration...   31 Updating kernel module dependencies...  4 Copying X Window System configuration into system...  6 Saving proxy configuration...   9 the package manager configuration...  ±±±±±41 Enabling random number generator...   ±±±43 Saving system settings...   ±46  x  => bootloader configuration...  48 kdump configuration...  51 x  => Writing YaST C 3%  Saving network configuration... 6%  hardware configuration... 8%  Writing Users Configuration...  60  Executing autoinstall scripts in the installation environment... 3  Writing automatic configuration...  5  x  => 8  Checking the installed system...  70  opying log files to installed system... 3  Unmounting all mounted devices...  5  x  100    Finished.  F10 Next 8 Back  F9 Abort  F10 Next [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxFinishing the installation...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj                            [?1l>[?1l%G[ 451.292172] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-97_lustre-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Mar 30 13:45:57 UTC 2017 (b708f71) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37756000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa81, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72320 r8192 d21888 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a81, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910920k/2097140k available (4576k kernel code, 460k absent, 185760k reserved, 7571k data, 1356k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013329] Security Framework initialized [ 0.014208] AppArmor: AppArmor initialized [ 0.015235] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017081] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020072] Mount-cache hash table entries: 256 [ 0.021201] Initializing cgroup subsys cpuacct [ 0.022103] Initializing cgroup subsys memory [ 0.023012] Initializing cgroup subsys devices [ 0.023915] Initializing cgroup subsys freezer [ 0.024006] Initializing cgroup subsys net_cls [ 0.024902] Initializing cgroup subsys blkio [ 0.025788] Initializing cgroup subsys perf_event [ 0.026849] Disabled fast string operations [ 0.028024] mce: CPU supports 10 MCE banks [ 0.031259] ACPI: Core revision 20110413 [ 0.033603] Enabling x2apic [ 0.034342] Enabled x2apic [ 0.035243] Switched APIC routing to physical x2apic. [ 0.037140] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.038233] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.060010] kvm-clock: cpu 1, msr 0:7fd10a81, secondary cpu clock [ 0.060025] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.060050] Brought up 2 CPUs [ 0.060052] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068126] devtmpfs: initialized [ 0.071019] print_constraints: dummy: [ 0.072057] Time: 15:40:53 Date: 04/04/17 [ 0.073057] NET: Registered protocol family 16 [ 0.074092] ACPI: bus type pci registered [ 0.075020] PCI: Using configuration type 1 for base access [ 0.076484] bio: create slab at 0 [ 0.080714] ACPI: Interpreter enabled [ 0.081534] ACPI: (supports S0 S5) [ 0.082441] ACPI: Using IOAPIC for interrupt routing [ 0.086903] ACPI: No dock devices found. [ 0.087757] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.088032] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.092114] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.093677] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.122342] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.128675] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.130093] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.131504] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.132838] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.134226] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.135605] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.136013] vgaarb: loaded [ 0.137005] PCI: Using ACPI for IRQ routing [ 0.137277] NetLabel: Initializing [ 0.140013] NetLabel: domain hash size = 128 [ 0.140905] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.141895] NetLabel: unlabeled traffic allowed by default [ 0.142940] Switching to clocksource kvm-clock [ 0.146218] AppArmor: AppArmor Filesystem Enabled [ 0.147244] pnp: PnP ACPI init [ 0.148103] ACPI: bus type pnp registered [ 0.149696] pnp: PnP ACPI: found 6 devices [ 0.150591] ACPI: ACPI bus type pnp unregistered [ 0.157911] NET: Registered protocol family 2 [ 0.159064] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.161265] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.164918] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.166628] TCP: Hash tables configured (established 262144 bind 65536) [ 0.167820] TCP reno registered [ 0.168637] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.169778] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.171163] NET: Registered protocol family 1 [ 0.172139] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.173277] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.174379] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.175628] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.176749] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.182174] pci 0000:00:01.2: PCI INT D disabled [ 0.183241] Unpacking initramfs... [ 0.316671] Freeing initrd memory: 8808k freed [ 0.330872] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.332088] Initialise module verification [ 0.333180] audit: initializing netlink socket (disabled) [ 0.334244] type=2000 audit(1491320454.332:1): initialized [ 0.352439] bounce pool size: 64 pages [ 0.353380] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.355565] VFS: Disk quotas dquot_6.5.2 [ 0.356528] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.357892] msgmni has been set to 937 [ 0.359214] alg: No test for stdrng (krng) [ 0.360152] Asymmetric key parser 'x509' registered [ 0.361216] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.362946] io scheduler noop registered [ 0.363829] io scheduler deadline registered (default) [ 0.364952] io scheduler cfq registered [ 0.366110] GHES: HEST is not enabled! [ 0.367015] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.368394] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.609236] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.645095] Non-volatile memory driver v1.3 [ 0.646065] Linux agpgart interface v0.103 [ 0.647066] Fixed MDIO Bus: probed [ 0.647951] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.650252] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.651238] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.652291] mousedev: PS/2 mouse device common for all mice [ 0.653539] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.655623] cpuidle: using governor ladder [ 0.656522] cpuidle: using governor menu [ 0.657385] EFI Variables Facility v0.08 2004-May-17 [ 0.658544] TCP cubic registered [ 0.659332] Registering the dns_resolver key type [ 0.660405] Loading module verification certificates [ 0.661407] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.663376] registered taskstats version 1 [ 0.664451] Magic number: 13:11:689 [ 0.667314] Freeing unused kernel memory: 1356k freed [ 0.668987] Write protecting the kernel read-only data: 10240k [ 0.673126] Freeing unused kernel memory: 1548k freed [ 0.676716] Freeing unused kernel memory: 840k freed doing fast boot [ 0.714793] Disabling lock debugging due to kernel taint [ 0.722022] SCSI subsystem initialized [ 0.746621] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.759709] scsi0 : ata_piix [ 0.760644] scsi1 : ata_piix [ 0.761450] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.762606] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.929384] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.951616] vda: vda1 vda2 [ 0.952821] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.953912] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.960617] alua: device handler registered [ 0.965518] hp_sw: device handler registered [ 0.970432] emc: device handler registered [ 0.975285] rdac: device handler registered Creating device nodes with udev [ 0.979157] udev: starting version 147 [ 0.998885] usbcore: registered new interface driver usbfs [ 0.999990] usbcore: registered new interface driver hub [ 1.018377] usbcore: registered new device driver usb [ 1.031567] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.036560] uhci_hcd: USB Universal Host Controller Interface driver [ 1.037881] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.039667] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.040767] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.042528] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.043708] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.044965] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.046608] usb usb1: Product: UHCI Host Controller [ 1.047643] usb usb1: Manufacturer: Linux 3.0.101-97_lustre-default uhci_hcd [ 1.048944] usb usb1: SerialNumber: 0000:00:01.2 [ 1.050021] hub 1-0:1.0: USB hub found [ 1.050940] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 15:40:55 2017 [ 1.085969] [drm] Initialized drm 1.1.0 20060810 [ 1.089008] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-97_lustre-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.317949] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 59088/1310720 files, 463081/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.350690] kjournald starting. Commit interval 15 seconds [ 1.352346] EXT3-fs (vda1): using internal journal [ 1.353033] Refined TSC clocksource calibration: 2593.637 MHz. [ 1.354414] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 15:40:55 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.913385] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.933949] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.935633] ACPI: Power Button [PWRF] [ 1.937228] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.941563] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.942954] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.944055] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.947866] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.959610] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 1.973676] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.988549] rtc_cmos 00:01: RTC can wake from S4 [ 1.989816] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.991061] rtc0: alarms up to one day, 114 bytes nvram [ 1.996117] NET: Registered protocol family 10 [ 1.997099] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 2.135449] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.570486] device-mapper: uevent: version 1.0.3 [ 2.571530] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.679627] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25h[?25h[?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25h[?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'linux'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump : No kdump initial ramdisk found.Tried to locate /boot/initrd-3.0.101-97_lustre-default-kdump skipped [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle boot.kdump System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25hStarting syslog services7[?25l done8[?25h unicode_start skipped on /dev/ttyS0 [?25lProbing connected terminal... 78 Initializing virtual console...  H H H H H H H H H [?25h Found an unknown terminal on /dev/ttyS0 (80 columns x 24 lines). Starting D-Bus daemon7[?25l done8[?25h Starting acpid 7[?25l done8[?25h Loading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h Setting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 28 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h 0 *** Starting YaST2 *** (B)0[?7h[?1h=[?1h=[?1h= YaST2 - installation @ trevis-41vm7   Initializing ...                     [Help] [Back] [Abort] [Next]                                           [Next]   [Next] [Help]      F1 Help  F9 Abort  Initializing...  Preparing the 1st system configuration...  YaST2 - installation @ trevis-41vm7    Initializing...                             Preparing the 1st system configuration...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort  YaST2 - installation @ trevis-41vm7    Initializing...                             Preparing the 1st system configuration...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort [Help] 8 Back  F9 Abort  F10 Next [Help] [Help] [Next] [Help][Next] [?1h=    ...                                        [Help]                                           [Next]    - Initialize the Target System - Read Installed Packages±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Loading the Package Manager... [Next] [Help] Loading the Package Manager...  =>  x   =>  x   100%     Initializing...    Preparing the 1st system configuration...  [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxInitializing repositories...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqj ing the 1st system configuratio  [?1h=    ...                                        [Help]                                           [Next]    - Load Sources - Refresh Sources - Rebuild Cache - Load Data±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Loading the Package Manager... [Next] [Help] Loading the Package Manager...  =>  x   =>  x  =>  4  25 x  =>  50 100%  x     Initializing...      Preparing the 1st system configuration... - Install required packages ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Preparing System for Automatic Installation  => Checking for required packages...  7 packages required for firewall...  15 host...   23 iscsi-client...  30 keyboard...   8 lan...   ±46 proxy...  53%  runlevel... 61%  timezone 9  users...  76  yast-language... 84  Adding found...  92  Running scripts...  100   x     9 Abort  F10 Next  8 Back  F9 Abort  F10 Next  ackage Installation     9 Abort   erform  lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx xx xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 8 Back  F9 Abort  F10 Next  ackage                                9 Abort   erform  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xMediaxSizexPackagesxTimex xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing...  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Packages...  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu Downloading desktop-data-SLED (download size 21.29 MB) Downloading desktop-data-SLED (download size 21.29 MB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sound-theme-freedesktop (download size 734.00 kB) Downloading sound-theme-freedesktop (download size 734.00 kB)  6  13  9  24  30  7  ±±±±43  48 54%  61%  7  72  8  85  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± audit (download size 227.00 kB)  Downloading audit (download size 227.00 kB)  6  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± -libs-32bit (download size 72.00 kB) Downloading audit-libs-32bit (download size 72.00 kB)  22  ±±±44  66%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse (download size 63.00 kB)   w   xDownloading fuse (download size 63.00 kB) v   25  50 75%  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines (download size 268.00 kB)  x w x   xDownloading gtk2-engines (download size 268.00 kB) v   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± keyutils-libs-32bit (download size 7.00 kB)  x w   xDownloading keyutils-libs-32bit (download size 7.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libbluetooth3 (download size 92.00 kB)   x w   xDownloading libbluetooth3 (download size 92.00 kB) v   17  34  51 69%  86  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cdio7 (download size 52.00 kB)   x w   xDownloading libcdio7 (download size 52.00 kB) v   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± db-4_5-32bit (download size 480.00 kB)  x w   xDownloading libdb-4_5-32bit (download size 480.00 kB) v   6  13  9  26  33  40  ±46  53%  60%  6  73  80  6  93  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± exif (download size 194.00 kB)   x w   xDownloading libexif (download size 194.00 kB) v   8  16  24  33  ±±±±±41  49 57%  66%  74  82  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse2 (download size 61  x w   xDownloading libfuse2 (download size 61.00 kB) v   26  52% 79%  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± idl (download size 94.00 kB)   x w   xDownloading libidl (download size 94.00 kB) v   16  33  51 68%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jpeg-32bit (download size 67.00 kB)  x w   xDownloading libjpeg-32bit (download size 67.00 kB) v   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pixman-1-0-32bit (download size 200.00 kB)  x x   xDownloading libpixman-1-0-32bit (download size 200.00 kB) q   7  15  23  32  40  48 56%  64%  72  80  8  96  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± roxy0 (download size 25.00 kB)   x   xDownloading libproxy0 (download size 25.00 kB) q   64%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± selinux1-32bit (download size 52.00 kB)  x   xDownloading libselinux1-32bit (download size 52.00 kB) q   3  62%  93  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± uuid1-32bit (download size 37.00 kB)   x   xDownloading libuuid1-32bit (download size 37.00 kB) q   ±±±±42  86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcrypt-32bit (download size 34.00 kB)  x   xDownloading libxcrypt-32bit (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± python-argparse42  x   xDownloading python-argparse (download size 42.00 kB) q   38  76%  100  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libICE-32bit (download size 41.00 kB)  x   xDownloading xorg-x11-libICE-32bit (download size 41.00 kB) q   38  78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xau8.00 kB)   x   xDownloading xorg-x11-libXau-32bit (download size 8.00 kB) q   100%   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-theme-SLED (download size 502.00 kB)   x   xDownloading gtk2-theme-SLED (download size 502.00 kB) q   6  11  7  24  30  7  ±±±±43  49 56%  62%  8  75  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_cdda0 (download size 20.00 kB)   x   xDownloading libcdio_cdda0 (download size 20.00 kB) q   79%  100   10 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-lang (download size 345.00 kB)  x   xDownloading libgphoto2-lang (download size 345.00 kB) q   9  18  23  32  7  ±46  51 60%  9  74  83  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± orbit2 (download size 170.00 kB)   x   xDownloading orbit2 (download size 170.00 kB) q   9  18  2  37  ±46  56%  65%  7  84  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libproxy0-config-gnome (download size 12.00 kB)  x   xDownloading libproxy0-config-gnome (download size 12.00 kB) q   100%   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libSM-32bit (download size 16.00 kB)   x   xDownloading xorg-x11-libSM-32bit (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcb-32bit (download size 101.00 kB)  x   xDownloading xorg-x11-libxcb-32bit (download size 101.00 kB) q   15  31  ±47  63%  79  95  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-branding-SLED (download size 5.00 kB)   x   xDownloading gtk2-branding-SLED (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_paranoia0 (download size 20  x   xDownloading libcdio_paranoia0 (download size 20.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2 (download size 705.00 kB)   x   xDownloading libgphoto2 (download size 705.00 kB) q   6  11  8  24  9  34  40  ±±45  52% 9%  65  70  7  83  8  93  9  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu openobex1 (download size 34.00 kB)  x   xDownloading libopenobex1 (download size 34.00 kB) q   ±46  94%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzmadec0 (download size 13.00 kB)   x   xDownloading liblzmadec0 (download size 13.00 kB) q   100%   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± obex-data-server (download size 93.00 kB)  x   xDownloading obex-data-server (download size 93.00 kB) q   17  34  51 69%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzma (download size 58.00 kB)   x   xDownloading lzma (download size 58.00 kB) q   27  55%  83%  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu ibtasn1-3 (download size 57.00 kB)  x   xDownloading libtasn1-3 (download size 57.00 kB) q   27  55%  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssh-helpers (download size 52.00 kB)  x   xDownloading openssh-helpers (download size 52.00 kB) q   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± readline-doc (download size 138.00 kB)   x   xDownloading readline-doc (download size 138.00 kB) q   11  23  34  ±46  58%  69%  81  93  100  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcares2 (download size 36.00 kB)   x   xDownloading libcares2 (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± om_err2-32bit (download size 41.00 kB)  x   xDownloading libcom_err2-32bit (download size 41.00 kB) q   38  77%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnutls26 (download size 283.00 kB)   x   xDownloading libgnutls26 (download size 283.00 kB) q   5  11  6  22  8  33  9  ±±45  50 6%  62%  7  73  9  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± expat1-32bit (download size 62.00 kB)  x   xDownloading libexpat1-32bit (download size 62.00 kB) q   25  51 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gcc_s56  x   xDownloading libgcc_s1-32bit (download size 56.00 kB) q   28  57%  86%  100   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dbus-1-32bit (download size 116.00 kB)   x   xDownloading dbus-1-32bit (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  82  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libavahi-common3 (download size 34.00 kB)  x   xDownloading libavahi-common3 (download size 34.00 kB) q   ±46  93%  100   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ref_array1 (download size 8.00 kB)   x   xDownloading libref_array1 (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhash1 (download size 9.00 kB)   x   xDownloading libdhash1 (download size 9.00 kB) q   100%  20 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bz2-1-32bit (download size 34.00 kB)  x   xDownloading libbz2-1-32bit (download size 34.00 kB) q   ±46 - 15.74 kB/s (download size 34.00 kB)  94%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sss_idmap0 (download size 23.00 kB)   x   xDownloading libsss_idmap0 (download size 23.00 kB) q   71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vim-data (download size 5.49 MB)   x   xDownloading vim-data (download size 5.49 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libwbclient0 (download size 223.00 kB)  x   xDownloading libwbclient0 (download size 223.00 kB) q   7  14  21  8  35  ±±±±43  50 7%  64%  71  9  86  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± augeas-lenses (download size 99  x   xDownloading augeas-lenses (download size 99.00 kB) q   16  32  48 64%  81  97  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtdb1 (download size 231.00 kB)   x   xDownloading libtdb1 (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zlib-32bit (download size 47.00 kB)  x   xDownloading zlib-32bit (download size 47.00 kB) q   33  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcollection2 (download size 21.00 kB)  x   xDownloading libcollection2 (download size 21.00 kB) q   76%  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± archive2 (download size 101.00 kB)   x   xDownloading libarchive2 (download size 101.00 kB) q   15  31  ±47  63%  79  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bash-doc (download size 394.00 kB)   x   xDownloading bash-doc (download size 394.00 kB) q   8  16  24  32  40  48 56%  65%  73  81  9  97  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtevent0 (download size 220.00 kB)  x   xDownloading libtevent0 (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  50 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± perl-doc (download size 5.78 MB)   x   xDownloading perl-doc (download size 5.78 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpath_utils1 (download size 9.00 kB)  x   xDownloading libpath_utils1 (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zypper-log (download size 75.00 kB)   x   xDownloading zypper-log (download size 75.00 kB) q   2  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-lang (download size 729.00 kB)  x   xDownloading gconf2-lang (download size 729.00 kB) q   6  11  6  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  - 582.00 kB/s (download size 729.00 kB) 8  93  9  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libX11-32bit (download size 575.00 kB)   x   xDownloading xorg-x11-libX11-32bit (download size 575.00 kB) q   5  11  6  22  7  33  8  ±±±44  50 5%  61%  6  72  7  83  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtasn1 (download size 103.00 kB)   x   xDownloading libtasn1 (download size 103.00 kB) q   15  31  ±46  62%  77  93  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± krb5-32bit (download size 497.00 kB)  x   xDownloading krb5-32bit (download size 497.00 kB) q   6  12  9  25  32  8  ±±45  51 7%  64%  70  7  83  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libstdc++6-32bit (download size 406.00 kB)  x   xDownloading libstdc++6-32bit (download size 406.00 kB) q   7  15  23  31  9  ±47  55%  63%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± avahi-glib1 (download size 10.00 kB)   x   xDownloading libavahi-glib1 (download size 10.00 kB) q   100%   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± client3 (download size 38.00 kB)  x   xDownloading libavahi-client3 (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssl0_9_8-32bit (download size 785.00 kB)  x   xDownloading libopenssl0_9_8-32bit (download size 785.00 kB) q   6  11  7  22  8  34  40  ±46  52% 8%  64%  71  7  83  9  95  100   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cracklib-32bit (download size 16.00 kB)   x   xDownloading cracklib-32bit (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpng12-0-32bit (download size 75.00 kB)  x   xDownloading libpng12-0-32bit (download size 75.00 kB) q   21  ±±±±42  64%  86  100   30 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xml2-32bit (download size 535.00 kB)   x   xDownloading libxml2-32bit (download size 535.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± freetype2-32bit (download size 255.00 kB)  x   xDownloading freetype2-32bit (download size 255.00 kB) q   6  12  8  25  31  7  ±±±±43  50 6%  62%  8  75  81  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libldb1 (download size 310.00 kB)   x   xDownloading libldb1 (download size 310.00 kB) q   5  10  5  20  5  30  6  ±±±±±41  ±46  51 6%  61%  7  72  7  82  7  92  8  100  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± smbclient0 (download size 1.69 MB)  x   xDownloading libsmbclient0 (download size 1.69 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ini_config2 (download size 17.00 kB)  x   xDownloading libini_config2 (download size 17.00 kB) q   97%  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2 (download size 273.00 kB)   x   xDownloading gconf2 (download size 273.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  82  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXfixes-32bit (download size 10.00 kB)  x   xDownloading xorg-x11-libXfixes-32bit (download size 10.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pm-32bit (download size 34.00 kB)   x   xDownloading xorg-x11-libXpm-32bit (download size 34.00 kB) q   ±47  94%  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± render-32bit (download size 18.00 kB)  x   xDownloading xorg-x11-libXrender-32bit (download size 18.00 kB) q   89%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtiff3-32bit (download size 116.00 kB)   x   xDownloading libtiff3-32bit (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pam-32bit (download size 174.00 kB)   x   xDownloading pam-32bit (download size 174.00 kB) q   9  18  27  36  ±±45  55%  64%  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fontconfig-32bit (download size 102.00 kB)  x   xDownloading fontconfig-32bit (download size 102.00 kB) q   15  31  ±46  62%  78  94  100  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd (download size 1.80 MB)   x   xDownloading sssd (download size 1.80 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu xorg-x11-libfontenc-32bit (download size 14.00 kB)  x   xDownloading xorg-x11-libfontenc-32bit (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xkbfile6  x   xDownloading xorg-x11-libxkbfile-32bit (download size 64.00 kB) q   24  50 75%  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xt-32bit (download size 148.00 kB)   x   xDownloading xorg-x11-libXt-32bit (download size 148.00 kB) q   1  21  32  ±±±±43  54%  64%  75  86  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ext-32bit (download size 26.00 kB)  x   xDownloading xorg-x11-libXext-32bit (download size 26.00 kB) q   62%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pcre-32bit (download size 98.00 kB)   x   xDownloading pcre-32bit (download size 98.00 kB) q   16  32  48 65%  81  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-lang (download size 301.00 kB)  x   xDownloading gnome-keyring-lang (download size 301.00 kB) q   5  10  5  21  6  31  7  ±±±±42  ±47  53%  8%  63%  8  74  9  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± conf2-branding-SLES (download size 101.00 kB)  x   xDownloading gconf2-branding-SLES (download size 101.00 kB) q   13  29  ±±45  61%  77  93  100   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cups-libs-32bit (download size 192.00 kB)   x   xDownloading cups-libs-32bit (download size 192.00 kB) q   8  16  24  33  ±±±±±41  49 58%  66%  75  83  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-32bit (download size 35.00 kB)   x   xDownloading sssd-32bit (download size 35.00 kB) q   ±±45  92%  100  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ConsoleKit-32bit (download size 15.00 kB)  x   xDownloading ConsoleKit-32bit (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXp-32bit (download size 16.00 kB)  x   xDownloading xorg-x11-libXp-32bit (download size 16.00 kB) q   100%   40 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mu-32bit (download size 47.00 kB)  x   xDownloading xorg-x11-libXmu-32bit (download size 47.00 kB) q   34  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring (download size 558.00 kB)   x   xDownloading gnome-keyring (download size 558.00 kB) q   5  11  7  22  8  34  40  ±±45  51 7%  63%  8  74  80  5  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXprintUtil-32bit (download size 16.00 kB)  x   xDownloading xorg-x11-libXprintUtil-32bit (download size 16.00 kB) q   98%  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs (download size 176.00 kB)   x   xDownloading gvfs (download size 176.00 kB) q   8  1  27  36  ±±45  54%  63%  72  81  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgvfscommon0 (download size 54.00 kB)  x   xDownloading libgvfscommon0 (download size 54.00 kB) q   29  58%  8  100   ±±±±42 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-lang (download size 343.00 kB)   x   xDownloading gvfs-lang (download size 343.00 kB) q   9  18  23  32  7  ±46  51 60%  5  70  9  84  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse26.00 kB)   x   xDownloading gvfs-fuse (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu xorg-x11-libXv-32bit (download size 11.00 kB)  x   xDownloading xorg-x11-libXv-32bit (download size 11.00 kB) q   100%  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libglib-2_0-0-32bit (download size 294  x   xDownloading libglib-2_0-0-32bit (download size 294.00 kB) q   5  10  6  21  7  32  8  ±±±±43  48 54%  9%  65  70  6  81  7  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± soup-2_4-1 (download size 176.00 kB)   x   xDownloading libsoup-2_4-1 (download size 176.00 kB) q   8  1  27  36  ±±45  54%  63%  72  81  90  10   ±±±44 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libs-32bit (download size 668.00 kB)  x   xDownloading xorg-x11-libs-32bit (download size 668.00 kB) q   7  12  9  24  31  6  ±±±±43  48 55%  60%  7  72  9  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgmodule-2_0-0-32bit (download size 39.00 kB)  x   xDownloading libgmodule-2_0-0-32bit (download size 39.00 kB) q   4  81%  100   ±±45 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± thread40  x   xDownloading libgthread-2_0-0-32bit (download size 40.00 kB) q   39  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± object123.00 kB)  x   xDownloading libgobject-2_0-0-32bit (download size 123.00 kB) q   12  25  39  52% 65%  78  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± io-2_0-0-32bit (download size 242.00 kB)   x   xDownloading libgio-2_0-0-32bit (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% 9%  66  72  9  85  92  9  100   ±46 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu croco-0_6-3-32bit (download size 77.00 kB)  x   xDownloading libcroco-0_6-3-32bit (download size 77.00 kB) q   2  ±±±±±41  62%  83  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-backends (download size 532.00 kB)   x   xDownloading gvfs-backends (download size 532.00 kB) q   5  11  8  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cairo-32bit (download size 224.00 kB)   x   xDownloading cairo-32bit (download size 224.00 kB) q   7  14  21  8  35  ±±±±42  50 7%  64%  71  8  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libatk-1_0-0-32bit (download size 39.00 kB)  x   xDownloading libatk-1_0-0-32bit (download size 39.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gsf-1-11492  x   xDownloading libgsf-1-114-32bit (download size 92.00 kB) q   17  34  52% 69%  87  100   48 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu pango-32bit (download size 272.00 kB)   x   xDownloading pango-32bit (download size 272.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±46  52% 8%  64%  70  6  82  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-32bit (download size 1.93 MB)   x   xDownloading gtk2-32bit (download size 1.93 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-48d2195axxx tu librsvg-32bit (download size 105.00 kB)  x   xDownloading librsvg-32bit (download size 105.00 kB) q   15  30  ±±45  60%  76  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-32bit (download size 156.00 kB)  x   xDownloading gtk2-engines-32bit (download size 156.00 kB) q   1  2  3  4  51 61%  7  82  9  100  50 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.50 (Remaining: 129.93 MB)  x   xInstalling desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  gtk2-engines-32bit (download size 156.00 kB) Installing desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.5 ~0 MB) Additional rpm output: Updating etc/sysconfig/windowmanager...  xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 105.43 MBx 119x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 30.89 MBx56x http-10.9.0.10-48d2195axxx tu 9%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 882 05.4 Additional rpm output: Updating etc/sysconfig/windowmanager...   Installing sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 8  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  3  9  ±47  57%  64%  9  74  80  6  93  9  100  03 5 4.57  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± audit-1.8-0.34.26.x86_64.rpm (installed size 927.00 kB)  4.57   Installing sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 8 ~82.00 kB) audit-1.8-0.34.26.x86_64.rpm (installed size 927.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  22  9  36  ±47  54%  63%  70  81  6  92  7  100  Updating etc/sysconfig/auditd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  29.12 4 3.66  7 60  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.00 3.66 insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Installing audit-libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.0  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  61%  99  100  8.98 3 51  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse-2.8.7-0.11.1.x86_64.rpm (installed size 170.00 kB)  51   Installing audit-libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.0 ~0 kB) fuse-2.8.7-0.11.1.x86_64.rpm (installed size 170.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  25  63%  80  9  9  100  setting /usr/bin/wall to root:tty 2755. (wrong permissions 0755) setting /usr/bin/write to root:tty 2755. (wrong permissions 0755) setting /usr/sbin/zypp-refresh-wrapper to root:root 4755. (wrong permiss ions 0755) Finished.  81 2 35  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-2.16.1-4.26.x86_64.rpm (installed size 1.10 MB) 35 ~ions 0755) Finished.   Installing gtk2-engines-2.16.1-4.26.x86_64.rpm (installed size 1.10 MB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  9  25  31  6  ±±±±±41  ±46  51 61%  6  72  9  85  94  9  100  7.7 1 2.24  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± keyutils-libs-32bit-1.2-107.29.15.x86_64.rpm (installed size 10 2.24  x   xInstalling keyutils-libs-32bit-1.2-107.29.15.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  97%  100  0 0 3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libbluetooth3-4.99-0.12.35.x86_64.rpm (installed size 203.00 kB 3  x   x~10.00 kB) x    xInstalling libbluetooth3-4.99-0.12.35.x86_64.rpm (installed size 203.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  54%  64%  9  9  100  5 49 0  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cdio7-0.80-6.11.x86_64.rpm (installed size 126.00 kB)  0  x   x~kB) x    xInstalling libcdio7-0.80-6.11.x86_64.rpm (installed size 126.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  51 99%  100  37 8 1.91  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± db-4_5-32bit-4.5.20-95.39.x86_64.rpm (installed size 1.21 MB 1.91  x   xInstalling libdb-4_5-32bit-4.5.20-95.39.x86_64.rpm (installed size 1.21 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±±41  ±46  51 7%  62%  7  72  7  82  8  93  8  100  6.1 7 0.7  0 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± exif-0.6.17-2.14.1.x86_64.rpm (installed size 938.00 kB)  0.7  x   x~MB) x    xInstalling libexif-0.6.17-2.14.1.x86_64.rpm (installed size 938.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  5  31  7  ±±±±42  ±47  56%  66%  7  83  8  95  100  5.25 6  99.79  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse2-2.8.7-0.1121 99.79 MB)   x   xInstalling libfuse2-2.8.7-0.11.1.x86_64.rpm (installed size 218.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  58%  8  99  100  04 5 58  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± idl-0.8.13-1.1.57.x86_64.rpm (installed size 258.00 kB)  58  x   xInstalling libidl-0.8.13-1.1.57.x86_64.rpm (installed size 258.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 67%  7  92  7  100  4.79 4 32  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jpeg-32bit-6.2.0-879.12.1.x86_64.rpm (installed size 138.00 32  x   xInstalling libjpeg-32bit-6.2.0-879.12.1.x86_64.rpm (installed size 138.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  92%  9  100  65 3 19  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pixman-1-0-32bit-0.24.4-0.15.1.x86_64.rpm (installed size 60 19  x   x~0 kB) x    xInstalling libpixman-1-0-32bit-0.24.4-0.15.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±42  53%  63%  74  85  9  100  06 2 8.60  5 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± roxy0-0.3.1-2.6.1.x86_64.rpm (installed size 66.00 kB)  8.60  x   x~602.00 kB) x    xInstalling libproxy0-0.3.1-2.6.1.x86_64.rpm (installed size 66.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  56%  65%  7  84  99  100  0 1 54  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± selinux1-32bit-2.0.91-4.2.1.x86_64.rpm (installed size 115.0 54  x   xInstalling libselinux1-32bit-2.0.91-4.2.1.x86_64.rpm (installed size 115q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  56%  99%  100  3.89 0 42  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± uuid1-32bit-2.19.1-6.72.1.x86_64.rpm (installed size 18.00 k 42  x   x~.00 kB) x    xInstalling libuuid1-32bit-2.19.1-6.72.1.x86_64.rpm (installed size 18.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7 39 1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcrypt-32bit-3.0.3-0.6.1.x86_64.rpm (installed size 131 1  x   x~ kB) x    xInstalling libxcrypt-32bit-3.0.3-0.6.1.x86_64.rpm (installed size 131.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  2  ±±±±43  59%  76  8  99  100  74 8 28  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± python-argparse-1.2.1-2.5.65.x86_64.rpm (installed size 158.00  28  x   x~ kB) x    xInstalling python-argparse-1.2.1-2.5.65.x86_64.rpm (installed size 158.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  57%  99%  100  59 7 12  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libICE-32bit-7.4-1.15.x86_64.rpm (installed size 95.00 12  x   x~0 kB) x    xInstalling xorg-x11-libICE-32bit-7.4-1.15.x86_64.rpm (installed size 95.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  99  100  0 6 03  99 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xau10 03  x   x~00 kB) x    xInstalling xorg-x11-libXau-32bit-7.4-1.15.x86_64.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  49 5 2  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-theme-SLED-11.1-1.29.82.noarch.rpm (installed size 623.00  2  x   x~00 kB) x    xInstalling gtk2-theme-SLED-11.1-1.29.82.noarch.rpm (installed size 623.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  2  6  32  7  ±±45  51 8%  63%  8  73  8  84  91  9  100  2.88 4 7.41  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_cdda0-0.80-6.11.x86_64.rpm (installed size 31.00 kB)  7.41  x   x~0 kB) x    xInstalling libcdio_cdda0-0.80-6.11.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  5 3 38  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-lang-2.4.3-3.27.1.x86_64.rpm (installed size 1.64 MB 38  x   xInstalling libgphoto2-lang-2.4.3-3.27.1.x86_64.rpm (installed size 1.64 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  7  32  7  ±±±±42  ±47  53%  61%  6  71  6  83  9  95  100  1.21 2 5.75  5 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± orbit2-2.14.17-2.7.8.x86_64.rpm (installed size 622.00 kB)  5.75  x   x~MB) x    xInstalling orbit2-2.14.17-2.7.8.x86_64.rpm (installed size 622.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  2  3  ±±±±4  53%  63%  7  80  93  9  100  0.60 1 14  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libproxy0-config-gnome-0.3.1-2.6.3.x86_64.rpm (installed size 1 14  x   xInstalling libproxy0-config-gnome-0.3.1-2.6.3.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  59 0 2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libSM-32bit-7.4-1.18.x86_64.rpm (installed size 30.00  2  x   x~ 15.00 kB) x    xInstalling xorg-x11-libSM-32bit-7.4-1.18.x86_64.rpm (installed size 30.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  6 29 09  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcb-32bit-7.4-1.29.1.x86_64.rpm (installed size 511 09  x   x~0 kB) x    xInstalling xorg-x11-libxcb-32bit-7.4-1.29.1.x86_64.rpm (installed size 5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  5  31  40  ±46  52% 60%  8  75  90  9  100  0 8 4.60  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-branding-SLED-11.1-1.26.noarch.rpm (installed size 2.00 kB 4.60  x   x~11.00 kB) x    xInstalling gtk2-branding-SLED-11.1-1.26.noarch.rpm (installed size 2.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  6  95%  100  7 59  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_paranoia0-0.80-6.11.x86_64.rpm (installed size 31 59  x   x~kB) x    xInstalling libcdio_paranoia0-0.80-6.11.x86_64.rpm (installed size 31.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 6 6  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-2.4.3-3.27.1.x86_64.rpm (installed size 5.96 MB)  6  x   x~kB) x    xInstalling libgphoto2-2.4.3-3.27.1.x86_64.rpm (installed size 5.96 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  61%  6  71  7  82  8  93  8  100  14.06 5 88.60  8 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 88.60 MBx 88x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x http-10.9.0.10-48d2195axxx tu openobex1-1.5-8.9.7.x86_64.rpm (installed size 86.00 k 88.60  x   xInstalling libopenobex1-1.5-8.9.7.x86_64.rpm (installed size 86.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  65%  9  100  SizexPackagesxTi88.60 MBx 88x x x 14.06 MBx 25x x x qu 51  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzmadec0-4.32.7-1.25.x86_64.rpm (installed size 15.00 kB) 51  x   xInstalling liblzmadec0-4.32.7-1.25.x86_64.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± obex-data-server-0.4.6-2.7.90.x86_64.rpm (installed size 279.00 0  x   xInstalling obex-data-server-0.4.6-2.7.90.x86_64.rpm (installed size 279.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±46  56%  62%  83  8  99  100  23  5 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzma-4.32.7-1.25.x86_64.rpm (installed size 133.00 kB)  23  x   x~00 kB) x    xInstalling lzma-4.32.7-1.25.x86_64.rpm (installed size 133.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  51 77%  85  92  8  100  10  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTiw Total x 88.10 MBx 84x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x v http-10.9.0.10-6075eb91xxx tu ibtasn1-3-1.5-1.34.1.x86_64.rpm (installed size 144.00 kB) 10  x   xInstalling libtasn1-3-1.5-1.34.1.x86_64.rpm (installed size 144.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  57%  81%  99  100  7.96  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssh-helpers-6.6p1-35.1.x86_64.rpm (installed size 66.00 kB) 7.96  x   xInstalling openssh-helpers-6.6p1-35.1.x86_64.rpm (installed size 66.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  91%  7  100  89  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± readline-doc-5.2-147.29.1.x86_64.rpm (installed size 590 89  x   x~B) x    xInstalling readline-doc-5.2-147.29.1.x86_64.rpm (installed size 590.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  6  27  38  49 60%  7  81  9  9  100  32  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcares2-1.7.4-7.9.1.x86_64.rpm (installed size 63.00 kB)  32  x   x~B) x    xInstalling libcares2-1.7.4-7.9.1.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  25  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± om_err2-32bit-1.41.9-2.16.1.x86_64.rpm (installed size 36.0 25  x   xInstalling libcom_err2-32bit-1.41.9-2.16.1.x86_64.rpm (installed size 36q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  99%  100  2 79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnutls26-2.4.1-24.39.67.1.x86_64.rpm (installed size 745.00  2  x   x~.00 kB) x    xInstalling libgnutls26-2.4.1-24.39.67.1.x86_64.rpm (installed size 745.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  34  ±±±±43  51 60%  8  77  85  94  9  100  6.49  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± expat1-32bit-2.0.1-88.41.1.x86_64.rpm (installed size 158.00 6.49  x   x~0 kB) x    xInstalling libexpat1-32bit-2.0.1-88.41.1.x86_64.rpm (installed size 158.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  81%  99  100  34  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gcc_s5.3.1+r233831-10.1.x86_64.rpm (installed size 1 34  x   x~00 kB) x    xInstalling libgcc_s1-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  99  100  23  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dbus-1-32bit-1.2.10-3.33.1.x86_64.rpm (installed size 259.00 kB 23  x   x~ 106.00 kB) x    xInstalling dbus-1-32bit-1.2.10-3.33.1.x86_64.rpm (installed size 259.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 74%  99  100  5.98  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libavahi-common3-0.6.23-11.34.3.x86_64.rpm (installed size 51.0 5.98  x   x~kB) x    xInstalling libavahi-common3-0.6.23-11.34.3.x86_64.rpm (installed size 51q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ref_array1-0.1.1-1.9.1.x86_64.rpm (installed size 11.00 kB)  3  x   x~.00 kB) x    xInstalling libref_array1-0.1.1-1.9.1.x86_64.rpm (installed size 11.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhash1-0.4.2-1.9.1.x86_64.rpm (installed size 15.00 kB)  2  x   x~) x    xInstalling libdhash1-0.4.2-1.9.1.x86_64.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bz2-1-32bit-1.0.5-34.255.1.x86_64.rpm (installed size 65.00 1  x   xInstalling libbz2-1-32bit-1.0.5-34.255.1.x86_64.rpm (installed size 65.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  84  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sss_idmap0-1.9.4-0.33.1.x86_64.rpm (installed size 19.00 kB) 84  x   x~0 kB) x    xInstalling libsss_idmap0-1.9.4-0.33.1.x86_64.rpm (installed size 19.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vim-data-7.2-8.17.1.x86_64.rpm (installed size 19.53 MB)  2  x   x~B) x    xInstalling vim-data-7.2-8.17.1.x86_64.rpm (installed size 19.53 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  66.30 69 74  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libwbclient0-3.6.3-87.1.x86_64.rpm (installed size 51.00 kB) 66.30  x   xInstalling libwbclient0-3.6.3-87.1.x86_64.rpm (installed size 51.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  25  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± augeas-lenses-0.9.0-3.17.2.x86_64.rpm (installed size 293.00 kB 25  x   xInstalling augeas-lenses-0.9.0-3.17.2.x86_64.rpm (installed size 293.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  9  24  30  5  40  ±±45  50 5%  60%  5  70  9  86  91  6  100  5.96  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtdb1-3.6.3-87.1.x86_64.rpm (installed size 67.00 kB)  5.96  x   x~kB) x    xInstalling libtdb1-3.6.3-87.1.x86_64.rpm (installed size 67.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  96%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zlib-32bit-1.2.7-0.14.1.x86_64.rpm (installed size 86.00 kB) 0  x   xInstalling zlib-32bit-1.2.7-0.14.1.x86_64.rpm (installed size 86.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  99  100  81  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcollection2-0.6.0-1.9.1.x86_64.rpm (installed size 47.00 kB) 81  x   xInstalling libcollection2-0.6.0-1.9.1.x86_64.rpm (installed size 47.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  77  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± archive2-2.5.5-9.1.x86_64.rpm (installed size 245.00 kB)  77  x   x~B) x    xInstalling libarchive2-2.5.5-9.1.x86_64.rpm (installed size 245.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 74%  8  91  6  100  53  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bash-doc-3.2-147.21.29 MB)  53  x   xInstalling bash-doc-3.2-147.29.1.x86_64.rpm (installed size 1.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  6  31  7  ±±±±42  51 6%  61%  6  71  80  5  91  9  100  4.2  2 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtevent0-3.6.3-8747.00 kB) 4.2  x   xInstalling libtevent0-3.6.3-87.1.x86_64.rpm (installed size 47.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  19  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± perl-doc-5.10.0-64.80.1.x86_64.rpm (installed size 9.78 MB) 19  x   xInstalling perl-doc-5.10.0-64.80.1.x86_64.rpm (installed size 9.78 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  54.40  0 9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpath_utils1-0.2.1-1.9.1.x86_64.rpm (installed size 15.00 kB) 54.40  x   xInstalling libpath_utils1-0.2.1-1.9.1.x86_64.rpm (installed size 15.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  39 59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zypper-log-1.6.334-26.3.x86_64.rpm (installed size 5.00 kB)  39  x   x~B) x    xInstalling zypper-log-1.6.334-26.3.x86_64.rpm (installed size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  87%  9  100  8  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-lang-2.28.0-1.8.1.x86_64.rpm (installed size 5.14 MB) 8  x   xInstalling gconf2-lang-2.28.0-1.8.1.x86_64.rpm (installed size 5.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±46  51 6%  61%  7  73  8  83  8  93  8  100  49.25  7 81  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libX11-32bit-7.4-5.11.68.1.x86_64.rpm (installed size 49.25  x   xInstalling xorg-x11-libX11-32bit-7.4-5.11.68.1.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±42  ±47  52% 7%  62%  7  73  8  83  8  93  9  100  8.04  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtasn1-1.5-1.34.1.x86_64.rpm (installed size 197.00 kB)  8.04  x   x~e 1.20 MB) x    xInstalling libtasn1-1.5-1.34.1.x86_64.rpm (installed size 197.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  31  48 61%  9  80  5  90  5  100  7.85  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± krb5-32bit-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.39 M 7.85  x   xInstalling krb5-32bit-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.39q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  8  25  30  6  ±±±±±41  ±46  53%  8%  67%  75  80  9  97  100  6.46  4 2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libstdc++6-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed size  6.46  x   x~ MB) x    xInstalling libstdc++6-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  30  8  ±±±±43  51 6%  64%  73  81  6  94  9  100  5.01  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± avahi-glib1-0.6.23-13.34.8.x86_64.rpm (installed size 15.00 5.01  x   x~e 1.45 MB) x    xInstalling libavahi-glib1-0.6.23-13.34.8.x86_64.rpm (installed size 15.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4.99  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± client3-0.6.23-11.34.3.x86_64.rpm (installed size 68.0 4.99  x   x~0 kB) x    xInstalling libavahi-client3-0.6.23-11.34.3.x86_64.rpm (installed size 68q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  95%  100  2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssl0_9_8-32bit-0.9.8j-0.105.1.x86_64.rpm (installed size 2  x   x~.00 kB) x    xInstalling libopenssl0_9_8-32bit-0.9.8j-0.105.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  23  9  35  ±±±±±41  48 54%  60%  7  73  9  86  93  9  100  2.94  0 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cracklib-32bit-2.8.12-56.13.1.x86_64.rpm (installed size 34.00  2.94  x   x~ze 1.98 MB) x    xInstalling cracklib-32bit-2.8.12-56.13.1.x86_64.rpm (installed size 34.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpng12-0-32bit-1.2.31-5.43.1.x86_64.rpm (installed size 162.0 1  x   x~0 kB) x    xInstalling libpng12-0-32bit-1.2.31-5.43.1.x86_64.rpm (installed size 162q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  79%  9  100  75  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xml2-32bit-2.7.6-0.64.1.x86_64.rpm (installed size 1.32 MB)  75  x   x~.00 kB) x    xInstalling libxml2-32bit-2.7.6-0.64.1.x86_64.rpm (installed size 1.32 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  23  8  33  ±±±±42  ±47  52% 61%  6  75  80  5  94  9  100  1.43  7 4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± freetype2-32bit-2.3.7-25.44.1.x86_64.rpm (installed size 511.00 1.43  x   x~) x    xInstalling freetype2-32bit-2.3.7-25.44.1.x86_64.rpm (installed size 511.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  37  50 62%  75  87  99  100  0.9  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libldb1-3.6.3-87.1.x86_64.rpm (installed size 360.00 kB)  0.9  x   x~00 kB) x    xInstalling libldb1-3.6.3-87.1.x86_64.rpm (installed size 360.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  22  8  34  50 68%  86  97  100  58  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± smbclient0-3.6.3-87.1.x86_64.rpm (installed size 5.74 MB) 58  x   xInstalling libsmbclient0-3.6.3-87.1.x86_64.rpm (installed size 5.74 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  6  91  6  100  34.84  4 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ini_config2-0.6.1-1.9.1.x86_64.rpm (installed size 35.00 kB) 34.84  x   xInstalling libini_config2-0.6.1-1.9.1.x86_64.rpm (installed size 35.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-2.28.0-1.8.1.x86_64.rpm (installed size 911.00 kB)  1  x   x~B) x    xInstalling gconf2-2.28.0-1.8.1.x86_64.rpm (installed size 911.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  8  25  32  8  ±±45  52% 9%  65  71  8  85  92  7  100   x   xAdditional rpm output: x    xMust specify some schema files to install q  3.92  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed size 3.92 Additional rpm output: Must specify some schema files to install   Installing xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed siz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pm-32bit-7.4-3.1.x86_64.rpm (installed size 67.00 0   Installing xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed siz ~e 18.00 kB)pm-32bit-7.4-3.1.x86_64.rpm (installed size 67.0  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  96%  100  83  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± render-32bit-7.4-1.23.1.x86_64.rpm (installed size 83  x   x~0 kB) x    xInstalling xorg-x11-libXrender-32bit-7.4-1.23.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtiff3-32bit-3.8.2-141.168.1.x86_64.rpm (installed size 372.0 0  x   x~ze 34.00 kB) x    xInstalling libtiff3-32bit-3.8.2-141.168.1.x86_64.rpm (installed size 372q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  34  51 68%  85  97  100  xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 33.44 MBx 38x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x v http-10.9.0.10-6075eb91xxx tu 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pam-32bit-1.1.5-0.17.2.x86_64.rpm (installed size 603.00 kB)  44  x   x~.00 kB) x    xInstalling pam-32bit-1.1.5-0.17.2.x86_64.rpm (installed size 603.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  21  6  32  7  ±±±±42  ±47  54%  9%  65  70  7  83  8  99  100  2.85  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fontconfig-32bit-2.6.0-10.19.1.x86_64.rpm (installed size 234.0 2.85  x   xInstalling fontconfig-32bit-2.6.0-10.19.1.x86_64.rpm (installed size 234q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  ±±45  73%  99  100  62  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-1.9.4-0.33.1.x86_64.rpm (installed size 7.48 MB)  62  x   x~.00 kB) x    xInstalling sssd-1.9.4-0.33.1.x86_64.rpm (installed size 7.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  25.14  5 90  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 25.14 MBx 35x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x v http-10.9.0.10-6075eb91xxx tu xorg-x11-libfontenc-32bit-7.4-1.15.x86_64.rpm (installed size 2 25.14  x   xInstalling xorg-x11-libfontenc-32bit-7.4-1.15.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 4 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xkbfile41 2  x   x~ 24.00 kB) x    xInstalling xorg-x11-libxkbfile-32bit-7.4-1.14.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  87%  99  100  3.89 3 4.97  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xt-32bit-7.4-1.19.8.x86_64.rpm (installed size 360. 4.97  x   x~ 147.00 kB) x    xInstalling xorg-x11-libXt-32bit-7.4-1.19.8.x86_64.rpm (installed size 36q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  71%  89  9  100  54 2 62  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ext-32bit-7.4-1.18.16.x86_64.rpm (installed size 6 62  x   x~0.00 kB) x    xInstalling xorg-x11-libXext-32bit-7.4-1.18.16.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  48 1 56  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pcre-32bit-7.8-2.18.x86_64.rpm (installed size 241.00 kB)  56  x   x~ 63.00 kB) x    xInstalling pcre-32bit-7.8-2.18.x86_64.rpm (installed size 241.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  79%  95  100  25 0 32  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-lang-2.28.2-1.14.30.x86_64.rpm (installed size 2. 32  x   xInstalling gnome-keyring-lang-2.28.2-1.14.30.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  32  7  ±±±±42  48 53%  8%  64%  9  74  9  84  90  6  100  1.09 19 2.16 29 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± conf2-branding-SLES-2.25-0.13.31.noarch13 2.16  x   x~2.16 MB) x    xInstalling gconf2-branding-SLES-2.25-0.13.31.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  22  ±±±±4  53%  99%  100  0.95 8 03  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cups-libs-32bit-1.3.9-8.46.56.1.x86_64.rpm (installed size 336. 03  x   x~138.00 kB) x    xInstalling cups-libs-32bit-1.3.9-8.46.56.1.x86_64.rpm (installed size 33q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  39  58%  69%  88  99  100  62 7 1.70  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-32bit-1.9.4-0.16.1.x86_64.rpm (installed size 60.00 kB)  1.70  x   x~6.00 kB) x    xInstalling sssd-32bit-1.9.4-0.16.1.x86_64.rpm (installed size 60.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 99%  100  56 6 64  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ConsoleKit-32bit-0.2.10-64.67.1.x86_64.rpm (installed size 23.0 64  x   xInstalling ConsoleKit-32bit-0.2.10-64.67.1.x86_64.rpm (installed size 23q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±±41  99%  100  4 5 2  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXp-32bit-7.4-1.18.734 2  x   x~.00 kB) x    xInstalling xorg-x11-libXp-32bit-7.4-1.18.7.x86_64.rpm (installed size 34q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 4 59  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mu-32bit-7.4-1.17.x86_64.rpm (installed size 109 59  x   x~.00 kB) x    xInstalling xorg-x11-libXmu-32bit-7.4-1.17.x86_64.rpm (installed size 109q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  87%  99  100  40 3 48  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-2.28.2-1.14.30.x86_64.rpm (installed size 2.57 MB 48  x   x~.00 kB) x    xInstalling gnome-keyring-2.28.2-1.14.30.x86_64.rpm (installed size 2.57 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  33  8  ±±±±43  48 53%  60%  5  70  5  82  7  93  8  100   7.83 2 18.91  2 2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXprintUtil-32bit-7.4-1.17.x86_64.rpm (installed siz 18.91  x   x~MB) x    xInstalling xorg-x11-libXprintUtil-32bit-7.4-1.17.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 1 88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-1.4.3-0.17.21.1.x86_64.rpm (installed size 614.00 kB)  88  x   x~ize 34.00 kB) x    xInstalling gvfs-1.4.3-0.17.21.1.x86_64.rpm (installed size 614.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  8  33  ±±±44  56%  67%  73  8  9  8  100  2 0 2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgvfscommon0-1.4.3-0.17.21.1.x86_64.rpm (installed size 128.0 2  x   xInstalling libgvfscommon0-1.4.3-0.17.21.1.x86_64.rpm (installed size 128q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  74%  99  100  07  9 15 19 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-lang-1.4.3-0.17.21.1.x86_64.rpm (installed size 2.10 MB)  15  x   x~.00 kB) x    xInstalling gvfs-lang-1.4.3-0.17.21.1.x86_64.rpm (installed size 2.10 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  33  8  ±±±±43  48 54%  9%  65  70  5  80  6  92  8  100  4.9 8 6.0  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse32.00 kB) 6.0  x   xInstalling gvfs-fuse-1.4.3-0.17.21.1.x86_64.rpm (installed size 32.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 7 2  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 16.02 MBx 17x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.94 MBx7x v http-10.9.0.10-6075eb91xxx tu xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18.0 2  x   x~) x    xInstalling xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  SizexPackagesxTim16.00 MBx 17x x x 4.94 MBx 7x x x qqu 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libglib-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed size  0 Installing gvfs-fuse-1.4.3-0.17.21.1.x86_64.rpm (installed size 32.00 kB ~) xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18 .00 kB) ng libglib-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed siz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  64%  72  80  9  97  100  5.23 5 4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± soup-2_4-1-2.32.2-4.15.1.x86_64.rpm (installed size 468.00 k 5.23  x   x~e 791.00 kB) x    xInstalling libsoup-2_4-1-2.32.2-4.15.1.x86_64.rpm (installed size 468.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  27  ±±±±±41  54%  68%  76  81  7  99  100  4.77 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libs-32bit-7.4-8.26.49.1.x86_64.rpm (installed size 2. 4.77  x   x~ kB) x    xInstalling xorg-x11-libs-32bit-7.4-8.26.49.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  8  33  9  ±±45  50 7%  62%  7  72  7  82  7  92  7  100  2.45 3 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgmodule-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed si 2.45  x   x~2.33 MB) x    xInstalling libgmodule-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± thread 4  x   x~size 14.00 kB) x    xInstalling libgthread-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± object 2  x   x~size 18.00 kB) x    xInstalling libgobject-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 74%  99  100  17 0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± io-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed size 6 17  x   x~size 259.00 kB) x    xInstalling libgio-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  3  ±47  57%  6  76  8  95  100  1.51  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 11.51 MBx 9x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.94 MBx7x v http-10.9.0.10-6075eb91xxx tu croco-0_6-3-32bit-0.6.1-120.16.x86_64.rpm (installed size 23 1.51  x   x~ 668.00 kB) x    xInstalling libcroco-0_6-3-32bit-0.6.1-120.16.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  82%  99  100  72 6 SizexPackagesxTim11.29 MBx 9x x x 4.72 MBx 6x x x qqu 8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-backends-1.4.3-0.17.21.1.x86_64.rpm (installed size 3.06 M 29  x   x~233.00 kB) x    xInstalling gvfs-backends-1.4.3-0.17.21.1.x86_64.rpm (installed size 3.06q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  33  8  ±±±44  50 7%  63%  9  74  9  85  90  5  100  1.65 5  8.22 7 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cairo-32bit-1.8.8-2.1.48.x86_64.rpm (installed size 515.00 kB)  8.22 MB)   x   x~ MB) x    xInstalling cairo-32bit-1.8.8-2.1.48.x86_64.rpm (installed size 515.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  24  37  49 62%  74  87  99  100  1 4 7.7 6 7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libatk-1_0-0-32bit-1.28.0-1.4.1.x86_64.rpm (installed size 115. 7.7  x   x~) x    xInstalling libatk-1_0-0-32bit-1.28.0-1.4.1.x86_64.rpm (installed size 11q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  56%  99%  100  04 3 61 5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gsf-1-11414.9-2.27.x86_64.rpm (installed size 232.0 61  x   x~5.00 kB) x    xInstalling libgsf-1-114-32bit-1.14.9-2.27.x86_64.rpm (installed size 232q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  82%  99  100  832.00 k 2 38 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTw Total x 7.38 MBx 4x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 832.00 kBx2x v http-10.9.0.10-6075eb91xxx tu pango-32bit-1.26.2-1.7.1.x86_64.rpm (installed size 789.00 kB)  38  x   x~.00 kB) x    xInstalling pango-32bit-1.26.2-1.7.1.x86_64.rpm (installed size 789.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  25  33  ±±±±4  51 9%  6  74  8  90  5  100  6.61  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-32bit-2.18.9-0.445.80 MB)  6.61  x   x~) x    xInstalling gtk2-32bit-2.18.9-0.44.1.x86_64.rpm (installed size 5.80 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 7%  62%  7  72  8  83  8  93  8  100  xSizexPackagesxTime  x        x     qq  xSizexPackagesx Total x 832.00 kBx 2x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 832.00 kBx2x tu 9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime         qq  xSizexPackagesx Total x 832.00 kBx 2x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 832.00 kBx2x tu librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 kB 832.00 kB / 00:10)  x   xInstalling librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  ±±±±±41  65%  89  9  100  569 1 SizexPackagesx 569.00 kBx 2x 0x x 569.00 kBx 1x 0qqqu 1  Tim 00: 00: qqqqu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569.0 569)   x   x~kB) x    xInstalling gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  9  3  50 61%  72  80  91  9  100   Done.   SizexPackagesx Time  x 1x00:00 x x Done.x x00:00 qq   100   Perform Installation lqqqqk xMedia x SizexPackagesx Time x xTotal x x x00:00 x xSUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109x x x  x xMedium 1 xDone.x x00:00 x x  x mqqqqj lActions performed:qqqqk x~)  x xInstalling gtk2-32bit-2.18.9-0.44.1.x86_64.rpm (installed size 5.80 MB) x xInstalling librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 x x~kB)  x xInstalling gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569 q mqqqqj Installing gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569.00 kB   Installing Packages... (Remaining: 569.00 kB)    F10 Next 8 Back  F9 Abort  F10 Next  Configuring System according to auto-install sett   Configuring System according to auto-install settings  lk  x  x  x x  x  x  x   x  xx  x x  x x  x x  x x  x x  x x  x x  x x  x x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Progress  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± [Help] [Back] [Abort] [Next] 9 Abort   2  4  6  Configuring lan  8  Configuring users  10  2 Configuring proxy  4  6  8  20  2  4  6  8  30 Configuring firewall  2  4 Configuring host  6  8  ±±±±±40 Configuring iscsi-client  ±±±±42  ±±44  ±46  48 51 Configuring keyboard 3%  5%  7%  9%  61  3  5  7  9  71  3  5  7  Configuring runlevel 9  81  3  5  7  9  91  3  5  Configuring timezone 7  Configuring yast-language  YaST2 - installation @ trevis-41vm7    Configuring System according to auto-install settings     lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk    xConfiguring lan x    xConfiguring users x    xConfiguring proxy x    xConfiguring firewall x    xConfiguring host x    xConfiguring iscsi-client x    xConfiguring keyboard x    xConfiguring runlevel x    xConfiguring timezone x    xConfiguring yast-language x    x x    x x    x x    x x    mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj    Progress     97% ±±   [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort 100  Executing Post-Scripts Finishing Configuration 10 Nex 9 Abort  F10 Next 8 Back  F9 Abort  F10 Next  Writing the sconfiguration...               ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±9 Abort   14  Setting up linker cache...  28  Running SuSEconfig.glib2  ±±±±42 Running SuSEconfig.groff  57%  Running SuSEconfig.permissions 71%  Running SuSEconfig.scpm 85  Running SuSEconfig.gtk2 100  Finishing SuSEconfig Done. 10 Nex 9 Abort  F10 Next 8 Back  F9 Abort  F10 Next [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxFinishing the installation...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj                            [?1l>Shutting down network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service network . . . . . . . . .7[?25l done8[?25h unicode_stop skipped on /dev/ttyS0 INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid 7[?25l done8[?25h [?25hD-Bus already started. Not starting. [?25hRe-Starting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25hRemoving old kernel packages... /sbin/purge-kernels: multiversion.kernels not configured in /etc/zypp/zypp.conf, exiting. 7[?25l done8[?25h [?25hInitializing random number generator7[?25l done8[?25h [?25hHAL already started. Not starting. [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA ED25519 Starting SSH daemon7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hExecuting AutoYaST script: /var/adm/autoinstall/init.d/chroot-scripts_temp-resolver7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/init-scripts_chef-solo7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/init-scripts_cobbler7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/load-correct-kernel7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/sanitize_logs7[?25l done8[?25h insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for script `munge' [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal [2017-04-04T15:43:45+00:00] WARN: ***************************************** [2017-04-04T15:43:45+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-04T15:43:45+00:00] WARN: ***************************************** Boot logging started on /dev/ttyS0((no tty)) at Tue Apr 4 15:43:46 2017 Master Resource Control: previous runlevel: 3, switching to runlevel: 6 Shutting down acpid 7[?25l done8[?25h [?25hNot unloading kdump during runlevel changes skipped [?25hShutting down CRON daemon7[?25l done8[?25h [?25hShutting down irqbalance 7[?25l done8[?25h [?25hShutting down mcelog... 7[?25l done8[?25h [?25h[?25hShutting down Name Service Cache Daemon7[?25l done8[?25h [?25h[?25hShutting down smartd 7[?25l done8[?25h [?25h[?25h[?25hShutting down SSH daemon *with all active connections*7[?25l done8[?25h [?25hShutting down auditd 7[?25l done8[?25h [?25h[?25hShutting down haveged daemon 7[?25l done8[?25h [?25h/etc/init.d/rc3.d/K02kbd stop7[?25l done8[?25h [?25hShutting down mail service (Postfix)7[?25l done8[?25h [?25hShutting down (remotefs) network interfaces: Shutting down service (remotefs) network . . . . . . . . .7[?25l done8[?25h [?25hSaving random seed7[?25l done8[?25h [?25hShutting down HAL daemon7[?25l done8[?25h [?25hShutting down NFS client services:7[?25l done8[?25h [?25hShutting down rpcbind 7[?25l done8[?25h [?25hShutting down syslog services7[?25l done8[?25h [?25h[?25hShutting down (localfs) network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service (localfs) network . . . . . . . . .7[?25l done8[?25h [?25hUnloading HCA driver:[ OK ] [?25hShutting down D-Bus daemon7[?25l done8[?25h [?25hRunning /etc/init.d/halt.local 7[?25l done8[?25hFuse filesystem not loaded7[?25l done8[?25h Not unloading kdump during runlevel changes skipped 7[?25l done8[?25h Turning off quota 7[?25l done8[?25h 7[?25l done8[?25h Set Hardware Clock to the current System Time7[?25l done8[?25h Turning off swap files Unmounting file systems 7[?25l done8[?25hStopping udevd: 7[?25l done8[?25h Sending all processes the TERM signal...  done Sending all processes the KILL signal...  done Please stand by while rebooting the system... [ 190.001724] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-97_lustre-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Mar 30 13:45:57 UTC 2017 (b708f71) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37756000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa81, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72320 r8192 d21888 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a81, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910920k/2097140k available (4576k kernel code, 460k absent, 185760k reserved, 7571k data, 1356k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013332] Security Framework initialized [ 0.014201] AppArmor: AppArmor initialized [ 0.016053] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.018349] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020154] Mount-cache hash table entries: 256 [ 0.021308] Initializing cgroup subsys cpuacct [ 0.022223] Initializing cgroup subsys memory [ 0.023137] Initializing cgroup subsys devices [ 0.024007] Initializing cgroup subsys freezer [ 0.024903] Initializing cgroup subsys net_cls [ 0.025833] Initializing cgroup subsys blkio [ 0.026718] Initializing cgroup subsys perf_event [ 0.027793] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.032078] ACPI: Core revision 20110413 [ 0.034551] Enabling x2apic [ 0.035287] Enabled x2apic [ 0.036007] Switched APIC routing to physical x2apic. [ 0.038132] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.039222] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.060011] kvm-clock: cpu 1, msr 0:7fd10a81, secondary cpu clock [ 0.060027] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.060054] Brought up 2 CPUs [ 0.060055] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068129] devtmpfs: initialized [ 0.071059] print_constraints: dummy: [ 0.072056] Time: 15:44:32 Date: 04/04/17 [ 0.073075] NET: Registered protocol family 16 [ 0.074133] ACPI: bus type pci registered [ 0.075081] PCI: Using configuration type 1 for base access [ 0.076483] bio: create slab at 0 [ 0.080901] ACPI: Interpreter enabled [ 0.081749] ACPI: (supports S0 S5) [ 0.082675] ACPI: Using IOAPIC for interrupt routing [ 0.087251] ACPI: No dock devices found. [ 0.088009] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.089832] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.092519] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.094110] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.117848] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.123940] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.124843] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.126212] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.128074] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.129431] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.130780] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.132012] vgaarb: loaded [ 0.132984] PCI: Using ACPI for IRQ routing [ 0.133226] NetLabel: Initializing [ 0.134006] NetLabel: domain hash size = 128 [ 0.136010] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.136978] NetLabel: unlabeled traffic allowed by default [ 0.138010] Switching to clocksource kvm-clock [ 0.138910] AppArmor: AppArmor Filesystem Enabled [ 0.139247] pnp: PnP ACPI init [ 0.140047] ACPI: bus type pnp registered [ 0.141634] pnp: PnP ACPI: found 6 devices [ 0.142531] ACPI: ACPI bus type pnp unregistered [ 0.149790] NET: Registered protocol family 2 [ 0.150895] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.153002] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.156616] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.158276] TCP: Hash tables configured (established 262144 bind 65536) [ 0.159436] TCP reno registered [ 0.160210] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.161326] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.162667] NET: Registered protocol family 1 [ 0.163604] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.164696] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.165793] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.167008] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.168085] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.173439] pci 0000:00:01.2: PCI INT D disabled [ 0.174479] Unpacking initramfs... [ 0.306433] Freeing initrd memory: 8808k freed [ 0.320532] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.321713] Initialise module verification [ 0.322756] audit: initializing netlink socket (disabled) [ 0.323796] type=2000 audit(1491320673.320:1): initialized [ 0.341886] bounce pool size: 64 pages [ 0.342740] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.344726] VFS: Disk quotas dquot_6.5.2 [ 0.345619] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.346894] msgmni has been set to 937 [ 0.348098] alg: No test for stdrng (krng) [ 0.348998] Asymmetric key parser 'x509' registered [ 0.350002] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.351625] io scheduler noop registered [ 0.352474] io scheduler deadline registered (default) [ 0.353533] io scheduler cfq registered [ 0.354598] GHES: HEST is not enabled! [ 0.355463] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.356760] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.592238] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.628110] Non-volatile memory driver v1.3 [ 0.628999] Linux agpgart interface v0.103 [ 0.629922] Fixed MDIO Bus: probed [ 0.630757] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.632950] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.633915] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.634933] mousedev: PS/2 mouse device common for all mice [ 0.636202] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.638207] cpuidle: using governor ladder [ 0.639061] cpuidle: using governor menu [ 0.639893] EFI Variables Facility v0.08 2004-May-17 [ 0.641030] TCP cubic registered [ 0.641807] Registering the dns_resolver key type [ 0.642834] Loading module verification certificates [ 0.643814] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.645769] registered taskstats version 1 [ 0.646805] Magic number: 13:561:739 [ 0.649648] Freeing unused kernel memory: 1356k freed [ 0.651281] Write protecting the kernel read-only data: 10240k [ 0.655362] Freeing unused kernel memory: 1548k freed [ 0.658922] Freeing unused kernel memory: 840k freed doing fast boot [ 0.694057] Disabling lock debugging due to kernel taint [ 0.700849] SCSI subsystem initialized [ 0.723806] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.736136] scsi0 : ata_piix [ 0.737063] scsi1 : ata_piix [ 0.737841] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.739017] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.903997] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.924168] vda: vda1 vda2 [ 0.925310] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.926355] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.932754] alua: device handler registered [ 0.937299] hp_sw: device handler registered [ 0.941912] emc: device handler registered [ 0.946497] rdac: device handler registered Creating device nodes with udev [ 0.950104] udev: starting version 147 [ 0.967922] usbcore: registered new interface driver usbfs [ 0.969044] usbcore: registered new interface driver hub [ 0.988865] usbcore: registered new device driver usb [ 1.001430] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.004335] uhci_hcd: USB Universal Host Controller Interface driver [ 1.006953] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.009129] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.010150] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.011840] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.012956] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.014204] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.015912] usb usb1: Product: UHCI Host Controller [ 1.016918] usb usb1: Manufacturer: Linux 3.0.101-97_lustre-default uhci_hcd [ 1.018190] usb usb1: SerialNumber: 0000:00:01.2 [ 1.019237] hub 1-0:1.0: USB hub found [ 1.020100] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 15:44:34 2017 [ 1.051559] [drm] Initialized drm 1.1.0 20060810 [ 1.054658] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-97_lustre-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.301159] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 77591/1310720 files, 547124/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.334582] kjournald starting. Commit interval 15 seconds [ 1.336200] EXT3-fs (vda1): using internal journal [ 1.337189] EXT3-fs (vda1): mounted filesystem with ordered data mode [ 1.341053] Refined TSC clocksource calibration: 2593.647 MHz. INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 15:44:34 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.782532] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.803553] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.805807] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.807474] ACPI: Power Button [PWRF] [ 1.812585] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.813995] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.815058] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.818892] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.827049] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 1.845763] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.849798] rtc_cmos 00:01: RTC can wake from S4 [ 1.850933] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.852131] rtc0: alarms up to one day, 114 bytes nvram [ 1.858070] NET: Registered protocol family 10 [ 1.864386] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 1.905016] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.559501] device-mapper: uevent: version 1.0.3 [ 2.560542] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.614068] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 3.125282] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25h[?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid acpid: starting up 7[?25l done8[?25h [?25hacpid: 3 rules loaded Starting D-Bus daemon7[?25l done8[?25h [?25hStarting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25h[?25hInitializing random number generator7[?25l done8[?25h [?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 28 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-97_lustre-default (console). trevis-41vm7 login: [2017-04-04T15:44:53+00:00] WARN: ***************************************** [2017-04-04T15:44:53+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-04T15:44:53+00:00] WARN: ***************************************** [2017-04-04T15:44:58+00:00] INFO: Forking chef instance to converge... [2017-04-04T15:44:58+00:00] INFO: *** Chef 12.4.3 *** [2017-04-04T15:44:58+00:00] INFO: Chef-client pid: 3680 [2017-04-04T15:44:59+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-04T15:44:59+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-04T15:44:59+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-04T15:44:59+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-04T15:44:59+00:00] INFO: Running start handlers [2017-04-04T15:44:59+00:00] INFO: Start handlers complete. [2017-04-04T15:45:00+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-04T15:45:00+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-04T15:45:00+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-04T15:45:00+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-04T15:45:00+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-04T15:45:00+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-04T15:45:00+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170404154500.435019 [2017-04-04T15:45:00+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-04T15:45:00+00:00] INFO: Processing zypper_package[sudo] action install (sudo::default line 20) [2017-04-04T15:45:00+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-04T15:45:00+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170404154500.859494 [2017-04-04T15:45:00+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-04T15:45:00+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-04T15:45:00+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-04T15:45:01+00:00] INFO: service[nscd] stopped [2017-04-04T15:45:01+00:00] INFO: Processing zypper_package[sssd] action install (sssd::default line 20) [2017-04-04T15:45:01+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-04T15:45:01+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170404154501.508159 [2017-04-04T15:45:01+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-04T15:45:01+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-04T15:45:01+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-04T15:45:01+00:00] INFO: service[sssd] restarted [2017-04-04T15:45:01+00:00] INFO: Processing template[/etc/nsswitch.conf] action create (sssd::default line 75) [2017-04-04T15:45:01+00:00] INFO: template[/etc/nsswitch.conf] backed up to /var/chef/backup/etc/nsswitch.conf.chef-20170404154501.838389 [2017-04-04T15:45:01+00:00] INFO: template[/etc/nsswitch.conf] updated file contents /etc/nsswitch.conf [2017-04-04T15:45:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-auth-pc] action create (sssd::default line 83) [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-auth-pc] backed up to /var/chef/backup/etc/pam.d/common-auth-pc.chef-20170404154501.841823 [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-auth-pc] updated file contents /etc/pam.d/common-auth-pc [2017-04-04T15:45:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-account-pc] action create (sssd::default line 83) [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-account-pc] backed up to /var/chef/backup/etc/pam.d/common-account-pc.chef-20170404154501.844812 [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-account-pc] updated file contents /etc/pam.d/common-account-pc [2017-04-04T15:45:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-password-pc] action create (sssd::default line 83) [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-password-pc] backed up to /var/chef/backup/etc/pam.d/common-password-pc.chef-20170404154501.847744 [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-password-pc] updated file contents /etc/pam.d/common-password-pc [2017-04-04T15:45:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-session-pc] action create (sssd::default line 83) [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-session-pc] backed up to /var/chef/backup/etc/pam.d/common-session-pc.chef-20170404154501.850739 [2017-04-04T15:45:01+00:00] INFO: cookbook_file[/etc/pam.d/common-session-pc] updated file contents /etc/pam.d/common-session-pc [2017-04-04T15:45:01+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-04T15:45:02+00:00] INFO: service[sssd] enabled [2017-04-04T15:45:02+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-04T15:45:02+00:00] INFO: Processing zypper_package[ntp] action install (ntp::default line 25) [2017-04-04T15:45:02+00:00] INFO: Processing service[ntp] action start (ntp::default line 27) [2017-04-04T15:45:02+00:00] INFO: service[ntp] started [2017-04-04T15:45:02+00:00] INFO: Processing service[ntp] action enable (ntp::default line 27) [2017-04-04T15:45:02+00:00] INFO: service[ntp] enabled [2017-04-04T15:45:02+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-04T15:45:02+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170404154502.661001 [2017-04-04T15:45:02+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-04T15:45:02+00:00] INFO: template[/etc/ntp.conf] group changed to 0 [2017-04-04T15:45:02+00:00] INFO: template[/etc/ntp.conf] mode changed to 644 [2017-04-04T15:45:02+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntp] (immediate) [2017-04-04T15:45:02+00:00] INFO: Processing service[ntp] action restart (ntp::default line 27) [2017-04-04T15:45:02+00:00] INFO: service[ntp] restarted [2017-04-04T15:45:02+00:00] INFO: Processing service[nfsserver] action nothing (nfs::default line 29) [2017-04-04T15:45:02+00:00] INFO: Processing zypper_package[nfs-client] action install (nfs::default line 35) [2017-04-04T15:45:03+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-04T15:45:03+00:00] INFO: service[rpcbind] enabled [2017-04-04T15:45:03+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-04T15:45:03+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-04T15:45:03+00:00] INFO: service[rpcbind] enabled [2017-04-04T15:45:03+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-04T15:45:03+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-04T15:45:03+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170404154503.539359 [2017-04-04T15:45:03+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-04T15:45:03+00:00] INFO: Processing zypper_package[autofs] action install (autofs::default line 17) [2017-04-04T15:45:03+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-04T15:45:03+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170404154503.834180 [2017-04-04T15:45:03+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-04T15:45:03+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-04T15:45:03+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-04T15:45:05+00:00] INFO: service[autofs] restarted [2017-04-04T15:45:05+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-04T15:45:05+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170404154505.656844 [2017-04-04T15:45:05+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-04T15:45:05+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-04T15:45:05+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-04T15:45:07+00:00] INFO: service[autofs] restarted [2017-04-04T15:45:07+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-04T15:45:07+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-04T15:45:07+00:00] INFO: service[autofs] enabled [2017-04-04T15:45:07+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-04T15:45:07+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-04T15:45:07+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-04T15:45:07+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170404154507.469126 [2017-04-04T15:45:07+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-04T15:45:07+00:00] INFO: Processing zypper_package[munge] action install (munge::default line 20) [2017-04-04T15:45:07+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-04T15:45:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] backed up to /var/chef/backup/etc/munge/munge.key.chef-20170404154507.771500 [2017-04-04T15:45:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-04T15:45:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-04T15:45:07+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-04T15:45:07+00:00] INFO: service[munge] restarted [2017-04-04T15:45:07+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-04T15:45:07+00:00] INFO: template[/etc/sysconfig/munge] backed up to /var/chef/backup/etc/sysconfig/munge.chef-20170404154507.887233 [2017-04-04T15:45:07+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-04T15:45:07+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-04T15:45:07+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-04T15:45:08+00:00] INFO: service[munge] restarted [2017-04-04T15:45:08+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-04T15:45:08+00:00] INFO: service[munge] enabled [2017-04-04T15:45:08+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-04T15:45:08+00:00] INFO: Processing zypper_package[mrsh-server] action install (mrsh::default line 10) [2017-04-04T15:45:08+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T15:45:09+00:00] INFO: zypper_package[mrsh-server] installed mrsh-server at 3:2.6-6.wc1 [2017-04-04T15:45:09+00:00] INFO: Processing zypper_package[mrsh] action install (mrsh::default line 11) [2017-04-04T15:45:10+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T15:45:10+00:00] INFO: zypper_package[mrsh] installed mrsh at 3:2.6-6.wc1 [2017-04-04T15:45:10+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170404154510.626099 [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-04T15:45:10+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170404154510.632466 [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-04T15:45:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-04T15:45:10+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-04T15:45:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:10+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-04T15:45:10+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-04T15:45:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:10+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-04T15:45:10+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-04T15:45:10+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-04T15:45:10+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-04T15:45:11+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T15:45:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:11+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-04T15:45:11+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-04T15:45:12+00:00] INFO: service[xinetd] started [2017-04-04T15:45:12+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-04T15:45:12+00:00] INFO: service[xinetd] enabled [2017-04-04T15:45:12+00:00] INFO: Processing cookbook_file[/etc/pam.d/mrsh] action create (mrsh::server line 77) [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] backed up to /var/chef/backup/etc/pam.d/mrsh.chef-20170404154512.823623 [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] updated file contents /etc/pam.d/mrsh [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] mode changed to 644 [2017-04-04T15:45:12+00:00] INFO: Processing cookbook_file[/etc/pam.d/mrlogin] action create (mrsh::server line 84) [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] backed up to /var/chef/backup/etc/pam.d/mrlogin.chef-20170404154512.827396 [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] updated file contents /etc/pam.d/mrlogin [2017-04-04T15:45:12+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] mode changed to 644 [2017-04-04T15:45:12+00:00] INFO: Processing zypper_package[openssh] action install (ssh::default line 9) [2017-04-04T15:45:13+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-04T15:45:13+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-04T15:45:13+00:00] INFO: service[sshd] enabled [2017-04-04T15:45:13+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-04T15:45:13+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170404154513.344346 [2017-04-04T15:45:13+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-04T15:45:13+00:00] INFO: template[/etc/ssh/sshd_config] mode changed to 600 [2017-04-04T15:45:13+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-04T15:45:13+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-04T15:45:13+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-04T15:45:13+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-04T15:45:13+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-04T15:45:13+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-04T15:45:13+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-04T15:45:13+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-04T15:45:13+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-04T15:45:13+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-04T15:45:13+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-04T15:45:13+00:00] INFO: Processing zypper_package[rsyslog] action install (rsyslog::client line 12) [2017-04-04T15:45:13+00:00] INFO: Processing zypper_package[ed] action install (rsyslog::client line 13) [2017-04-04T15:45:13+00:00] INFO: Processing cookbook_file[/etc/sysconfig/syslog] action create (rsyslog::client line 17) [2017-04-04T15:45:13+00:00] INFO: cookbook_file[/etc/sysconfig/syslog] backed up to /var/chef/backup/etc/sysconfig/syslog.chef-20170404154513.932741 [2017-04-04T15:45:13+00:00] INFO: cookbook_file[/etc/sysconfig/syslog] updated file contents /etc/sysconfig/syslog [2017-04-04T15:45:13+00:00] INFO: Processing service[syslog] action nothing (rsyslog::client line 22) [2017-04-04T15:45:13+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-04T15:45:13+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:14+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-04T15:45:14+00:00] INFO: Processing zypper_package[rsh] action install (rsh::default line 9) [2017-04-04T15:45:14+00:00] INFO: Processing zypper_package[rsh-server] action install (rsh::server line 11) [2017-04-04T15:45:14+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T15:45:15+00:00] INFO: zypper_package[rsh-server] installed rsh-server at 0.17-706.22.1 [2017-04-04T15:45:15+00:00] INFO: Processing execute[enable rsh] action run (rsh::server line 22) [2017-04-04T15:45:16+00:00] INFO: execute[enable rsh] ran successfully [2017-04-04T15:45:16+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-04T15:45:16+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:16+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-04T15:45:16+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-04T15:45:16+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:16+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-04T15:45:16+00:00] INFO: Processing execute[enable rlogin] action run (rsh::server line 22) [2017-04-04T15:45:18+00:00] INFO: execute[enable rlogin] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T15:45:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T15:45:18+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-04T15:45:18+00:00] INFO: Processing zypper_package[openmpi] action install (openmpi::default line 23) [2017-04-04T15:45:18+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-04T15:45:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-04T15:45:18+00:00] INFO: Processing zypper_package[cvs] action install (test_node::default line 12) [2017-04-04T15:45:19+00:00] INFO: Processing zypper_package[bc] action install (test_node::default line 13) [2017-04-04T15:45:19+00:00] INFO: Processing zypper_package[expect] action install (test_node::default line 14) [2017-04-04T15:45:19+00:00] INFO: Processing zypper_package[make] action install (test_node::default line 15) [2017-04-04T15:45:20+00:00] INFO: Processing zypper_package[gcc] action install (test_node::default line 16) [2017-04-04T15:45:20+00:00] INFO: Processing zypper_package[sg3_utils] action install (test_node::default line 17) [2017-04-04T15:45:20+00:00] INFO: Processing zypper_package[patch] action install (test_node::default line 18) [2017-04-04T15:45:20+00:00] INFO: Processing zypper_package[kexec-tools] action install (test_node::default line 19) [2017-04-04T15:45:21+00:00] INFO: Processing zypper_package[setools-console] action install (test_node::default line 101) [2017-04-04T15:45:21+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T15:45:22+00:00] INFO: zypper_package[setools-console] installed setools-console at 3.3.6-1.4.1 [2017-04-04T15:45:22+00:00] INFO: Processing execute[enable_unsupported_modules] action run (test_node::default line 104) [2017-04-04T15:45:22+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T15:45:22+00:00] INFO: execute[enable_unsupported_modules] ran successfully [2017-04-04T15:45:22+00:00] INFO: Processing service[openibd] action enable (test_node::default line 120) [2017-04-04T15:45:23+00:00] INFO: service[openibd] enabled [2017-04-04T15:45:23+00:00] INFO: Processing service[openibd] action start (test_node::default line 120) [2017-04-04T15:45:23+00:00] INFO: service[openibd] started [2017-04-04T15:45:23+00:00] INFO: Processing bash[setup kernel crash dumps] action run (test_node::default line 125) [2017-04-04T15:45:23+00:00] INFO: bash[setup kernel crash dumps] ran successfully [2017-04-04T15:45:23+00:00] INFO: Processing service[boot.kdump] action restart (test_node::default line 136) [2017-04-04T15:45:37+00:00] INFO: service[boot.kdump] restarted [2017-04-04T15:45:37+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-04T15:45:37+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-04T15:45:37+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-04T15:45:37+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-04T15:45:38+00:00] INFO: group[quota_usr] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-04T15:45:38+00:00] INFO: user[quota_usr] created [2017-04-04T15:45:38+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-04T15:45:38+00:00] INFO: group[quota_2usr] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-04T15:45:38+00:00] INFO: user[quota_2usr] created [2017-04-04T15:45:38+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-04T15:45:38+00:00] INFO: group[sanityusr] created [2017-04-04T15:45:38+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-04T15:45:38+00:00] INFO: user[sanityusr] created [2017-04-04T15:45:38+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-04T15:45:38+00:00] INFO: group[sanityusr1] created [2017-04-04T15:45:38+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-04T15:45:38+00:00] INFO: user[sanityusr1] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_1] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_2] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_3] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_4] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_5] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_6] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_7] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_8] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-04T15:45:38+00:00] INFO: user[quota15_9] created [2017-04-04T15:45:38+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_10] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_11] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_12] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_13] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_14] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_15] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_16] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_17] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_18] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_19] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_20] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_21] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_22] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_23] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_24] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_25] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_26] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_27] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_28] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_29] created [2017-04-04T15:45:39+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-04T15:45:39+00:00] INFO: user[quota15_30] created [2017-04-04T15:45:39+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-04T15:45:39+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-04T15:45:39+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-04T15:45:39+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-04T15:45:39+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-04T15:45:39+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-04T15:45:39+00:00] INFO: bash[set_hostid] ran successfully [2017-04-04T15:45:39+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-04T15:45:39+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-04T15:45:39+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-04T15:45:40+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-04T15:45:40+00:00] INFO: Processing bash[add_panic_on_oom] action run (test_node::default line 243) [2017-04-04T15:45:40+00:00] INFO: bash[add_panic_on_oom] ran successfully [2017-04-04T15:45:40+00:00] INFO: Processing bash[set_kernel_printk] action run (test_node::default line 249) [2017-04-04T15:45:40+00:00] INFO: bash[set_kernel_printk] ran successfully [2017-04-04T15:45:40+00:00] INFO: Processing execute[restart_xinetd] action run (test_node::default line 266) [2017-04-04T15:45:41+00:00] INFO: execute[restart_xinetd] ran successfully [2017-04-04T15:45:41+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-04T15:45:41+00:00] INFO: Processing zypper_package[mcelog] action install (mcelog::default line 9) [2017-04-04T15:45:41+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-04T15:45:41+00:00] INFO: user[mpiuser] created [2017-04-04T15:45:41+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 20) [2017-04-04T15:45:41+00:00] INFO: group[mpiuser] created [2017-04-04T15:45:41+00:00] INFO: Processing bash[usermod_add_mpiuser_to_mpiuser] action run (openmpi::test_node line 24) [2017-04-04T15:45:41+00:00] INFO: bash[usermod_add_mpiuser_to_mpiuser] ran successfully [2017-04-04T15:45:41+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-04T15:45:41+00:00] INFO: directory[/localhome/mpiuser] created directory /localhome/mpiuser [2017-04-04T15:45:41+00:00] INFO: directory[/localhome/mpiuser] owner changed to 532 [2017-04-04T15:45:41+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-04T15:45:41+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-04T15:45:41+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-04T15:45:41+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-04T15:45:41+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-04T15:45:41+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-04T15:45:41+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-04T15:45:42+00:00] INFO: execute[newaliases] ran successfully [2017-04-04T15:45:42+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-04T15:45:42+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-04T15:45:43+00:00] INFO: service[sshd] restarted [2017-04-04T15:45:43+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[syslog] (delayed) [2017-04-04T15:45:43+00:00] INFO: Processing service[syslog] action restart (rsyslog::client line 22) [2017-04-04T15:45:44+00:00] INFO: service[syslog] restarted [2017-04-04T15:45:44+00:00] INFO: Chef Run complete in 44.355550445 seconds [2017-04-04T15:45:44+00:00] INFO: Skipping removal of unused files from the cache [2017-04-04T15:45:44+00:00] INFO: Running report handlers [2017-04-04T15:45:44+00:00] INFO: Creating JSON run report [2017-04-04T15:45:44+00:00] INFO: Report handlers complete INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal Boot logging started on /dev/ttyS0((no tty)) at Tue Apr 4 16:18:23 2017 Master Resource Control: previous runlevel: 3, switching to runlevel: 6 Shutting down acpid 7[?25l done8[?25h [?25hShutting down automount (force) 7[?25l done8[?25h [?25hNot unloading kdump during runlevel changes skipped [?25h[?25hShutting down CRON daemon7[?25l done8[?25h [?25hShutting down irqbalance 7[?25l done8[?25h [?25hShutting down mcelog... 7[?25l done8[?25h [?25h[?25hStopping MUNGE: munged7[?25l done8[?25h [?25hShutting down Name Service Cache Daemon7[?25l done8[?25h [?25h[?25hShutting down smartd 7[?25l done8[?25h [?25h[?25h[?25hShutting down SSH daemon *with all active connections*7[?25l done8[?25h [?25hShutting down sssd 7[?25l done8[?25h [?25hShutting down xinetd: (waiting for all children to terminate) 7[?25l done8[?25h [?25hShutting down auditd 7[?25l done8[?25h [?25h[?25hShutting down haveged daemon 7[?25l done8[?25h [?25h/etc/init.d/rc3.d/K02kbd stop7[?25l done8[?25h [?25hShutting down network time protocol daemon (NTPD)7[?25l done8[?25h [?25hShutting down mail service (Postfix)7[?25l done8[?25h [?25hShutting down (remotefs) network interfaces: Shutting down service (remotefs) network . . . . . . . . .7[?25l done8[?25h [?25hSaving random seed7[?25l done8[?25h [?25hShutting down HAL daemon7[?25l done8[?25h [?25hShutting down NFS client services: rpc.statd7[?25l done8[?25h [?25hShutting down rpcbind 7[?25l done8[?25h [?25hShutting down syslog services7[?25l done8[?25h [?25h[?25hShutting down (localfs) network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service (localfs) network . . . . . . . . .7[?25l done8[?25h [?25hUnloading HCA driver:[ OK ] [?25hShutting down D-Bus daemon7[?25l done8[?25h [?25hRunning /etc/init.d/halt.local 7[?25l done8[?25hUnmounting fuse control filesystem7[?25l done8[?25h Not unloading kdump during runlevel changes skipped 7[?25l done8[?25h Turning off quota 7[?25l done8[?25h 7[?25l done8[?25h The System Time is in sync with Hardware Clock good Turning off swap files Unmounting file systems 7[?25l done8[?25hStopping udevd: 7[?25l done8[?25h Sending all processes the TERM signal...  done Sending all processes the KILL signal...  done Please stand by while rebooting the system... [ 2047.815965] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-97_lustre-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Mar 30 13:45:57 UTC 2017 (b708f71) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37756000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa81, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72320 r8192 d21888 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a81, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910920k/2097140k available (4576k kernel code, 460k absent, 185760k reserved, 7571k data, 1356k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013346] Security Framework initialized [ 0.014226] AppArmor: AppArmor initialized [ 0.015254] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017081] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020038] Mount-cache hash table entries: 256 [ 0.021169] Initializing cgroup subsys cpuacct [ 0.022077] Initializing cgroup subsys memory [ 0.022989] Initializing cgroup subsys devices [ 0.023897] Initializing cgroup subsys freezer [ 0.024006] Initializing cgroup subsys net_cls [ 0.024900] Initializing cgroup subsys blkio [ 0.025784] Initializing cgroup subsys perf_event [ 0.026839] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.031269] ACPI: Core revision 20110413 [ 0.033610] Enabling x2apic [ 0.034398] Enabled x2apic [ 0.035300] Switched APIC routing to physical x2apic. [ 0.037138] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.038237] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.064011] kvm-clock: cpu 1, msr 0:7fd10a81, secondary cpu clock [ 0.064028] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.064054] Brought up 2 CPUs [ 0.064055] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068549] devtmpfs: initialized [ 0.071040] print_constraints: dummy: [ 0.072057] Time: 16:19:09 Date: 04/04/17 [ 0.073071] NET: Registered protocol family 16 [ 0.074132] ACPI: bus type pci registered [ 0.075054] PCI: Using configuration type 1 for base access [ 0.076506] bio: create slab at 0 [ 0.081285] ACPI: Interpreter enabled [ 0.082143] ACPI: (supports S0 S5) [ 0.083071] ACPI: Using IOAPIC for interrupt routing [ 0.087579] ACPI: No dock devices found. [ 0.088010] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.089891] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.093015] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.094606] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.118400] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.124880] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.126292] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.127682] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.128943] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.130304] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.131670] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.132012] vgaarb: loaded [ 0.133005] PCI: Using ACPI for IRQ routing [ 0.133251] NetLabel: Initializing [ 0.136013] NetLabel: domain hash size = 128 [ 0.136888] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.137850] NetLabel: unlabeled traffic allowed by default [ 0.138911] Switching to clocksource kvm-clock [ 0.142267] AppArmor: AppArmor Filesystem Enabled [ 0.143276] pnp: PnP ACPI init [ 0.144079] ACPI: bus type pnp registered [ 0.145670] pnp: PnP ACPI: found 6 devices [ 0.146551] ACPI: ACPI bus type pnp unregistered [ 0.153743] NET: Registered protocol family 2 [ 0.154856] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.156961] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.160566] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.162229] TCP: Hash tables configured (established 262144 bind 65536) [ 0.163399] TCP reno registered [ 0.164174] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.165297] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.166647] NET: Registered protocol family 1 [ 0.167586] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.168692] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.169780] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.170998] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.172081] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.177427] pci 0000:00:01.2: PCI INT D disabled [ 0.178470] Unpacking initramfs... [ 0.315250] Freeing initrd memory: 8808k freed [ 0.329492] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.330661] Initialise module verification [ 0.331720] audit: initializing netlink socket (disabled) [ 0.332782] type=2000 audit(1491322750.332:1): initialized [ 0.350919] bounce pool size: 64 pages [ 0.351799] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.353841] VFS: Disk quotas dquot_6.5.2 [ 0.354787] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.356101] msgmni has been set to 937 [ 0.357371] alg: No test for stdrng (krng) [ 0.358278] Asymmetric key parser 'x509' registered [ 0.359298] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.360987] io scheduler noop registered [ 0.361881] io scheduler deadline registered (default) [ 0.362955] io scheduler cfq registered [ 0.364083] GHES: HEST is not enabled! [ 0.364980] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.366299] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.605249] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.641114] Non-volatile memory driver v1.3 [ 0.642058] Linux agpgart interface v0.103 [ 0.643037] Fixed MDIO Bus: probed [ 0.643915] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.646231] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.647266] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.648383] mousedev: PS/2 mouse device common for all mice [ 0.649694] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.651862] cpuidle: using governor ladder [ 0.652820] cpuidle: using governor menu [ 0.653724] EFI Variables Facility v0.08 2004-May-17 [ 0.654937] TCP cubic registered [ 0.655778] Registering the dns_resolver key type [ 0.656891] Loading module verification certificates [ 0.657930] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.659996] registered taskstats version 1 [ 0.661177] Magic number: 13:255:337 [ 0.662040] tty tty58: hash matches [ 0.665098] Freeing unused kernel memory: 1356k freed [ 0.666810] Write protecting the kernel read-only data: 10240k [ 0.671010] Freeing unused kernel memory: 1548k freed [ 0.674687] Freeing unused kernel memory: 840k freed doing fast boot [ 0.711178] Disabling lock debugging due to kernel taint [ 0.718026] SCSI subsystem initialized [ 0.741475] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.753765] scsi0 : ata_piix [ 0.754619] scsi1 : ata_piix [ 0.755372] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.756539] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.922833] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.945828] vda: vda1 vda2 vda4 [ 0.947073] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.948172] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.956247] alua: device handler registered [ 0.961896] hp_sw: device handler registered [ 0.968225] emc: device handler registered [ 0.973711] rdac: device handler registered Creating device nodes with udev [ 0.979195] udev: starting version 147 [ 1.022473] usbcore: registered new interface driver usbfs [ 1.023546] usbcore: registered new interface driver hub [ 1.039175] usbcore: registered new device driver usb [ 1.042956] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.047534] uhci_hcd: USB Universal Host Controller Interface driver [ 1.050858] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.053517] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.054608] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.056486] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.058876] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.060667] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.063074] usb usb1: Product: UHCI Host Controller [ 1.064330] usb usb1: Manufacturer: Linux 3.0.101-97_lustre-default uhci_hcd [ 1.065727] usb usb1: SerialNumber: 0000:00:01.2 [ 1.066966] hub 1-0:1.0: USB hub found [ 1.067979] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 16:19:11 2017 [ 1.197579] [drm] Initialized drm 1.1.0 20060810 [ 1.200740] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-97_lustre-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.317576] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 80096/1310720 files, 553620/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.338309] kjournald starting. Commit interval 15 seconds [ 1.338846] EXT3-fs (vda1): using internal journal [ 1.338851] EXT3-fs (vda1): mounted filesystem with ordered data mode [ 1.349033] Refined TSC clocksource calibration: 2593.643 MHz. INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 16:19:11 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.477773] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.499819] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.503335] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.505072] ACPI: Power Button [PWRF] [ 1.507970] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.509423] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.510635] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.514624] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.525453] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 1.539525] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.547171] rtc_cmos 00:01: RTC can wake from S4 [ 1.548440] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.549733] rtc0: alarms up to one day, 114 bytes nvram [ 1.559749] NET: Registered protocol family 10 [ 1.565344] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 1.702883] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.553586] device-mapper: uevent: version 1.0.3 [ 2.554643] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.608746] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 2.877764] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... [?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid acpid: starting up 7[?25l done8[?25h [?25hacpid: 3 rules loaded Starting D-Bus daemon7[?25l done8[?25h [?25hStarting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25h[?25hInitializing random number generator7[?25l done8[?25h [?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 27 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting network time protocol daemon (NTPD)7[?25l done8[?25h [?25hStarting MUNGE: munged7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting sssd 7[?25l done8[?25h [?25hStarting automount 7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hStarting INET services. (xinetd)7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-97_lustre-default (console). trevis-41vm7 login: [ 92.679769] RPC: Registered named UNIX socket transport module. [ 92.681270] RPC: Registered udp transport module. [ 92.682282] RPC: Registered tcp transport module. [ 92.683211] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 92.685920] FS-Cache: Loaded [ 92.689698] FS-Cache: Netfs 'nfs' registered for caching [ 4599.408976] SysRq : Show State [ 4599.409760] task PC stack pid father [ 4599.410780] init S ffff88007b455ad8 0 1 0 0x00000000 [ 4599.411955] ffff88007b455948 0000000000000086 ffff88007b454010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b455fd8 [ 4599.412786] ffff88007b455fd8 0000000000010b80 ffff88007b452040 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fc5aaf10943>] 0x7fc5aaf10942 [ 4599.412786] kthreadd S ffff88007b458080 0 2 0 0x00000000 [ 4599.412786] ffff88007b45bea0 0000000000000046 ffff88007b45a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b45bfd8 [ 4599.412786] ffff88007b45bfd8 0000000000010b80 ffff88007b458080 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] kthreadd+0x2a1/0x2b0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 [ 4599.412786] ffff88007b461ea0 0000000000000046 ffff88007b460010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b461fd8 [ 4599.412786] ffff88007b461fd8 0000000000010b80 ffff88007b45e0c0 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] run_ksoftirqd+0x10d/0x170 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/u:0 S 0000000000011600 0 5 2 0x00000000 [ 4599.412786] ffff88007b469e70 0000000000000046 ffff88007b468010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b469fd8 [ 4599.412786] ffff88007b469fd8 0000000000010b80 ffff88007b466140 ffff880037856080 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] migration/0 S ffff88007fc0cee8 0 6 2 0x00000000 [ 4599.412786] ffff88007b46fe00 0000000000000046 ffff88007b46e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b46ffd8 [ 4599.412786] ffff88007b46ffd8 0000000000010b80 ffff88007b46c180 ffff8800378840c0 [ 4599.412786] Call Trace: [ 4599.412786] [] cpu_stopper_thread+0x10d/0x180 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] watchdog/0 S 0000000000000000 0 7 2 0x00000000 [ 4599.412786] ffff88007b473ec0 0000000000000046 ffff88007b472010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b473fd8 [ 4599.412786] ffff88007b473fd8 0000000000010b80 ffff88007b4701c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] watchdog+0x9f/0xf0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] migration/1 S ffff88007fd0cee8 0 8 2 0x00000000 [ 4599.412786] ffff88007b477e00 0000000000000046 ffff88007b476010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b477fd8 [ 4599.412786] ffff88007b477fd8 0000000000010b80 ffff88007b474200 ffff880079f961c0 [ 4599.412786] Call Trace: [ 4599.412786] [] cpu_stopper_thread+0x10d/0x180 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/1:0 S 0000000000011600 0 9 2 0x00000000 [ 4599.412786] ffff88007b47fe70 0000000000000046 ffff88007b47e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b47ffd8 [ 4599.412786] ffff88007b47ffd8 0000000000010b80 ffff88007b47c240 ffff880079538580 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ksoftirqd/1 S 0000000000000000 0 10 2 0x00000000 [ 4599.412786] ffff88007b4a7ea0 0000000000000046 ffff88007b4a6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4a7fd8 [ 4599.412786] ffff88007b4a7fd8 0000000000010b80 ffff88007b4a4280 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] run_ksoftirqd+0x10d/0x170 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/0:1 S 0000000000011600 0 11 2 0x00000000 [ 4599.412786] ffff88007b4abe70 0000000000000046 ffff88007b4aa010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4abfd8 [ 4599.412786] ffff88007b4abfd8 0000000000010b80 ffff88007b4a82c0 ffff8800378840c0 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] watchdog/1 S 0000000000000000 0 12 2 0x00000000 [ 4599.412786] ffff88007b4b5ec0 0000000000000046 ffff88007b4b4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4b5fd8 [ 4599.412786] ffff88007b4b5fd8 0000000000010b80 ffff88007b4b2340 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] watchdog+0x9f/0xf0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] cpuset S ffff88007b499860 0 13 2 0x00000000 [ 4599.412786] ffff88007b4dfe60 0000000000000046 ffff88007b4de010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4dffd8 [ 4599.412786] ffff88007b4dffd8 0000000000010b80 ffff88007b4dc380 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] khelper S ffff88007b4997e0 0 14 2 0x00000000 [ 4599.412786] ffff88007b4e5e60 0000000000000046 ffff88007b4e4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4e5fd8 [ 4599.412786] ffff88007b4e5fd8 0000000000010b80 ffff88007b4e23c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] netns S ffff88007b4991e0 0 15 2 0x00000000 [ 4599.412786] ffff88007b4ede60 0000000000000046 ffff88007b4ec010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b4edfd8 [ 4599.412786] ffff88007b4edfd8 0000000000010b80 ffff88007b4ea400 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] sync_supers S 0000000000000000 0 16 2 0x00000000 [ 4599.412786] ffff88007b571ec0 0000000000000046 ffff88007b570010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b571fd8 [ 4599.412786] ffff88007b571fd8 0000000000010b80 ffff88007b56e440 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] bdi_sync_supers+0x40/0x60 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] bdi-default S ffffffff81a36160 0 17 2 0x00000000 [ 4599.412786] ffff88007b577d70 0000000000000046 ffff88007b576010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b577fd8 [ 4599.412786] ffff88007b577fd8 0000000000010b80 ffff88007b574480 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x1b0/0x2a0 [ 4599.412786] [] bdi_forker_thread+0x205/0x4a0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kintegrityd S ffff88007b563ce0 0 18 2 0x00000000 [ 4599.412786] ffff88007b57be60 0000000000000046 ffff88007b57a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b57bfd8 [ 4599.412786] ffff88007b57bfd8 0000000000010b80 ffff88007b5784c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kblockd S ffff88007b5631e0 0 19 2 0x00000000 [ 4599.412786] ffff88007b681e60 0000000000000046 ffff88007b680010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b681fd8 [ 4599.412786] ffff88007b681fd8 0000000000010b80 ffff88007b64e500 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] md S ffff88007b73c9e0 0 20 2 0x00000000 [ 4599.412786] ffff88007b755e60 0000000000000046 ffff88007b754010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007b755fd8 [ 4599.412786] ffff88007b755fd8 0000000000010b80 ffff88007b752540 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] khungtaskd S 0000000000000000 0 21 2 0x00000000 [ 4599.412786] ffff88007953be40 0000000000000046 ffff88007953a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007953bfd8 [ 4599.412786] ffff88007953bfd8 0000000000010b80 ffff880079538580 ffff88007953e5c0 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] watchdog+0x3a/0x50 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/1:1 S 0000000000011600 0 22 2 0x00000000 [ 4599.412786] ffff880079561e70 0000000000000046 ffff880079560010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079561fd8 [ 4599.412786] ffff880079561fd8 0000000000010b80 ffff88007953e5c0 ffff880078dbe040 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kswapd0 S ffff88007ffbc068 0 23 2 0x00000000 [ 4599.412786] ffff880079567e00 0000000000000046 ffff880079566010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079567fd8 [ 4599.412786] ffff880079567fd8 0000000000010b80 ffff880079564600 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] kswapd_try_to_sleep+0x185/0x1c0 [ 4599.412786] [] kswapd+0xd0/0x230 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ksmd S ffff8800795a6640 0 24 2 0x00000000 [ 4599.412786] ffff8800795a9e60 0000000000000046 ffff8800795a8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800795a9fd8 [ 4599.412786] ffff8800795a9fd8 0000000000010b80 ffff8800795a6640 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] ksm_scan_thread+0x295/0x2e0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] khugepaged S ffff8800795ac040 0 25 2 0x00000000 [ 4599.412786] ffff8800795afdb0 0000000000000046 ffff8800795ae010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800795affd8 [ 4599.412786] ffff8800795affd8 0000000000010b80 ffff8800795ac040 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x1b0/0x2a0 [ 4599.412786] [] khugepaged_loop+0x1f0/0x3c0 [ 4599.412786] [] khugepaged+0x4a/0xb0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] fsnotify_mark S ffff8800795f2080 0 26 2 0x00000000 [ 4599.412786] ffff8800795f5e60 0000000000000046 ffff8800795f4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800795f5fd8 [ 4599.412786] ffff8800795f5fd8 0000000000010b80 ffff8800795f2080 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] fsnotify_mark_destroy+0x119/0x150 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] crypto S ffff88007959d4e0 0 27 2 0x00000000 [ 4599.412786] ffff880079673e60 0000000000000046 ffff880079672010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079673fd8 [ 4599.412786] ffff880079673fd8 0000000000010b80 ffff8800796700c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kthrotld S ffff88007959d260 0 31 2 0x00000000 [ 4599.412786] ffff8800796fbe60 0000000000000046 ffff8800796fa010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800796fbfd8 [ 4599.412786] ffff8800796fbfd8 0000000000010b80 ffff8800796f81c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kpsmoused S ffff88007b73c360 0 32 2 0x00000000 [ 4599.412786] ffff88007976de60 0000000000000046 ffff88007976c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007976dfd8 [ 4599.412786] ffff88007976dfd8 0000000000010b80 ffff88007976a200 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/0:2 S 0000000000011600 0 33 2 0x00000000 [ 4599.412786] ffff880079799e70 0000000000000046 ffff880079798010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079799fd8 [ 4599.412786] ffff880079799fd8 0000000000010b80 ffff88007979c240 ffff8800375ca080 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ata_sff S ffff8800797596e0 0 93 2 0x00000000 [ 4599.412786] ffff880037a0be60 0000000000000046 ffff880037a0a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037a0bfd8 [ 4599.412786] ffff880037a0bfd8 0000000000010b80 ffff8800378264c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] scsi_eh_0 S 0000000000000000 0 147 2 0x00000000 [ 4599.412786] ffff880037ac3eb0 0000000000000046 ffff880037ac2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037ac3fd8 [ 4599.412786] ffff880037ac3fd8 0000000000010b80 ffff880037810380 ffff8800378ea2c0 [ 4599.412786] Call Trace: [ 4599.412786] [] scsi_error_handler+0x7e/0x1d0 [scsi_mod] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] scsi_eh_1 S 0000000000000000 0 148 2 0x00000000 [ 4599.412786] ffff880037acdeb0 0000000000000046 ffff880037acc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037acdfd8 [ 4599.412786] ffff880037acdfd8 0000000000010b80 ffff8800378ea2c0 ffff8800378365c0 [ 4599.412786] Call Trace: [ 4599.412786] [] scsi_error_handler+0x7e/0x1d0 [scsi_mod] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kworker/u:1 S 0000000000011600 0 149 2 0x00000000 [ 4599.412786] ffff8800379fde70 0000000000000046 ffff8800379fc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800379fdfd8 [ 4599.412786] ffff8800379fdfd8 0000000000010b80 ffff8800378365c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] worker_thread+0x258/0x410 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kmpath_aluad S ffff880037a96260 0 173 2 0x00000000 [ 4599.412786] ffff880037b0fe60 0000000000000046 ffff880037b0e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037b0ffd8 [ 4599.412786] ffff880037b0ffd8 0000000000010b80 ffff880037860640 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kmpath_rdacd S ffff8800378242e0 0 195 2 0x00000000 [ 4599.412786] ffff880037b27e60 0000000000000046 ffff880037b26010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037b27fd8 [ 4599.412786] ffff880037b27fd8 0000000000010b80 ffff88003781e440 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] khubd S ffff8800791a1e98 0 277 2 0x00000000 [ 4599.412786] ffff8800791a1e70 0000000000000046 ffff8800791a0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800791a1fd8 [ 4599.412786] ffff8800791a1fd8 0000000000010b80 ffff8800375a4640 ffff88007953e5c0 [ 4599.412786] Call Trace: [ 4599.412786] [] hub_thread+0xf6/0x1a0 [usbcore] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kjournald S ffff880079019e80 0 404 2 0x00000000 [ 4599.412786] ffff880079019e60 0000000000000046 ffff880079018010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079019fd8 [ 4599.412786] ffff880079019fd8 0000000000010b80 ffff8800790f4240 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] kjournald+0x20b/0x230 [jbd] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] flush-253:0 S ffff880037a577c8 0 410 2 0x00000000 [ 4599.412786] ffff88003769ddd0 0000000000000046 ffff88003769c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88003769dfd8 [ 4599.412786] ffff88003769dfd8 0000000000010b80 ffff88003781a400 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x1b0/0x2a0 [ 4599.412786] [] bdi_writeback_thread+0x218/0x240 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] udevd S ffff8800375a9e3c 0 497 1 0x00000000 [ 4599.412786] ffff8800375a9a28 0000000000000082 ffff8800375a8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800375a9fd8 [ 4599.412786] ffff8800375a9fd8 0000000000010b80 ffff8800375f6100 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f4054aa315f>] 0x7f4054aa315e [ 4599.412786] udevd S ffff88007a331ddc 0 587 497 0x00000000 [ 4599.412786] ffff88007a3319e8 0000000000000082 ffff88007a330010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a331fd8 [ 4599.412786] ffff88007a331fd8 0000000000010b80 ffff88007a0be580 ffff8800375f6100 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_ppoll+0x4f/0x180 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f4054aa3267>] 0x7f4054aa3266 [ 4599.412786] udevd S ffff880037591ddc 0 610 497 0x00000000 [ 4599.412786] ffff8800375919e8 0000000000000086 ffff880037590010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037591fd8 [ 4599.412786] ffff880037591fd8 0000000000010b80 ffff88003754e040 ffff88007ab88580 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_ppoll+0x4f/0x180 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f4054aa3267>] 0x7f4054aa3266 [ 4599.412786] vballoon S ffff880037763478 0 617 2 0x00000000 [ 4599.412786] ffff88007a0f1e50 0000000000000046 ffff88007a0f0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a0f1fd8 [ 4599.412786] ffff88007a0f1fd8 0000000000010b80 ffff880037b685c0 ffff8800377aa3c0 [ 4599.412786] Call Trace: [ 4599.412786] [] balloon+0x20b/0x290 [virtio_balloon] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kauditd S 0000000000000000 0 685 2 0x00000000 [ 4599.412786] ffff88003780de80 0000000000000046 ffff88003780c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88003780dfd8 [ 4599.412786] ffff88003780dfd8 0000000000010b80 ffff880078cb81c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] kauditd_thread+0x185/0x1a0 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] acpid S ffff88007a1a5e24 0 1254 1 0x00000000 [ 4599.412786] ffff88007a1a5a28 0000000000000082 ffff88007a1a4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1a5fd8 [ 4599.412786] ffff88007a1a5fd8 0000000000010b80 ffff88007aad0280 ffff880037b74340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fec602e115f>] 0x7fec602e115e [ 4599.412786] dbus-daemon S ffff88007a18de3c 0 1267 1 0x00000000 [ 4599.412786] ffff88007a18da28 0000000000000086 ffff88007a18c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a18dfd8 [ 4599.412786] ffff88007a18dfd8 0000000000010b80 ffff8800797d0340 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007efc07c1315f>] 0x7efc07c1315e [ 4599.412786] hald S ffff88007aa7fe7c 0 1356 1 0x00000000 [ 4599.412786] ffff88007aa7fa28 0000000000000082 ffff88007aa7e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007aa7ffd8 [ 4599.412786] ffff88007aa7ffd8 0000000000010b80 ffff88007ab88580 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fc90ecaf196>] 0x7fc90ecaf195 [ 4599.412786] hald S 0000000000800000 0 1361 1 0x00000000 [ 4599.412786] ffff88007a1cfcf8 0000000000000082 ffff88007a1ce010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1cffd8 [ 4599.412786] ffff88007a1cffd8 0000000000010b80 ffff88007a05e200 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] pipe_wait+0x52/0x70 [ 4599.412786] [] pipe_read+0x428/0x550 [ 4599.412786] [] do_sync_read+0xd7/0x120 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fc90ecaabbd>] 0x7fc90ecaabbc [ 4599.412786] console-kit-dae S ffff88007a3bfe34 0 1359 1 0x00000000 [ 4599.412786] ffff88007a3bfa28 0000000000000082 ffff88007a3be010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3bffd8 [ 4599.412786] ffff88007a3bffd8 0000000000010b80 ffff88007a1a8540 ffff88007ab88580 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49d196>] 0x7fd59f49d195 [ 4599.412786] console-kit-dae S ffff88007a1a1ea0 0 1362 1 0x00000000 [ 4599.412786] ffff88007a1a1d48 0000000000000082 ffff88007a1a0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1a1fd8 [ 4599.412786] ffff88007a1a1fd8 0000000000010b80 ffff88007a19e400 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f74f66c>] 0x7fd59f74f66b [ 4599.412786] console-kit-dae S 0000000000000000 0 1363 1 0x00000000 [ 4599.412786] ffff88007a3c3cf8 0000000000000082 ffff88007a3c2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3c3fd8 [ 4599.412786] ffff88007a3c3fd8 0000000000010b80 ffff88007aa680c0 ffff8800794265c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1364 1 0x00000000 [ 4599.412786] ffff88007a3c5cf8 0000000000000082 ffff88007a3c4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3c5fd8 [ 4599.412786] ffff88007a3c5fd8 0000000000010b80 ffff88007a804340 ffff88007941c540 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1365 1 0x00000000 [ 4599.412786] ffff88007a3c7cf8 0000000000000082 ffff88007a3c6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3c7fd8 [ 4599.412786] ffff88007a3c7fd8 0000000000010b80 ffff8800377aa3c0 ffff88007a3f8300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1366 1 0x00000000 [ 4599.412786] ffff88007a3c9cf8 0000000000000082 ffff88007a3c8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3c9fd8 [ 4599.412786] ffff88007a3c9fd8 0000000000010b80 ffff88007a198040 ffff88007a3d4100 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1367 1 0x00000000 [ 4599.412786] ffff88007a3cbcf8 0000000000000082 ffff88007a3ca010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3cbfd8 [ 4599.412786] ffff88007a3cbfd8 0000000000010b80 ffff88007a644480 ffff88007a3f42c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1368 1 0x00000000 [ 4599.412786] ffff88007a3cdcf8 0000000000000082 ffff88007a3cc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3cdfd8 [ 4599.412786] ffff88007a3cdfd8 0000000000010b80 ffff88007aa62440 ffff88007a1a8540 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1369 1 0x00000000 [ 4599.412786] ffff88007a3cfcf8 0000000000000082 ffff88007a3ce010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3cffd8 [ 4599.412786] ffff88007a3cffd8 0000000000010b80 ffff88007a078300 ffff880079458280 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1370 1 0x00000000 [ 4599.412786] ffff88007a3d1cf8 0000000000000082 ffff88007a3d0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3d1fd8 [ 4599.412786] ffff88007a3d1fd8 0000000000010b80 ffff88007a1bc580 ffff88007a3d4100 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1371 1 0x00000000 [ 4599.412786] ffff88007a3d3cf8 0000000000000082 ffff88007a3d2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3d3fd8 [ 4599.412786] ffff88007a3d3fd8 0000000000010b80 ffff88007a1a6480 ffff880079476440 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1372 1 0x00000000 [ 4599.412786] ffff88007a3d7cf8 0000000000000082 ffff88007a3d6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3d7fd8 [ 4599.412786] ffff88007a3d7fd8 0000000000010b80 ffff88007a3d4100 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1373 1 0x00000000 [ 4599.412786] ffff88007a3dbcf8 0000000000000082 ffff88007a3da010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3dbfd8 [ 4599.412786] ffff88007a3dbfd8 0000000000010b80 ffff88007a3d8140 ffff880079464340 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1374 1 0x00000000 [ 4599.412786] ffff88007a3ddcf8 0000000000000082 ffff88007a3dc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3ddfd8 [ 4599.412786] ffff88007a3ddfd8 0000000000010b80 ffff88007a38a640 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1375 1 0x00000000 [ 4599.412786] ffff88007a3dfcf8 0000000000000082 ffff88007a3de010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3dffd8 [ 4599.412786] ffff88007a3dffd8 0000000000010b80 ffff88007a05a1c0 ffff880079432040 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1376 1 0x00000000 [ 4599.412786] ffff88007a3e3cf8 0000000000000082 ffff88007a3e2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3e3fd8 [ 4599.412786] ffff88007a3e3fd8 0000000000010b80 ffff88007a3e0180 ffff88007943a0c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1377 1 0x00000000 [ 4599.412786] ffff88007a3e7cf8 0000000000000082 ffff88007a3e6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3e7fd8 [ 4599.412786] ffff88007a3e7fd8 0000000000010b80 ffff88007a3e41c0 ffff88007940c440 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1378 1 0x00000000 [ 4599.412786] ffff88007a3ebcf8 0000000000000082 ffff88007a3ea010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3ebfd8 [ 4599.412786] ffff88007a3ebfd8 0000000000010b80 ffff88007a3e8200 ffff880079410480 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1379 1 0x00000000 [ 4599.412786] ffff88007a3efcf8 0000000000000082 ffff88007a3ee010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3effd8 [ 4599.412786] ffff88007a3effd8 0000000000010b80 ffff88007a3ec240 ffff88007947e4c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1380 1 0x00000000 [ 4599.412786] ffff88007a3f3cf8 0000000000000082 ffff88007a3f2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3f3fd8 [ 4599.412786] ffff88007a3f3fd8 0000000000010b80 ffff88007a3f0280 ffff880079460300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1381 1 0x00000000 [ 4599.412786] ffff88007a3f7cf8 0000000000000082 ffff88007a3f6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3f7fd8 [ 4599.412786] ffff88007a3f7fd8 0000000000010b80 ffff88007a3f42c0 ffff88007aa680c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1382 1 0x00000000 [ 4599.412786] ffff88007a3fbcf8 0000000000000082 ffff88007a3fa010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3fbfd8 [ 4599.412786] ffff88007a3fbfd8 0000000000010b80 ffff88007a3f8300 ffff88007a3f0280 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1383 1 0x00000000 [ 4599.412786] ffff88007a3ffcf8 0000000000000082 ffff88007a3fe010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3fffd8 [ 4599.412786] ffff88007a3fffd8 0000000000010b80 ffff88007a3fc340 ffff88007942e640 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1384 1 0x00000000 [ 4599.412786] ffff880079403cf8 0000000000000082 ffff880079402010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079403fd8 [ 4599.412786] ffff880079403fd8 0000000000010b80 ffff880079400380 ffff880079436080 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1385 1 0x00000000 [ 4599.412786] ffff880079407cf8 0000000000000082 ffff880079406010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079407fd8 [ 4599.412786] ffff880079407fd8 0000000000010b80 ffff8800794043c0 ffff88007943e100 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1386 1 0x00000000 [ 4599.412786] ffff88007940bcf8 0000000000000082 ffff88007940a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007940bfd8 [ 4599.412786] ffff88007940bfd8 0000000000010b80 ffff880079408400 ffff88007a078300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1387 1 0x00000000 [ 4599.412786] ffff88007940fcf8 0000000000000082 ffff88007940e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007940ffd8 [ 4599.412786] ffff88007940ffd8 0000000000010b80 ffff88007940c440 ffff880079442140 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1388 1 0x00000000 [ 4599.412786] ffff880079413cf8 0000000000000082 ffff880079412010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079413fd8 [ 4599.412786] ffff880079413fd8 0000000000010b80 ffff880079410480 ffff880079448180 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1389 1 0x00000000 [ 4599.412786] ffff880079417cf8 0000000000000082 ffff880079416010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079417fd8 [ 4599.412786] ffff880079417fd8 0000000000010b80 ffff8800794144c0 ffff88007aa62440 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1390 1 0x00000000 [ 4599.412786] ffff88007a1c5cf8 0000000000000082 ffff88007a1c4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1c5fd8 [ 4599.412786] ffff88007a1c5fd8 0000000000010b80 ffff88007a380180 ffff88007946e3c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1391 1 0x00000000 [ 4599.412786] ffff88007a183cf8 0000000000000082 ffff88007a182010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a183fd8 [ 4599.412786] ffff88007a183fd8 0000000000010b80 ffff8800379ee180 ffff880079422580 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1392 1 0x00000000 [ 4599.412786] ffff88007941bcf8 0000000000000082 ffff88007941a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007941bfd8 [ 4599.412786] ffff88007941bfd8 0000000000010b80 ffff880079418500 ffff8800794144c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1393 1 0x00000000 [ 4599.412786] ffff88007941fcf8 0000000000000082 ffff88007941e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007941ffd8 [ 4599.412786] ffff88007941ffd8 0000000000010b80 ffff88007941c540 ffff880079418500 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1394 1 0x00000000 [ 4599.412786] ffff880079425cf8 0000000000000082 ffff880079424010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079425fd8 [ 4599.412786] ffff880079425fd8 0000000000010b80 ffff880079422580 ffff8800377aa3c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1395 1 0x00000000 [ 4599.412786] ffff880079429cf8 0000000000000082 ffff880079428010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079429fd8 [ 4599.412786] ffff880079429fd8 0000000000010b80 ffff8800794265c0 ffff88007a804340 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1396 1 0x00000000 [ 4599.412786] ffff88007942dcf8 0000000000000082 ffff88007942c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007942dfd8 [ 4599.412786] ffff88007942dfd8 0000000000010b80 ffff88007942a600 ffff88007a3fc340 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1397 1 0x00000000 [ 4599.412786] ffff880079431cf8 0000000000000082 ffff880079430010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079431fd8 [ 4599.412786] ffff880079431fd8 0000000000010b80 ffff88007942e640 ffff88007a05a1c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1398 1 0x00000000 [ 4599.412786] ffff880079435cf8 0000000000000082 ffff880079434010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079435fd8 [ 4599.412786] ffff880079435fd8 0000000000010b80 ffff880079432040 ffff880079400380 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1399 1 0x00000000 [ 4599.412786] ffff880079439cf8 0000000000000082 ffff880079438010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079439fd8 [ 4599.412786] ffff880079439fd8 0000000000010b80 ffff880079436080 ffff88007a3e0180 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1400 1 0x00000000 [ 4599.412786] ffff88007943dcf8 0000000000000082 ffff88007943c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007943dfd8 [ 4599.412786] ffff88007943dfd8 0000000000010b80 ffff88007943a0c0 ffff8800794043c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1401 1 0x00000000 [ 4599.412786] ffff880079441cf8 0000000000000082 ffff880079440010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079441fd8 [ 4599.412786] ffff880079441fd8 0000000000010b80 ffff88007943e100 ffff88007a3e41c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1402 1 0x00000000 [ 4599.412786] ffff880079445cf8 0000000000000082 ffff880079444010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079445fd8 [ 4599.412786] ffff880079445fd8 0000000000010b80 ffff880079442140 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1403 1 0x00000000 [ 4599.412786] ffff88007944bcf8 0000000000000082 ffff88007944a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007944bfd8 [ 4599.412786] ffff88007944bfd8 0000000000010b80 ffff880079448180 ffff88007944c1c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1404 1 0x00000000 [ 4599.412786] ffff88007944fcf8 0000000000000082 ffff88007944e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007944ffd8 [ 4599.412786] ffff88007944ffd8 0000000000010b80 ffff88007944c1c0 ffff880079408400 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1405 1 0x00000000 [ 4599.412786] ffff880079453cf8 0000000000000082 ffff880079452010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079453fd8 [ 4599.412786] ffff880079453fd8 0000000000010b80 ffff880079450200 ffff88007a3ec240 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1406 1 0x00000000 [ 4599.412786] ffff880079457cf8 0000000000000082 ffff880079456010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079457fd8 [ 4599.412786] ffff880079457fd8 0000000000010b80 ffff880079454240 ffff880079472400 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1407 1 0x00000000 [ 4599.412786] ffff88007945bcf8 0000000000000082 ffff88007945a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007945bfd8 [ 4599.412786] ffff88007945bfd8 0000000000010b80 ffff880079458280 ffff880079442140 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1408 1 0x00000000 [ 4599.412786] ffff88007945fcf8 0000000000000082 ffff88007945e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007945ffd8 [ 4599.412786] ffff88007945ffd8 0000000000010b80 ffff88007945c2c0 ffff88007a3e8200 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1409 1 0x00000000 [ 4599.412786] ffff880079463cf8 0000000000000082 ffff880079462010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079463fd8 [ 4599.412786] ffff880079463fd8 0000000000010b80 ffff880079460300 ffff88007a3f42c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1410 1 0x00000000 [ 4599.412786] ffff880079467cf8 0000000000000082 ffff880079466010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079467fd8 [ 4599.412786] ffff880079467fd8 0000000000010b80 ffff880079464340 ffff8800379ee180 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1411 1 0x00000000 [ 4599.412786] ffff88007946bcf8 0000000000000082 ffff88007946a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007946bfd8 [ 4599.412786] ffff88007946bfd8 0000000000010b80 ffff880079468380 ffff88007a1a6480 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1412 1 0x00000000 [ 4599.412786] ffff880079471cf8 0000000000000082 ffff880079470010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079471fd8 [ 4599.412786] ffff880079471fd8 0000000000010b80 ffff88007946e3c0 ffff880079c06540 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1413 1 0x00000000 [ 4599.412786] ffff880079475cf8 0000000000000082 ffff880079474010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079475fd8 [ 4599.412786] ffff880079475fd8 0000000000010b80 ffff880079472400 ffff880079c0a580 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1414 1 0x00000000 [ 4599.412786] ffff880079479cf8 0000000000000082 ffff880079478010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079479fd8 [ 4599.412786] ffff880079479fd8 0000000000010b80 ffff880079476440 ffff88007945c2c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1415 1 0x00000000 [ 4599.412786] ffff88007947dcf8 0000000000000082 ffff88007947c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007947dfd8 [ 4599.412786] ffff88007947dfd8 0000000000010b80 ffff88007947a480 ffff880079c02500 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1416 1 0x00000000 [ 4599.412786] ffff880079c01cf8 0000000000000082 ffff880079c00010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c01fd8 [ 4599.412786] ffff880079c01fd8 0000000000010b80 ffff88007947e4c0 ffff880079c0e5c0 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1417 1 0x00000000 [ 4599.412786] ffff880079c05cf8 0000000000000082 ffff880079c04010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c05fd8 [ 4599.412786] ffff880079c05fd8 0000000000010b80 ffff880079c02500 ffff880079450200 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1418 1 0x00000000 [ 4599.412786] ffff880079c09cf8 0000000000000082 ffff880079c08010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c09fd8 [ 4599.412786] ffff880079c09fd8 0000000000010b80 ffff880079c06540 ffff880079468380 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1419 1 0x00000000 [ 4599.412786] ffff880079c0dcf8 0000000000000082 ffff880079c0c010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c0dfd8 [ 4599.412786] ffff880079c0dfd8 0000000000010b80 ffff880079c0a580 ffff88007a380180 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1420 1 0x00000000 [ 4599.412786] ffff880079c11cf8 0000000000000082 ffff880079c10010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c11fd8 [ 4599.412786] ffff880079c11fd8 0000000000010b80 ffff880079c0e5c0 ffff880079454240 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1421 1 0x00000000 [ 4599.412786] ffff880079c17cf8 0000000000000082 ffff880079c16010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c17fd8 [ 4599.412786] ffff880079c17fd8 0000000000010b80 ffff880079c14600 ffff880079c18640 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1422 1 0x00000000 [ 4599.412786] ffff880079c1bcf8 0000000000000082 ffff880079c1a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c1bfd8 [ 4599.412786] ffff880079c1bfd8 0000000000010b80 ffff880079c18640 ffff880079c1c040 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] console-kit-dae S 0000000000000000 0 1423 1 0x00000000 [ 4599.412786] ffff880079c1fcf8 0000000000000082 ffff880079c1e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c1ffd8 [ 4599.412786] ffff880079c1ffd8 0000000000010b80 ffff880079c1c040 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] vt_event_wait+0xa6/0x130 [ 4599.412786] [] vt_waitactive+0x2d/0x60 [ 4599.412786] [] vt_ioctl+0x1044/0x1cf0 [ 4599.412786] [] tty_ioctl+0x377/0x9a0 [ 4599.412786] [] do_vfs_ioctl+0x8b/0x3b0 [ 4599.412786] [] sys_ioctl+0xa1/0xb0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd59f49e9a7>] 0x7fd59f49e9a6 [ 4599.412786] hald-runner S ffff88007a3c1e1c 0 1360 1356 0x00000000 [ 4599.412786] ffff88007a3c1a28 0000000000000082 ffff88007a3c0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a3c1fd8 [ 4599.412786] ffff88007a3c1fd8 0000000000010b80 ffff88007a382500 ffff88007ab88580 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5a8fcf215f>] 0x7f5a8fcf215e [ 4599.412786] hald-addon-inpu S ffff880079c39e2c 0 1448 1360 0x00000000 [ 4599.412786] ffff880079c39a28 0000000000000086 ffff880079c38010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079c39fd8 [ 4599.412786] ffff880079c39fd8 0000000000010b80 ffff88007a6ca340 ffff88007ab88580 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f0e2771d15f>] 0x7f0e2771d15e [ 4599.412786] hald-addon-acpi S ffff88007a913478 0 1468 1360 0x00000000 [ 4599.412786] ffff88007a6e1b68 0000000000000086 ffff88007a6e0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a6e1fd8 [ 4599.412786] ffff88007a6e1fd8 0000000000010b80 ffff88007a6921c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] unix_stream_data_wait+0xab/0x110 [ 4599.412786] [] unix_stream_recvmsg+0x3f3/0x530 [ 4599.412786] [] sock_aio_read+0x121/0x130 [ 4599.412786] [] do_sync_read+0xd7/0x120 [ 4599.412786] [] vfs_read+0x125/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f27c570dba0>] 0x7f27c570db9f [ 4599.412786] mthca_catas S ffff88007ab1d1e0 0 1492 2 0x00000000 [ 4599.412786] ffff88007a937e60 0000000000000046 ffff88007a936010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a937fd8 [ 4599.412786] ffff88007a937fd8 0000000000010b80 ffff88007aad8480 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] mlx4 S ffff88007a91bc60 0 1494 2 0x00000000 [ 4599.412786] ffff88007a1b7e60 0000000000000046 ffff88007a1b6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1b7fd8 [ 4599.412786] ffff88007a1b7fd8 0000000000010b80 ffff8800375ca080 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ib_mcast S ffff88007a91bae0 0 1496 2 0x00000000 [ 4599.412786] ffff88007a9b3e60 0000000000000046 ffff88007a9b2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a9b3fd8 [ 4599.412786] ffff88007a9b3fd8 0000000000010b80 ffff88007a07a5c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] mlx4_ib S ffff88007a91b460 0 1497 2 0x00000000 [ 4599.412786] ffff88007a6ade60 0000000000000046 ffff88007a6ac010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a6adfd8 [ 4599.412786] ffff88007a6adfd8 0000000000010b80 ffff880079c66100 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] mlx4_ib_mcg S ffff88007a91b360 0 1498 2 0x00000000 [ 4599.412786] ffff88007a9b5e60 0000000000000046 ffff88007a9b4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a9b5fd8 [ 4599.412786] ffff88007a9b5fd8 0000000000010b80 ffff880079c6a140 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] mlx5_core_wq S ffff88007a91bf60 0 1503 2 0x00000000 [ 4599.412786] ffff88007a9b9e60 0000000000000046 ffff88007a9b8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a9b9fd8 [ 4599.412786] ffff88007a9b9fd8 0000000000010b80 ffff880079c4e3c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] qib_cq S ffff88007a9313e0 0 1506 2 0x00000000 [ 4599.412786] ffff88007a7f5e60 0000000000000046 ffff88007a7f4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a7f5fd8 [ 4599.412786] ffff88007a7f5fd8 0000000000010b80 ffff880079c40080 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] iw_cxgb3 S ffff88007a9ebb60 0 1510 2 0x00000000 [ 4599.412786] ffff88007a207e60 0000000000000046 ffff88007a206010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a207fd8 [ 4599.412786] ffff88007a207fd8 0000000000010b80 ffff880079c440c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ib_cm S ffff88007a9eb0e0 0 1516 2 0x00000000 [ 4599.412786] ffff88007a7ede60 0000000000000046 ffff88007a7ec010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a7edfd8 [ 4599.412786] ffff88007a7edfd8 0000000000010b80 ffff88007a1f2080 ffff88007ab88580 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ipoib S ffff88007a75bae0 0 1517 2 0x00000000 [ 4599.412786] ffff88007a26fe60 0000000000000046 ffff88007a26e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a26ffd8 [ 4599.412786] ffff88007a26ffd8 0000000000010b80 ffff88007a26c500 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] ib_addr S ffff88007a75b7e0 0 1524 2 0x00000000 [ 4599.412786] ffff88007a2bde60 0000000000000046 ffff88007a2bc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a2bdfd8 [ 4599.412786] ffff88007a2bdfd8 0000000000010b80 ffff88007a6b8240 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] iw_cm_wq S ffff88007a9eb260 0 1525 2 0x00000000 [ 4599.412786] ffff88007a277e60 0000000000000046 ffff88007a276010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a277fd8 [ 4599.412786] ffff88007a277fd8 0000000000010b80 ffff88007a6bc280 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] rdma_cm S ffff88007a75b3e0 0 1526 2 0x00000000 [ 4599.412786] ffff88007a2dbe60 0000000000000046 ffff88007a2da010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a2dbfd8 [ 4599.412786] ffff88007a2dbfd8 0000000000010b80 ffff880079c6c440 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] dhcpcd S ffff880078d21e1c 0 2476 1 0x00000000 [ 4599.412786] ffff880078d21a28 0000000000000086 ffff880078d20010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078d21fd8 [ 4599.412786] ffff880078d21fd8 0000000000010b80 ffff88007a694500 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fe465bd615f>] 0x7fe465bd615e [ 4599.412786] rsyslogd S ffff88007a2cbad8 0 2862 1 0x00000000 [ 4599.412786] ffff88007a2cb948 0000000000000086 ffff88007a2ca010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a2cbfd8 [ 4599.412786] ffff88007a2cbfd8 0000000000010b80 ffff88007aae4280 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd98376d963>] 0x7fd98376d962 [ 4599.412786] rs:main Q:Reg S ffff88007a279ea0 0 2863 1 0x00000000 [ 4599.412786] ffff88007a279d48 0000000000000086 ffff88007a278010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a279fd8 [ 4599.412786] ffff88007a279fd8 0000000000010b80 ffff880079d38600 ffff88007ab0a080 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd983e2a66c>] 0x7fd983e2a66b [ 4599.412786] rsyslogd S ffff880079d99ad8 0 2864 1 0x00000000 [ 4599.412786] ffff880079d99948 0000000000000086 ffff880079d98010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079d99fd8 [ 4599.412786] ffff880079d99fd8 0000000000010b80 ffff880079d10380 ffff880079d38600 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd98376d963>] 0x7fd98376d962 [ 4599.412786] rsyslogd S ffff880079d53ad8 0 2865 1 0x00000000 [ 4599.412786] ffff880079d53948 0000000000000086 ffff880079d52010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079d53fd8 [ 4599.412786] ffff880079d53fd8 0000000000010b80 ffff880078d52580 ffff880079d38600 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd98376d963>] 0x7fd98376d962 [ 4599.412786] rsyslogd S 00007fd9813ff5d0 0 2866 1 0x00000000 [ 4599.412786] ffff880079d6fe38 0000000000000086 ffff880079d6e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079d6ffd8 [ 4599.412786] ffff880079d6ffd8 0000000000010b80 ffff88007a2c25c0 ffff880079d38600 [ 4599.412786] Call Trace: [ 4599.412786] [] do_syslog+0x461/0x4d0 [ 4599.412786] [] kmsg_read+0x2e/0x70 [ 4599.412786] [] proc_reg_read+0x77/0xc0 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd983e2d7fd>] 0x7fd983e2d7fc [ 4599.412786] auditd S 00000000038f9b17 0 2878 1 0x00000000 [ 4599.412786] ffff88007a2c7db8 0000000000000082 ffff88007a2c6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a2c7fd8 [ 4599.412786] ffff88007a2c7fd8 0000000000010b80 ffff880079d84340 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fee6fe34d03>] 0x7fee6fe34d02 [ 4599.412786] auditd S ffff88007ab8fea0 0 2879 1 0x00000000 [ 4599.412786] ffff88007ab8fd48 0000000000000082 ffff88007ab8e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007ab8ffd8 [ 4599.412786] ffff88007ab8ffd8 0000000000010b80 ffff88007a2e8280 ffff88007aba8540 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fee7035666c>] 0x7fee7035666b [ 4599.412786] audispd S ffff880037b8bea0 0 2880 2878 0x00000000 [ 4599.412786] ffff880037b8bd48 0000000000000082 ffff880037b8a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037b8bfd8 [ 4599.412786] ffff880037b8bfd8 0000000000010b80 ffff88007aba8540 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f33f055966c>] 0x7f33f055966b [ 4599.412786] audispd S ffff88007a275e24 0 2881 2878 0x00000000 [ 4599.412786] ffff88007a275a28 0000000000000082 ffff88007a274010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a275fd8 [ 4599.412786] ffff88007a275fd8 0000000000010b80 ffff880078d220c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f33f02a7196>] 0x7f33f02a7195 [ 4599.412786] haveged S 0000000000000000 0 2899 1 0x00000000 [ 4599.412786] ffff880079dc7948 0000000000000086 ffff880079dc6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079dc7fd8 [ 4599.412786] ffff880079dc7fd8 0000000000010b80 ffff88007a6ce380 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f504ad1a943>] 0x7f504ad1a942 [ 4599.412786] rpcbind S ffff880078d63e54 0 2911 1 0x00000000 [ 4599.412786] ffff880078d63a28 0000000000000086 ffff880078d62010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078d63fd8 [ 4599.412786] ffff880078d63fd8 0000000000010b80 ffff880079d9e040 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f4b2c00b15f>] 0x7f4b2c00b15e [ 4599.412786] irqbalance S 00007fef0dcd13b4 0 2999 1 0x00000000 [ 4599.412786] ffff880079df5e78 0000000000000082 ffff880079df4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079df5fd8 [ 4599.412786] ffff880079df5fd8 0000000000010b80 ffff88007a2c8080 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] do_nanosleep+0x9d/0xd0 [ 4599.412786] [] hrtimer_nanosleep+0xa8/0x140 [ 4599.412786] [] sys_nanosleep+0x67/0x80 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fef0d353540>] 0x7fef0d35353f [ 4599.412786] mcelog S ffff880079dd7de4 0 3013 1 0x00000000 [ 4599.412786] ffff880079dd79e8 0000000000000082 ffff880079dd6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079dd7fd8 [ 4599.412786] ffff880079dd7fd8 0000000000010b80 ffff880079d6a640 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_ppoll+0x4f/0x180 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2cc318a267>] 0x7f2cc318a266 [ 4599.412786] sshd S ffff88007a2f3ad8 0 3310 1 0x00000000 [ 4599.412786] ffff88007a2f3948 0000000000000086 ffff88007a2f2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a2f3fd8 [ 4599.412786] ffff88007a2f3fd8 0000000000010b80 ffff880079d92480 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fa7a48f9943>] 0x7fa7a48f9942 [ 4599.412786] ntpd S ffff880079dedad8 0 3324 1 0x00000000 [ 4599.412786] ffff880079ded948 0000000000000086 ffff880079dec010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079dedfd8 [ 4599.412786] ffff880079dedfd8 0000000000010b80 ffff880078de6540 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f8066568943>] 0x7f8066568942 [ 4599.412786] ntpd S ffff88007a241238 0 3325 3324 0x00000000 [ 4599.412786] ffff880037b75b68 0000000000000082 ffff880037b74010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037b75fd8 [ 4599.412786] ffff880037b75fd8 0000000000010b80 ffff880079da8140 ffff88007aa3c4c0 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] unix_stream_data_wait+0xab/0x110 [ 4599.412786] [] unix_stream_recvmsg+0x3f3/0x530 [ 4599.412786] [] sock_aio_read+0x121/0x130 [ 4599.412786] [] do_sync_read+0xd7/0x120 [ 4599.412786] [] vfs_read+0x125/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f8066561ba0>] 0x7f8066561b9f [ 4599.412786] munged S ffff880078debde0 0 3347 1 0x00000000 [ 4599.412786] ffff880078debce8 0000000000000086 ffff880078dea010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078debfd8 [ 4599.412786] ffff880078debfd8 0000000000010b80 ffff88007a272480 ffff880078d26080 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] __skb_recv_datagram+0x264/0x270 [ 4599.412786] [] skb_recv_datagram+0x1c/0x30 [ 4599.412786] [] unix_accept+0x7f/0x140 [ 4599.412786] [] sys_accept4+0x177/0x260 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2d01a7fadd>] 0x7f2d01a7fadc [ 4599.412786] munged S ffff880078dffea0 0 3348 1 0x00000000 [ 4599.412786] ffff880078dffd48 0000000000000086 ffff880078dfe010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078dfffd8 [ 4599.412786] ffff880078dfffd8 0000000000010b80 ffff880078daa0c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2d01a7c9fc>] 0x7f2d01a7c9fb [ 4599.412786] munged S ffff880079d43ea0 0 3349 1 0x00000000 [ 4599.412786] ffff880079d43d48 0000000000000086 ffff880079d42010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079d43fd8 [ 4599.412786] ffff880079d43fd8 0000000000010b80 ffff88007aa3c4c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2d01a7c66c>] 0x7f2d01a7c66b [ 4599.412786] munged S ffff880078d15ea0 0 3350 1 0x00000000 [ 4599.412786] ffff880078d15d48 0000000000000086 ffff880078d14010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078d15fd8 [ 4599.412786] ffff880078d15fd8 0000000000010b80 ffff880078d26080 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2d01a7c66c>] 0x7f2d01a7c66b [ 4599.412786] nscd S 0000000001c965bf 0 3363 1 0x00000000 [ 4599.412786] ffff880078d69db8 0000000000000082 ffff880078d68010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078d69fd8 [ 4599.412786] ffff880078d69fd8 0000000000010b80 ffff880078d9e240 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40b23d03>] 0x7f5e40b23d02 [ 4599.412786] nscd S ffff880078dfdea0 0 3364 1 0x00000000 [ 4599.412786] ffff880078dfdd48 0000000000000082 ffff880078dfc010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078dfdfd8 [ 4599.412786] ffff880078dfdfd8 0000000000010b80 ffff880078d645c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe49fc>] 0x7f5e40fe49fb [ 4599.412786] nscd S ffff880078dafea0 0 3365 1 0x00000000 [ 4599.412786] ffff880078dafd48 0000000000000082 ffff880078dae010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078daffd8 [ 4599.412786] ffff880078daffd8 0000000000010b80 ffff880079dbe140 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe49fc>] 0x7f5e40fe49fb [ 4599.412786] nscd S ffff880079de5ea0 0 3366 1 0x00000000 [ 4599.412786] ffff880079de5d48 0000000000000082 ffff880079de4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079de5fd8 [ 4599.412786] ffff880079de5fd8 0000000000010b80 ffff880078d182c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe49fc>] 0x7f5e40fe49fb [ 4599.412786] nscd S ffff880079db7ea0 0 3367 1 0x00000000 [ 4599.412786] ffff880079db7d48 0000000000000082 ffff880079db6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079db7fd8 [ 4599.412786] ffff880079db7fd8 0000000000010b80 ffff880078d5a600 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] nscd S ffff880079defea0 0 3368 1 0x00000000 [ 4599.412786] ffff880079defd48 0000000000000082 ffff880079dee010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079deffd8 [ 4599.412786] ffff880079deffd8 0000000000010b80 ffff880079d503c0 ffff880078dbe040 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] nscd S ffff880079dd1ea0 0 3369 1 0x00000000 [ 4599.412786] ffff880079dd1d48 0000000000000082 ffff880079dd0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079dd1fd8 [ 4599.412786] ffff880079dd1fd8 0000000000010b80 ffff880078d38540 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] nscd S ffff880078dd5ea0 0 3370 1 0x00000000 [ 4599.412786] ffff880078dd5d48 0000000000000082 ffff880078dd4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078dd5fd8 [ 4599.412786] ffff880078dd5fd8 0000000000010b80 ffff880078d1e100 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] nscd S ffff88007a7e1ea0 0 7477 1 0x00000000 [ 4599.412786] ffff88007a7e1d48 0000000000000082 ffff88007a7e0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a7e1fd8 [ 4599.412786] ffff88007a7e1fd8 0000000000010b80 ffff88007925e300 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] nscd S ffff88007a209ea0 0 7478 1 0x00000000 [ 4599.412786] ffff88007a209d48 0000000000000082 ffff88007a208010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a209fd8 [ 4599.412786] ffff88007a209fd8 0000000000010b80 ffff8800790da480 ffff880078dbe040 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f5e40fe466c>] 0x7f5e40fe466b [ 4599.412786] sssd S 0000000000987357 0 3382 1 0x00000000 [ 4599.412786] ffff880078dd1db8 0000000000000082 ffff880078dd0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078dd1fd8 [ 4599.412786] ffff880078dd1fd8 0000000000010b80 ffff880078dd2280 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fb322353ce3>] 0x7fb322353ce2 [ 4599.412786] sssd_be S 0000000005f5e100 0 3383 3382 0x00000000 [ 4599.412786] ffff88007a469db8 0000000000000082 ffff88007a468010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a469fd8 [ 4599.412786] ffff88007a469fd8 0000000000010b80 ffff880078d7c240 ffff880078dd2280 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fde2faf9ce3>] 0x7fde2faf9ce2 [ 4599.412786] sssd_nss S 0000000001c9c380 0 3384 3382 0x00000000 [ 4599.412786] ffff88007a4c9db8 0000000000000082 ffff88007a4c8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a4c9fd8 [ 4599.412786] ffff88007a4c9fd8 0000000000010b80 ffff880079dd4200 ffff880078dba340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f7076273ce3>] 0x7f7076273ce2 [ 4599.412786] sssd_pam S 0000000001c9c37f 0 3385 3382 0x00000000 [ 4599.412786] ffff88007a4e9db8 0000000000000086 ffff88007a4e8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a4e9fd8 [ 4599.412786] ffff88007a4e9fd8 0000000000010b80 ffff880078d1a200 ffff880078d7c240 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f0a3453cce3>] 0x7f0a3453cce2 [ 4599.412786] sssd_ssh S 0000000001c9c37f 0 3386 3382 0x00000000 [ 4599.412786] ffff88007a529db8 0000000000000086 ffff88007a528010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a529fd8 [ 4599.412786] ffff88007a529fd8 0000000000010b80 ffff880078dba340 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f43751bdce3>] 0x7f43751bdce2 [ 4599.412786] automount S 00007ffc56b4f5cc 0 3404 1 0x00000000 [ 4599.412786] ffff880078df1dd8 0000000000000086 ffff880078df0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880078df1fd8 [ 4599.412786] ffff880078df1fd8 0000000000010b80 ffff880078d8c2c0 ffff88007a588400 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] do_sigtimedwait+0x110/0x1c0 [ 4599.412786] [] sys_rt_sigtimedwait+0x94/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd943681527>] 0x7fd943681526 [ 4599.412786] automount S ffff88007a58bea0 0 3405 1 0x00000000 [ 4599.412786] ffff88007a58bd48 0000000000000086 ffff88007a58a010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a58bfd8 [ 4599.412786] ffff88007a58bfd8 0000000000010b80 ffff880078dc6640 ffff88007a5823c0 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd94367d9fc>] 0x7fd94367d9fb [ 4599.412786] automount S ffff88007a581ea0 0 3406 1 0x00000000 [ 4599.412786] ffff88007a581d48 0000000000000086 ffff88007a580010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a581fd8 [ 4599.412786] ffff88007a581fd8 0000000000010b80 ffff88007a5823c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] futex_wait_queue_me+0xc5/0x110 [ 4599.412786] [] futex_wait+0x117/0x250 [ 4599.412786] [] do_futex+0xb8/0x1c0 [ 4599.412786] [] sys_futex+0x82/0x170 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd94367d66c>] 0x7fd94367d66b [ 4599.412786] automount S ffff88007a585e2c 0 3409 1 0x00000000 [ 4599.412786] ffff88007a585a28 0000000000000086 ffff88007a584010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a585fd8 [ 4599.412786] ffff88007a585fd8 0000000000010b80 ffff88007a57c380 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd9427b3196>] 0x7fd9427b3195 [ 4599.412786] automount S ffff88007a5bfe2c 0 3412 1 0x00000000 [ 4599.412786] ffff88007a5bfa28 0000000000000086 ffff88007a5be010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a5bffd8 [ 4599.412786] ffff88007a5bffd8 0000000000010b80 ffff880078d9c4c0 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd9427b3196>] 0x7fd9427b3195 [ 4599.412786] automount S ffff88007a5c3e2c 0 3413 1 0x00000000 [ 4599.412786] ffff88007a5c3a28 0000000000000086 ffff88007a5c2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a5c3fd8 [ 4599.412786] ffff88007a5c3fd8 0000000000010b80 ffff88007a5c0440 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd9427b3196>] 0x7fd9427b3195 [ 4599.412786] automount S ffff88007a5bbe2c 0 3414 1 0x00000000 [ 4599.412786] ffff88007a5bba28 0000000000000086 ffff88007a5ba010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a5bbfd8 [ 4599.412786] ffff88007a5bbfd8 0000000000010b80 ffff88007a7f6080 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_poll+0x1b2/0x2a0 [ 4599.412786] [] do_sys_poll+0x18e/0x200 [ 4599.412786] [] sys_poll+0x66/0x100 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fd9427b3196>] 0x7fd9427b3195 [ 4599.412786] master S 00000000039386ff 0 3495 1 0x00000000 [ 4599.412786] ffff88007d2b7db8 0000000000000086 ffff88007d2b6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007d2b7fd8 [ 4599.412786] ffff88007d2b7fd8 0000000000010b80 ffff88007d2b2240 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007efee1c50ce3>] 0x7efee1c50ce2 [ 4599.412786] pickup S 0000000005f5e0ff 0 3508 3495 0x00000000 [ 4599.412786] ffff880079ea9db8 0000000000000082 ffff880079ea8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079ea9fd8 [ 4599.412786] ffff880079ea9fd8 0000000000010b80 ffff88007a5f8280 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fb3a4fbdce3>] 0x7fb3a4fbdce2 [ 4599.412786] qmgr S 0000000005f5e100 0 3510 3495 0x00000000 [ 4599.412786] ffff880079ecfdb8 0000000000000086 ffff880079ece010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079ecffd8 [ 4599.412786] ffff880079ecffd8 0000000000010b80 ffff880078d0c480 ffff88007d2b2240 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 4599.412786] [] ep_poll+0x315/0x3a0 [ 4599.412786] [] sys_epoll_wait+0xd4/0xe0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f2402750ce3>] 0x7f2402750ce2 [ 4599.412786] cron S 0000000000000000 0 3520 1 0x00000000 [ 4599.412786] ffff88007d2bbe78 0000000000000086 ffff88007d2ba010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007d2bbfd8 [ 4599.412786] ffff88007d2bbfd8 0000000000010b80 ffff88007b798140 ffff8800797d0340 [ 4599.412786] Call Trace: [ 4599.412786] [] do_nanosleep+0x9d/0xd0 [ 4599.412786] [] hrtimer_nanosleep+0xa8/0x140 [ 4599.412786] [] sys_nanosleep+0x67/0x80 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f7aac129540>] 0x7f7aac12953f [ 4599.412786] xinetd S ffff88007aee7ad8 0 3545 1 0x00000000 [ 4599.412786] ffff88007aee7948 0000000000000086 ffff88007aee6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007aee7fd8 [ 4599.412786] ffff88007aee7fd8 0000000000010b80 ffff88007a5c8540 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fa520a0d943>] 0x7fa520a0d942 [ 4599.412786] mingetty S 00007ffd4bf0c6d8 0 7342 1 0x00000000 [ 4599.412786] ffff880079f17d18 0000000000000082 ffff880079f16010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079f17fd8 [ 4599.412786] ffff880079f17fd8 0000000000010b80 ffff8800790a2100 ffff88007b4a82c0 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f6ba071aba0>] 0x7f6ba071ab9f [ 4599.412786] mingetty S 00007ffd38fcb558 0 7343 1 0x00000000 [ 4599.412786] ffff880037afbd18 0000000000000082 ffff880037afa010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037afbfd8 [ 4599.412786] ffff880037afbfd8 0000000000010b80 ffff88007ab38580 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007ff3fd704ba0>] 0x7ff3fd704b9f [ 4599.412786] mingetty S 00007ffe204bded8 0 7344 1 0x00000000 [ 4599.412786] ffff88007a8b7d18 0000000000000086 ffff88007a8b6010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a8b7fd8 [ 4599.412786] ffff88007a8b7fd8 0000000000010b80 ffff880079f445c0 ffff880079f205c0 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f82b5207ba0>] 0x7f82b5207b9f [ 4599.412786] mingetty S 00007ffd2362d898 0 7345 1 0x00000000 [ 4599.412786] ffff880079e89d18 0000000000000082 ffff880079e88010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079e89fd8 [ 4599.412786] ffff880079e89fd8 0000000000010b80 ffff880079f205c0 ffff88007a6ce380 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f63a8835ba0>] 0x7f63a8835b9f [ 4599.412786] mingetty S 00007ffcf7bb5758 0 7346 1 0x00000000 [ 4599.412786] ffff88007a5dfd18 0000000000000086 ffff88007a5de010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a5dffd8 [ 4599.412786] ffff88007a5dffd8 0000000000010b80 ffff88007a466640 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007fb4308b0ba0>] 0x7fb4308b0b9f [ 4599.412786] mingetty S 00007ffe768f1878 0 7347 1 0x00000000 [ 4599.412786] ffff88007d2bfd18 0000000000000086 ffff88007d2be010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007d2bffd8 [ 4599.412786] ffff88007d2bffd8 0000000000010b80 ffff88007a5d2600 ffff88007a466640 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007efe26f3aba0>] 0x7efe26f3ab9f [ 4599.412786] agetty S 00007ffe61e17227 0 7348 1 0x00000000 [ 4599.412786] ffff88007a6afd18 0000000000000086 ffff88007a6ae010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a6affd8 [ 4599.412786] ffff88007a6affd8 0000000000010b80 ffff880079efe040 ffff88007b4a82c0 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] n_tty_read+0x603/0x950 [ 4599.412786] [] tty_read+0xb5/0x110 [ 4599.412786] [] vfs_read+0xc7/0x130 [ 4599.412786] [] sys_read+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f251da5dba0>] 0x7f251da5db9f [ 4599.412786] rpc.statd S ffff8800377a5ad8 0 7523 1 0x00000000 [ 4599.412786] ffff8800377a5948 0000000000000082 ffff8800377a4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800377a5fd8 [ 4599.412786] ffff8800377a5fd8 0000000000010b80 ffff8800376c21c0 ffff88007911e340 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f726ba7c943>] 0x7f726ba7c942 [ 4599.412786] rpciod S ffff88007a5b0ae0 0 7532 2 0x00000000 [ 4599.412786] ffff880079e67e60 0000000000000046 ffff880079e66010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079e67fd8 [ 4599.412786] ffff880079e67fd8 0000000000010b80 ffff8800791a6380 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] nfsiod S ffff88007919f560 0 7533 2 0x00000000 [ 4599.412786] ffff880037521e60 0000000000000046 ffff880037520010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880037521fd8 [ 4599.412786] ffff880037521fd8 0000000000010b80 ffff8800790d22c0 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff880037b413e0 0 7867 2 0x00000000 [ 4599.412786] ffff8800793c9e60 0000000000000046 ffff8800793c8010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800793c9fd8 [ 4599.412786] ffff8800793c9fd8 0000000000010b80 ffff88007a0b24c0 ffff8800793ce100 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff880078dcdc60 0 7928 2 0x00000000 [ 4599.412786] ffff88003747fe60 0000000000000046 ffff88003747e010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88003747ffd8 [ 4599.412786] ffff88003747ffd8 0000000000010b80 ffff880079eec280 ffff8800793ce100 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff880078dcd1e0 0 7989 2 0x00000000 [ 4599.412786] ffff8800791f1e60 0000000000000046 ffff8800791f0010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800791f1fd8 [ 4599.412786] ffff8800791f1fd8 0000000000010b80 ffff88003754a080 ffff88007ab34480 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff88007a5adee0 0 8050 2 0x00000000 [ 4599.412786] ffff880079e47e60 0000000000000046 ffff880079e46010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff880079e47fd8 [ 4599.412786] ffff880079e47fd8 0000000000010b80 ffff8800790962c0 ffff8800790b0500 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff880037824760 0 8111 2 0x00000000 [ 4599.412786] ffff8800791a3e60 0000000000000046 ffff8800791a2010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800791a3fd8 [ 4599.412786] ffff8800791a3fd8 0000000000010b80 ffff8800790b0500 ffff88007a5bc240 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] kdmflush S ffff880079f603e0 0 8173 2 0x00000000 [ 4599.412786] ffff88007a589e60 0000000000000046 ffff88007a588010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a589fd8 [ 4599.412786] ffff88007a589fd8 0000000000010b80 ffff8800378964c0 ffff880078c98440 [ 4599.412786] Call Trace: [ 4599.412786] [] rescuer_thread+0x2db/0x310 [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] sshd S ffff88007a1dfad8 0 8968 3310 0x00000000 [ 4599.412786] ffff88007a1df948 0000000000000082 ffff88007a1de010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff88007a1dffd8 [ 4599.412786] ffff88007a1dffd8 0000000000010b80 ffff880078dbe040 ffffffff81a11020 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 4599.412786] [] poll_schedule_timeout+0x44/0x70 [ 4599.412786] [] do_select+0x5fb/0x680 [ 4599.412786] [] core_sys_select+0x19c/0x2f0 [ 4599.412786] [] sys_select+0x51/0x110 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007f92d25ce943>] 0x7f92d25ce942 [ 4599.412786] sh R running task 0 8970 8968 0x00000000 [ 4599.412786] ffffffff00000018 ffff880037ac1998 ffff880037ac1958 ffffffff8185b1c2 [ 4599.412786] ffffffff8185b1c4 ffffffff810a43ae 00007f92d25ce942 ffff880037ac19b0 [ 4599.412786] ffffffff812641d6 0000000000000000 ffff8800b7ac1b47 ffffffff812644b8 [ 4599.412786] Call Trace: [ 4599.412786] [] dump_trace+0x75/0x300 [ 4599.412786] [] show_stack_log_lvl+0xef/0x150 [ 4599.412786] [] show_state_filter+0x82/0x110 [ 4599.412786] [] __handle_sysrq+0x11d/0x170 [ 4599.412786] [] write_sysrq_trigger+0x38/0x40 [ 4599.412786] [] proc_reg_write+0x77/0xc0 [ 4599.412786] [] vfs_write+0xce/0x140 [ 4599.412786] [] sys_write+0x53/0xa0 [ 4599.412786] [] system_call_fastpath+0x16/0x1b [ 4599.412786] [<00007ff33a3ddc00>] 0x7ff33a3ddbff [ 4599.412786] lockd S ffff88007a690e40 0 9010 2 0x00000000 [ 4599.412786] ffff8800793c5d60 0000000000000046 ffff8800793c4010 0000000000010b80 [ 4599.412786] 0000000000010b80 0000000000010b80 0000000000010b80 ffff8800793c5fd8 [ 4599.412786] ffff8800793c5fd8 0000000000010b80 ffff88007b464200 ffff88007b4ae300 [ 4599.412786] Call Trace: [ 4599.412786] [] schedule_timeout+0x205/0x2a0 [ 4599.412786] [] svc_recv+0x4e0/0x7c0 [sunrpc] [ 4599.412786] [] lockd+0x140/0x1e0 [lockd] [ 4599.412786] [] kthread+0x96/0xa0 [ 4599.412786] [] kernel_thread_helper+0x4/0x10 [ 4599.412786] Sched Debug Version: v0.10, 3.0.101-97_lustre-default #1 [ 4599.412786] ktime : 4601809.779884 [ 4599.412786] sched_clk : 7398056.461408 [ 4599.412786] cpu_clk : 4599412.786973 [ 4599.412786] jiffies : 4296042148 [ 4599.412786] sched_clock_stable : 0 [ 4599.412786] [ 4599.412786] sysctl_sched [ 4599.412786] .sysctl_sched_latency : 12.000000 [ 4599.412786] .sysctl_sched_min_granularity : 4.000000 [ 4599.412786] .sysctl_sched_wakeup_granularity : 5.000000 [ 4599.412786] .sysctl_sched_child_runs_first : 0 [ 4599.412786] .sysctl_sched_features : 15471 [ 4599.412786] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 4599.412786] [ 4599.412786] cpu#0, 2593.768 MHz [ 4599.412786] .nr_running : 1 [ 4599.412786] .load : 1024 [ 4599.412786] .nr_switches : 91011 [ 4599.412786] .nr_load_updates : 32519 [ 4599.412786] .nr_uninterruptible : -164 [ 4599.412786] .next_balance : 4296.042152 [ 4599.412786] .curr->pid : 8970 [ 4599.412786] .clock : 4599408.844897 [ 4599.412786] .cpu_load[0] : 2048 [ 4599.412786] .cpu_load[1] : 1176 [ 4599.412786] .cpu_load[2] : 839 [ 4599.412786] .cpu_load[3] : 697 [ 4599.412786] .cpu_load[4] : 538 [ 4599.412786] .yld_count : 0 [ 4599.412786] .sched_switch : 0 [ 4599.412786] .sched_count : 95651 [ 4599.412786] .sched_goidle : 36130 [ 4599.412786] .avg_idle : 113044 [ 4599.412786] .ttwu_count : 48764 [ 4599.412786] .ttwu_local : 44212 [ 4599.412786] [ 4599.412786] cfs_rq[0]:/ [ 4599.412786] .exec_clock : 7120.797824 [ 4599.412786] .MIN_vruntime : 0.000001 [ 4599.412786] .min_vruntime : 27840.201408 [ 4599.412786] .max_vruntime : 0.000001 [ 4599.412786] .spread : 0.000000 [ 4599.412786] .spread0 : 0.000000 [ 4599.412786] .nr_spread_over : 4714 [ 4599.412786] .nr_running : 1 [ 4599.412786] .load : 1024 [ 4599.412786] .load_avg : 0.000000 [ 4599.412786] .load_period : 0.000000 [ 4599.412786] .load_contrib : 0 [ 4599.412786] .load_tg : 0 [ 4599.412786] .tg->cfs_bandwidth.timer_active: 0 [ 4599.412786] .throttled : 0 [ 4599.412786] .throttle_count : 0 [ 4599.412786] [ 4599.412786] rt_rq[0]:/ [ 4599.412786] .rt_nr_running : 0 [ 4599.412786] .rt_throttled : 0 [ 4599.412786] .rt_time : 0.000000 [ 4599.412786] .rt_runtime : 950.000000 [ 4599.412786] [ 4599.412786] runnable tasks: [ 4599.412786] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 4599.412786] ---------------------------------------------------------------------------------------------------------- [ 4599.412786] R sh 8970 27836.066458 63 120 27836.066458 25.285097 72.750507 / [ 4599.412786] [ 4599.412786] cpu#1, 2593.768 MHz [ 4599.412786] .nr_running : 0 [ 4599.412786] .load : 0 [ 4599.412786] .nr_switches : 66507 [ 4599.412786] .nr_load_updates : 20857 [ 4599.412786] .nr_uninterruptible : 164 [ 4599.412786] .next_balance : 4296.042149 [ 4599.412786] .curr->pid : 0 [ 4599.412786] .clock : 4601873.008354 [ 4599.412786] .cpu_load[0] : 0 [ 4599.412786] .cpu_load[1] : 256 [ 4599.412786] .cpu_load[2] : 201 [ 4599.412786] .cpu_load[3] : 168 [ 4599.412786] .cpu_load[4] : 146 [ 4599.412786] .yld_count : 4 [ 4599.412786] .sched_switch : 0 [ 4599.412786] .sched_count : 71125 [ 4599.412786] .sched_goidle : 26550 [ 4599.412786] .avg_idle : 1000000 [ 4599.412786] .ttwu_count : 33778 [ 4599.412786] .ttwu_local : 29280 [ 4599.412786] [ 4599.412786] cfs_rq[1]:/ [ 4599.412786] .exec_clock : 15395.621567 [ 4599.412786] .MIN_vruntime : 0.000001 [ 4599.412786] .min_vruntime : 40261.602815 [ 4599.412786] .max_vruntime : 0.000001 [ 4599.412786] .spread : 0.000000 [ 4599.412786] .spread0 : 12421.401407 [ 4599.412786] .nr_spread_over : 4099 [ 4599.412786] .nr_running : 0 [ 4599.412786] .load : 0 [ 4599.412786] .load_avg : 0.000000 [ 4599.412786] .load_period : 0.000000 [ 4599.412786] .load_contrib : 0 [ 4599.412786] .load_tg : 0 [ 4599.412786] .tg->cfs_bandwidth.timer_active: 0 [ 4599.412786] .throttled : 0 [ 4599.412786] .throttle_count : 0 [ 4599.412786] [ 4599.412786] rt_rq[1]:/ [ 4599.412786] .rt_nr_running : 0 [ 4599.412786] .rt_throttled : 0 [ 4599.412786] .rt_time : 0.000000 [ 4599.412786] .rt_runtime : 950.000000 [ 4599.412786] [ 4599.412786] runnable tasks: [ 4599.412786] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 4599.412786] ---------------------------------------------------------------------------------------------------------- [ 4599.412786] Console [trevis-41vm7] disconnected from at 04-04 17:37. Console [trevis-41vm7] connected to at 04-04 17:37. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-97_lustre-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Mar 30 13:45:57 UTC 2017 (b708f71) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37756000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa81, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72320 r8192 d21888 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a81, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910920k/2097140k available (4576k kernel code, 460k absent, 185760k reserved, 7571k data, 1356k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013492] Security Framework initialized [ 0.014370] AppArmor: AppArmor initialized [ 0.016200] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.019284] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020994] Mount-cache hash table entries: 256 [ 0.022206] Initializing cgroup subsys cpuacct [ 0.023118] Initializing cgroup subsys memory [ 0.024044] Initializing cgroup subsys devices [ 0.024942] Initializing cgroup subsys freezer [ 0.025846] Initializing cgroup subsys net_cls [ 0.026747] Initializing cgroup subsys blkio [ 0.028019] Initializing cgroup subsys perf_event [ 0.029115] Disabled fast string operations [ 0.030007] mce: CPU supports 10 MCE banks [ 0.033618] ACPI: Core revision 20110413 [ 0.036239] Enabling x2apic [ 0.036969] Enabled x2apic [ 0.037899] Switched APIC routing to physical x2apic. [ 0.040002] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.040005] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.060011] kvm-clock: cpu 1, msr 0:7fd10a81, secondary cpu clock [ 0.060029] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.064006] Brought up 2 CPUs [ 0.064008] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068169] devtmpfs: initialized [ 0.071111] print_constraints: dummy: [ 0.072064] Time: 17:37:57 Date: 04/04/17 [ 0.073117] NET: Registered protocol family 16 [ 0.074163] ACPI: bus type pci registered [ 0.075107] PCI: Using configuration type 1 for base access [ 0.076557] bio: create slab at 0 [ 0.081340] ACPI: Interpreter enabled [ 0.082175] ACPI: (supports S0 S5) [ 0.083102] ACPI: Using IOAPIC for interrupt routing [ 0.087596] ACPI: No dock devices found. [ 0.088010] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.089841] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.092835] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.094395] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.122330] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.128799] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.130206] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.131580] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.132835] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.134189] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.135550] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.136013] vgaarb: loaded [ 0.136981] PCI: Using ACPI for IRQ routing [ 0.137252] NetLabel: Initializing [ 0.140013] NetLabel: domain hash size = 128 [ 0.140895] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.141868] NetLabel: unlabeled traffic allowed by default [ 0.142902] Switching to clocksource kvm-clock [ 0.146599] AppArmor: AppArmor Filesystem Enabled [ 0.147604] pnp: PnP ACPI init [ 0.148379] ACPI: bus type pnp registered [ 0.149956] pnp: PnP ACPI: found 6 devices [ 0.150817] ACPI: ACPI bus type pnp unregistered [ 0.158185] NET: Registered protocol family 2 [ 0.159309] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.161666] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.166802] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.168812] TCP: Hash tables configured (established 262144 bind 65536) [ 0.169967] TCP reno registered [ 0.170734] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.171829] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.173207] NET: Registered protocol family 1 [ 0.174123] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.175191] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.176239] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.177450] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.178501] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.183785] pci 0000:00:01.2: PCI INT D disabled [ 0.184812] Unpacking initramfs... [ 0.325996] Freeing initrd memory: 8808k freed [ 0.340134] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.341303] Initialise module verification [ 0.342363] audit: initializing netlink socket (disabled) [ 0.343380] type=2000 audit(1491327479.340:1): initialized [ 0.361530] bounce pool size: 64 pages [ 0.362384] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.364581] VFS: Disk quotas dquot_6.5.2 [ 0.365532] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.366838] msgmni has been set to 937 [ 0.368122] alg: No test for stdrng (krng) [ 0.368998] Asymmetric key parser 'x509' registered [ 0.370034] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.371694] io scheduler noop registered [ 0.372536] io scheduler deadline registered (default) [ 0.373630] io scheduler cfq registered [ 0.374738] GHES: HEST is not enabled! [ 0.375608] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.376879] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.617236] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.653093] Non-volatile memory driver v1.3 [ 0.653987] Linux agpgart interface v0.103 [ 0.654921] Fixed MDIO Bus: probed [ 0.655748] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.657984] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.658950] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.659992] mousedev: PS/2 mouse device common for all mice [ 0.661253] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.663320] cpuidle: using governor ladder [ 0.664200] cpuidle: using governor menu [ 0.665044] EFI Variables Facility v0.08 2004-May-17 [ 0.666215] TCP cubic registered [ 0.667002] Registering the dns_resolver key type [ 0.668063] Loading module verification certificates [ 0.669050] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.670997] registered taskstats version 1 [ 0.672070] Magic number: 13:670:642 [ 0.674924] Freeing unused kernel memory: 1356k freed [ 0.676583] Write protecting the kernel read-only data: 10240k [ 0.680688] Freeing unused kernel memory: 1548k freed [ 0.684247] Freeing unused kernel memory: 840k freed doing fast boot [ 0.720046] Disabling lock debugging due to kernel taint [ 0.726852] SCSI subsystem initialized [ 0.749974] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.762390] scsi0 : ata_piix [ 0.763255] scsi1 : ata_piix [ 0.764042] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.765181] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.928043] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.945871] vda: vda1 vda2 vda4 [ 0.947139] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.948225] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.954619] alua: device handler registered [ 0.959220] hp_sw: device handler registered [ 0.963852] emc: device handler registered [ 0.968464] rdac: device handler registered Creating device nodes with udev [ 0.972180] udev: starting version 147 [ 0.989133] usbcore: registered new interface driver usbfs [ 0.990213] usbcore: registered new interface driver hub [ 1.009826] usbcore: registered new device driver usb [ 1.025783] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.030267] uhci_hcd: USB Universal Host Controller Interface driver [ 1.031498] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.033217] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.034217] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.035860] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.036960] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.038179] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.039816] usb usb1: Product: UHCI Host Controller [ 1.040829] usb usb1: Manufacturer: Linux 3.0.101-97_lustre-default uhci_hcd [ 1.042111] usb usb1: SerialNumber: 0000:00:01.2 [ 1.043247] hub 1-0:1.0: USB hub found [ 1.044202] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 17:38:00 2017 [ 1.069862] [drm] Initialized drm 1.1.0 20060810 [ 1.072854] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-97_lustre-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.326252] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: recovering journal [ 1.361079] Refined TSC clocksource calibration: 2593.649 MHz. /dev/vda1: clean, 80150/1310720 files, 553800/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.600131] kjournald starting. Commit interval 15 seconds [ 1.602697] EXT3-fs (vda1): using internal journal [ 1.603666] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 17:38:00 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.737876] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.766890] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.768632] ACPI: Power Button [PWRF] [ 1.770270] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.772511] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.773850] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.774909] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.778931] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 1.779510] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.790886] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.813753] rtc_cmos 00:01: RTC can wake from S4 [ 1.814874] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.816123] rtc0: alarms up to one day, 114 bytes nvram [ 1.821184] NET: Registered protocol family 10 [ 1.828352] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 1.859330] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.533031] device-mapper: uevent: version 1.0.3 [ 2.534073] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hWaiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 6 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h [?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.691868] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 2.953892] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... 6 logical volume(s) in volume group "lvm-Role_MDS" monitored 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... [?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting syslog services7[?25l done8[?25h [?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25hInitializing random number generator7[?25l done8[?25h [?25hWaiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 6 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h [?25hStarting acpid 7[?25l done8[?25h [?25hStarting D-Bus daemon7[?25l done8[?25h [?25h[?25h[?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hLoading kdump 7[?25l done8[?25h [?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting network time protocol daemon (NTPD)7[?25l done8[?25h [?25hStarting MUNGE: munged7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting sssd 7[?25l done8[?25h [?25hStarting automount 7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hStarting INET services. (xinetd)7[?25l done8[?25h [?25hMaster Resource Control: Running /etc/init.d/after.local Waiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 6 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h 7[?25l done8[?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-97_lustre-default (console). trevis-41vm7 login: Console [trevis-41vm7] departed by on pts/56 at 04-04 17:40. Console [trevis-41vm7] joined by on pts/55 at 04-04 17:41. Console [trevis-41vm7] disconnected from at 04-04 19:03. Console [trevis-41vm7] connected to at 04-04 19:03. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-63-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Tue Jun 23 16:02:31 UTC 2015 (4b89d0c) [ 0.000000] Command line: initrd=/images/SLES-11.4-x86_64/initrd ksdevice=bootif lang= console=tty0 console=ttyS0,115200 install=http://10.9.0.10/cblr/links/SLES-11.4-x86_64 text autoyast=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/SLES-11.4-x86_64/linux BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] No AGP bridge found [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 7d9c5000 - 7fea5000 [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1bee801, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007d600000 s71680 r8192 d22528 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7d610801, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/SLES-11.4-x86_64/initrd ksdevice=bootif lang= console=tty0 console=ttyS0,115200 install=http://10.9.0.10/cblr/links/SLES-11.4-x86_64 text autoyast=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/SLES-11.4-x86_64/linux BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Memory: 2013056k/2097140k available (4556k kernel code, 460k absent, 83624k reserved, 7590k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013410] Security Framework initialized [ 0.014230] AppArmor: AppArmor initialized [ 0.015259] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017809] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020616] Mount-cache hash table entries: 256 [ 0.021753] Initializing cgroup subsys cpuacct [ 0.022608] Initializing cgroup subsys memory [ 0.023500] Initializing cgroup subsys devices [ 0.024007] Initializing cgroup subsys freezer [ 0.024857] Initializing cgroup subsys net_cls [ 0.025711] Initializing cgroup subsys blkio [ 0.026568] Initializing cgroup subsys perf_event [ 0.027614] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.032038] ACPI: Core revision 20110413 [ 0.034631] Enabling x2apic [ 0.035361] Enabled x2apic [ 0.036007] Switched APIC routing to physical x2apic. [ 0.038043] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.039070] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.060011] kvm-clock: cpu 1, msr 0:7d710801, secondary cpu clock [ 0.060030] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.060065] Brought up 2 CPUs [ 0.060067] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068020] devtmpfs: initialized [ 0.071317] print_constraints: dummy: [ 0.072049] Time: 19:04:06 Date: 04/04/17 [ 0.073045] NET: Registered protocol family 16 [ 0.074047] ACPI: bus type pci registered [ 0.074933] PCI: Using configuration type 1 for base access [ 0.076516] bio: create slab at 0 [ 0.080831] ACPI: Interpreter enabled [ 0.081617] ACPI: (supports S0 S5) [ 0.082478] ACPI: Using IOAPIC for interrupt routing [ 0.086940] ACPI: No dock devices found. [ 0.087760] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.088026] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.090406] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.092015] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.122214] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.128466] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.129891] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.131208] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.132543] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.133832] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.135126] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.136013] vgaarb: loaded [ 0.136947] PCI: Using ACPI for IRQ routing [ 0.137173] NetLabel: Initializing [ 0.137916] NetLabel: domain hash size = 128 [ 0.140010] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.140945] NetLabel: unlabeled traffic allowed by default [ 0.141926] Switching to clocksource kvm-clock [ 0.142778] AppArmor: AppArmor Filesystem Enabled [ 0.143145] pnp: PnP ACPI init [ 0.143869] ACPI: bus type pnp registered [ 0.145381] pnp: PnP ACPI: found 6 devices [ 0.146198] ACPI: ACPI bus type pnp unregistered [ 0.153470] NET: Registered protocol family 2 [ 0.157792] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.160051] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.164871] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.166789] TCP: Hash tables configured (established 262144 bind 65536) [ 0.167872] TCP reno registered [ 0.168592] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.169631] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.170926] NET: Registered protocol family 1 [ 0.171796] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.172799] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.173807] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.174940] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.175954] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.177548] pci 0000:00:01.2: PCI INT D disabled [ 0.178512] Unpacking initramfs... [ 0.821780] Freeing initrd memory: 37760k freed [ 0.842532] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.843656] Initialise module verification [ 0.844651] audit: initializing netlink socket (disabled) [ 0.845635] type=2000 audit(1491332647.844:1): initialized [ 0.862610] bounce pool size: 64 pages [ 0.863418] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.865359] VFS: Disk quotas dquot_6.5.2 [ 0.866221] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867455] msgmni has been set to 1001 [ 0.868655] alg: No test for stdrng (krng) [ 0.869501] Asymmetric key parser 'x509' registered [ 0.870446] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.871991] io scheduler noop registered [ 0.872788] io scheduler deadline registered [ 0.873714] io scheduler cfq registered (default) [ 0.874832] GHES: HEST is not enabled! [ 0.875651] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.876850] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.177227] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.213097] Non-volatile memory driver v1.3 [ 1.213940] Linux agpgart interface v0.103 [ 1.214824] Fixed MDIO Bus: probed [ 1.215593] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.217701] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.218643] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.219648] mousedev: PS/2 mouse device common for all mice [ 1.220865] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 1.222854] cpuidle: using governor ladder [ 1.223691] cpuidle: using governor menu [ 1.224505] EFI Variables Facility v0.08 2004-May-17 [ 1.225621] TCP cubic registered [ 1.226348] Registering the dns_resolver key type [ 1.227334] Loading module verification certificates [ 1.228314] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 1.230172] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 1.232059] registered taskstats version 1 [ 1.233069] Magic number: 13:814:90 [ 1.235904] Freeing unused kernel memory: 1360k freed [ 1.237517] Write protecting the kernel read-only data: 10240k [ 1.241635] Freeing unused kernel memory: 1568k freed [ 1.245311] Freeing unused kernel memory: 916k freed Moving into tmpfs... done. [ 1.430853] SCSI subsystem initialized [ 1.435833] alua: device handler registered [ 1.438408] emc: device handler registered [ 1.440918] hp_sw: device handler registered [ 1.443445] rdac: device handler registered >>> SUSE Linux Enterprise Server 11 installation program v3.3.108 (c) 1996-2010 SUSE Linux Products GmbH <<< Starting udev... [ 1.455070] udev: starting version 147 [ 1.470107] usbcore: registered new interface driver usbfs [ 1.471134] usbcore: registered new interface driver hub [ 1.477187] [drm] Initialized drm 1.1.0 20060810 [ 1.479478] This driver is only used in secure boot mode as default [ 1.482562] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.483819] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.484836] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 [ 1.490423] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 1.492002] ACPI: Power Button [PWRF] [ 1.505363] usbcore: registered new device driver usb [ 1.515245] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000902000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.520950] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.523192] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.524192] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 1.527318] rtc_cmos 00:01: RTC can wake from S4 [ 1.528384] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.529548] rtc0: alarms up to one day, 114 bytes nvram [ 1.533254] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.534982] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.538380] uhci_hcd: USB Universal Host Controller Interface driver [ 1.539621] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.541313] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.542263] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.543873] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.556415] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.557559] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.558148] FDC 0 is a S82078B [ 1.560032] usb usb1: Product: UHCI Host Controller [ 1.560949] usb usb1: Manufacturer: Linux 3.0.101-63-default uhci_hcd [ 1.562038] usb usb1: SerialNumber: 0000:00:01.2 [ 1.563011] hub 1-0:1.0: USB hub found [ 1.563825] hub 1-0:1.0: 2 ports detected [ 1.571760] vda: vda1 vda2 vda4 [ 1.572366] scsi0 : ata_piix [ 1.575030] scsi1 : ata_piix [ 1.575849] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.576999] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 ok [?1l%G78[ 1.860038] Refined TSC clocksource calibration: 2593.642 MHz. [ 1.880693] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [9;0]Loading basic drivers... ok Starting hardware detection... ok (If a driver is not working for you, try booting with brokenmodules=driver_name.) Red Hat Qemu virtual machine drivers: ata_piix*, ata_generic, pata_acpi Red Hat Virtio block device drivers: virtio_pci* Virtio Storage 0 drivers: virtio_blk* Activating usb devices... ok Realtek RTL-8139/8139C/8139C+ drivers: 8139cp*, 8139too Sending DHCP request to eth0... Loading Installation System (1/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (2/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (3/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (4/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (5/6) -  0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16% 17% 18% 19% 20% 21% 22% 23% 24% 25% 26% 27% 28% 29% 30% 31% 32% 33% 34% 35% 36% 37% 38% 39% 40% 41% 42% 43% 44% 45% 46% 47% 48% 49% 50% 51% 52% 53% 54% 55% 56% 57% 58% 59% 60% 61% 62% 63% 64% 65% 66% 67% 68% 69% 70% 71% 72% 73% 74% 75% 76% 77% 78% 79% 80% 81% 82% 83% 84% 85% 86% 87% 88% 89% 90% 91% 92% 93% 94% 95% 96% 97% 98% 99%100% Loading Installation System (6/6) -  - missing (optional) Starting haveged daemon haveged: haveged starting up haveged: arch: x86 vendor: intel generic: 0 i_cache: 32 d_cache: 32 loop_idx: 30 loop_idxmax: 40 loop_sz: 31836 loop_szmax: 124334 etime: 10770 havege_ndpt 0 7[?25l done8[?25h starting hald... ok starting syslogd (logging to /dev/tty4)... ok starting klogd... ok starting yast... > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.1.1: open serial > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > pci.1: sysfs drivers > pci.2: get sysfs pci data > pci.4: build list > pci.3: macio > pci.4: vio > pci.5: xen > pci.6: ps3 > pci.7: platform > pci.8: of_platform > pci.9: vm > pci.10: virtio > pci.11: ibmebus > pci.12: uisvirtpci > serial.1: read info > serial.2: build list > misc.5: misc data > usb.1: sysfs drivers > usb.2: usb > usb.3.1: joydev mod > usb.3.2: evdev mod > usb.3.3: input > usb.3.4: lp > usb.3.5: serial > braille.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent > hal.1: read hal data > cpu.1: cpuinfo > cpu.2: assign udi > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > pci.1: sysfs drivers > pci.2: get sysfs pci data > pci.4: build list > pci.3: macio > pci.4: vio > pci.5: xen > pci.6: ps3 > pci.7: platform > pci.8: of_platform > pci.9: vm > pci.10: virtio > pci.11: ibmebus > pci.12: uisvirtpci > misc.5: misc data > misc.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent > hal.1: read hal data > bios.1: cmdline > bios.1.1: apm > bios.2: ram > bios.2: rom > bios.3: smp > bios.5: 32 > bios.6: acpi > misc.9: kernel log > misc.1: misc data > misc.2.1: io > misc.2.2: dma > misc.2.3: irq > misc.3: FPU > misc.3.1: DMA > misc.3.2: PIC > misc.3.3: timer > misc.3.4: RTC > cpu.1: cpuinfo > misc.5: misc data > usb.1: sysfs drivers > usb.2: usb > usb.3.1: joydev mod > usb.3.2: evdev mod > usb.3.3: input > usb.3.4: lp > usb.3.5: serial > input.1: joydev mod > input.1.1: evdev mod > input.2: input > kbd.2: uml > cpu.1: cpuinfo > kbd.3: serial console > kbd.2: assign udi > int.2: cdrom > int.3: media > int.4: floppy > int.5: edd > int.5.1: bios > int.6: mouse > int.15: system info > int.7: hdb > int.7.1: modules > int.8: usbscsi > int.9: hotplug > int.10: modem > int.11: wlan > int.12: udev > int.13: device names > int.14: soft raid > int.15: geo > int.16: parent [?25lProbing connected terminal... 78 Initializing virtual console...  H H H H H H H H H [?25h Found an unknown terminal on /dev/console (80 columns x 24 lines). 0 *** Starting YaST2 *** (B)0[?7h[?1h=[?1h=[?1h= YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com   Initializing ...                     [Help] [Back] [Abort] [Next]                                           [Next]   [Next] [Help]      F1 Help  F9 Abort [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkx Downloading installation system language extension... x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj     YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com                                                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com                                                              [Help] [Back] [Abort] [Next]    F1 Help  F9 AbortInitializing... Initializing the installation environment...  YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Initializing...                             Initializing the installation environment...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort  - Probe hardware - Retrieve & Read Control File - Process Profiles and Rules - Parse control file ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Preparing System for Automatic Installation  => Preprobing stage obing hardware...  Analyzing the Computer           - Probe USB devices - Probe FireWire devices - Probe floppy disks devices - Probe hard disk controllers - Load kernel modules for hard disk controllers - Probe hard disks - Search for Linux partitions - Search for system files - Evaluate update possibility - Initialize package manager±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  System Probing  Probing USB devices...  =>  1  x  FireWire devices...  =>  2  x  floppy disks devices... =>  3 x  hard disk controllers... =>  4 x  Loading kernel modules for hard disk controllers... =>  50 x  Probing hard disks...  => 60%  x  Searching for Linux partitions... => 7  x  system files...  => 8  x  Evaluating update possibility... => 9  x  Initializing package manager...  => [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxInitializing repositories...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqjartitions files ssibility manager    Adding the Repository... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading: http://10.9.0.10/cblr/.../SLES-11.4-x86_64/media.1/products  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± links/SLES-11.4-x86_64/media.1/media   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± content.asc   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± key  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±    100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../descr/32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Basis-Devel  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/Dom0-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Dom0-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± _KVM-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/Minimal-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Minimal-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/WBEM-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± WBEM-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± apparmor-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/base-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± base-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhcp_dns_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.p  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± irectory_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ocumentation-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± file_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gateway_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pa  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/gnome-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/jeos-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jeos-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/kde-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± kde-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vm_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lamp_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/laptop-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± laptop-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mail_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.g  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/ofed-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ofed-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± racle_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cblr/links/.../setup/descr/packages.DU.gz   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± en  8  13  22  31  9  ±±±44  53%  62%  71  80  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± use/setup/descr/packages.gz  6  12  9  25  30  6  ±±±±±41  48 54%  9%  65  70  7  83  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± tterns   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../print_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sap_server-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± le-manager_client-1-0.17.30  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-1-0.17.30.x86_64.  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± descr/x11-11-38.47.37.x86_64.pat.gz   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± x11-32bit  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± en_server-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 32bit-11-38.47.37.x86_64.pat.gz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cblr/links/SLES-11.4-x86_64/license.tar.gz   22  ±±46  69%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± .../gpg-pubkey-307e3d54-53287cdc.asc   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 9db7c82-510a966b  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± d25d3d9-36e12d04  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 50a3dd1c-50f35137  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 9c800aca-53287d18  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± b37b98a992f  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Building repository 'SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109' cache  100%  7  9  100  Initializing package manager...    ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading: http://10.9.0.10/.../SLES-11.4-x86_64/add_on_products.xml cblr/links/SLES-11.4-x86_64/add_on_products x    Retrieving control file (/cblr/svc/op/ks/system/trevis-41vm7) from HTTP serv Processing Profiles and Rules  arsing control file  Initial Configuration    F10 Next 8 Back  F9 Abort  F10 Next [Help] [Help] [Help] [Next] MM- Configure General Settings - Execute pre-install user scripts - Set up language - Create partition plans   - Configure Software selections    - Configure Bootloader  - Configure runlevel         Preparing System for Automated Installation [Next] [Help] Configuring general settings...  =>  14  x  Executing pre-install user scripts...  =>  28  x  Setting up language  => Configuring language... [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkx Downloading installation system language extension... x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjigure runlevel     YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Preparing System for Automated Installation        x Configure General Settings    x Execute pre-install user scripts    => Set up language    - Create partition plans    - Configure Software selections    - Configure Bootloader    - Configure runlevel                      Configuring language...     ±±±±±±±±±±±±±±±28%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±         [Help] [Back] [Abort] [Next]    F1 Help  F8 Back  F9 Abort  F10 Next YaST2 - installation @ trevis-41vm7.trevis.hpdd.intel.com    Preparing System for Automated Installation        x Configure General Settings    x Execute pre-install user scripts    => Set up language    - Create partition plans    - Configure Software selections    - Configure Bootloader    - Configure runlevel                      Configuring language...     ±±±±±±±±±±±±±±±28%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±         [Help] [Back] [Abort] [Next]    F1 Help  F8 Back  F9 Abort  F10 Next ±±±±42x reating partition plans...=> 57% x onfiguring Software selections...=>[?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Error x x These packages could not be found in the software repositories: x x compat-glibc-headersx x policycoreutils-pythonx x10x xx x[OK] [Stop]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj9 lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj8lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj7lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj6lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj5lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj4lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj3lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj2lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj1lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjlqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx x[OK]x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj0 Create partition plans  Configure Software selections  Configure Bootloader  Configure runlevel              71%  x Bootloader...  =>85  x runlevel...  =>100  x   9 Abort  F10 Next  8 Back  F9 Abort  F10 Next  ackage Installation           9 Abort   erform  lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx xx xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  F10 Next 8 Back  F9 Abort  F10 Next Preparing disks... Deleting logical volume /dev/lvm-Role_MDS/forcemnt  Deleting logical volume /dev/lvm-Role_MDS/forcemnt S1  Deleting logical volume /dev/lvm-Role_MDS/S1  1 P4 Deleting logical volume /dev/lvm-Role_MDS/P4 2 3 Deleting logical volume /dev/lvm-Role_MDS/P3  3 2 Deleting logical volume /dev/lvm-Role_MDS/P2 1 Deleting logical volume /dev/lvm-Role_MDS/P1 4 Removing volume group lvm-Role_MDS  Removing volume group lvm-Role_MDS  5 Deleting partition /dev/vda4  Deleting partition /dev/vda4  6 2 Deleting partition /dev/vda2  7 1 Deleting partition /dev/vda1 8 Setting disk label of /dev/vda to MSDOS Setting disk label of /dev/vda to MSDOS Creating volume /dev/vda1   w   xCreating volume /dev/vda1 v   9 2  x wx   xCreating volume /dev/vda2 v   10 Setting type of partition /dev/vda2 to 82  x wx   xSetting type of partition /dev/vda2 to 82 v   1 Formatting partition /dev/vda1 (19.99 GB) with ext3  x wx   xFormatting partition /dev/vda1 (19.99 GB) with ext3 v   2  3 4  5  6  7 8  9  10  1 2  3  4  5 6  7  8  9 20  1  2  3 4  5  6 7  8  9  30 1  2  3  4 5  6  7  8 9  40  1  ±±±±42 3  ±±±44  ±±45  ±46 7  48 49 50 1 2% 3%  4 5%  6%  7%  8 9%  60  1  2 3  4  5  6 7  8  9  70 1  2  3  4 5  6  7 8  9  80  1 2  3  4  5 6  7  8  9 90  1  2  3 4  5  6  7 100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 2 (13.67swap  x wx   xFormatting partition /dev/vda2 (13.67 GB) with swap v   3 Mounting /dev/vda2 to swap   x wx   xMounting /dev/vda2 to swap v  Adding entry for mount point swap to /etc/fstab  x wx   xAdding entry for mount point swap to /etc/fstab v   4 Mounting /dev/vda1 to /   x wx   xMounting /dev/vda1 to / v   5 Adding entry for mount point / to /etc/fstab  x w x   xAdding entry for mount point / to /etc/fstab v  6  ackage                                9 Abort   erform  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xMediaxSizexPackagesxTimex xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Adding entry for mount point / to /etc/fstab  100% Preparing disks...   ±±±±±±±±±±±±±±±±±±±±±±±±16%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2 Installing Packages...  xSizexPackagesxTq Total x 1.09 GBx 643x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 403.54 MBx 433x http-10.9.0.10-ae2fc03cxxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 1.09 GBx 643x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 403.54 MBx 433x http-10.9.0.10-ae2fc03cxxx tu Downloading master-boot-code (download size 5.00 kB) Downloading master-boot-code (download size 5.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing master-boot-code-1.14-70.11.i586.rpm (installed size 3.00 kB) (Remaining: 1.09 GB) Installing master-boot-code-1.14-70.11.i586.rpm (installed size 3.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  96%  100  2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading adaptec-firmware (download size 22.00 kB)  Downloading adaptec-firmware (download size 22.00 kB)  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing adaptec-firmware-1.35-2.15.4.noarch.rpm (installed size 27.00 k Installing adaptec-firmware-1.35-2.15.4.noarch.rpm (installed size 27.00  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  80%  99  100  1 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading branding-SLES (download size 85.00 kB)   w   x~ kB) x    xDownloading branding-SLES (download size 85.00 kB) v   18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing branding-SLES-11-3.20.30.noarch.rpm (installed size 2.00 kB)  x w   xInstalling branding-SLES-11-3.20.30.noarch.rpm (installed size 2.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 1.09 GBx 640x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 403.51 MBx 430x http-10.9.0.10-ae2fc03cxxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading brocade-firmware (download size 999.00 kB)   x w   xDownloading brocade-firmware (download size 999.00 kB) v   5  11  6  21  7  32  7  ±±±±43  48 53%  9%  64  9  74  80  5  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing brocade-firmware-3.2.3.0-0.11.35.noarch.rpm (installed size 3.0  x w   xInstalling brocade-firmware-3.2.3.0-0.11.35.noarch.rpm (installed size 3v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  9  24  30  6  ±±±±42  48 54%  9%  65  72  8  84  9  95  100  0.44 29  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading desktop-translations (download size 2.42 MB)   x x   x~.06 MB) w    xDownloading desktop-translations (download size 2.42 MB) v   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing desktop-translations-11.1-10.50.1.noarch.rpm (installed size 17  x w   xInstalling desktop-translations-11.1-10.50.1.noarch.rpm (installed size v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  383 8 7  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2-branding-SLES (download size 6.00 kB)   x x   x~17.00 MB) w    xDownloading glib2-branding-SLES (download size 6.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-branding-SLES-2.16-47.3.noarch.rpm (installed size 4.00 k 7  x x   xInstalling glib2-branding-SLES-2.16-47.3.noarch.rpm (installed size 4.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  97%  100  7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gnome-icon-theme (download size 3.52 MB)   x   x~ kB) x    xDownloading gnome-icon-theme (download size 3.52 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gnome-icon-theme-2.28.0-1.2.11.noarch.rpm (installed size 10.43  x   xInstalling gnome-icon-theme-2.28.0-1.2.11.noarch.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  73.01 6 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hicolor-icon-theme (download size 35.00 kB)   x   x~43 MB) x    xDownloading hicolor-icon-theme (download size 35.00 kB) q   ±±45  92%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hicolor-icon-theme-0.10-102.22.noarch.rpm (installed size 48.00 6  x   xInstalling hicolor-icon-theme-0.10-102.22.noarch.rpm (installed size 48.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  23  8  33  8  ±±±±43  48 53%  8%  63%  8  95  100  2.96 5  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icom-firmware (download size 18.00 kB)   x   x~00 kB) x    xDownloading icom-firmware (download size 18.00 kB) q   88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icom-firmware-1.0-124.28.4.noarch.rpm (installed size 47.00 kB)  x   xInstalling icom-firmware-1.0-124.28.4.noarch.rpm (installed size 47.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  16  99%  100  1 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kernel-firmware (download size 15.15 MB)   x   x~B) x    xDownloading kernel-firmware (download size 15.15 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-firmware-20110923-0.57.21.noarch.rpm (installed size 39.  x   xInstalling kernel-firmware-20110923-0.57.21.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  33.77 3 2  3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading latex2html-pngicons (download size 13.00 kB)   x   x~9.14 MB) x    xDownloading latex2html-pngicons (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing latex2html-pngicons-2002.2.1-604.31.noarch.rpm (installed size 2  x   xInstalling latex2html-pngicons-2002.2.1-604.31.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  33  9  ±46  54%  62%  9  77  82  8  95  100  6 2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading licenses (download size 340.00 kB)   x   x~e 8.00 kB) x    xDownloading licenses (download size 340.00 kB) q   9  14  23  8  37  ±±±±42  ±47  56%  61%  70  5  80  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing licenses-20070810-2.15.noarch.rpm (installed size 3.29 MB)  x   xInstalling licenses-20070810-2.15.noarch.rpm (installed size 3.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  6  81  6  91  6  100  0.48 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man-pages (download size 2.20 MB)   x   xDownloading man-pages (download size 2.20 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-pages-3.15-2.29.1.noarch.rpm (installed size 2.03 MB)  x   xInstalling man-pages-3.15-2.29.1.noarch.rpm (installed size 2.03 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  28.45 0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man-pages-posix (download size 2.33 MB)   x   xDownloading man-pages-posix (download size 2.33 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-pages-posix-2003a-3.7.noarch.rpm (installed size 2.20 MB)  x   xInstalling man-pages-posix-2003a-3.7.noarch.rpm (installed size 2.20 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  6.2 19 1  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading providers (download size 24.00 kB)   x   xDownloading providers (download size 24.00 kB) q   66%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing providers-2006.11.21-105.21.noarch.rpm (installed size 136.00 k 1  x   xInstalling providers-2006.11.21-105.21.noarch.rpm (installed size 136.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  25  31  79%  9  100  11 8  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-sam-data (download size 5.06 MB)   x   x~ kB) x    xDownloading suse-sam-data (download size 5.06 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-sam-data-0.8.5-0.4.9.noarch.rpm (installed size 5.01 MB)  x   xInstalling suse-sam-data-0.8.5-0.4.9.noarch.rpm (installed size 5.01 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  91  6  100  1.10 7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-trans-en_US (download size 129.00 kB)   x   xDownloading yast2-trans-en_US (download size 129.00 kB) q   12  24  37  49 62%  74  87  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-trans-en_US-2.17.35-0.7.37.noarch.rpm (installed size 584  x   xInstalling yast2-trans-en_US-2.17.35-0.7.37.noarch.rpm (installed size 5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  33  8  ±±±44  50 61%  8  73  80  5  96  100  0.53 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-trans-stats (download size 9.00 kB)   x   x~84.00 kB) x    xDownloading yast2-trans-stats (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-trans-stats-2.15.0-2.17.noarch.rpm (installed size 1.00 k  x   xInstalling yast2-trans-stats-2.15.0-2.17.noarch.rpm (installed size 1.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  5  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading boost-license (download size 23.00 kB)   x   x~ kB) x    xDownloading boost-license (download size 23.00 kB) q   70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing boost-license-1.36.0-12.6.49.x86_64.rpm (installed size 2.00 kB  x   xInstalling boost-license-1.36.0-12.6.49.x86_64.rpm (installed size 2.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  92%  100  4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cracklib-dict-full (download size 3.60 MB)   x   x~kB) x    xDownloading cracklib-dict-full (download size 3.60 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cracklib-dict-full-2.8.12-43.16.x86_64.rpm (installed size 10.4  x   xInstalling cracklib-dict-full-2.8.12-43.16.x86_64.rpm (installed size 10q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  10.07 3 1021.32 M  3  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading filesystem (download size 51.00 kB)   x   x~.46 MB) x    xDownloading filesystem (download size 51.00 kB) q   31  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing filesystem-11.1-3.5.3.x86_64.rpm (installed size 0.00 B) 021.32 MB)  x   xInstalling filesystem-11.1-3.5.3.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ofed-doc (download size 220.00 kB)   x   xDownloading ofed-doc (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-doc-1.5.4.1-20.26.x86_64.rpm (installed size 829.00 kB)  x   xInstalling ofed-doc-1.5.4.1-20.26.x86_64.rpm (installed size 829.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  8  34  ±±±±±41  ±46  51 6%  62%  7  7  84  90  7  100  09.26 1 0.51  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sash (download size 317.00 kB)   x   xDownloading sash (download size 317.00 kB) q   1  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sash-3.7-170.1.30.x86_64.rpm (installed size 769.00 kB) 0.51  x   xInstalling sash-3.7-170.1.30.x86_64.rpm (installed size 769.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  ±±±±±41  49 58%  66%  74  83  91  8  100  8.51 0 19.76  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-release-DVD (download size 4.00 kB)   x   xDownloading sles-release-DVD (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-release-DVD-11.4-1.109.x86_64.rpm (installed size 1.00 kB) 19.76  x   xInstalling sles-release-DVD-11.4-1.109.x86_64.rpm (installed size 1.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  75%  100  09  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading terminfo (download size 397.00 kB)   x   x~B) x    xDownloading terminfo (download size 397.00 kB) q   7  12  20  8  36  ±±±44  52% 60%  8  76  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing terminfo-5.6-90.55.x86_64.rpm (installed size 1.53 MB)  x   xInstalling terminfo-5.6-90.55.x86_64.rpm (installed size 1.53 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  6.97 8 8.22  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading terminfo-base (download size 114.00 kB)   x   xDownloading terminfo-base (download size 114.00 kB) q   16  30  ±±±44  58%  73%  87  100   30 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing terminfo-base-5.6-90.55.x86_64.rpm (installed size 996.00 kB) 8.22  x   xInstalling terminfo-base-5.6-90.55.x86_64.rpm (installed size 996.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  9  25  31  8  ±±±44  50 7%  63%  9  76  82  8  94  9  100  00 7 7.25  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-country-data (download size 44.00 kB)   x   xDownloading yast2-country-data (download size 44.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-country-data-2.17.55-0.7.1.x86_64.rpm (installed size 166 7.25  x   xInstalling yast2-country-data-2.17.55-0.7.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  81%  99  100  5.84 6 09  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxq Total x 1017.09 MBx 616x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-ae2fc03cxxx tu Downloading pam-doc (download size 496.00 kB)   x   x~66.00 kB) x    xDownloading pam-doc (download size 496.00 kB) q   6  11  8  24  31  7  ±±±±43  50 6%  63%  9  76  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-doc-1.1.5-0.17.2.x86_64.rpm (installed size 1.62 MB) 09  x   xInstalling pam-doc-1.1.5-0.17.2.x86_64.rpm (installed size 1.62 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  9  24  31  6  ±±±±±41  ±46  51 6%  61%  6  73  80  6  93  9  100  SizexPackagesxT1015.47 MBx 616x x x 305.84 MBx 406x x x qu  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading release-notes-sles (download size 517.00 kB)   x   xDownloading release-notes-sles (download size 517.00 kB) q   6  12 - 47.74 kB/s (download size 517.00 kB)  8  24  30  7  ±±±±43  49 55%  61%  8  74  80  6  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing release-notes-sles-11.4.26-0.35.1.x86_64.rpm (installed size 1. 5.47  x   xInstalling release-notes-sles-11.4.26-0.35.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  26  33  40  ±±45  52% 9%  65  71  8  85  92  7  100  3.69  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading snmp-mibs (download size 263.00 kB)   x   x~1.78 MB) x    xDownloading snmp-mibs (download size 263.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  9  85  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing snmp-mibs-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 1.54 MB) 3.69  x   xInstalling snmp-mibs-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 1.54 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  9  24  30  8  ±±45  50 5%  60%  8  73  8  84  90  7  100  2.15  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-i18ndata (download size 3.14 MB)   x   x~B) x    xDownloading glibc-i18ndata (download size 3.14 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-i18ndata-2.11.3-17.102.1.x86_64.rpm (installed size 10.45 2.15  x   xInstalling glibc-i18ndata-2.11.3-17.102.1.x86_64.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  01.69  2 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-schema (download size 72.00 kB)   x   x~45 MB) x    xDownloading yast2-schema (download size 72.00 kB) q   21  ±±±44  66%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-schema-2.17.7-8.1.x86_64.rpm (installed size 925.00 kB) 01.69  x   xInstalling yast2-schema-2.17.7-8.1.x86_64.rpm (installed size 925.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  36  ±±±±43  50 7%  64%  71  7  84  91  8  100  0.7  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-admin_en-pdf (download size 3.07 MB)   x   xDownloading sles-admin_en-pdf (download size 3.07 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-admin_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4.09 M 0.7  x   xInstalling sles-admin_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4.09q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  9  35  ±±±±±41  ±47  53%  8%  64%  70  6  81  7  93  9  100   996.70  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-manuals_en (download size 14.49 MB)   x   x~ MB) x    xDownloading sles-manuals_en (download size 14.49 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-manuals_en-11.4-0.33.1.noarch.rpm (installed size 22.03 MB 996.70 MB)   x   xInstalling sles-manuals_en-11.4-0.33.1.noarch.rpm (installed size 22.03 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  74.67  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-storage_en-pdf (download size 2.90 MB)   x   x~MB) x    xDownloading sles-storage_en-pdf (download size 2.90 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-storage_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.84 74.67  x   xInstalling sles-storage_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  ±47  52% 8%  63%  8  73  8  84  9  94  9  100  0.83  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-installquick_en-pdf (download size 4.22 MB)   x   x~84 MB) x    xDownloading sles-installquick_en-pdf (download size 4.22 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-installquick_en-pdf-11.4-0.33.1.noarch.rpm (installed size 0.83  x   xInstalling sles-installquick_en-pdf-11.4-0.33.1.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  31  6  ±±±±±41  ±46  52% 8%  63%  9  74  9  84  9  94  9  100  66.41  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-deployment_en-pdf (download size 3.65 MB)   x   x~ze 4.42 MB) x    xDownloading sles-deployment_en-pdf (download size 3.65 MB) q   5  10  5  20  5  30  5 - 1.29 MB/s (download size 3.65 MB)  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-deployment_en-pdf-11.4-0.33.1.noarch.rpm (installed size 4 66.41  x   xInstalling sles-deployment_en-pdf-11.4-0.33.1.noarch.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  6  ±±±±±41  ±46  51 6%  62%  7  72  7  82  8  93  8  100  xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 962.08 MBx 606x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-ae2fc03cxxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-tuning_en-pdf (download size 706.00 kB)   x   x~ 4.33 MB) x    xDownloading sles-tuning_en-pdf (download size 706.00 kB) q   5  10  6  21  7  32  8  ±±±±43  48 54%  9%  64  9  75  80  6  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-tuning_en-pdf-11.4-0.33.1.noarch.rpm (installed size 1.04 2.08  x   xInstalling sles-tuning_en-pdf-11.4-0.33.1.noarch.rpm (installed size 1.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  1.04  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-security_en-pdf (download size 2.19 MB)   x   x~4 MB) x    xDownloading sles-security_en-pdf (download size 2.19 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-security_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3.0 1.04  x   xInstalling sles-security_en-pdf-11.4-0.33.1.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  7  ±±±±43  49 55%  61%  7  72  8  84  90  6  100  58.00  4  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfs-doc (download size 108.00 kB)   x   x~.04 MB) x    xDownloading nfs-doc (download size 108.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-doc-1.2.3-18.47.1.x86_64.rpm (installed size 363.00 kB) 58.00  x   xInstalling nfs-doc-1.2.3-18.47.1.x86_64.rpm (installed size 363.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  29  ±47  61%  6  72  80  7  92  8  100  7.65  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc (download size 1.86 MB)   x   xDownloading glibc (download size 1.86 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-2.11.3-17.102.1.x86_64.rpm (installed size 5.30 MB) 7.65  x   xInstalling glibc-2.11.3-17.102.1.x86_64.rpm (installed size 5.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  71  6  81  6  91  6  100  2.3  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zlib (download size 45.00 kB)   x   xDownloading zlib (download size 45.00 kB) q   35  70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zlib-1.2.7-0.14.1.x86_64.rpm (installed size 87.00 kB) 2.3  x   xInstalling zlib-1.2.7-0.14.1.x86_64.rpm (installed size 87.00 kB) [0;1;7mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  73%  99  100  27  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-base (download size 1.10 MB)   x   xDownloading perl-base (download size 1.10 MB) q   6  11  6  21  6  31  6  ±±±±±41  ±46  51 7%  63%  8  73  8  84  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-base-5.10.0-64.80.1.x86_64.rpm (installed size 3.30 MB) 27  x   xInstalling perl-base-5.10.0-64.80.1.x86_64.rpm (installed size 3.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  32  7  ±±±±42  ±47  52% 7%  62%  8  73  9  85  90  6  100  48.96  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dosfstools (download size 85.00 kB)   x   xDownloading dosfstools (download size 85.00 kB) q   18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dosfstools-3.0.26-3.1.x86_64.rpm (installed size 217.00 kB) 48.96  x   xInstalling dosfstools-3.0.26-3.1.x86_64.rpm (installed size 217.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±47  60%  85  91  6  100  75 599 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ethtool (download size 94.00 kB)   x   xDownloading ethtool (download size 94.00 kB) q   16  33  50 67%  84  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ethtool-6.2.6.39-0.29.3.x86_64.rpm (installed size 262.00 kB) 75  x   xInstalling ethtool-6.2.6.39-0.29.3.x86_64.rpm (installed size 262.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 73%  8  90  7  100  50  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgomp1 (download size 66.00 kB)   x   xDownloading libgomp1 (download size 66.00 kB) q   24  48 72%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgomp1-5.3.1+r233831-10.1.x86_64.rpm (installed size 140.00 k 50  x   xInstalling libgomp1-5.3.1+r233831-10.1.x86_64.rpm (installed size 140.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  91%  9  100  36  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsqlite3-0 (download size 288.00 kB)   x   x~ kB) x    xDownloading libsqlite3-0 (download size 288.00 kB) q   5  11  6  22  7  33  8  ±±±44  50 5%  61%  6  72  7  83  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsqlite3-0-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 766.00 36  x   xInstalling libsqlite3-0-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 766.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  25  33  ±±±±±41  50 8%  66%  75  83  91  9  100  7.61  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libbz2-1 (download size 44.00 kB)   x   x~0 kB) x    xDownloading libbz2-1 (download size 44.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libbz2-1-1.0.5-34.255.1.x86_64.rpm (installed size 82.00 kB) 7.61  x   xInstalling libbz2-1-1.0.5-34.255.1.x86_64.rpm (installed size 82.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  88  99  100  53  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libquadmath0 (download size 142.00 kB)   x   xDownloading libquadmath0 (download size 142.00 kB) q   11  22  33  ±±45  56%  67%  78  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libquadmath0-5.3.1+r233831-10.1.x86_64.rpm (installed size 252. 53  x   xInstalling libquadmath0-5.3.1+r233831-10.1.x86_64.rpm (installed size 25q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  50 76%  99  100  28  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libusb-1_0-0 (download size 91.00 kB)   x   x~2.00 kB) x    xDownloading libusb-1_0-0 (download size 91.00 kB) q   17  34  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libusb-1_0-0-1.0.9-8.4.4.1.x86_64.rpm (installed size 239.00 kB 28  x   xInstalling libusb-1_0-0-1.0.9-8.4.4.1.x86_64.rpm (installed size 239.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  37  64%  90  8  100  05  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc_s1 (download size 49.00 kB)   x   x~kB) x    xDownloading libgcc_s1 (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc_s1-5.3.1+r233831-10.1.x86_64.rpm (installed size 91.00 k 05  x   xInstalling libgcc_s1-5.3.1+r233831-10.1.x86_64.rpm (installed size 91.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  70%  99  100  6.96  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpng12-0 (download size 72.00 kB)   x   x~ kB) x    xDownloading libpng12-0 (download size 72.00 kB) q   22  ±±±44  67%  89  100   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpng12-0-1.2.31-5.43.1.x86_64.rpm (installed size 160.00 kB) 6.96  x   xInstalling libpng12-0-1.2.31-5.43.1.x86_64.rpm (installed size 160.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  80%  99  100  81  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgfortran3 (download size 273.00 kB)   x   x~) x    xDownloading libgfortran3 (download size 273.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgfortran3-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.14 81  x   xInstalling libgfortran3-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  5.67  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++6 (download size 383.00 kB)   x   x~14 MB) x    xDownloading libstdc++6 (download size 383.00 kB) q   8  16  25  33  ±±±±±41  50 8%  66%  71  9  87  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++6-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.50 M 5.67  x   xInstalling libstdc++6-5.3.1+r233831-10.1.x86_64.rpm (installed size 1.50q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  25  33  ±±±±±41  50 8%  66%  75  83  91  9  100  4.1  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libicu (download size 4.24 MB)   x   x~ MB) x    xDownloading libicu (download size 4.24 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libicu-4.0-46.1.x86_64.rpm (installed size 16.52 MB) 4.1  x   xInstalling libicu-4.0-46.1.x86_64.rpm (installed size 16.52 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  27.65  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 927.65 MBx 588x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 305.84 MBx 406x http-10.9.0.10-ae2fc03cxxx tu Downloading xorg-x11-libXdmcp (download size 11.00 kB)   x   xDownloading xorg-x11-libXdmcp (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXdmcp-7.4-1.15.x86_64.rpm (installed size 23.00 kB) 27.65  x   xInstalling xorg-x11-libXdmcp-7.4-1.15.x86_64.rpm (installed size 23.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 5 3  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXau (download size 8.00 kB)   x   x~B) x    xDownloading xorg-x11-libXau (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXau-7.4-1.15.x86_64.rpm (installed size 15.00 kB) 3  x   xInstalling xorg-x11-libXau-7.4-1.15.x86_64.rpm (installed size 15.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 4 2  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libICE (download size 41.00 kB)   x   xDownloading xorg-x11-libICE (download size 41.00 kB) q   38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libICE-7.4-1.15.x86_64.rpm (installed size 101.00 kB) 2  x   xInstalling xorg-x11-libICE-7.4-1.15.x86_64.rpm (installed size 101.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  63%  99  100  7 3 5  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vlan (download size 34.00 kB)   x   x~) x    xDownloading vlan (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vlan-1.9-62.24.1.x86_64.rpm (installed size 98.00 kB) 5  x   xInstalling vlan-1.9-62.24.1.x86_64.rpm (installed size 98.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  30  72%  90  6  100  61 2 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading utempter (download size 19.00 kB)   x   xDownloading utempter (download size 19.00 kB) q   86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing utempter-0.5.5-106.20.1.x86_64.rpm (installed size 42.00 kB) 4  x   xInstalling utempter-0.5.5-106.20.1.x86_64.rpm (installed size 42.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  40  98%  100  56 1 38  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tunctl (download size 7.00 kB)   x   xDownloading tunctl (download size 7.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tunctl-20070815-10.22.x86_64.rpm (installed size 11.00 kB) 38  x   xInstalling tunctl-20070815-10.22.x86_64.rpm (installed size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  5 0 7  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcpd (download size 78.00 kB)   x   xDownloading tcpd (download size 78.00 kB) q   2  ±±±±±41  61%  82  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcpd-7.6-856.13.x86_64.rpm (installed size 210.00 kB) 7  x   xInstalling tcpd-7.6-856.13.x86_64.rpm (installed size 210.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  24  9  40  50 5%  65%  88  95  100  3 399 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sdpnetstat (download size 47.00 kB)   x   xDownloading sdpnetstat (download size 47.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sdpnetstat-1.60-32.26.x86_64.rpm (installed size 119.00 kB) 1  x   xInstalling sdpnetstat-1.60-32.26.x86_64.rpm (installed size 119.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  84%  99  100  23 8 05  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rds-tools (download size 34.00 kB)   x   xDownloading rds-tools (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rds-tools-2.0.4-6.18.x86_64.rpm (installed size 83.00 kB) 05  x   xInstalling rds-tools-2.0.4-6.18.x86_64.rpm (installed size 83.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  40  92%  9  100  15 7 6.97  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading prctl (download size 8.00 kB)   x   xDownloading prctl (download size 8.00 kB) q   100%  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing prctl-1.3-488.22.x86_64.rpm (installed size 12.00 kB) 6.97  x   xInstalling prctl-1.3-488.22.x86_64.rpm (installed size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  99  100  4 6 6  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading popt (download size 46.00 kB)   x   xDownloading popt (download size 46.00 kB) q   34  69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing popt-1.7-37.63.64.1.x86_64.rpm (installed size 69.00 kB) 6  x   xInstalling popt-1.7-37.63.64.1.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  48 53%  60%  6  71  6  81  6  92  7  100  07 5 89  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pax (download size 58.00 kB)   x   xDownloading pax (download size 58.00 kB) q   27  54%  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pax-3.4-135.21.x86_64.rpm (installed size 118.00 kB) 89  x   xInstalling pax-3.4-135.21.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  91%  9  100  4.96 4 77  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading patch (download size 63.00 kB)   x   xDownloading patch (download size 63.00 kB) q   25  50 76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing patch-2.5.9-252.22.2.x86_64.rpm (installed size 118.00 kB) 77  x   xInstalling patch-2.5.9-252.22.2.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  80%  8  99  100  84 3 66  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netdate (download size 13.00 kB)   x   xDownloading netdate (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netdate-1.2-713.22.x86_64.rpm (installed size 19.00 kB) 66  x   xInstalling netdate-1.2-713.22.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  78%  87  99  100  2 2 4  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netcat (download size 59.00 kB)   x   xDownloading netcat (download size 59.00 kB) q   26  54%  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netcat-1.10-1005.22.x86_64.rpm (installed size 129.00 kB) 4  x   xInstalling netcat-1.10-1005.22.x86_64.rpm (installed size 129.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  64%  70  5  80  6  94  9  100  70 1 51  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading net-tools (download size 241.00 kB)   x   xDownloading net-tools (download size 241.00 kB) q   6  13  9  26  33  9  ±46  53%  9%  66  73  9  86  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing net-tools-1.60-725.35.1.x86_64.rpm (installed size 758.00 kB) 51  x   xInstalling net-tools-1.60-725.35.1.x86_64.rpm (installed size 758.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  24  32  7  ±±±44  51 9%  65  71  7  83  9  94  9  100  3.96 0 5.77  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mingetty (download size 15.00 kB)   x   xDownloading mingetty (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mingetty-1.0.7s-92.22.x86_64.rpm (installed size 25.00 kB) 5.77  x   xInstalling mingetty-1.0.7s-92.22.x86_64.rpm (installed size 25.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  90%  9  100  3 89 5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lsscsi (download size 30.00 kB)   x   xDownloading lsscsi (download size 30.00 kB) q   53%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsscsi-0.23-0.12.1.x86_64.rpm (installed size 63.00 kB) 5  x   xInstalling lsscsi-0.23-0.12.1.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  92  9  100  87 8 69  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxtables9 (download size 33.00 kB)   x   xDownloading libxtables9 (download size 33.00 kB) q   ±47  96%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxtables9-1.4.16.3-1.37.x86_64.rpm (installed size 47.00 kB) 69  x   xInstalling libxtables9-1.4.16.3-1.37.x86_64.rpm (installed size 47.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 7 4  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxcrypt (download size 38.00 kB)   x   x~) x    xDownloading libxcrypt (download size 38.00 kB) q   ±±±±42  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxcrypt-3.0.3-0.6.1.x86_64.rpm (installed size 132.00 kB) 4  x   xInstalling libxcrypt-3.0.3-0.6.1.x86_64.rpm (installed size 132.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  27  ±±±±±41  55%  68%  79  90  6  100  70 6 51  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libx86 (download size 214.00 kB)   x   xDownloading libx86 (download size 214.00 kB) q   7  14  22  9  37  ±±±44  52% 9%  67  74  82  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libx86-1.1-13.17.x86_64.rpm (installed size 1.24 MB) 51  x   xInstalling libx86-1.1-13.17.x86_64.rpm (installed size 1.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  6  91  6  100  2.46 5 4.28  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libusb-0_1-4 (download size 27.00 kB)   x   xDownloading libusb-0_1-4 (download size 27.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libusb-0_1-4-0.1.12-139.1.1.x86_64.rpm (installed size 67.00 kB 4.28  x   xInstalling libusb-0_1-4-0.1.12-139.1.1.x86_64.rpm (installed size 67.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  92%  9  100  39 4 1  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsepol1 (download size 96.00 kB)   x   x~kB) x    xDownloading libsepol1 (download size 96.00 kB) q   16  33  50 67%  83  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsepol1-2.0.41-2.2.9.x86_64.rpm (installed size 240.00 kB) 1  x   xInstalling libsepol1-2.0.41-2.2.9.x86_64.rpm (installed size 240.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  80%  99  100  16 3 3.98  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libselinux1 (download size 52.00 kB)   x   xDownloading libselinux1 (download size 52.00 kB) q   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libselinux1-2.0.91-4.2.1.x86_64.rpm (installed size 116.00 kB) 3.98  x   xInstalling libselinux1-2.0.91-4.2.1.x86_64.rpm (installed size 116.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  99%  100  04 2 86  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpixman-1-0 (download size 181.00 kB)   x   x~) x    xDownloading libpixman-1-0 (download size 181.00 kB) q   8  17  26  35  ±±±44  53%  62%  70  9  88  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpixman-1-0-0.24.4-0.15.1.x86_64.rpm (installed size 541.00 k 86  x   xInstalling libpixman-1-0-0.24.4-0.15.1.x86_64.rpm (installed size 541.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  35  ±47  59%  70  82  94  9  100  1.52 1 33  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpcap0 (download size 99.00 kB)   x   x~ kB) x    xDownloading libpcap0 (download size 99.00 kB) q   15  32  48 64%  80  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpcap0-0.9.8-50.10.1.x86_64.rpm (installed size 245.00 kB) 33  x   xInstalling libpcap0-0.9.8-50.10.1.x86_64.rpm (installed size 245.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 78%  86  93  9  100  28 0 09  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenct1 (download size 21.00 kB)   x   xDownloading libopenct1 (download size 21.00 kB) q   78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenct1-0.6.15-5.20.x86_64.rpm (installed size 39.00 kB) 09  x   xInstalling libopenct1-0.6.15-5.20.x86_64.rpm (installed size 39.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 79 6  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnuma1 (download size 27.00 kB)   x   xDownloading libnuma1 (download size 27.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnuma1-2.0.10-0.9.31.x86_64.rpm (installed size 44.00 kB) 6  x   xInstalling libnuma1-2.0.10-0.9.31.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 8 1  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnscd (download size 16.00 kB)   x   xDownloading libnscd (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnscd-2.0.2-73.18.x86_64.rpm (installed size 33.00 kB) 1  x   xInstalling libnscd-2.0.2-73.18.x86_64.rpm (installed size 33.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  96%  100  16 7 2.98  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnl (download size 121.00 kB)   x   xDownloading libnl (download size 121.00 kB) q   13  26  39  52% 66%  79  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnl-1.1-49.22.6.1.x86_64.rpm (installed size 362.00 kB) 2.98  x   xInstalling libnl-1.1-49.22.6.1.x86_64.rpm (installed size 362.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  70%  88  96  100  0.81 6 63  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnfnetlink0 (download size 22.00 kB)   x   xDownloading libnfnetlink0 (download size 22.00 kB) q   73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnfnetlink0-1.0.0+git1-9.5.56.x86_64.rpm (installed size 45.0 63  x   xInstalling libnfnetlink0-1.0.0+git1-9.5.56.x86_64.rpm (installed size 45q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  57%  96%  100  77 5 58  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses6 (download size 241.00 kB)   x   x~.00 kB) x    xDownloading libncurses6 (download size 241.00 kB) q   6  13  9  26  33  9  ±46  53%  9%  66  73  9  86  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses6-5.6-90.55.x86_64.rpm (installed size 856.00 kB) 58  x   xInstalling libncurses6-5.6-90.55.x86_64.rpm (installed size 856.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  37  ±±±44  52% 9%  65  72  80  7  98  100  299.93 4 1.75  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses5 (download size 242.00 kB)   x   xDownloading libncurses5 (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% 9%  66  72  9  86  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses5-5.6-90.55.x86_64.rpm (installed size 915.00 kB) 1.75  x   xInstalling libncurses5-5.6-90.55.x86_64.rpm (installed size 915.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  20  7  38  ±±45  52% 9%  66  74  81  8  9  100  04 3 0.8  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading liblzo2-2 (download size 45.00 kB)   x   xDownloading liblzo2-2 (download size 45.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblzo2-2-2.03-12.3.1.x86_64.rpm (installed size 135.00 kB) 0.8  x   xInstalling liblzo2-2-2.03-12.3.1.x86_64.rpm (installed size 135.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±47  95%  100  8.91 2 72  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading liblzma5 (download size 80.00 kB)   x   xDownloading liblzma5 (download size 80.00 kB) q   19  40  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblzma5-5.0.3-0.12.1.x86_64.rpm (installed size 139.00 kB) 72  x   xInstalling liblzma5-5.0.3-0.12.1.x86_64.rpm (installed size 139.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  92%  9  100  77 1 59  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libltdl7 (download size 21.00 kB)   x   xDownloading libltdl7 (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libltdl7-2.2.6-2.131.1.x86_64.rpm (installed size 39.00 kB) 59  x   xInstalling libltdl7-2.2.6-2.131.1.x86_64.rpm (installed size 39.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 0 5  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libjpeg (download size 67.00 kB)   x   xDownloading libjpeg (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libjpeg-6.2.0-879.12.1.x86_64.rpm (installed size 143.00 kB) 5  x   xInstalling libjpeg-6.2.0-879.12.1.x86_64.rpm (installed size 143.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  89%  9  100  59 69 41  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libiptc0 (download size 33.00 kB)   x   xDownloading libiptc0 (download size 33.00 kB) q   48 98%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libiptc0-1.4.16.3-1.37.x86_64.rpm (installed size 65.00 kB) 41  x   xInstalling libiptc0-1.4.16.3-1.37.x86_64.rpm (installed size 65.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  87%  99  100  3 8 35  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libibverbs (download size 69.00 kB)   x   xDownloading libibverbs (download size 69.00 kB) q   22  ±46  69%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibverbs-1.1.4-0.16.37.x86_64.rpm (installed size 234.00 kB) 35  x   xInstalling libibverbs-1.1.4-0.16.37.x86_64.rpm (installed size 234.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  2  3  ±±±±42  52% 77%  86  95  100  30 7 12  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libibumad3 (download size 43.00 kB)   x   x~) x    xDownloading libibumad3 (download size 43.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibumad3-1.3.7-10.26.x86_64.rpm (installed size 62.00 kB) 12  x   xInstalling libibumad3-1.3.7-10.26.x86_64.rpm (installed size 62.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  69%  74  9  84  9  94  9  100  24 6 06  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgssglue1 (download size 21.00 kB)   x   xDownloading libgssglue1 (download size 21.00 kB) q   76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgssglue1-0.1-20.2.1.x86_64.rpm (installed size 40.00 kB) 06  x   xInstalling libgssglue1-0.1-20.2.1.x86_64.rpm (installed size 40.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 5 2  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libevent-1_4-2 (download size 64.00 kB)   x   xDownloading libevent-1_4-2 (download size 64.00 kB) q   24  49 74%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libevent-1_4-2-1.4.5-24.24.1.x86_64.rpm (installed size 226.00 2  x   xInstalling libevent-1_4-2-1.4.5-24.24.1.x86_64.rpm (installed size 226.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  ±±45  63%  91  9  100  7.98 4 19.80  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libelf1 (download size 44.00 kB)   x   x~0 kB) x    xDownloading libelf1 (download size 44.00 kB) q   36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libelf1-0.152-4.9.17.x86_64.rpm (installed size 83.00 kB) 19.80  x   xInstalling libelf1-0.152-4.9.17.x86_64.rpm (installed size 83.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  77%  99  100  0 3 72  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libelf0 (download size 36.00 kB)   x   xDownloading libelf0 (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libelf0-0.8.12-1.1.52.x86_64.rpm (installed size 101.00 kB) 72  x   xInstalling libelf0-0.8.12-1.1.52.x86_64.rpm (installed size 101.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  63%  91  9  100  8 2 6  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libdb-4_5 (download size 470.00 kB)   x   xDownloading libdb-4_5 (download size 470.00 kB) q   6  13  20  7  34  40  ±47  54%  61%  8  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libdb-4_5-4.5.20-95.39.x86_64.rpm (installed size 1.22 MB) 6  x   xInstalling libdb-4_5-4.5.20-95.39.x86_64.rpm (installed size 1.22 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  6.58 1 8.40  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libattr (download size 11.00 kB)   x   xDownloading libattr (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libattr-2.4.43-1.18.x86_64.rpm (installed size 20.00 kB) 8.40  x   xInstalling libattr-2.4.43-1.18.x86_64.rpm (installed size 20.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  6 0 38  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libaio (download size 15.00 kB)   x   xDownloading libaio (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libaio-0.3.109-0.1.46.x86_64.rpm (installed size 32.00 kB) 38  x   xInstalling libaio-0.3.109-0.1.46.x86_64.rpm (installed size 32.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  98%  100  3 59 5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libHX13 (download size 24.00 kB)   x   xDownloading libHX13 (download size 24.00 kB) q   68%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libHX13-1.23-4.13.1.x86_64.rpm (installed size 47.00 kB) 5  x   xInstalling libHX13-1.23-4.13.1.x86_64.rpm (installed size 47.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  48 8 0  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading keyutils-libs (download size 16.00 kB)   x   xDownloading keyutils-libs (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing keyutils-libs-1.2-107.29.15.x86_64.rpm (installed size 36.00 kB 0  x   xInstalling keyutils-libs-1.2-107.29.15.x86_64.rpm (installed size 36.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  99%  100  5 7 27  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading initviocons (download size 25.00 kB)   x   x~kB) x    xDownloading initviocons (download size 25.00 kB) q   63%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing initviocons-0.5-55.21.x86_64.rpm (installed size 56.00 kB) 27  x   xInstalling initviocons-0.5-55.21.x86_64.rpm (installed size 56.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  34  68%  80  96  100  39 6 1  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading infinipath-psm (download size 160.00 kB)   x   xDownloading infinipath-psm (download size 160.00 kB) q   9  1  2  40  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing infinipath-psm-3.1-10.24.x86_64.rpm (installed size 381.00 kB) 1  x   xInstalling infinipath-psm-3.1-10.24.x86_64.rpm (installed size 381.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  31  48 64%  81  98  100  02 5 7.84  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpart (download size 40.00 kB)   x   x~) x    xDownloading gpart (download size 40.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpart-0.1h-565.22.x86_64.rpm (installed size 78.00 kB) 7.84  x   xInstalling gpart-0.1h-565.22.x86_64.rpm (installed size 78.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  77%  92  9  100  5.95 4 76  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gdbm (download size 26.00 kB)   x   xDownloading gdbm (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gdbm-1.8.3-374.25.x86_64.rpm (installed size 55.00 kB) 76  x   xInstalling gdbm-1.8.3-374.25.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  58%  90%  6  100  89 3 1  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fping (download size 25.00 kB)   x   xDownloading fping (download size 25.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fping-2.4b2-97.5.x86_64.rpm (installed size 44.00 kB) 1  x   xInstalling fping-2.4b2-97.5.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  86  92  9  100  5 2 67  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fillup (download size 20.00 kB)   x   xDownloading fillup (download size 20.00 kB) q   70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fillup-1.42-242.21.x86_64.rpm (installed size 37.00 kB) 67  x   xInstalling fillup-1.42-242.21.x86_64.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  1 1 3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dos2unix (download size 11.00 kB)   x   xDownloading dos2unix (download size 11.00 kB) q   100%  - 10.84 kB/s (download size 11.00 kB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dos2unix-3.1-438.22.1.x86_64.rpm (installed size 22.00 kB) 3  x   xInstalling dos2unix-3.1-438.22.1.x86_64.rpm (installed size 22.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  84%  92  8  100  79 0 1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dmapi (download size 27.00 kB)   x   xDownloading dmapi (download size 27.00 kB) q   59%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dmapi-2.2.10-8.7.2.x86_64.rpm (installed size 80.00 kB) 1  x   xInstalling dmapi-2.2.10-8.7.2.x86_64.rpm (installed size 80.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  95%  100  1 49 53  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpufrequtils (download size 50.00 kB)   x   xDownloading cpufrequtils (download size 50.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpufrequtils-004-35.20.x86_64.rpm (installed size 195.00 kB) 53  x   xInstalling cpufrequtils-004-35.20.x86_64.rpm (installed size 195.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  52% 63%  75  84  9  94  9  100  52 8 34  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bridge-utils (download size 51.00 kB)   x   xDownloading bridge-utils (download size 51.00 kB) q   31  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bridge-utils-1.4-23.18.1.x86_64.rpm (installed size 161.00 kB) 34  x   xInstalling bridge-utils-1.4-23.18.1.x86_64.rpm (installed size 161.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  57%  76%  87  93  8  100  37 7 18  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading audit-libs (download size 72.00 kB)   x   x~) x    xDownloading audit-libs (download size 72.00 kB) q   21  ±±±44  66%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing audit-libs-1.8-0.34.26.x86_64.rpm (installed size 158.00 kB) 18  x   xInstalling audit-libs-1.8-0.34.26.x86_64.rpm (installed size 158.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  62%  99  100  21 6 03  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libfontenc (download size 14.00 kB)   x   xDownloading xorg-x11-libfontenc (download size 14.00 kB) q   100%   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libfontenc-7.4-1.15.x86_64.rpm (installed size 30.00 k 03  x   xInstalling xorg-x11-libfontenc-7.4-1.15.x86_64.rpm (installed size 30.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  18 5 0  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading file (download size 266.00 kB)   x   x~ kB) x    xDownloading file (download size 266.00 kB) q   5  11  7  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing file-4.24-43.27.1.x86_64.rpm (installed size 2.17 MB) 0  x   xInstalling file-4.24-43.27.1.x86_64.rpm (installed size 2.17 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  33  9  ±±45  51 6%  62%  8  74  9  85  91  7  100  3.01 4 4.83  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading strace (download size 98.00 kB)   x   xDownloading strace (download size 98.00 kB) q   16  32  48 65%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing strace-4.5.18-10.26.19.x86_64.rpm (installed size 313.00 kB) 4.83  x   xInstalling strace-4.5.18-10.26.19.x86_64.rpm (installed size 313.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  4  61%  8  9  6  100  2.70 3 52  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pptp (download size 61.00 kB)   x   xDownloading pptp (download size 61.00 kB) q   25  52% 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pptp-1.7.2-1.22.x86_64.rpm (installed size 134.00 kB) 52  x   xInstalling pptp-1.7.2-1.22.x86_64.rpm (installed size 134.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  60%  5  79  84  90  5  100  57 2 39  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading checkmedia (download size 37.00 kB)   x   xDownloading checkmedia (download size 37.00 kB) q   ±±±±43  87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing checkmedia-3.0-0.9.1.x86_64.rpm (installed size 94.00 kB) 39  x   xInstalling checkmedia-3.0-0.9.1.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  9  100  48 1 0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading update-alternatives (download size 15.00 kB)   x   xDownloading update-alternatives (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing update-alternatives-1.14.19-1.22.noarch.rpm (installed size 28. 0  x   xInstalling update-alternatives-1.14.19-1.22.noarch.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  83%  99  100  5 0 27  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading convmv (download size 29.00 kB)   x   x~8.00 kB) x    xDownloading convmv (download size 29.00 kB) q   55%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing convmv-1.12-1.22.noarch.rpm (installed size 51.00 kB) 27  x   xInstalling convmv-1.12-1.22.noarch.rpm (installed size 51.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  52% 86%  99  100  0 39 2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atmel-firmware (download size 166.00 kB)   x   xDownloading atmel-firmware (download size 166.00 kB) q   9  1  28  3  4 57%  6  7  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atmel-firmware-1.3-139.15.4.noarch.rpm (installed size 752.00 k 2  x   xInstalling atmel-firmware-1.3-139.15.4.noarch.rpm (installed size 752.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  21  6  34  ±±±±42  ±47  56%  61%  70  5  80  5  94  9  100  1.67 8 3.49  0 9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zip (download size 253.00 kB)   x   x~ kB) x    xDownloading zip (download size 253.00 kB) q   6  12  8  25  31  7  ±±±44  50 6%  63%  9  75  82  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zip-3.0-4.9.1.x86_64.rpm (installed size 795.00 kB) 3.49  x   xInstalling zip-3.0-4.9.1.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  8  ±46  57%  70%  8  86  91  6  100  0.89 7 2.71  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc43 (download size 4.00 kB)   x   xDownloading libgcc43 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) 2.71  x   xInstalling libgcc43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  74%  100  6  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pcre (download size 381.00 kB)   x   xDownloading pcre (download size 381.00 kB) q   8  16  25  33  ±±±±42  50 8%  63%  71  9  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pcre-7.8-2.18.x86_64.rpm (installed size 1.30 MB)  x   xInstalling pcre-7.8-2.18.x86_64.rpm (installed size 1.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  8  33  8  ±±±±43  50 5%  63%  8  76  81  6  94  9  100  89.5 5 1.4  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++43 (download size 4.00 kB)   x   xDownloading libstdc++43 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kB) 1.4  x   xInstalling libstdc++43-4.6.9-0.14.1.9.x86_64.rpm (installed size 1.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  74%  100  4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsmbios2 (download size 155.00 kB)   x   x~) x    xDownloading libsmbios2 (download size 155.00 kB) q   1  2  3  ±±±±±41  51 61%  72  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsmbios2-2.0.2-11.8.4.x86_64.rpm (installed size 646.00 kB)  x   xInstalling libsmbios2-2.0.2-11.8.4.x86_64.rpm (installed size 646.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  2  3  4 59%  6  7  8  9  100  8.96 3 0.78  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libboost_regex1_36_0 (download size 237.00 kB)   x   xDownloading libboost_regex1_36_0 (download size 237.00 kB) q   6  13  20  7  33  40  ±47  54%  60%  7  74  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libboost_regex1_36_0-1.36.0-12.6.49.x86_64.rpm (installed size 0.78  x   xInstalling libboost_regex1_36_0-1.36.0-12.6.49.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  07 2 09.89  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libxcb (download size 97.00 kB)   x   x~e 910.00 kB) x    xDownloading xorg-x11-libxcb (download size 97.00 kB) q   16  32  49 65%  82  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libxcb-7.4-1.29.1.x86_64.rpm (installed size 528.00 kB 09.89  x   xInstalling xorg-x11-libxcb-7.4-1.29.1.x86_64.rpm (installed size 528.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  3  9  ±46  51 6%  61%  9  76  91  9  100  7.55 1 37  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading delayacct-utils (download size 18.00 kB)   x   x~kB) x    xDownloading delayacct-utils (download size 18.00 kB) q   90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing delayacct-utils-2.6.32.12-3.3.38.x86_64.rpm (installed size 35. 37  x   xInstalling delayacct-utils-2.6.32.12-3.3.38.x86_64.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±±41  91%  9  100  2 0 4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procinfo (download size 37.00 kB)   x   x~5.00 kB) x    xDownloading procinfo (download size 37.00 kB) q   ±±±±43  88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procinfo-18-198.16.1.x86_64.rpm (installed size 72.00 kB) 4  x   xInstalling procinfo-18-198.16.1.x86_64.rpm (installed size 72.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  66%  71  9  9  100  45 29 27  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ncurses-utils (download size 73.00 kB)   x   xDownloading ncurses-utils (download size 73.00 kB) q   21  ±±±±43  66%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ncurses-utils-5.6-90.55.x86_64.rpm (installed size 103.00 kB) 27  x   xInstalling ncurses-utils-5.6-90.55.x86_64.rpm (installed size 103.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  20  34  52% 60%  9  9  100  3 8 1  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading joe (download size 320.00 kB)   x   xDownloading joe (download size 320.00 kB) q   9  14  9  24  9  35  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing joe-3.5-92.22.x86_64.rpm (installed size 1.20 MB) 1  x   xInstalling joe-3.5-92.22.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  24  9  34  9  4 55%  61%  7  73  8  84  91  6  100  6.1 7 7.9  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi-libs (download size 1.50 MB)   x   xDownloading openmpi-libs (download size 1.50 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-libs-1.8.4-1.26.x86_64.rpm (installed size 5.33 MB) 7.9  x   xInstalling openmpi-libs-1.8.4-1.26.x86_64.rpm (installed size 5.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  61%  6  71  6  82  7  92  7  100  0.83 6 2.64  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librdmacm (download size 58.00 kB)   x   xDownloading librdmacm (download size 58.00 kB) q   27  55%  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librdmacm-1.0.15-12.29.x86_64.rpm (installed size 175.00 kB) 2.64  x   xInstalling librdmacm-1.0.15-12.29.x86_64.rpm (installed size 175.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  20  6  32  ±±±44  55%  63%  8  94  9  100  66 5 47  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmthca-rdmav2 (download size 18.00 kB)   x   xDownloading libmthca-rdmav2 (download size 18.00 kB) q   87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmthca-rdmav2-1.0.5-5.27.1.x86_64.rpm (installed size 35.00 k 47  x   xInstalling libmthca-rdmav2-1.0.5-5.27.1.x86_64.rpm (installed size 35.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 4 4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmlx4 (download size 28.00 kB)   x   x~ kB) x    xDownloading libmlx4-rdmav2 (download size 28.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmlx4-rdmav2-1.0.1-15.30.x86_64.rpm (installed size 53.00 kB) 4  x   xInstalling libmlx4-rdmav2-1.0.1-15.30.x86_64.rpm (installed size 53.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  95%  100  57 3 39  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcxgb3-rdmav2 (download size 13.00 kB)   x   x~B) x    xDownloading libcxgb3-rdmav2 (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcxgb3-rdmav2-1.2.5-0.8.1.x86_64.rpm (installed size 23.00 kB 39  x   xInstalling libcxgb3-rdmav2-1.2.5-0.8.1.x86_64.rpm (installed size 23.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  5 2 6  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ibvexdmtools (download size 14.00 kB)   x   x~kB) x    xDownloading ibvexdmtools (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ibvexdmtools-0.0.1-75.23.23.x86_64.rpm (installed size 23.00 kB 6  x   xInstalling ibvexdmtools-0.0.1-75.23.23.x86_64.rpm (installed size 23.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  97%  100  2 1 4  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librpcsecgss (download size 130.00 kB)   x   x~kB) x    xDownloading librpcsecgss (download size 130.00 kB) q   12  24  36  49 61%  73  86  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librpcsecgss-0.18-1.15.x86_64.rpm (installed size 586.00 kB) 4  x   xInstalling librpcsecgss-0.18-1.15.x86_64.rpm (installed size 586.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  27  38  49 60%  71  82  8  99  100  79.95 0 1.77  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libdw1 (download size 98.00 kB)   x   xDownloading libdw1 (download size 98.00 kB) q   16  32  49 65%  82  98  100   40 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libdw1-0.152-4.9.17.x86_64.rpm (installed size 212.00 kB) 1.77  x   xInstalling libdw1-0.152-4.9.17.x86_64.rpm (installed size 212.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  60%  9  9  100  7 19 56  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libasm1 (download size 24.00 kB)   x   xDownloading libasm1 (download size 24.00 kB) q   66%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libasm1-0.152-4.9.17.x86_64.rpm (installed size 31.00 kB) 56  x   xInstalling libasm1-0.152-4.9.17.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 8 3  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading db-utils (download size 74.00 kB)   x   xDownloading db-utils (download size 74.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing db-utils-4.5.20-95.39.x86_64.rpm (installed size 280.00 kB) 3  x   xInstalling db-utils-4.5.20-95.39.x86_64.rpm (installed size 280.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  31  7  ±±±±42  49 58%  81%  6  92  9  100  44 7 26 499 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcap2 (download size 12.00 kB)   x   xDownloading libcap2 (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcap2-2.11-2.17.1.x86_64.rpm (installed size 19.00 kB) 26  x   xInstalling libcap2-2.11-2.17.1.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 6 4  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libacl (download size 18.00 kB)   x   xDownloading libacl (download size 18.00 kB) q   88%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libacl-2.2.47-30.36.1.x86_64.rpm (installed size 31.00 kB) 4  x   xInstalling libacl-2.2.47-30.36.1.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  39 5 1  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading attr (download size 52.00 kB)   x   xDownloading attr (download size 52.00 kB) q   3  62%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing attr-2.4.43-1.18.x86_64.rpm (installed size 149.00 kB) 1  x   xInstalling attr-2.4.43-1.18.x86_64.rpm (installed size 149.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  2  61%  8  76  84  91  6  100  25 4 06  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading genisoimage (download size 348.00 kB)   x   xDownloading genisoimage (download size 348.00 kB) q   9  18  27  32  ±±±±±41  50 5%  64%  9  78  87  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing genisoimage-1.1.8-3.32.3.x86_64.rpm (installed size 1.61 MB) 06  x   xInstalling genisoimage-1.1.8-3.32.3.x86_64.rpm (installed size 1.61 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  37  ±±45  53%  9%  64  72  80  8  93  9  100  7.64 3 899.45  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcc46 (download size 4.00 kB)   x   xDownloading libgcc46 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcc46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB) 899.45  x   xInstalling libgcc46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  74%  100  3 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++46 (download size 4.00 kB)   x   xDownloading libstdc++46 (download size 4.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB)  x   xInstalling libstdc++46-4.6.9-0.13.22.x86_64.rpm (installed size 1.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  74%  100  1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading makedumpfile (download size 160.00 kB)   x   xDownloading makedumpfile (download size 160.00 kB) q   9  1  2  40  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing makedumpfile-1.5.6-2.29.x86_64.rpm (installed size 372.00 kB)  x   xInstalling makedumpfile-1.5.6-2.29.x86_64.rpm (installed size 372.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  34  51 68%  81  7  93  8  100  27 0 09  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libebl1 (download size 71.00 kB)   x   xDownloading libebl1 (download size 71.00 kB) q   22  ±±45  68%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libebl1-0.152-4.9.17.x86_64.rpm (installed size 251.00 kB) 09  x   xInstalling libebl1-0.152-4.9.17.x86_64.rpm (installed size 251.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  36  ±±±±43  52% 9%  6  78  99  100  03 09 8.84  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading acl (download size 68.00 kB)   x   xDownloading acl (download size 68.00 kB) q   23  ±47  71%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing acl-2.2.47-30.36.1.x86_64.rpm (installed size 192.00 kB) 8.84  x   xInstalling acl-2.2.47-30.36.1.x86_64.rpm (installed size 192.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  19  37  64%  70  5  80  9  94  9  100  6.84 8 66  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 898.66 MBx 490x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-ae2fc03cxxx tu Downloading fsx (download size 14.00 kB)   x   xDownloading fsx (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fsx-1.0-1.wc1.x86_64.rpm (installed size 37.00 kB) 66  x   xInstalling fsx-1.0-1.wc1.x86_64.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading connectathon (download size 107.00 kB)   x   xDownloading connectathon (download size 107.00 kB) q   14  29  ±±±44  59%  74  89  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing connectathon-1.19-1.intel.x86_64.rpm (installed size 975.00 kB) 2  x   xInstalling connectathon-1.19-1.intel.x86_64.rpm (installed size 975.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  24  31  7  ±±±±42  50 6%  61%  7  72  7  84  9  95  100  7.67  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-rsh (download size 12.00 kB)   x   x~B) x    xDownloading pdsh-rcmd-rsh (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-rsh-2.29-1.wc1.x86_64.rpm (installed size 25.00 kB) 7.67  x   xInstalling pdsh-rcmd-rsh-2.29-1.wc1.x86_64.rpm (installed size 25.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading byacc (download size 89.00 kB)   x   xDownloading byacc (download size 89.00 kB) q   17  35  53%  71%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing byacc-1.9.20070509-7.wc1.x86_64.rpm (installed size 255.00 kB) 4  x   xInstalling byacc-1.9.20070509-7.wc1.x86_64.rpm (installed size 255.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  50 75%  93  8  100  warning: group jenkins does not exist - using root warning: user jenkins does not exist - using root warning: group jenkins does not exist - using root warning: user jenkins does not exist - using root warning: group jenkins does not exist - using root  39  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-ssh (download size 13.00 kB)  user jenkins does not exist - using root group jenkins does not exist - using root   Downloading pdsh-rcmd-ssh (download size 13.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-ssh-2.29-1.wc1.x86_64.rpm (installed size 27.00 kB) 39  x   xInstalling pdsh-rcmd-ssh-2.29-1.wc1.x86_64.rpm (installed size 27.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbench (download size 1.05 MB)   x   xDownloading dbench (download size 1.05 MB) q   5  10  5  20  6  32  8  ±±±44  50 6%  62%  8  74  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbench-4.00-1.8b5143b.wc1.x86_64.rpm (installed size 25.32 MB) 7  x   xInstalling dbench-4.00-1.8b5143b.wc1.x86_64.rpm (installed size 25.32 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  72.05  4  ±±±±42 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesxTq Total x 872.05 MBx 484x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-ae2fc03cxxx tu Downloading lsof (download size 314.00 kB)   x   x~) x    xDownloading lsof (download size 314.00 kB) q   5  10  5  20  5  30  5  40  ±±45  50 6%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsof-4.80-1.33.1.x86_64.rpm (installed size 925.00 kB) 72.05  x   xInstalling lsof-4.80-1.33.1.x86_64.rpm (installed size 925.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  24  31  ±±±±±4  ±47  54%  61%  6  73  8  9  94  9  100  1.14  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcpdump (download size 290.00 kB)   x   xDownloading tcpdump (download size 290.00 kB) q   5  10  6  21  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcpdump-3.9.8-1.29.1.x86_64.rpm (installed size 702.00 kB) 1.14  x   xInstalling tcpdump-3.9.8-1.29.1.x86_64.rpm (installed size 702.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  27  36  ±±45  54%  63%  72  81  8  93  9  100  0.46  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libreadline5 (download size 106.00 kB)   x   xDownloading libreadline5 (download size 106.00 kB) q   14  30  ±±45  60%  75  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libreadline5-5.2-147.29.1.x86_64.rpm (installed size 292.00 kB) 0.46  x   xInstalling libreadline5-5.2-147.29.1.x86_64.rpm (installed size 292.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  33  55%  77%  99  100  17  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtiff3 (download size 117.00 kB)   x   x~B) x    xDownloading libtiff3 (download size 117.00 kB) q   13  27  40  54%  68%  82  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtiff3-3.8.2-141.168.1.x86_64.rpm (installed size 386.00 kB) 17  x   xInstalling libtiff3-3.8.2-141.168.1.x86_64.rpm (installed size 386.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  33  49 66%  82  96  100  69.80  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtirpc1 (download size 71.00 kB)   x   x~) x    xDownloading libtirpc1 (download size 71.00 kB) q   22  ±±45  67%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtirpc1-0.2.1-1.9.1.x86_64.rpm (installed size 159.00 kB) 69.80  x   xInstalling libtirpc1-0.2.1-1.9.1.x86_64.rpm (installed size 159.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  81%  99  100  64  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl (download size 6.87 MB)   x   xDownloading perl (download size 6.87 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-5.10.0-64.80.1.x86_64.rpm (installed size 41.58 MB) 64  x   xInstalling perl-5.10.0-64.80.1.x86_64.rpm (installed size 41.58 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  28.06  8 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading star (download size 342.00 kB)   x   xDownloading star (download size 342.00 kB) q   9  18  23  32  7  ±46  51 60%  5  74  9  84  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing star-1.5final-28.23.27.1.x86_64.rpm (installed size 1.03 MB) 28.06  x   xInstalling star-1.5final-28.23.27.1.x86_64.rpm (installed size 1.03 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  9  25  31  7  ±±±±42  48 54%  9%  64  70  6  82  8  93  9  100  7.03  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sqlite3 (download size 49.00 kB)   x   xDownloading sqlite3 (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sqlite3-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 61.00 kB) 7.03  x   xInstalling sqlite3-3.7.6.3-1.4.6.1.x86_64.rpm (installed size 61.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  94%  9  100  6.97  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bash (download size 296.00 kB)   x   xDownloading bash (download size 296.00 kB) q   5  10  6  21  6  32  7  ±±±±43  48 53%  9%  64  70  5  81  6  91  7  100   ±±±44 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bash-3.2-147.29.1.x86_64.rpm (installed size 864.00 kB) 6.97  x   xInstalling bash-3.2-147.29.1.x86_64.rpm (installed size 864.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  9  36  ±±±±43  51 8%  65%  70  9  85  93  8  100  13  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxml2 (download size 607.00 kB)   x   xDownloading libxml2 (download size 607.00 kB) q   5  10  5  21  6  31  6  ±±±±42  ±47  52% 7%  63%  8  73  9  84  9  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxml2-2.7.6-0.64.1.x86_64.rpm (installed size 1.55 MB) 13  x   xInstalling libxml2-2.7.6-0.64.1.x86_64.rpm (installed size 1.55 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  25  33  ±±±±±41  49 57%  65%  73  81  9  96  100  4.58  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Module-Build (download size 194.00 kB)   x   xDownloading perl-Module-Build (download size 194.00 kB) q   8  16  24  32  ±±±±±41  49 57%  65%  74  82  90  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Module-Build-0.2808.01-0.80.1.x86_64.rpm (installed size 4 4.58  x   xInstalling perl-Module-Build-0.2808.01-0.80.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  20  38  51 6%  62%  7  72  9  84  90  5  100  17  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-32bit (download size 1.06 MB)   x   x~ 415.00 kB) x    xDownloading glibc-32bit (download size 1.06 MB) q   5  10  5  20  5  31  7  ±±±±43  49 54%  9%  65  71  7  83  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-32bit-2.11.3-17.102.1.x86_64.rpm (installed size 2.88 MB) 17  x   xInstalling glibc-32bit-2.11.3-17.102.1.x86_64.rpm (installed size 2.88 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  32  8  ±±45  51 8%  64%  70  5  80  5  92  8  100  1.29  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading freetype2 (download size 360.00 kB)   x   x~B) x    xDownloading freetype2 (download size 360.00 kB) q   8  13  22  31  9  ±±±44  53%  62%  71  80  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing freetype2-2.3.7-25.44.1.x86_64.rpm (installed size 891.00 kB) 1.29  x   xInstalling freetype2-2.3.7-25.44.1.x86_64.rpm (installed size 891.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  50 7%  6  73  81  7  92  9  100  xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 820.42 MBx 471x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-ae2fc03cxxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenssl0_9_8 (download size 772.00 kB)   x   xDownloading libopenssl0_9_8 (download size 772.00 kB) q   6  11  7  23  9  35  ±±±±±41  ±46  52% 8%  64%  70  5  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenssl0_9_8-0.9.8j-0.105.1.x86_64.rpm (installed size 2.12 0.42 MB / 04:23)  x   xInstalling libopenssl0_9_8-0.9.8j-0.105.1.x86_64.rpm (installed size 2.1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  8  34  40  ±46  52% 8%  63%  9  75  81  6  92  8  100  18.31  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pmtools (download size 515.00 kB)   x   x~2 MB) x    xDownloading pmtools (download size 515.00 kB) q   6  12  8  24  31  7  ±±±±43  49 55%  62%  8  74  80  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pmtools-20071116-44.35.1.x86_64.rpm (installed size 1.63 MB) 18.312  x   xInstalling pmtools-20071116-44.35.1.x86_64.rpm (installed size 1.63 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  21  9  34  ±±±±±41  49 56%  64%  71  9  87  92  7  100  6.67  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libaugeas0 (download size 164.00 kB)   x   xDownloading libaugeas0 (download size 164.00 kB) q   9  1  2  38  4 58%  6  77  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libaugeas0-0.9.0-3.17.2.x86_64.rpm (installed size 334.00 kB) 6.67  x   xInstalling libaugeas0-0.9.0-3.17.2.x86_64.rpm (installed size 334.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  38  57%  76%  98  100  35  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libpython2_6-1_0 (download size 563.00 kB)   x   xDownloading libpython2_6-1_0 (download size 563.00 kB) q   5  11  7  22  8  34  9  ±±45  51 6%  62%  8  73  9  85  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libpython2_6-1_0-2.6.9-39.1.x86_64.rpm (installed size 1.54 MB) 35  x   xInstalling libpython2_6-1_0-2.6.9-39.1.x86_64.rpm (installed size 1.54 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  61%  9  77  85  93  9  100  4.81  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cracklib (download size 56.00 kB)   x   x~B) x    xDownloading cracklib (download size 56.00 kB) q   28  57%  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cracklib-2.8.12-56.13.1.x86_64.rpm (installed size 183.00 kB) 4.811  x   xInstalling cracklib-2.8.12-56.13.1.x86_64.rpm (installed size 183.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  9  ±±±±43  49 54%  9%  64  9  75  80  5  90  5  100  63  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libexpat1 (download size 62.00 kB)   x   xDownloading libexpat1 (download size 62.00 kB) q   22  48 74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libexpat1-2.0.1-88.41.1.x86_64.rpm (installed size 167.00 kB) 63  x   xInstalling libexpat1-2.0.1-88.41.1.x86_64.rpm (installed size 167.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  76%  99  100  47  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sg3_utils (download size 652.00 kB)   x   xDownloading sg3_utils (download size 652.00 kB) q   7  12  7  22  9  34  9  ±±±44  49 54%  61%  6  71  6  81  8  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) 47  x   xInstalling sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  71  7  83  8  93  8  100  sg3_utils-1.40-0.29.1.x86_64.rpm (installed size 1.81 MB) Additional rpm output: /var/tmp/rpm-tmp.24391: line 2: ln: command not found /var/tmp/rpm-tmp.24391: line 3: ln: command not found /var/tmp/rpm-tmp.24391: line 4: ln: command not found  2.65  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vim-base (download size 170.00 kB)  /var/tmp/rpm-tmp.24391: line 3: ln: command not found /var/tmp/rpm-tmp.24391: line 4: ln: command not found   Downloading vim-base (download size 170.00 kB)   9  18  2  37  ±4  56%  66%  75  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vim-base-7.2-8.17.1.x86_64.rpm (installed size 322.00 kB) 2.65  x   xInstalling vim-base-7.2-8.17.1.x86_64.rpm (installed size 322.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  15  20  36  ±±±±±41  ±46  51 7%  62%  79  96  100  34  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bzip2 (download size 47.00 kB)   x   xDownloading bzip2 (download size 47.00 kB) q   33  68%  100   ±±45 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bzip2-1.0.5-34.255.1.x86_64.rpm (installed size 90.00 kB) 34  x   xInstalling bzip2-1.0.5-34.255.1.x86_64.rpm (installed size 90.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  ±±±±42  58%  65%  77  8  96  100  25  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading unzip (download size 260.00 kB)   x   xDownloading unzip (download size 260.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  86  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing unzip-6.00-11.17.1.x86_64.rpm (installed size 721.00 kB) 25  x   xInstalling unzip-6.00-11.17.1.x86_64.rpm (installed size 721.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  20  5  34  9  ±47  54%  60%  5  74  9  8  95  100  1.5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Test-Simple (download size 90.00 kB)   x   xDownloading perl-Test-Simple (download size 90.00 kB) q   17  35  53%  70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Test-Simple-0.72-0.80.1.x86_64.rpm (installed size 167.00 1.5  x   xInstalling perl-Test-Simple-0.72-0.80.1.x86_64.rpm (installed size 167.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  36  59%  74  80  5  92  9  100  38  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libssh2-1 (download size 68.00 kB)   x   x~0 kB) x    xDownloading libssh2-1 (download size 68.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libssh2-1-1.2.9-4.2.11.1.x86_64.rpm (installed size 164.00 kB) 380  x   xInstalling libssh2-1-1.2.9-4.2.11.1.x86_64.rpm (installed size 164.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  78%  99  100  22  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssl (download size 536.00 kB)   x   x~) x    xDownloading openssl (download size 536.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssl-0.9.8j-0.105.1.x86_64.rpm (installed size 1.13 MB) 22  x   xInstalling openssl-0.9.8j-0.105.1.x86_64.rpm (installed size 1.13 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  ±47  53%  8%  64%  9  76  81  6  91  6  100  0.10  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libesmtp (download size 62.00 kB)   x   xDownloading libesmtp (download size 62.00 kB) q   25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libesmtp-1.0.4-157.17.2.x86_64.rpm (installed size 180.00 kB) 0.10  x   xInstalling libesmtp-1.0.4-157.17.2.x86_64.rpm (installed size 180.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  56%  62%  7  97  100  09.92  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtalloc2 (download size 220.00 kB)   x   xDownloading libtalloc2 (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  50 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtalloc2-3.6.3-87.1.x86_64.rpm (installed size 53.00 kB) 09.92  x   xInstalling libtalloc2-3.6.3-87.1.x86_64.rpm (installed size 53.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  99%  100  87  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-base (download size 4.29 MB)   x   xDownloading python-base (download size 4.29 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-base-2.6.9-39.1.x86_64.rpm (installed size 19.59 MB) 87  x   xInstalling python-base-2.6.9-39.1.x86_64.rpm (installed size 19.59 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  790.28  5  ±46 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam (download size 450.00 kB)   x   xDownloading pam (download size 450.00 kB) q   7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-1.1.5-0.17.2.x86_64.rpm (installed size 1.50 MB) 790.2814  x   xInstalling pam-1.1.5-0.17.2.x86_64.rpm (installed size 1.50 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  88.7  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading expat (download size 54.00 kB)   x   xDownloading expat (download size 54.00 kB) q   29  59%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing expat-2.0.1-88.41.1.x86_64.rpm (installed size 149.00 kB) 88.7  x   xInstalling expat-2.0.1-88.41.1.x86_64.rpm (installed size 149.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  24  9  34  77%  94  9  100  64  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fontconfig (download size 274.00 kB)   x   xDownloading fontconfig (download size 274.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  81  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fontconfig-2.6.0-10.19.1.x86_64.rpm (installed size 889.00 kB) 64  x   xInstalling fontconfig-2.6.0-10.19.1.x86_64.rpm (installed size 889.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  23  30  7  ±±45  53%  60%  7  74  81  8  95  100  7.77  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading elilo (download size 122.00 kB)   x   x~) x    xDownloading elilo (download size 122.00 kB) q   12  26  39  52% 65%  78  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing elilo-3.14-0.38.1.x86_64.rpm (installed size 404.00 kB) 7.77  x   xInstalling elilo-3.14-0.38.1.x86_64.rpm (installed size 404.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  27  ±±±±43  59%  75  80  6  91  6  100  38  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssl-certs (download size 177.00 kB)   x   xDownloading openssl-certs (download size 177.00 kB) q   8  17  26  3  ±±45  54%  63%  72  81  90  9  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssl-certs-2.7-0.6.1.noarch.rpm (installed size 261.00 kB) 383  x   xInstalling openssl-certs-2.7-0.6.1.noarch.rpm (installed size 261.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  12  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-xml (download size 190.00 kB)   x   xDownloading python-xml (download size 190.00 kB) q   8  16  25  33  ±±±±±41  50 8%  67%  75  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-xml-2.6.9-39.1.x86_64.rpm (installed size 838.00 kB) 12  x   xInstalling python-xml-2.6.9-39.1.x86_64.rpm (installed size 838.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  9  37  ±46  54%  61%  6  74  80  5  91  6  100  6.30  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gdb (download size 3.28 MB)   x   xDownloading gdb (download size 3.28 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gdb-7.9.1-8.2.x86_64.rpm (installed size 9.14 MB) 6.30  x   xInstalling gdb-7.9.1-8.2.x86_64.rpm (installed size 9.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  77.16  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python (download size 269.00 kB)   x   xDownloading python (download size 269.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-2.6.9-39.1.x86_64.rpm (installed size 1.14 MB) 77.160  x   xInstalling python-2.6.9-39.1.x86_64.rpm (installed size 1.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  22  31  6  ±±±±43  49 55%  60%  5  71  6  81  6  92  7  100  6.02  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ppp (download size 324.00 kB)   x   xDownloading ppp (download size 324.00 kB) q   5  10  5  20  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ppp-2.4.5.git-2.31.7.x86_64.rpm (installed size 853.00 kB) 6.02  x   xInstalling ppp-2.4.5.git-2.31.7.x86_64.rpm (installed size 853.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  34  ±±±±±41  49 56%  63%  7  80  5  90  5  100  5.19  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxml2-python (download size 162.00 kB)   x   xDownloading libxml2-python (download size 162.00 kB) q   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxml2-python-2.7.6-0.64.4.x86_64.rpm (installed size 832.00 k 5.19  x   xInstalling libxml2-python-2.7.6-0.64.4.x86_64.rpm (installed size 832.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  24  32  9  48 56%  64%  71  9  85  90  5  100  4.38  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 774.38 MBx 445x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-ae2fc03cxxx tu Downloading libibumad-devel (download size 34.00 kB)   x   x~ kB) x    xDownloading libibumad-devel (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibumad-devel-1.3.7-10.26.x86_64.rpm (installed size 112.00 k 4.3802  x   xInstalling libibumad-devel-1.3.7-10.26.x86_64.rpm (installed size 112.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  62%  98  100  27  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 774.27 MBx 444x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 276.84 MBx 308x http-10.9.0.10-ae2fc03cxxx tu Downloading liblcms1 (download size 93.00 kB)   x   x~ kB) x    xDownloading liblcms1 (download size 93.00 kB) q   17  34  51 68%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing liblcms1-1.17-77.16.1.x86_64.rpm (installed size 218.00 kB) 27  x   xInstalling liblcms1-1.17-77.16.1.x86_64.rpm (installed size 218.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  58%  8  99  100  62 7 05  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yp-tools (download size 65.00 kB)   x   xDownloading yp-tools (download size 65.00 kB) q   24  49 73%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yp-tools-2.12-0.2.5.x86_64.rpm (installed size 206.00 kB) 05  x   xInstalling yp-tools-2.12-0.2.5.x86_64.rpm (installed size 206.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  1  24  32  7  ±46  55%  61%  70  7  82  94  9  100  4 6 3.8  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-libyui (download size 192.00 kB)   x   xDownloading yast2-libyui (download size 192.00 kB) q   8  16  24  33  ±±±±±41  49 58%  66%  75  83  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-libyui-2.21.1-2.5.93.x86_64.rpm (installed size 679.00 kB 3.8  x   xInstalling yast2-libyui-2.21.1-2.5.93.x86_64.rpm (installed size 679.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  2  37  ±4  56%  66%  75  84  9  9  100  5.76 5 19  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xz (download size 107.00 kB)   x   x~kB) x    xDownloading xz (download size 107.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xz-5.0.3-0.12.1.x86_64.rpm (installed size 228.00 kB) 191  x   xInstalling xz-5.0.3-0.12.1.x86_64.rpm (installed size 228.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  33  8  49 57%  65%  71  89  97  100  54 4 2.97  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysfsutils (download size 75.00 kB)   x   xDownloading sysfsutils (download size 75.00 kB) q   21  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysfsutils-2.1.0-102.25.1.x86_64.rpm (installed size 337.00 kB) 2.97  x   xInstalling sysfsutils-2.1.0-102.25.1.x86_64.rpm (installed size 337.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  20  5  ±±±44  63%  82  93  9  100  21 3 64  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sles-release (download size 16.00 kB)   x   x~B) x    xDownloading sles-release (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB) 64  x   xInstalling sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  15 2 58  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading qlvnictools (download size 31.00 kB)   x   xDownloading qlvnictools (download size 31.00 kB) q   52% 100%   48 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 kB) 58  x   xInstalling qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  2  64%  96  100  06 1 4  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pth (download size 216.00 kB)  Installing sles-release-11.4-1.109.x86_64.rpm (installed size 63.00 kB)qlvnictools (download size 31.00 kB) qlvnictools-0.0.1-75.23.23.x86_64.rpm (installed size 94.00 k ~B) Downloading pth (download size 216.00 kB)   7  14  22  9  36  ±±±44  51 9%  66  74  81  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pth-2.0.7-102.22.x86_64.rpm (installed size 856.00 kB) 4  x   xInstalling pth-2.0.7-102.22.x86_64.rpm (installed size 856.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  25  32  40  ±47  55%  62%  70  8  84  9  94  9  100  4.22 0 1.65  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mtools (download size 250.00 kB)   x   xDownloading mtools (download size 250.00 kB) q   6  12  9  25  31  8  ±±±44  51 7%  64%  70  6  83  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mtools-3.9.11-165.31.x86_64.rpm (installed size 542.00 kB) 1.65  x   xInstalling mtools-3.9.11-165.31.x86_64.rpm (installed size 542.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  34  ±±±4  58%  70%  81  90  5  100  3.69 299 12  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mstflint (download size 254.00 kB)   x   xDownloading mstflint (download size 254.00 kB) q   6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mstflint-3.6.0-1.8.x86_64.rpm (installed size 1.38 MB) 12  x   xInstalling mstflint-3.6.0-1.8.x86_64.rpm (installed size 1.38 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  21  30  5  ±±±±±41  ±47  53%  9%  64  9  74  82  91  9  100  2.31 8 69.73  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libzio (download size 10.00 kB)   x   xDownloading libzio (download size 10.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libzio-0.9-5.77.x86_64.rpm (installed size 19.00 kB) 69.730  x   xInstalling libzio-0.9-5.77.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  29 7 2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libuuid1 (download size 37.00 kB)   x   xDownloading libuuid1 (download size 37.00 kB) q   ±±±±43  87%  100  - 36.41 kB/s (download size 37.00 kB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libuuid1-2.19.1-6.72.1.x86_64.rpm (installed size 19.00 kB) 2  x   xInstalling libuuid1-2.19.1-6.72.1.x86_64.rpm (installed size 19.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7 6 0  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++33 (download size 198.00 kB)   x   xDownloading libstdc++33 (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++33-3.3.3-11.9.x86_64.rpm (installed size 833.00 kB) 0  x   xInstalling libstdc++33-3.3.3-11.9.x86_64.rpm (installed size 833.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  23  30  8  ±46  53%  61%  9  76  84  92  9  100  1.46 5 8.88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libnet (download size 218.00 kB)   x   xDownloading libnet (download size 218.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  7  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libnet-1.1.2.1-140.24.1.x86_64.rpm (installed size 1.61 MB) 8.88  x   xInstalling libnet-1.1.2.1-140.24.1.x86_64.rpm (installed size 1.61 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  33  ±±±±±41  49 54%  62%  8  73  8  83  8  93  8  100  69.85 4 7.27  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgpg-error0 (download size 42.00 kB)   x   xDownloading libgpg-error0 (download size 42.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgpg-error0-1.10-0.7.29.x86_64.rpm (installed size 162.00 kB) 7.27  x   xInstalling libgpg-error0-1.10-0.7.29.x86_64.rpm (installed size 162.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  19  2  38  4 63%  71  8  92  9  100  69 3 12  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading less (download size 121.00 kB)   x   x~B) x    xDownloading less (download size 121.00 kB) q   13  26  39  53%  66%  79  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing less-424b-10.24.1.x86_64.rpm (installed size 266.00 kB) 12  x   xInstalling less-424b-10.24.1.x86_64.rpm (installed size 266.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 56%  66%  81  90  8  100  43 2 6.86  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading keyutils (download size 54.00 kB)   x   xDownloading keyutils (download size 54.00 kB) q   29  59%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing keyutils-1.2-107.29.15.x86_64.rpm (installed size 87.00 kB) 6.86  x   xInstalling keyutils-1.2-107.29.15.x86_64.rpm (installed size 87.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  ±47  66%  74  80  5  90  5  100  34 1 77  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iptables (download size 263.00 kB)   x   xDownloading iptables (download size 263.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  54%  60%  6  72  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iptables-1.4.16.3-1.37.x86_64.rpm (installed size 1.25 MB) 77  x   xInstalling iptables-1.4.16.3-1.37.x86_64.rpm (installed size 1.25 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  7  82  7  93  8  100  8.10 0 5.53  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading insserv (download size 39.00 kB)   x   xDownloading insserv (download size 39.00 kB) q   4  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing insserv-1.12.0-25.11.47.x86_64.rpm (installed size 69.00 kB) 5.533:59  x   xInstalling insserv-1.12.0-25.11.47.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  93%  9  100  03 89 46  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ifplugd (download size 42.00 kB)   x   xDownloading ifplugd (download size 42.00 kB) q   37  76%  100  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ifplugd-0.28-210.20.x86_64.rpm (installed size 100.00 kB) 46  x   xInstalling ifplugd-0.28-210.20.x86_64.rpm (installed size 100.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  56%  74%  90  5  100  7.9 8 3  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icedax (download size 174.00 kB)   x   xDownloading icedax (download size 174.00 kB) q   9  18  27  36  ±±45  55%  64%  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icedax-1.1.8-3.32.3.x86_64.rpm (installed size 387.00 kB) 3  x   xInstalling icedax-1.1.8-3.32.3.x86_64.rpm (installed size 387.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  34  50 67%  81  7  93  9  100  56 7 4.98  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gmp (download size 137.00 kB)   x   xDownloading gmp (download size 137.00 kB) q   11  23  34  ±46  58%  70%  81  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gmp-4.2.3-10.99.x86_64.rpm (installed size 310.00 kB) 4.98  x   xInstalling gmp-4.2.3-10.99.x86_64.rpm (installed size 310.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  ±±±±±41  61%  82  7  93  8  100  25 6 6  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading elfutils (download size 260.00 kB)   x   xDownloading elfutils (download size 260.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  86  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing elfutils-0.152-4.9.17.x86_64.rpm (installed size 795.00 kB) 6  x   xInstalling elfutils-0.152-4.9.17.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  18  23  3  ±±±±±41  ±46  52% 7%  65%  73  8  86  91  8  100  6.48 5 3.90  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xdg-utils (download size 49.00 kB)   x   xDownloading xdg-utils (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xdg-utils-1.0.2-36.18.noarch.rpm (installed size 183.00 kB) 3.90  x   xInstalling xdg-utils-1.0.2-36.18.noarch.rpm (installed size 183.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  28  37  49 64%  70  8  5  90  5  100  30 4 73  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lsb-release (download size 9.00 kB)   x   xDownloading lsb-release (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lsb-release-2.0-1.2.18.noarch.rpm (installed size 15.00 kB) 73  x   xInstalling lsb-release-2.0-1.2.18.noarch.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  99  100  28 3 1  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libncurses5-32bit (download size 242.00 kB)   x   xDownloading libncurses5-32bit (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% 9%  66  72  9  86  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libncurses5-32bit-5.6-90.55.x86_64.rpm (installed size 810.00 k 1  x   xInstalling libncurses5-32bit-5.6-90.55.x86_64.rpm (installed size 810.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  23  9  37  ±±45  53%  60%  5  72  80  7  98  100  5.49 2 2.92  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wireless-tools (download size 168.00 kB)   x   x~ kB) x    xDownloading wireless-tools (download size 168.00 kB) q   9  18  2  3  ±47  57%  66%  7  85  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wireless-tools-30.pre7-3.13.1.x86_64.rpm (installed size 408.00 2.92  x   xInstalling wireless-tools-30.pre7-3.13.1.x86_64.rpm (installed size 408.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  23  30  7  ±±45  53%  60%  71  6  81  7  93  8  100  0 1 5  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ruby (download size 1.76 MB)   x   x~00 kB) x    xDownloading ruby (download size 1.76 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ruby-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 6.43 MB) 58  x   xInstalling ruby-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 6.43 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  58.66 0 56.09  6 50 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openslp (download size 61.00 kB)   x   xDownloading openslp (download size 61.00 kB) q   26  52% 79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openslp-1.2.0-172.24.1.x86_64.rpm (installed size 137.00 kB) 56.097  x   xInstalling openslp-1.2.0-172.24.1.x86_64.rpm (installed size 137.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  19  66%  92  8  100  53 79 5.96  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libblocxx6 (download size 349.00 kB)   x   xDownloading libblocxx6 (download size 349.00 kB) q   9  18  27  36  ±±±±±41  50 9%  64  73  82  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libblocxx6-2.1.0.342-122.6.x86_64.rpm (installed size 1.15 MB) 5.96  x   xInstalling libblocxx6-2.1.0.342-122.6.x86_64.rpm (installed size 1.15 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  32  8  ±±±±43  48 54%  9%  65  70  6  81  6  92  7  100  7.38 8 4.81  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-gettext (download size 21.00 kB)   x   x~) x    xDownloading perl-gettext (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-gettext-1.05-91.95.x86_64.rpm (installed size 44.00 kB) 4.816  x   xInstalling perl-gettext-1.05-91.95.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  76%  91  8  100  4 7 76  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Writer (download size 24.00 kB)   x   xDownloading perl-XML-Writer (download size 24.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Writer-0.604-2.10.x86_64.rpm (installed size 52.00 kB) 76  x   xInstalling perl-XML-Writer-0.604-2.10.x86_64.rpm (installed size 52.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  72%  84  99  100  29 6 1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Parser (download size 231.00 kB)   x   x~B) x    xDownloading perl-XML-Parser (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Parser-2.36-1.18.x86_64.rpm (installed size 688.00 kB) 1  x   xInstalling perl-XML-Parser-2.36-1.18.x86_64.rpm (installed size 688.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  23  9  34  9  ±±±44  56%  65%  71  80  9  94  9  100  6.61 5 04  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-NamespaceSupport (download size 16.00 kB)   x   x~B) x    xDownloading perl-XML-NamespaceSupport (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-NamespaceSupport-1.09-1.22.x86_64.rpm (installed size 04  x   xInstalling perl-XML-NamespaceSupport-1.09-1.22.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  80  98  100  59 4 1  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-LibXML-Common (download size 17.00 kB)   x   x~e 29.00 kB) x    xDownloading perl-XML-LibXML-Common (download size 17.00 kB) q   95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-LibXML-Common-0.13-70.7.x86_64.rpm (installed size 36. 1  x   xInstalling perl-XML-LibXML-Common-0.13-70.7.x86_64.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  84%  90  8  100  5 3 3.98  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-URI (download size 111.00 kB)   x   x~6.00 kB) x    xDownloading perl-URI (download size 111.00 kB) q   14  28  ±±±±43  57%  72%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-URI-1.37-1.22.x86_64.rpm (installed size 252.00 kB) 3.98  x   xInstalling perl-URI-1.37-1.22.x86_64.rpm (installed size 252.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  6  21  6  33  8  ±±±±43  52% 77%  84  90  5  100  30 2 73  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-TimeDate (download size 34.00 kB)   x   xDownloading perl-TimeDate (download size 34.00 kB) q   ±47  95%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-TimeDate-1.16-71.22.x86_64.rpm (installed size 76.00 kB) 73  x   xInstalling perl-TimeDate-1.16-71.22.x86_64.rpm (installed size 76.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  7  23  9  37  ±±±±42  ±47  52% 63%  7  87  92  9  100  23 1 66  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Socket6 (download size 24.00 kB)   x   xDownloading perl-Socket6 (download size 24.00 kB) q   67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Socket6-0.22-1.24.1.x86_64.rpm (installed size 61.00 kB) 66  x   xInstalling perl-Socket6-0.22-1.24.1.x86_64.rpm (installed size 61.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  78%  94  9  100  17 0 0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Parse-RecDescent (download size 91.00 kB)   x   xDownloading perl-Parse-RecDescent (download size 91.00 kB) q   17  35  52% 70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Parse-RecDescent-1.96.0-1.22.x86_64.rpm (installed size 23 0  x   xInstalling perl-Parse-RecDescent-1.96.0-1.22.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  83%  99  100  5.95 69 37  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-HTML-Tagset (download size 15.00 kB)   x   x~230.00 kB) x    xDownloading perl-HTML-Tagset (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-HTML-Tagset-3.20-1.22.x86_64.rpm (installed size 20.00 kB) 37  x   xInstalling perl-HTML-Tagset-3.20-1.22.x86_64.rpm (installed size 20.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  9  76  97  100  3 8 5  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-SHA1 (download size 48.00 kB)   x   x~B) x    xDownloading perl-Digest-SHA1 (download size 48.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-SHA1-2.11-1.22.x86_64.rpm (installed size 112.00 kB 5  x   xInstalling perl-Digest-SHA1-2.11-1.22.x86_64.rpm (installed size 112.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  33  ±46  54%  95%  100  82 7 2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-MD4 (download size 23.00 kB)   x   x~kB) x    xDownloading perl-Digest-MD4 (download size 23.00 kB) q   69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-MD4-1.5-1.22.x86_64.rpm (installed size 44.00 kB) 2  x   xInstalling perl-Digest-MD4-1.5-1.22.x86_64.rpm (installed size 44.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  83%  98  100  77 6 0  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Config-IniFiles (download size 38.00 kB)   x   xDownloading perl-Config-IniFiles (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Config-IniFiles-2.39-1.24.1.x86_64.rpm (installed size 84. 0  x   xInstalling perl-Config-IniFiles-2.39-1.24.1.x86_64.rpm (installed size 8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  75%  82  99  100  69 5 12  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Config-Crontab (download size 50.00 kB)   x   x~4.00 kB) x    xDownloading perl-Config-Crontab (download size 50.00 kB) q   31  63%  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Config-Crontab-1.21-73.21.x86_64.rpm (installed size 115.0 12  x   xInstalling perl-Config-Crontab-1.21-73.21.x86_64.rpm (installed size 115q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  55%  76%  82  99  100  58 4 01  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Carp-Clan (download size 24.00 kB)   x   x~.00 kB) x    xDownloading perl-Carp-Clan (download size 24.00 kB) q   67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Carp-Clan-6.00-1.16.x86_64.rpm (installed size 46.00 kB) 01  x   xInstalling perl-Carp-Clan-6.00-1.16.x86_64.rpm (installed size 46.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  24  39  ±±45  82%  91  8  100  4 3 2.96 399 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading numactl (download size 64.00 kB)   x   xDownloading numactl (download size 64.00 kB) q   24  49 75%  100  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing numactl-2.0.10-0.9.31.x86_64.rpm (installed size 154.00 kB) 2.96  x   xInstalling numactl-2.0.10-0.9.31.x86_64.rpm (installed size 154.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  2  ±±±±±41  66%  84  9  9  100  38 2 81  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading login (download size 74.00 kB)   x   xDownloading login (download size 74.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing login-3.41-0.4.2.x86_64.rpm (installed size 291.00 kB) 81  x   xInstalling login-3.41-0.4.2.x86_64.rpm (installed size 291.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  9  25  31  6  ±±±±±41  ±46  53%  8%  64%  9  75  80  5  91  6  100  10 1 53  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lcms (download size 49.00 kB)   x   xDownloading lcms (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lcms-1.17-77.16.1.x86_64.rpm (installed size 156.00 kB) 53  x   xInstalling lcms-1.17-77.16.1.x86_64.rpm (installed size 156.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  18  3  52% 67%  86  93  8  100  4.95 0 38  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xz-lang (download size 36.00 kB)   x   xDownloading xz-lang (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xz-lang-5.0.3-0.12.1.x86_64.rpm (installed size 108.00 kB) 38  x   xInstalling xz-lang-5.0.3-0.12.1.x86_64.rpm (installed size 108.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  3  60%  8  99  100  84 59 27  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsensors3 (download size 73.00 kB)   x   xDownloading libsensors3 (download size 73.00 kB) q   21  ±±±±43  65%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsensors3-2.10.6-10.15.x86_64.rpm (installed size 436.00 kB) 27  x   xInstalling libsensors3-2.10.6-10.15.x86_64.rpm (installed size 436.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  9  33  48 63%  77  92  9  100  42 8 1.84  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iputils (download size 80.00 kB)   x   x~) x    xDownloading iputils (download size 80.00 kB) q   19  40  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iputils-ss021109-292.28.1.x86_64.rpm (installed size 219.00 kB) 1.845  x   xInstalling iputils-ss021109-292.28.1.x86_64.rpm (installed size 219.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  33  ±±±±42  50 9%  71  8  85  90  5  100  20 7 63  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading info (download size 159.00 kB)   x   x~B) x    xDownloading info (download size 159.00 kB) q   9  1  30  4  50 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing info-4.12-1.86.x86_64.rpm (installed size 278.00 kB) 63  x   xInstalling info-4.12-1.86.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  37  60%  72  81  90  8  100  3.93 6 36  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libSM (download size 16.00 kB)   x   xDownloading xorg-x11-libSM (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libSM-7.4-1.18.x86_64.rpm (installed size 35.00 kB) 36  x   xInstalling xorg-x11-libSM-7.4-1.18.x86_64.rpm (installed size 35.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 5 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading syslinux (download size 403.00 kB)   x   xDownloading syslinux (download size 403.00 kB) q   7  15  23  31  9  ±47  55%  63%  71  9  87  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing syslinux-3.82-8.17.6.x86_64.rpm (installed size 1.07 MB) 3  x   xInstalling syslinux-3.82-8.17.6.x86_64.rpm (installed size 1.07 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  7  22  7  32  9  4 56%  61%  7  75  81  6  92  7  100  2.83 4 0.26  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading reiserfs (download size 228.00 kB)   x   xDownloading reiserfs (download size 228.00 kB) q   6  13  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing reiserfs-3.6.21-1.15.1.x86_64.rpm (installed size 1016.00 kB) 0.26  x   xInstalling reiserfs-3.6.21-1.15.1.x86_64.rpm (installed size 1016.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  27  33  ±±±±4  49 54%  61%  7  76  83  9  95  100  1.84 3 49.27  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libblkid1 (download size 90.00 kB)   x   xDownloading libblkid1 (download size 90.00 kB) q   17  35  53%  71%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libblkid1-2.19.1-6.72.1.x86_64.rpm (installed size 153.00 kB) 49.27  x   xInstalling libblkid1-2.19.1-6.72.1.x86_64.rpm (installed size 153.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  84%  99  100  69 2 12  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmpfr1 (download size 125.00 kB)   x   xDownloading libmpfr1 (download size 125.00 kB) q   12  25  38  51 63%  76  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmpfr1-2.3.2-3.118.2.x86_64.rpm (installed size 285.00 kB) 12  x   xInstalling libmpfr1-2.3.2-3.118.2.x86_64.rpm (installed size 285.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  ±±45  67%  90  9  100  41 1 8.84  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading facter (download size 52.00 kB)   x   xDownloading facter (download size 52.00 kB) q   3  61%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing facter-1.6.18-0.3.1.x86_64.rpm (installed size 146.00 kB) 8.84  x   xInstalling facter-1.6.18-0.3.1.x86_64.rpm (installed size 146.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±42  ±47  52% 60%  6  74  82  8  93  8  100  27 0 70  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal (download size 137.00 kB)   x   xDownloading limal (download size 137.00 kB) q   11  23  34  ±46  58%  70%  81  93  100  2% ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-1.5.3-0.2.11.x86_64.rpm (installed size 538.00 kB) 70  x   xInstalling limal-1.5.3-0.2.11.x86_64.rpm (installed size 538.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  35  ±47  56%  61%  6  72  7  83  8  94  9  100  0.74 49 17  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-XPath (download size 76.00 kB)   x   xDownloading perl-XML-XPath (download size 76.00 kB) q   2  ±±±±±41  63%  84  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-XPath-1.13-74.15.x86_64.rpm (installed size 191.00 kB) 174  x   xInstalling perl-XML-XPath-1.13-74.15.x86_64.rpm (installed size 191.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  21  7  35  ±±±±43  48 60%  9  75  80  5  90  5  100  56 8 7.98  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-Simple (download size 71.00 kB)   x   x~B) x    xDownloading perl-XML-Simple (download size 71.00 kB) q   22  ±±45  68%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-Simple-2.18-1.15.x86_64.rpm (installed size 156.00 kB) 7.98  x   xInstalling perl-XML-Simple-2.18-1.15.x86_64.rpm (installed size 156.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  60%  72  80  93  9  100  40 7 83  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-SAX (download size 75.00 kB)   x   x~B) x    xDownloading perl-XML-SAX (download size 75.00 kB) q   21  ±±±±42  64%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-SAX-0.96-2.7.x86_64.rpm (installed size 260.00 kB) 83  x   xInstalling perl-XML-SAX-0.96-2.7.x86_64.rpm (installed size 260.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  48 56%  66%  72  7  82  7  92  8  100  15 6 58  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-X500-DN (download size 14.00 kB)   x   xDownloading perl-X500-DN (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-X500-DN-0.29-78.22.x86_64.rpm (installed size 16.00 kB) 58  x   xInstalling perl-X500-DN-0.29-78.22.x86_64.rpm (installed size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  29  3  48 53%  60%  7  83  97  100  3 5 6  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-HTML-Parser (download size 109.00 kB)   x   xDownloading perl-HTML-Parser (download size 109.00 kB) q   14  29  ±±±±43  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-HTML-Parser-3.56-1.18.1.x86_64.rpm (installed size 248.00 6  x   xInstalling perl-HTML-Parser-3.56-1.18.1.x86_64.rpm (installed size 248.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  26  33  59%  65  79  84  9  96  100  49.89 4 32  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Digest-HMAC (download size 21.00 kB)   x   x~0 kB) x    xDownloading perl-Digest-HMAC (download size 21.00 kB) q   77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Digest-HMAC-1.01-436.17.x86_64.rpm (installed size 34.00 k 32  x   xInstalling perl-Digest-HMAC-1.01-436.17.x86_64.rpm (installed size 34.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  20  80%  6  92  8  100  6 3 29  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Crypt-SmbHash (download size 15.00 kB)   x   x~ kB) x    xDownloading perl-Crypt-SmbHash (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Crypt-SmbHash-0.12-132.17.x86_64.rpm (installed size 25.00 29  x   xInstalling perl-Crypt-SmbHash-0.12-132.17.x86_64.rpm (installed size 25.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  80  5  98  100  3 2 6  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Bit-Vector (download size 198.00 kB)   x   x~00 kB) x    xDownloading perl-Bit-Vector (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Bit-Vector-6.4-1.14.x86_64.rpm (installed size 1.29 MB) 6  x   xInstalling perl-Bit-Vector-6.4-1.14.x86_64.rpm (installed size 1.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  9  24  9  38  ±±±±43  48 53%  8%  67%  72  7  82  7  92  8  100  8.54 1 5.97  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libmng (download size 183.00 kB)   x   xDownloading libmng (download size 183.00 kB) q   8  17  26  34  ±±±±43  52% 61%  70  8  87  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libmng-1.0.10-46.18.x86_64.rpm (installed size 566.00 kB) 5.97  x   xInstalling libmng-1.0.10-46.18.x86_64.rpm (installed size 566.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±45  56%  67%  79  90  9  100  7.99 0 42  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zsh (download size 2.58 MB)   x   xDownloading zsh (download size 2.58 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zsh-4.3.6-67.9.1.x86_64.rpm (installed size 7.97 MB) 42  x   xInstalling zsh-4.3.6-67.9.1.x86_64.rpm (installed size 7.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  0.02 39 37.45  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading screen (download size 453.00 kB)   x   xDownloading screen (download size 453.00 kB) q   7  13  20  7  34  ±±±±±41  48 55%  62%  9  76  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing screen-4.0.2-162.19.2.1.x86_64.rpm (installed size 725.00 kB) 37.451  x   xInstalling screen-4.0.2-162.19.2.1.x86_64.rpm (installed size 725.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  26  35  ±±±±43  48 57%  63%  71  9  86  95  100  39.31 8 6.74  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading recode (download size 744.00 kB)   x   xDownloading recode (download size 744.00 kB) q   5  11  7  23  8  34  ±±±±±41  ±47  54%  60%  6  71  7  82  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing recode-3.6-635.18.x86_64.rpm (installed size 2.23 MB) 6.74  x   xInstalling recode-3.6-635.18.x86_64.rpm (installed size 2.23 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  34  40  ±46  51 7%  63%  8  74  9  84  9  94  9  100  7.08 7 4.51  3 3%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pinentry (download size 60.00 kB)   x   xDownloading pinentry (download size 60.00 kB) q   26  53%  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pinentry-0.7.5-61.33.1.x86_64.rpm (installed size 136.00 kB) 4.510  x   xInstalling pinentry-0.7.5-61.33.1.x86_64.rpm (installed size 136.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  35  ±47  86%  92  9  100  6.95 6 38  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading make (download size 356.00 kB)   x   xDownloading make (download size 356.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing make-3.81-128.20.x86_64.rpm (installed size 985.00 kB) 38  x   xInstalling make-3.81-128.20.x86_64.rpm (installed size 985.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  23  30  5  ±±±±42  50 7%  63%  9  75  82  7  93  9  100  5.99 5 3.42  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading m4 (download size 234.00 kB)   x   xDownloading m4 (download size 234.00 kB) q   6  13  20  7  34  ±±±±±41  ±47  54%  61%  8  75  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing m4-1.4.12-1.19.x86_64.rpm (installed size 441.00 kB) 3.42  x   xInstalling m4-1.4.12-1.19.x86_64.rpm (installed size 441.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  33  48 62%  9  75  92  9  100  56 4 2.99  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libtool (download size 514.00 kB)   x   xDownloading libtool (download size 514.00 kB) q   6  11  7  24  30  6  ±±±±42  48 55%  61%  7  73  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libtool-2.2.6-2.131.1.x86_64.rpm (installed size 2.33 MB) 2.99  x   xInstalling libtool-2.2.6-2.131.1.x86_64.rpm (installed size 2.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  9  34  40  ±46  51 7%  62%  8  73  9  85  90  5  100  3.23 3 0.66  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libreiserfs (download size 43.00 kB)   x   xDownloading libreiserfs (download size 43.00 kB) q   36  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libreiserfs-0.3.0.5-143.3.x86_64.rpm (installed size 111.00 kB) 0.6649  x   xInstalling libreiserfs-0.3.0.5-143.3.x86_64.rpm (installed size 111.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  67%  99  100  12 2 55  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gzip (download size 74.00 kB)   x   x~B) x    xDownloading gzip (download size 74.00 kB) q   21  ±±±±42  64%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gzip-1.3.12-69.23.1.x86_64.rpm (installed size 116.00 kB) 55  x   xInstalling gzip-1.3.12-69.23.1.x86_64.rpm (installed size 116.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  61%  7  74  89  95  100  01 1 44  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading grep (download size 232.00 kB)   x   xDownloading grep (download size 232.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing grep-2.7-5.7.1.x86_64.rpm (installed size 835.00 kB) 44  x   xInstalling grep-2.7-5.7.1.x86_64.rpm (installed size 835.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  3  40  ±47  55%  60%  5  70  5  80  5  90  5  100  2.19 0 29.62  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gettext-runtime (download size 699.00 kB)   x   xDownloading gettext-runtime (download size 699.00 kB) q   6  11  6  22  7  32  8  ±±±±43  48 54%  9%  64  70  5  80  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gettext-runtime-0.17-61.40.x86_64.rpm (installed size 2.18 MB) 29.62  x   xInstalling gettext-runtime-0.17-61.40.x86_64.rpm (installed size 2.18 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  7  33  9  ±±±44  50 6%  62%  8  74  81  7  92  7  100  0.02 29 7.44  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gawk (download size 773.00 kB)   x   x~) x    xDownloading gawk (download size 773.00 kB) q   6  11  7  23  9  35  40  ±46  52% 8%  64%  9  75  81  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gawk-3.1.8-2.6.1.x86_64.rpm (installed size 1.92 MB) 7.448  x   xInstalling gawk-3.1.8-2.6.1.x86_64.rpm (installed size 1.92 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  9  25  31  8  ±±±±43  49 55%  61%  8  74  80  5  90  5  100  28.10 8 5.53  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading findutils (download size 321.00 kB)   x   xDownloading findutils (download size 321.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing findutils-4.4.0-38.26.1.x86_64.rpm (installed size 896.00 kB) 5.53  x   xInstalling findutils-4.4.0-38.26.1.x86_64.rpm (installed size 896.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  22  9  38  ±±±44  49 54%  9%  64  9  75  80  5  91  6  100  7.23 7 4.65  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fastjar (download size 78.00 kB)   x   xDownloading fastjar (download size 78.00 kB) q   2  4  61%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fastjar-0.95-1.24.1.x86_64.rpm (installed size 175.00 kB) 4.65  x   xInstalling fastjar-0.95-1.24.1.x86_64.rpm (installed size 175.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  60%  73  9  8  100  05 6 48  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ed (download size 45.00 kB)   x   xDownloading ed (download size 45.00 kB) q   35  71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ed-0.2-1001.30.1.x86_64.rpm (installed size 71.00 kB) 48  x   xInstalling ed-0.2-1001.30.1.x86_64.rpm (installed size 71.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  76%  89  9  100  6.99 5 1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading diffutils (download size 289.00 kB)   x   xDownloading diffutils (download size 289.00 kB) q   5  11  6  22  7  33  8  ±±±44  49 55%  60%  6  72  7  83  8  94  9  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing diffutils-2.8.7-143.23.1.x86_64.rpm (installed size 982.00 kB) 1  x   xInstalling diffutils-2.8.7-143.23.1.x86_64.rpm (installed size 982.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  21  7  34  9  ±46  52% 7%  63%  9  74  80  5  92  7  100  03 4 3.45  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading coreutils (download size 1.19 MB)   x   x~) x    xDownloading coreutils (download size 1.19 MB) q   5  10  5  21  6  31  6  ±±±±42  ±47  52% 7%  63%  8  73  8  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing coreutils-8.12-6.25.32.33.1.x86_64.rpm (installed size 4.60 MB) 3.457  x   xInstalling coreutils-8.12-6.25.32.33.1.x86_64.rpm (installed size 4.60 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  31  6  ±±±±±41  ±46  52% 7%  62%  7  73  8  83  8  94  9  100  1.42 3 18.8  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bc (download size 108.00 kB)   x   x~B) x    xDownloading bc (download size 108.00 kB) q   12  27  ±±±±42  57%  72%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bc-1.06-838.15.x86_64.rpm (installed size 239.00 kB) 18.86  x   xInstalling bc-1.06-838.15.x86_64.rpm (installed size 239.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  32  50 8%  69%  83  91  7  100  19 2 62  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading FirmwareUpdateKit (download size 117.00 kB)   x   xDownloading FirmwareUpdateKit (download size 117.00 kB) q   13  27  40  54%  68%  81  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing FirmwareUpdateKit-1.1-2.17.x86_64.rpm (installed size 178.00 kB 62  x   xInstalling FirmwareUpdateKit-1.1-2.17.x86_64.rpm (installed size 178.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  ±±±44  70%  9  9  100  02 1 44  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpp43 (download size 2.80 MB)   x   x~kB) x    xDownloading cpp43 (download size 2.80 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpp43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.83 MB 44  x   xInstalling cpp43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.83 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  13.18 0 0.61  6 5%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-perl (download size 105.00 kB)   x   x~MB) x    xDownloading limal-perl (download size 105.00 kB) q   15  30  ±±45  61%  76  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-perl-1.5.3-0.2.11.x86_64.rpm (installed size 374.00 kB) 0.614  x   xInstalling limal-perl-1.5.3-0.2.11.x86_64.rpm (installed size 374.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  37  54%  71%  88  99  100  2.82 19 25  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-nfs-server (download size 115.00 kB)   x   xDownloading limal-nfs-server (download size 115.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-nfs-server-1.5.3-0.2.11.x86_64.rpm (installed size 487.00 25  x   xInstalling limal-nfs-server-1.5.3-0.2.11.x86_64.rpm (installed size 487.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  26  39  52% 7%  63%  8  73  9  84  9  95  100  34 8 09.77  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-ca-mgm (download size 660.00 kB)   x   x~00 kB) x    xDownloading limal-ca-mgm (download size 660.00 kB) q   7  12  7  22  7  34  9  ±46  51 6%  63%  8  73  80  5  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-ca-mgm-1.5.24-0.3.2.x86_64.rpm (installed size 2.65 MB) 09.77  x   xInstalling limal-ca-mgm-1.5.24-0.3.2.x86_64.rpm (installed size 2.65 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  40  ±47  54%  61%  8  73  8  83  8  93  8  100  09.70 7 7.12  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-XML-LibXML (download size 358.00 kB)   x   xDownloading perl-XML-LibXML (download size 358.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-XML-LibXML-1.66-1.14.x86_64.rpm (installed size 1.11 MB) 7.123  x   xInstalling perl-XML-LibXML-1.66-1.14.x86_64.rpm (installed size 1.11 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  33  8  ±±±44  49 55%  61%  6  72  7  83  8  93  8  100  8.58 6 6.01  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-NetxAP (download size 54.00 kB)   x   xDownloading perl-NetxAP (download size 54.00 kB) q   29  59%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-NetxAP-0.02-604.22.x86_64.rpm (installed size 135.00 kB) 6.012  x   xInstalling perl-NetxAP-0.02-604.22.x86_64.rpm (installed size 135.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  57%  69%  75  84  96  100  45 5 5.88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Date-Calc (download size 258.00 kB)   x   xDownloading perl-Date-Calc (download size 258.00 kB) q   6  12  8  24  31  7  ±±±±43  49 55%  62%  8  74  - 191.74 kB/s (download size 258.00 kB) 80  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Date-Calc-5.4-137.14.x86_64.rpm (installed size 1.03 MB) 5.88  x   xInstalling perl-Date-Calc-5.4-137.14.x86_64.rpm (installed size 1.03 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  8  25  31  6  ±±±44  50 6%  62%  8  73  8  83  8  93  8  9  100  xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 704.85 MBx 350x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 207.42 MBx 214x http-10.9.0.10-ae2fc03cxxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcsh (download size 357.00 kB)   x   xDownloading tcsh (download size 357.00 kB) q   8  16  25  34  ±±±±43  52% 61%  70  9  88  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcsh-6.18.01-0.9.1.x86_64.rpm (installed size 1.01 MB) 4.854:10  x   xInstalling tcsh-6.18.01-0.9.1.x86_64.rpm (installed size 1.01 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  5  ±±±±±41  ±47  53%  62%  71  6  84  9  9  100  6.40 3 3.83  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tcl (download size 1.69 MB)   x   xDownloading tcl (download size 1.69 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tcl-8.5.5-2.81.x86_64.rpm (installed size 5.05 MB) 3.83  x   xInstalling tcl-8.5.5-2.81.x86_64.rpm (installed size 5.05 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 7%  62%  7  72  7  83  8  93  8  100  1.35 2 698.78  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysvinit (download size 204.00 kB)   x   xDownloading sysvinit (download size 204.00 kB) q   7  15  23  31  9  ±47  54%  62%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysvinit-2.86-219.37.x86_64.rpm (installed size 642.00 kB) 698.7808  x   xInstalling sysvinit-2.86-219.37.x86_64.rpm (installed size 642.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  23  9  3  ±±45  50 6%  61%  7  72  80  6  91  6  100  0.72 1 15  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading srptools (download size 42.00 kB)   x   xDownloading srptools (download size 42.00 kB) q   37  76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing srptools-0.0.4-11.26.x86_64.rpm (installed size 94.00 kB) 15  x   xInstalling srptools-0.0.4-11.26.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  72%  96  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  63 0 06  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sharutils (download size 163.00 kB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading sharutils (download size 163.00 kB)  9  1  2  3  48 58%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sharutils-4.6.3-3.19.x86_64.rpm (installed size 577.00 kB) 06  x   xInstalling sharutils-4.6.3-3.19.x86_64.rpm (installed size 577.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  24  31  8  ±±±44  50 5%  61%  9  75  82  9  96  100  07 09 7.50  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opensm (download size 502.00 kB)   x   xDownloading opensm (download size 502.00 kB) q   6  12  9  25  31  8  ±±±44  51 7%  63%  70  6  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opensm-3.3.13-12.26.x86_64.rpm (installed size 1.34 MB) 7.50  x   xInstalling opensm-3.3.13-12.26.x86_64.rpm (installed size 1.34 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  31  6  ±±45  54%  9%  64  73  80  5  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/opensm... q  198.73 8 6.15  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lukemftp (download size 129.00 kB)  Additional rpm output: Updating etc/sysconfig/opensm...   Downloading lukemftp (download size 129.00 kB)  12  24  37  49 62%  74  86  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lukemftp-1.5-725.17.x86_64.rpm (installed size 278.00 kB) 6.157  x   xInstalling lukemftp-1.5-725.17.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±±45  68%  7  90  9  100  46 7 5.88  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ib-bonding (download size 11.00 kB)   x   xDownloading ib-bonding (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ib-bonding-0.9.0-138.35.x86_64.rpm (installed size 18.00 kB) 5.88  x   xInstalling ib-bonding-0.9.0-138.35.x86_64.rpm (installed size 18.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  55%  93%  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  4 6 7  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading groff (download size 1.92 MB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading groff (download size 1.92 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing groff-1.18.1.1-167.18.61.x86_64.rpm (installed size 7.24 MB) 7  x   xInstalling groff-1.18.1.1-167.18.61.x86_64.rpm (installed size 7.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/suseconfig... q  1.20 5 88.62  1 6%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading coreutils-lang (download size 1.38 MB)  Additional rpm output: Updating etc/sysconfig/suseconfig...   Downloading coreutils-lang (download size 1.38 MB)  5  10  5  20  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing coreutils-lang-8.12-6.25.32.33.1.x86_64.rpm (installed size 7.9 88.625  x   xInstalling coreutils-lang-8.12-6.25.32.33.1.x86_64.rpm (installed size 7q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  83.27 4 0.70  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading blktrace (download size 236.00 kB)   x   x~.93 MB) x    xDownloading blktrace (download size 236.00 kB) q   6  13  20  7  33  40  ±47  54%  61%  7  74  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing blktrace-1.0.3-0.7.1.x86_64.rpm (installed size 414.00 kB) 0.702  x   xInstalling blktrace-1.0.3-0.7.1.x86_64.rpm (installed size 414.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  9  25  30  9  55%  60%  75  90  5  100  2.8 3 29  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-theme-SLE (download size 2.72 MB)   x   xDownloading yast2-theme-SLE (download size 2.72 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-theme-SLE-2.17.27-0.18.9.noarch.rpm (installed size 2.87 29  x   xInstalling yast2-theme-SLE-2.17.27-0.18.9.noarch.rpm (installed size 2.8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  8  33  8  ±±±±43  48 54%  9%  64  9  74  9  84  9  94  9  100  0.00 2 77.43  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ulimit (download size 9.00 kB)   x   x~7 MB) x    xDownloading ulimit (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ulimit-1.2-2.4.11.noarch.rpm (installed size 13.00 kB) 77.431  x   xInstalling ulimit-1.2-2.4.11.noarch.rpm (installed size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  99%  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/ulimit... q  79.99 1 2  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading linux-kernel-headers (download size 813.00 kB)  Additional rpm output: Updating etc/sysconfig/ulimit...   Downloading linux-kernel-headers (download size 813.00 kB)  5  10  5  21  7  33  9  ±±45  51 7%  63%  9  75  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.95 2  x   xInstalling linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.9q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  Downloading linux-kernel-headers (download size 813.00 kB) Installing linux-kernel-headers-3.0-19.18.noarch.rpm (installed size 2.9 ~5 MB) Additional rpm output: post linux-kernel-headers-3.0-19.18 /var/tmp/rpm-tmp.39930 1  7.04 0 4.46  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpp (download size 5.00 kB)  Additional rpm output: post linux-kernel-headers-3.0-19.18 /var/tmp/rpm-tmp.39930 1   Downloading cpp (download size 5.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpp-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB) 4.460  x   xInstalling cpp-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  33  54%  79%  100  3 199  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-nfs-server-perl (download size 135.00 kB)   x   xDownloading limal-nfs-server-perl (download size 135.00 kB) q   11  23  35  ±47  59%  71  83  95  100  7%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-nfs-server-perl-1.5.3-0.2.11.x86_64.rpm (installed size 6  x   xInstalling limal-nfs-server-perl-1.5.3-0.2.11.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  25  3  ±46  56%  66%  77  8  98  100  6.44 8 3.87  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading limal-ca-mgm-perl (download size 351.00 kB)   x   x~ 610.00 kB) x    xDownloading limal-ca-mgm-perl (download size 351.00 kB) q   9  18  27  36  ±±±±±41  50 9%  68  77  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing limal-ca-mgm-perl-1.5.24-0.3.2.x86_64.rpm (installed size 1.83 3.87  x   xInstalling limal-ca-mgm-perl-1.5.24-0.3.2.x86_64.rpm (installed size 1.8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  21  8  35  ±±±±42  49 56%  62%  9  76  83  90  7  100  4.61 7 2.04  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mail-plugins (download size 23.00 kB)   x   x~3 MB) x    xDownloading yast2-mail-plugins (download size 23.00 kB) q   70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mail-plugins-2.17.7-0.15.1.noarch.rpm (installed size 65. 2.043:59  x   xInstalling yast2-mail-plugins-2.17.7-0.15.1.noarch.rpm (installed size 6q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  25  53%  99%  100  55 6 1.98  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mpi-selector (download size 24.00 kB)   x   x~5.00 kB) x    xDownloading mpi-selector (download size 24.00 kB) q   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mpi-selector-1.0.3-0.3.1.x86_64.rpm (installed size 49.00 kB) 1.98  x   xInstalling mpi-selector-1.0.3-0.3.1.x86_64.rpm (installed size 49.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  71%  6  81  6  99  100  0 5 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cvs (download size 387.00 kB)   x   xDownloading cvs (download size 387.00 kB) q   8  16  24  33  ±±±±±41  49 57%  62%  70  8  86  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cvs-1.12.12-144.23.5.1.x86_64.rpm (installed size 899.00 kB) 3  x   xInstalling cvs-1.12.12-144.23.5.1.x86_64.rpm (installed size 899.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  5  80  7  92  8  100  3.62 4 05  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading expect (download size 277.00 kB)   x   xDownloading expect (download size 277.00 kB) q   5  11  7  23  8  34  40  ±46  51 7%  63%  9  75  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing expect-5.44.1.11-1.241.1.x86_64.rpm (installed size 668.00 kB) 05  x   xInstalling expect-5.44.1.11-1.241.1.x86_64.rpm (installed size 668.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  26  35  ±±4  52% 62%  7  8  90  7  100  2.97 3 0.40  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading metamail (download size 153.00 kB)   x   x~) x    xDownloading metamail (download size 153.00 kB) q   1  2  31  ±±±±±4  52% 62%  73  84  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing metamail-2.7.19-1244.20.x86_64.rpm (installed size 422.00 kB) 0.40  x   xInstalling metamail-2.7.19-1244.20.x86_64.rpm (installed size 422.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  30  7  ±±±44  50 5%  60%  78  89  94  9  100  56 2 69.99  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading man (download size 569.00 kB)   x   xDownloading man (download size 569.00 kB) q   5  10  5  20  6  32  7  ±±±±43  48 54%  60%  5  71  7  82  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing man-2.5.2-17.16.x86_64.rpm (installed size 1.93 MB) 69.99  x   xInstalling man-2.5.2-17.16.x86_64.rpm (installed size 1.93 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  35  ±±±±±41  ±47  52% 7%  63%  9  74  9  84  9  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/cron... q  0.63 1 8.06  7 8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-branding-SLES (download size 10.00 kB)  Additional rpm output: Updating etc/sysconfig/cron...   Downloading yast2-branding-SLES (download size 10.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-branding-SLES-2.17.0-1.11.noarch.rpm (installed size 18.0 8.068  x   xInstalling yast2-branding-SLES-2.17.0-1.11.noarch.rpm (installed size 18q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 0 4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-common (download size 8.00 kB)   x   x~.00 kB) x    xDownloading yast2-nfs-common (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-common-2.17.8-0.2.1.noarch.rpm (installed size 4.00 k 4  x   xInstalling yast2-nfs-common-2.17.8-0.2.1.noarch.rpm (installed size 4.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  64%  73  96  100  89 3  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi (download size 1.13 MB)   x   x~ kB) x    xDownloading openmpi (download size 1.13 MB) q   6  11  7  22  7  32  8  ±±±±43  49 54%  60%  5  71  6  82  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-1.8.4-1.26.x86_64.rpm (installed size 3.93 MB) 3  x   xInstalling openmpi-1.8.4-1.26.x86_64.rpm (installed size 3.93 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 6%  61%  6  71  6  81  6  91  7  100  66.67 8 4.10  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx q Total x 664.10 MBx 324x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-ae2fc03cxxx tu Downloading powerman (download size 254.00 kB)   x   xDownloading powerman (download size 254.00 kB) q   6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing powerman-2.3.24-1.02.wc1.x86_64.rpm (installed size 1.16 MB) 4.104:12  x   xInstalling powerman-2.3.24-1.02.wc1.x86_64.rpm (installed size 1.16 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  48 54%  9%  65  70  5  81  6  95  100  2.94  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fdtree (download size 12.00 kB)   x   xDownloading fdtree (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fdtree-1.0.2-1.wc1.x86_64.rpm (installed size 32.00 kB) 2.9405  x   xInstalling fdtree-1.0.2-1.wc1.x86_64.rpm (installed size 32.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  87%  99  100  1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iozone (download size 690.00 kB)   x   xDownloading iozone (download size 690.00 kB) q   6  11  6  23  30  7  ±±±44  50 5%  60%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iozone-3-373.wc1.x86_64.rpm (installed size 1.00 MB) 1  x   xInstalling iozone-3-373.wc1.x86_64.rpm (installed size 1.00 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  23  9  38  ±±45  51 9%  65  70  6  83  9  95  100  1  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading posix (download size 3.54 MB)   x   xDownloading posix (download size 3.54 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing posix-1.1-intel.x86_64.rpm (installed size 4.08 MB) 1  x   xInstalling posix-1.1-intel.x86_64.rpm (installed size 4.08 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±42  48 54%  60%  5  71  6  82  7  93  9  100  57.83  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading munge-libs (download size 42.00 kB)   x   xDownloading munge-libs (download size 42.00 kB) q   38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing munge-libs-0.5.10-1.wc1.x86_64.rpm (installed size 120.00 kB) 57.834  x   xInstalling munge-libs-0.5.10-1.wc1.x86_64.rpm (installed size 120.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  99%  100  71  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bonnie++ (download size 34.00 kB)   x   xDownloading bonnie++ (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bonnie++-1.03e-1.wc1.x86_64.rpm (installed size 77.00 kB) 71  x   xInstalling bonnie++-1.03e-1.wc1.x86_64.rpm (installed size 77.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  61%  80  92  7  100  63  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading compilebench (download size 248.00 kB)   x   xDownloading compilebench (download size 248.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing compilebench-0.6-1.wc1.x86_64.rpm (installed size 2.98 MB) 63  x   xInstalling compilebench-0.6-1.wc1.x86_64.rpm (installed size 2.98 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  23  8  34  40  ±±45  50 7%  63%  9  75  80  6  92  8  100  4.65  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading simul (download size 23.00 kB)   x   xDownloading simul (download size 23.00 kB) q   69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing simul-1.14-1.wc1.x86_64.rpm (installed size 62.00 kB) 4.653  x   xInstalling simul-1.14-1.wc1.x86_64.rpm (installed size 62.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  69%  96  100  59  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading IOR (download size 78.00 kB)   x   xDownloading IOR (download size 78.00 kB) q   2  4  61%  82  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing IOR-2.10.3-wc1.x86_64.rpm (installed size 278.00 kB) 59  x   xInstalling IOR-2.10.3-wc1.x86_64.rpm (installed size 278.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  ±±45  68%  7  83  99  100  32  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mdtest (download size 43.00 kB)   x   xDownloading mdtest (download size 43.00 kB) q   37  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mdtest-1.8.3-1.wc1.x86_64.rpm (installed size 127.00 kB) 322  x   xInstalling mdtest-1.8.3-1.wc1.x86_64.rpm (installed size 127.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 54%  65%  70  5  94  9  100  20  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading metabench (download size 29.00 kB)   x   xDownloading metabench (download size 29.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing metabench-1.0-1.wc2.x86_64.rpm (installed size 69.00 kB) 20  x   xInstalling metabench-1.0-1.wc2.x86_64.rpm (installed size 69.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  90%  9  100  13  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading munge (download size 221.00 kB)   x   xDownloading munge (download size 221.00 kB) q   7  14  21  8  36  ±±±±43  50 7%  65%  72  9  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing munge-0.5.10-1.wc1.x86_64.rpm (installed size 795.00 kB) 13  x   xInstalling munge-0.5.10-1.wc1.x86_64.rpm (installed size 795.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  29  3  ±47  55%  63%  71  9  87  92  7  100  3.35  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh-rcmd-mrsh (download size 16.00 kB)   x   xDownloading pdsh-rcmd-mrsh (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-rcmd-mrsh-2.29-1.wc1.x86_64.rpm (installed size 35.00 kB) 3.35  x   xInstalling pdsh-rcmd-mrsh-2.29-1.wc1.x86_64.rpm (installed size 35.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pdsh (download size 208.00 kB)   x   x~) x    xDownloading pdsh (download size 208.00 kB) q   7  15  23  30  8  ±46  53%  61%  9  77  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pdsh-2.29-1.wc1.x86_64.rpm (installed size 1.59 MB) 2  x   xInstalling pdsh-2.29-1.wc1.x86_64.rpm (installed size 1.59 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  31  6  ±±±44  51 9%  67  75  83  91  6  100  1.73  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x           qq  xSizexPackagesx w Total x 651.73 MBx 310x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-ae2fc03cxxx tu Downloading libcom_err2 (download size 60.00 kB)   x   xDownloading libcom_err2 (download size 60.00 kB) q   26  53%  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcom_err2-1.42.13.wc5-7suse.x86_64.rpm (installed size 126.00 1.739  x   xInstalling libcom_err2-1.42.13.wc5-7suse.x86_64.rpm (installed size 126.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  80%  99  100  61  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libext2fs2 (download size 389.00 kB)   x   x~00 kB) x    xDownloading libext2fs2 (download size 389.00 kB) q   8  16  24  32  7  ±±45  53%  61%  9  74  82  90  8  100  9%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libext2fs2-1.42.13.wc5-7suse.x86_64.rpm (installed size 1.35 MB 611  x   xInstalling libext2fs2-1.42.13.wc5-7suse.x86_64.rpm (installed size 1.35 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  26  31  40  49 54%  63%  72  7  86  91  9  100  0.25  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading e2fsprogs (download size 1.29 MB)   x   x~MB) x    xDownloading e2fsprogs (download size 1.29 MB) q   6  11  6  21  6  31  7  ±±±±42  48 54%  9%  65  70  6  82  7  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing e2fsprogs-1.42.13.wc5-7suse.x86_64.rpm (installed size 5.13 MB) 0.250  x   xInstalling e2fsprogs-1.42.13.wc5-7suse.x86_64.rpm (installed size 5.13 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  33  8  ±±±±43  48 53%  8%  63%  9  74  9  85  91  6  100  45.12  7 60  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 645.12 MBx 307x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e6784b1axxx tu Downloading ruby-devel (download size 1021.00 kB)   x   x~B) x    xDownloading ruby-devel (download size 1021.00 kB) q   6  11  6  22  7  33  8  ±±±44  49 55%  61%  6  72  7  83  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ruby-devel-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 4.46 45.123:59  x   xInstalling ruby-devel-1.8.7.p357-0.9.17.1.x86_64.rpm (installed size 4.4q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  32  7  ±±±±42  ±47  53%  8%  63%  8  74  9  84  9  95  100  0.66  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 640.66 MBx 306x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e6784b1axxx tu Downloading yast2-core (download size 1009.00 kB)   x   x~6 MB) x    xDownloading yast2-core (download size 1009.00 kB) q   5  10  5  20  6  31  7  ±±±±42  ±47  53%  8%  64%  9  75  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-core-2.17.47-7.1.x86_64.rpm (installed size 3.28 MB) 0.664:03  x   xInstalling yast2-core-2.17.47-7.1.x86_64.rpm (installed size 3.28 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  32  7  ±±±±43  48 53%  9%  65  71  6  82  8  94  9  100  37.38  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-storage-lib (download size 718.00 kB)   x   xDownloading yast2-storage-lib (download size 718.00 kB) q   6  11  7  24  30  5  ±±±±±41  ±46  52% 7%  64%  70  5  81  6  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-storage-lib-2.17.161-5.1.x86_64.rpm (installed size 1.99 37.382  x   xInstalling yast2-storage-lib-2.17.161-5.1.x86_64.rpm (installed size 1.9q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  25  31  7  ±±±44  50 6%  62%  9  75  81  8  94  9  100  5.40  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libidn (download size 223.00 kB)   x   x~9 MB) x    xDownloading libidn (download size 223.00 kB) q   7  14  21  8  35  ±±±±42  50 7%  64%  71  8  86  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libidn-1.10-6.1.x86_64.rpm (installed size 788.00 kB) 5.401  x   xInstalling libidn-1.10-6.1.x86_64.rpm (installed size 788.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  20  8  36  ±±±44  52% 60%  8  75  8  90  5  100  4.63  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading binutils (download size 5.13 MB)   x   xDownloading binutils (download size 5.13 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing binutils-2.25.0-5.5.x86_64.rpm (installed size 33.97 MB) 4.63  x   xInstalling binutils-2.25.0-5.5.x86_64.rpm (installed size 33.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  00.65  2 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tar (download size 753.00 kB)   x   xDownloading tar (download size 753.00 kB) q   6  11  6  22  8  33  9  ±±45  50 6%  62%  7  73  9  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tar-1.26-1.2.13.1.x86_64.rpm (installed size 2.48 MB) 00.653:48  x   xInstalling tar-1.26-1.2.13.1.x86_64.rpm (installed size 2.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  7  ±±±±42  49 54%  9%  64  9  74  9  84  90  5  100  598.17  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sed (download size 160.00 kB)   x   xDownloading sed (download size 160.00 kB) q   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sed-4.1.5-85.32.1.x86_64.rpm (installed size 491.00 kB) 598.177  x   xInstalling sed-4.1.5-85.32.1.x86_64.rpm (installed size 491.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  7  23  40  52% 9%  64  9  74  9  85  90  5  100  7.69  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libksba (download size 104.00 kB)   x   xDownloading libksba (download size 104.00 kB) q   15  30  ±46  61%  77  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libksba-1.0.4-1.25.1.x86_64.rpm (installed size 271.00 kB) 7.69  x   xInstalling libksba-1.0.4-1.25.1.x86_64.rpm (installed size 271.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±47  70%  7  90  8  100  42 299 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpio-lang (download size 56.00 kB)   x   xDownloading cpio-lang (download size 56.00 kB) q   24  53%  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpio-lang-2.9-75.80.1.x86_64.rpm (installed size 225.00 kB) 42  x   xInstalling cpio-lang-2.9-75.80.1.x86_64.rpm (installed size 225.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  26  33  ±±±±±41  50 60%  7  76  88  9  100  21  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wget (download size 414.00 kB)   x   xDownloading wget (download size 414.00 kB) q   7  13  21  9  37  ±±±44  52% 60%  8  75  83  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wget-1.11.4-1.40.1.x86_64.rpm (installed size 1.46 MB) 21  x   xInstalling wget-1.11.4-1.40.1.x86_64.rpm (installed size 1.46 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  8  25  30  6  ±±±±42  48 55%  60%  5  71  6  82  8  95  100  5.74  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xfsprogs (download size 730.00 kB)   x   xDownloading xfsprogs (download size 730.00 kB) q   6  11  7  22  8  33  9  ±±±44  50 5%  61%  6  72  7  83  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xfsprogs-3.1.8-0.10.2.x86_64.rpm (installed size 3.30 MB) 5.746  x   xInstalling xfsprogs-3.1.8-0.10.2.x86_64.rpm (installed size 3.30 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  9  35  40  ±±45  50 6%  61%  6  71  6  82  8  93  8  100  2.4  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pciutils-ids (download size 206.00 kB)   x   xDownloading pciutils-ids (download size 206.00 kB) q   7  15  23  31  8  ±46  54%  62%  70  7  85  93  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pciutils-ids-2015.10.07-3.1.noarch.rpm (installed size 974.00 k 2.45  x   xInstalling pciutils-ids-2015.10.07-3.1.noarch.rpm (installed size 974.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  33  9  ±46  52% 9%  66  72  9  85  92  8  100  1.49  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-locale (download size 11.36 MB)   x   x~ kB) x    xDownloading glibc-locale (download size 11.36 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-locale-2.11.3-17.102.1.x86_64.rpm (installed size 106.13 1.49  x   xInstalling glibc-locale-2.11.3-17.102.1.x86_64.rpm (installed size 106.1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  485.36  4 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vim (download size 835.00 kB)   x   x~3 MB) x    xDownloading vim (download size 835.00 kB) q   5  10  5  21  7  32  8  ±±±44  50 5%  61%  7  73  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vim-7.2-8.17.1.x86_64.rpm (installed size 1.77 MB) 485.3606  x   xInstalling vim-7.2-8.17.1.x86_64.rpm (installed size 1.77 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100  3.59  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading timezone (download size 358.00 kB)   x   xDownloading timezone (download size 358.00 kB) q   8  13  22  31  40  49 58%  67%  76  85  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing timezone-2017b-0.51.1.x86_64.rpm (installed size 1.21 MB) 3.595  x   xInstalling timezone-2017b-0.51.1.x86_64.rpm (installed size 1.21 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 482.38 MBx 292x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e6784b1axxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading usbutils (download size 229.00 kB)   x   xDownloading usbutils (download size 229.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing usbutils-004-1.18.1.x86_64.rpm (installed size 696.00 kB) 2.382:3  x   xInstalling usbutils-004-1.18.1.x86_64.rpm (installed size 696.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  15  21  32  ±±±±±41  50 9%  68  7  87  96  100  1.70  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libX11 (download size 726.00 kB)   x   xDownloading xorg-x11-libX11 (download size 726.00 kB) q   5  10  5  21  6  32  7  ±±±±43  48 54%  9%  65  70  7  83  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libX11-7.4-5.11.68.1.x86_64.rpm (installed size 3.00 M 1.704  x   xInstalling xorg-x11-libX11-7.4-5.11.68.1.x86_64.rpm (installed size 3.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  9  35  ±±±±±41  ±46  51 7%  63%  8  73  9  85  91  6  100  78  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glibc-devel (download size 4.87 MB)   x   x~ MB) x    xDownloading glibc-devel (download size 4.87 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glibc-devel-2.11.3-17.102.1.x86_64.rpm (installed size 31.28 MB 78  x   xInstalling glibc-devel-2.11.3-17.102.1.x86_64.rpm (installed size 31.28 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  47.41  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-Bootloader (download size 161.00 kB)   x   x~MB) x    xDownloading perl-Bootloader (download size 161.00 kB) q   9  1  2  3  4 59%  6  7  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-Bootloader-0.4.89.72-3.1.x86_64.rpm (installed size 369.00 47.412  x   xInstalling perl-Bootloader-0.4.89.72-3.1.x86_64.rpm (installed size 369.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  25  ±±±±42  51 8%  66%  81  6  91  6  100  05  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading crash (download size 2.50 MB)   x   x~00 kB) x    xDownloading crash (download size 2.50 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing crash-7.0.9-26.1.x86_64.rpm (installed size 6.95 MB) 05  x   xInstalling crash-7.0.9-26.1.x86_64.rpm (installed size 6.95 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  0.10  7 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading supportutils (download size 126.00 kB)   x   xDownloading supportutils (download size 126.00 kB) q   12  25  38  50 63%  76  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing supportutils-1.20-121.1.noarch.rpm (installed size 292.00 kB) 0.102  x   xInstalling supportutils-1.20-121.1.noarch.rpm (installed size 292.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  18  40  62%  84  92  7  100  39.82  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading krb5 (download size 521.00 kB)   x   xDownloading krb5 (download size 521.00 kB) q   6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  85  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing krb5-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.49 MB) 39.82  x   xInstalling krb5-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.49 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  20  8  34  9  ±±±44  49 56%  65%  73  81  90  7  100  8.33  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpcbind (download size 48.00 kB)   x   xDownloading rpcbind (download size 48.00 kB) q   32  66%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpcbind-0.1.6+git20080930-6.24.1.x86_64.rpm (installed size 101 8.331  x   xInstalling rpcbind-0.1.6+git20080930-6.24.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  62%  9  9  100  Updating etc/sysconfig/rpcbind... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl (download size 314.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading cyrus-sasl (download size 314.00 kB)  5  10  5  20  5  30  5  40  ±±45  50 5%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-2.1.22-182.23.1.x86_64.rpm (installed size 1.21 MB) 2  x   xInstalling cyrus-sasl-2.1.22-182.23.1.x86_64.rpm (installed size 1.21 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  9  35  40  ±±45  52% 7%  65%  71  7  83  92  8  100  7.02  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2-lang (download size 499.00 kB)   x   x~) x    xDownloading glib2-lang (download size 499.00 kB) q   6  12  9  25  32  8  ±±±44  51 7%  64%  70  6  83  9  96  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-lang-2.22.5-0.8.26.1.x86_64.rpm (installed size 3.20 MB) 7.02  x   xInstalling glib2-lang-2.22.5-0.8.26.1.x86_64.rpm (installed size 3.20 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  6  81  6  92  7  100  3.8  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading open-iscsi (download size 354.00 kB)   x   x~) x    xDownloading open-iscsi (download size 354.00 kB) q   8  13  8  27  36  ±±45  54%  63%  72  81  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing open-iscsi-2.0.873-0.41.3.x86_64.rpm (installed size 1.67 MB) 3.80  x   xInstalling open-iscsi-2.0.873-0.41.3.x86_64.rpm (installed size 1.67 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  9  37  ±±±44  51 9%  66  74  81  6  94  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  2.14  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsync (download size 362.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading rsync (download size 362.00 kB)  8  13  22  30  5  ±±±44  52% 7%  66%  75  83  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsync-3.0.4-2.52.1.x86_64.rpm (installed size 713.00 kB) 2.14  x   xInstalling rsync-3.0.4-2.52.1.x86_64.rpm (installed size 713.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  18  27  36  ±±45  54%  63%  7  82  9  98  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  1.45  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nscd (download size 99.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading nscd (download size 99.00 kB)  15  32  48 64%  80  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nscd-2.11.3-17.102.1.x86_64.rpm (installed size 138.00 kB) 1.4519  x   xInstalling nscd-2.11.3-17.102.1.x86_64.rpm (installed size 138.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 96%  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  31  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading microcode_ctl (download size 1.33 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading microcode_ctl (download size 1.33 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  91  6  100  - 1.30 MB/s (download size 1.33 MB) ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing microcode_ctl-1.17-102.82.1.x86_64.rpm (installed size 2.73 MB) 31  x   xInstalling microcode_ctl-1.17-102.82.1.x86_64.rpm (installed size 2.73 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  8  25  31  6  ±±±±43  49 54%  9%  65  70  5  80  5  90  5  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  28.58  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading irqbalance (download size 29.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading irqbalance (download size 29.00 kB)  54%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing irqbalance-1.0.4-0.23.2.x86_64.rpm (installed size 46.00 kB) 28.588  x   xInstalling irqbalance-1.0.4-0.23.2.x86_64.rpm (installed size 46.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  92%  9  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/irqbalance...  4  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpio (download size 80.00 kB)  ~ script `munge' Updating etc/sysconfig/irqbalance...   Downloading cpio (download size 80.00 kB)  19  3  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpio-2.9-75.80.1.x86_64.rpm (installed size 177.00 kB) 4  x   xInstalling cpio-2.9-75.80.1.x86_64.rpm (installed size 177.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  71%  94  9  100  36  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pciutils (download size 77.00 kB)   x   xDownloading pciutils (download size 77.00 kB) q   2  ±±±±±41  62%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pciutils-3.1.7-11.13.1.x86_64.rpm (installed size 155.00 kB) 36  x   xInstalling pciutils-3.1.7-11.13.1.x86_64.rpm (installed size 155.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  71%  89  94  9  100  21  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXrender (download size 19.00 kB)   x   xDownloading xorg-x11-libXrender (download size 19.00 kB) q   84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXrender-7.4-1.23.1.x86_64.rpm (installed size 39.00 21  x   xInstalling xorg-x11-libXrender-7.4-1.23.1.x86_64.rpm (installed size 39.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  17  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXfixes (download size 11.00 kB)   x   x~00 kB) x    xDownloading xorg-x11-libXfixes (download size 11.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXfixes-7.4-1.20.1.x86_64.rpm (installed size 23.00 17  x   xInstalling xorg-x11-libXfixes-7.4-1.20.1.x86_64.rpm (installed size 23.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  5  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kexec-tools (download size 140.00 kB)   x   x~0 kB) x    xDownloading kexec-tools (download size 140.00 kB) q   11  22  34  ±±45  57%  68%  80  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kexec-tools-2.0.3-0.20.2.x86_64.rpm (installed size 388.00 kB) 5  x   xInstalling kexec-tools-2.0.3-0.20.2.x86_64.rpm (installed size 388.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  36  52% 69%  75  80  5  93  8  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  7.77  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading crash-eppic (download size 167.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading crash-eppic (download size 167.00 kB)  9  1  28  3  ±47  57%  6  76  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing crash-eppic-7.0.9-26.1.x86_64.rpm (installed size 358.00 kB) 7.77  x   xInstalling crash-eppic-7.0.9-26.1.x86_64.rpm (installed size 358.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  71%  89  97  100  42  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mailx (download size 323.00 kB)   x   xDownloading mailx (download size 323.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mailx-12.5-1.11.2.x86_64.rpm (installed size 527.00 kB) 42  x   xInstalling mailx-12.5-1.11.2.x86_64.rpm (installed size 527.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  24  36  48 60%  6  72  85  91  9  100  6.91  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bind-libs (download size 1.12 MB)   x   xDownloading bind-libs (download size 1.12 MB) q   5  10  6  21  7  32  7  ±±±±42  48 53%  9%  64  9  74  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bind-libs-9.9.6P1-0.39.1.x86_64.rpm (installed size 3.66 MB) 6.91  x   xInstalling bind-libs-9.9.6P1-0.39.1.x86_64.rpm (installed size 3.66 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  6  31  6  ±±±±±41  ±46  51 6%  62%  7  73  8  83  8  93  9  100  3.25  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-digestmd5 (download size 27.00 kB)   x   xDownloading cyrus-sasl-digestmd5 (download size 27.00 kB) q   59%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-digestmd5-2.1.22-182.23.1.x86_64.rpm (installed size 3.257  x   xInstalling cyrus-sasl-digestmd5-2.1.22-182.23.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-gssapi (download size 18.00 kB)   x   x~ze 52.00 kB) x    xDownloading cyrus-sasl-gssapi (download size 18.00 kB) q   89%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-gssapi-2.1.22-182.23.1.x86_64.rpm (installed size 31 0  x   xInstalling cyrus-sasl-gssapi-2.1.22-182.23.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  17  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libldap-2_4-2 (download size 221.00 kB)   x   x~31.00 kB) x    xDownloading libldap-2_4-2 (download size 221.00 kB) q   6  13  20  8  35  ±±±±42  49 56%  64%  71  8  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libldap-2_4-2-2.4.26-0.65.2.x86_64.rpm (installed size 667.00 k 17  x   xInstalling libldap-2_4-2-2.4.26-0.65.2.x86_64.rpm (installed size 667.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  1  2  37  ±4  52% 62%  71  8  9  9  100  2.52  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-plain (download size 13.00 kB)   x   x~ kB) x    xDownloading cyrus-sasl-plain (download size 13.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-plain-2.1.22-182.23.1.x86_64.rpm (installed size 19. 2.52  x   xInstalling cyrus-sasl-plain-2.1.22-182.23.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading glib2 (download size 160.00 kB)   x   x~9.00 kB) x    xDownloading glib2 (download size 160.00 kB) q   9  1  2  3  4 60%  7  8  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing glib2-2.22.5-0.8.26.1.x86_64.rpm (installed size 497.00 kB) 0  x   xInstalling glib2-2.22.5-0.8.26.1.x86_64.rpm (installed size 497.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  24  36  49 62%  74  87  98  100  01  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bind-utils (download size 264.00 kB)   x   xDownloading bind-utils (download size 264.00 kB) q   5  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bind-utils-9.9.6P1-0.39.1.x86_64.rpm (installed size 1.05 MB) 016  x   xInstalling bind-utils-9.9.6P1-0.39.1.x86_64.rpm (installed size 1.05 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  24  31  7  ±±±±42  51 7%  63%  70  5  84  90  5  100  0.96  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sudo (download size 439.00 kB)   x   xDownloading sudo (download size 439.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sudo-1.7.6p2-0.29.1.x86_64.rpm (installed size 1.23 MB) 0.96  x   xInstalling sudo-1.7.6p2-0.29.1.x86_64.rpm (installed size 1.23 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  22  9  37  ±±±±42  ±47  53%  8%  63%  8  73  8  83  8  93  8  100  19.73  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcurl4 (download size 153.00 kB)   x   xDownloading libcurl4 (download size 153.00 kB) q   1  2  31  ±±±±±4  52% 62%  73  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcurl4-7.19.7-1.64.1.x86_64.rpm (installed size 457.00 kB) 19.73  x   xInstalling libcurl4-7.19.7-1.64.1.x86_64.rpm (installed size 457.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  28  ±±±±42  56%  70%  84  98  100  28  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cyrus-sasl-saslauthd (download size 49.00 kB)   x   xDownloading cyrus-sasl-saslauthd (download size 49.00 kB) q   32  65%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed size 28  x   xInstalling cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  68%  84  96  100  Downloading cyrus-sasl-saslauthd (download size 49.00 kB) Installing cyrus-sasl-saslauthd-2.1.22-182.23.3.x86_64.rpm (installed si ~ze 121.00 kB) Additional rpm output: Updating etc/sysconfig/saslauthd...  17  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openldap2-client (download size 164.00 kB)  Additional rpm output: Updating etc/sysconfig/saslauthd...   Downloading openldap2-client (download size 164.00 kB)  9  1  2  38  4 58%  6  7  87  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openldap2-client-2.4.26-0.65.2.x86_64.rpm (installed size 560.0 17  x   xInstalling openldap2-client-2.4.26-0.65.2.x86_64.rpm (installed size 560q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±43  55%  65%  77  83  9  8  100  8.62  59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libglib-2_0-0 (download size 299.00 kB)   x   x~.00 kB) x    xDownloading libglib-2_0-0 (download size 299.00 kB) q   5  10  5  21  6  32  7  ±±±±42  48 53%  8%  64%  9  74  80  5  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libglib-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 789.00 8.625  x   xInstalling libglib-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 789.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  64%  73  81  9  97  100  7.85  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading curl (download size 168.00 kB)   x   x~00 kB) x    xDownloading curl (download size 168.00 kB) q   9  18  2  3  ±47  57%  66%  7  85  9  100  70 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing curl-7.19.7-1.64.1.x86_64.rpm (installed size 309.00 kB) 7.85  x   xInstalling curl-7.19.7-1.64.1.x86_64.rpm (installed size 309.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  ±±±±±41  ±46  63%  76  85  92  9  100  5  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgthread-2_0-0 (download size 41.00 kB)   x   xDownloading libgthread-2_0-0 (download size 41.00 kB) q   39  78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgthread-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 19. 5  x   xInstalling libgthread-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgmodule-2_0-0 (download size 40.00 kB)   x   x~9.00 kB) x    xDownloading libgmodule-2_0-0 (download size 40.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgmodule-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 15. 3  x   xInstalling libgmodule-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgobject-2_0-0 (download size 128.00 kB)   x   x~5.00 kB) x    xDownloading libgobject-2_0-0 (download size 128.00 kB) q   12  24  37  49 62%  74  87  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgobject-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 264 1  x   xInstalling libgobject-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  48 72%  97  100  26  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgio-2_0-0 (download size 244.00 kB)   x   x~64.00 kB) x    xDownloading libgio-2_0-0 (download size 244.00 kB) q   6  13  9  26  32  9  ±±45  52% 9%  65  72  8  85  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgio-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 671.00 26  x   xInstalling libgio-2_0-0-2.22.5-0.8.26.1.x86_64.rpm (installed size 671.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  3  ±47  57%  66%  7  85  9  100  6.60  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 416.60 MBx 253x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 166.67 MBx 188x http-10.9.0.10-e6784b1axxx tu Downloading yast2-ycp-ui-bindings (download size 218.00 kB)   x   x~0 kB) x    xDownloading yast2-ycp-ui-bindings (download size 218.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ycp-ui-bindings-2.21.1-2.5.107.x86_64.rpm (installed size 6.600  x   xInstalling yast2-ycp-ui-bindings-2.21.1-2.5.107.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  37  ±4  56%  66%  75  8  94  9  100  01 7 5.94  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-slp (download size 59.00 kB)   x   x~ze 676.00 kB) x    xDownloading yast2-slp (download size 59.00 kB) q   26  54%  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-slp-2.17.0-0.7.55.x86_64.rpm (installed size 284.00 kB) 5.9409  x   xInstalling yast2-slp-2.17.0-0.7.55.x86_64.rpm (installed size 284.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  9  38  ±±±±43  53%  8%  6  73  9  84  90  5  100  5.73 6 66  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ksymoops (download size 739.00 kB)   x   xDownloading ksymoops (download size 739.00 kB) q   5  10  5  21  6  32  9  ±±45  52% 8%  65%  71  8  84  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ksymoops-2.4.11-117.22.x86_64.rpm (installed size 2.19 MB) 66  x   xInstalling ksymoops-2.4.11-117.22.x86_64.rpm (installed size 2.19 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  34  9  ±±45  51 7%  62%  8  74  9  85  91  7  100  3.55 5 3.47  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading smartmontools (download size 451.00 kB)   x   xDownloading smartmontools (download size 451.00 kB) q   7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing smartmontools-6.3-0.8.18.x86_64.rpm (installed size 1.66 MB) 3.47  x   xInstalling smartmontools-6.3-0.8.18.x86_64.rpm (installed size 1.66 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  20  8  35  ±±±±42  49 57%  64%  70  8  85  91  6  100  Updating etc/sysconfig/smartmontools... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  1.89 4 1.82  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading setserial (download size 23.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading setserial (download size 23.00 kB)  70%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing setserial-2.17-716.22.x86_64.rpm (installed size 44.00 kB) 1.828  x   xInstalling setserial-2.17-716.22.x86_64.rpm (installed size 44.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  74%  83  99  100  4 3 77  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading scpm (download size 757.00 kB)   x   xDownloading scpm (download size 757.00 kB) q   6  12  8  23  9  35  40  ±46  52% 9%  65  71  6  82  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing scpm-1.1.7-1.19.x86_64.rpm (installed size 2.83 MB) 77  x   xInstalling scpm-1.1.7-1.19.x86_64.rpm (installed size 2.83 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  8  24  9  35  ±±±±42  48 53%  9%  64  70  5  81  6  91  7  100  Updating etc/sysconfig/scpm... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  59.02 2 08.95  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading psmisc (download size 75.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading psmisc (download size 75.00 kB)  21  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing psmisc-22.7-7.3.1.x86_64.rpm (installed size 257.00 kB) 08.957  x   xInstalling psmisc-22.7-7.3.1.x86_64.rpm (installed size 257.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  22  35  40  ±±45  53%  61%  8  73  80  5  93  8  100  8.77 1 69  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procps (download size 203.00 kB)   x   xDownloading procps (download size 203.00 kB) q   7  15  23  31  9  ±47  55%  63%  71  9  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procps-3.2.7-151.28.1.x86_64.rpm (installed size 601.00 kB) 69  x   xInstalling procps-3.2.7-151.28.1.x86_64.rpm (installed size 601.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  2  7  32  ±±±±±41  51 8%  64%  72  8  83  8  93  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  18 0 11  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading procmail (download size 134.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading procmail (download size 134.00 kB)  11  23  35  ±47  59%  71  83  95  100  1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing procmail-3.22-240.8.1.x86_64.rpm (installed size 271.00 kB) 11  x   xInstalling procmail-3.22-240.8.1.x86_64.rpm (installed size 271.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  20  ±46  54%  63%  8  73  80  6  92  8  100  7.91 79 7.84  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openslp-server (download size 260.00 kB)   x   xDownloading openslp-server (download size 260.00 kB) q   6  12 - 31.74 kB/s (download size 260.00 kB)  8  24  30  6  ±±±±43  49 55%  61%  7  73  80  6  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openslp-server-1.2.0-172.24.1.x86_64.rpm (installed size 897.00 7.84  x   xInstalling openslp-server-1.2.0-172.24.1.x86_64.rpm (installed size 897.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  24  33  40  ±±45  53%  62%  9  76  85  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  04 8 6.97  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openct (download size 206.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading openct (download size 206.00 kB)  7  15  23  31  8  ±46  54%  62%  9  77  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openct-0.6.15-5.20.x86_64.rpm (installed size 874.00 kB) 6.97  x   xInstalling openct-0.6.15-5.20.x86_64.rpm (installed size 874.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  16  23  30  7  ±±±44  51 8%  64%  71  8  83  8  93  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  6.18 7 11  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kbd (download size 1.09 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading kbd (download size 1.09 MB)  6  11  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kbd-1.14.1-16.33.1.x86_64.rpm (installed size 2.41 MB) 11  x   xInstalling kbd-1.14.1-16.33.1.x86_64.rpm (installed size 2.41 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/keyboard...  3.77 6 3.70  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading iproute2 (download size 952.00 kB)  ~ script `munge' Updating etc/sysconfig/keyboard...   Downloading iproute2 (download size 952.00 kB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing iproute2-3.0-4.30.x86_64.rpm (installed size 1.69 MB) 3.706  x   xInstalling iproute2-3.0-4.30.x86_64.rpm (installed size 1.69 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  25  31  8  ±46  51 6%  63%  70  8  84  9  94  9  100  arpd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  2.09 5 2.02  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hdparm (download size 91.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading hdparm (download size 91.00 kB)  17  34  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hdparm-9.27-1.6.39.x86_64.rpm (installed size 161.00 kB) 2.025  x   xInstalling hdparm-9.27-1.6.39.x86_64.rpm (installed size 161.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  60%  74  87  99  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/ide... q  1.93 4 1.86  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpm (download size 197.00 kB)  Additional rpm output: Updating etc/sysconfig/ide...   Downloading gpm (download size 197.00 kB)  8  16  24  32  40  48 56%  65%  73  81  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpm-1.20.1-452.83.x86_64.rpm (installed size 546.00 kB) 1.86  x   xInstalling gpm-1.20.1-452.83.x86_64.rpm (installed size 546.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  5  27  33  ±±±44  53%  8%  69%  76  81  6  91  7  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/mouse...  40 3 33  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading fbset (download size 41.00 kB)  ~ script `munge' Updating etc/sysconfig/mouse...   Downloading fbset (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing fbset-2.1-919.22.x86_64.rpm (installed size 127.00 kB) 33  x   xInstalling fbset-2.1-919.22.x86_64.rpm (installed size 127.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  ±±45  65%  83  91  6  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/console...  27 2 20  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading acpid (download size 38.00 kB)  ~ script `munge' Updating etc/sysconfig/console...   Downloading acpid (download size 38.00 kB)  ±±±±±41  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing acpid-1.0.6-91.25.20.x86_64.rpm (installed size 77.00 kB) 20  x   xInstalling acpid-1.0.6-91.25.20.x86_64.rpm (installed size 77.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  34  53%  67%  72  7  82  94  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  0 1 13  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pm-profiler (download size 19.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading pm-profiler (download size 19.00 kB)  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pm-profiler-0.1-28.7.1.noarch.rpm (installed size 37.00 kB) 13  x   xInstalling pm-profiler-0.1-28.7.1.noarch.rpm (installed size 37.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  25  37  ±±45  52% 60%  7  74  82  7  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/pm-profiler... q  16 0 09  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ncurses (download size 312.00 kB)  Additional rpm output: Updating etc/sysconfig/pm-profiler...   Downloading yast2-ncurses (download size 312.00 kB)  5  10  5  20  5  30  5  40  ±46  51 6%  61%  6  71  6  82  7  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ncurses-2.17.23-1.14.x86_64.rpm (installed size 1.03 MB) 09  x   xInstalling yast2-ncurses-2.17.23-1.14.x86_64.rpm (installed size 1.03 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  0.13 69 0.06  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libxkbfile (download size 63.00 kB)   x   x~) x    xDownloading xorg-x11-libxkbfile (download size 63.00 kB) q   25  50 75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libxkbfile-7.4-1.14.x86_64.rpm (installed size 149.00 0.06  x   xInstalling xorg-x11-libxkbfile-7.4-1.14.x86_64.rpm (installed size 149.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  86%  99  100  49.98 8 399.91  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXt (download size 155.00 kB)   x   x~0 kB) x    xDownloading xorg-x11-libXt (download size 155.00 kB) q   1  2  3  ±±±±±41  51 61%  72  8  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXt-7.4-1.19.8.x86_64.rpm (installed size 404.00 kB) 399.91  x   xInstalling xorg-x11-libXt-7.4-1.19.8.x86_64.rpm (installed size 404.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  31  ±47  63%  79  95  100  59 7 52  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXext (download size 27.00 kB)   x   x~B) x    xDownloading xorg-x11-libXext (download size 27.00 kB) q   60%  10  2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXext-7.4-1.18.16.x86_64.rpm (installed size 72.00 k 52  x   xInstalling xorg-x11-libXext-7.4-1.18.16.x86_64.rpm (installed size 72.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  88%  99  100  2 6 45  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++43-devel (download size 2.25 MB)   x   x~ kB) x    xDownloading libstdc++43-devel (download size 2.25 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++43-devel-4.3.4_20091019-0.37.30.x86_64.rpm (installed 45  x   xInstalling libstdc++43-devel-4.3.4_20091019-0.37.30.x86_64.rpm (installeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  31.45 5 81.38  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc43 (download size 2.08 MB)   x   x~d size 18.06 MB) x    xDownloading gcc43 (download size 2.08 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.21 MB 81.380  x   xInstalling gcc43-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 7.21 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  24.24 4 74.17  29 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cups-libs (download size 408.00 kB)   x   x~MB) x    xDownloading cups-libs (download size 408.00 kB) q   7  15  23  31  9  ±47  54%  62%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cups-libs-1.3.9-8.46.56.1.x86_64.rpm (installed size 2.02 MB) 74.171:57  x   xInstalling cups-libs-1.3.9-8.46.56.1.x86_64.rpm (installed size 2.02 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  23  8  35  40  ±47  53%  9%  66  71  9  84  90  7  100  2.22 3 2.15  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gfxboot (download size 29.00 kB)   x   xDownloading gfxboot (download size 29.00 kB) q   55%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gfxboot-4.1.34-0.5.44.x86_64.rpm (installed size 75.00 kB) 2.15  x   xInstalling gfxboot-4.1.34-0.5.44.x86_64.rpm (installed size 75.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  85%  97  100  15 2 08  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading x86info (download size 62.00 kB)   x   xDownloading x86info (download size 62.00 kB) q   25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing x86info-1.29-0.5.34.x86_64.rpm (installed size 227.00 kB) 08  x   xInstalling x86info-1.29-0.5.34.x86_64.rpm (installed size 227.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  ±±±±43  71%  97  100  1.93 1 1.86  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading efibootmgr (download size 38.00 kB)   x   xDownloading efibootmgr (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing efibootmgr-0.6.0-2.23.x86_64.rpm (installed size 74.00 kB) 1.86  x   xInstalling efibootmgr-0.6.0-2.23.x86_64.rpm (installed size 74.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  64%  88  95  100  85 0 78  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cpupower (download size 82.00 kB)   x   xDownloading cpupower (download size 82.00 kB) q   19  38  58%  7  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cpupower-3.19-2.23.3.x86_64.rpm (installed size 206.00 kB) 78  x   xInstalling cpupower-3.19-2.23.3.x86_64.rpm (installed size 206.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  55%  68%  73  80  8  93  8  100  6 59 5  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfsidmap (download size 32.00 kB)   x   xDownloading nfsidmap (download size 32.00 kB) q   50 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfsidmap-0.25-0.13.1.x86_64.rpm (installed size 82.00 kB) 5  x   xInstalling nfsidmap-0.25-0.13.1.x86_64.rpm (installed size 82.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  37  49 57%  89%  96  100  57 8 0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libldapcpp1 (download size 87.00 kB)   x   xDownloading libldapcpp1 (download size 87.00 kB) q   18  36  55%  73%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libldapcpp1-0.3.0-0.9.29.x86_64.rpm (installed size 274.00 kB) 0  x   xInstalling libldapcpp1-0.3.0-0.9.29.x86_64.rpm (installed size 274.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±46  70%  93  9  100  31 7 23  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-WWW-Curl (download size 53.00 kB)   x   x~) x    xDownloading perl-WWW-Curl (download size 53.00 kB) q   29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-WWW-Curl-4.09-2.1.72.x86_64.rpm (installed size 142.00 kB) 23  x   xInstalling perl-WWW-Curl-4.09-2.1.72.x86_64.rpm (installed size 142.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  59%  80  9  5  100  17 6 10  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcroco-0_6-3 (download size 79.00 kB)   x   x~B) x    xDownloading libcroco-0_6-3 (download size 79.00 kB) q   2  4  61%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcroco-0_6-3-0.6.1-120.16.x86_64.rpm (installed size 241.00 k 10  x   xInstalling libcroco-0_6-3-0.6.1-120.16.x86_64.rpm (installed size 241.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  79%  9  100  0.93 5 0.86  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-transfer (download size 60.00 kB)   x   x~ kB) x    xDownloading yast2-transfer (download size 60.00 kB) q   26  53%  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-transfer-2.16.3-0.5.133.x86_64.rpm (installed size 322.00 0.866  x   xInstalling yast2-transfer-2.16.3-0.5.133.x86_64.rpm (installed size 322.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  29  3  ±±±44  49 58%  63%  9  76  81  7  92  8  100  62 4 55  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atk (download size 47.00 kB)   x   x~00 kB) x    xDownloading atk (download size 47.00 kB) q   33  67%  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atk-1.28.0-1.4.1.x86_64.rpm (installed size 152.00 kB) 55  x   xInstalling atk-1.28.0-1.4.1.x86_64.rpm (installed size 152.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  58%  85%  98  100  47 3 40  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgsf-1-114 (download size 89.00 kB)   x   xDownloading libgsf-1-114 (download size 89.00 kB) q   17  35  54%  72%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgsf-1-114-1.14.9-2.27.x86_64.rpm (installed size 243.00 kB) 40  x   xInstalling libgsf-1-114-1.14.9-2.27.x86_64.rpm (installed size 243.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 79%  9  100  23 2 16  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-python-bindings (download size 93.00 kB)   x   x~) x    xDownloading yast2-python-bindings (download size 93.00 kB) q   17  34  51 68%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-python-bindings-2.17.4-1.32.115.x86_64.rpm (installed siz 16  x   xInstalling yast2-python-bindings-2.17.4-1.32.115.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  3  63%  82  93  9  100  19.91 1 69.84  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-perl-bindings (download size 99.00 kB)   x   x~ize 326.00 kB) x    xDownloading yast2-perl-bindings (download size 99.00 kB) q   16  32  48 64%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-perl-bindings-2.17.2-1.32.121.x86_64.rpm (installed size 69.84  x   xInstalling yast2-perl-bindings-2.17.2-1.32.121.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  35  53%  71%  6  82  7  93  8  100  58 0 51  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ipoibtools (download size 19.00 kB)   x   x~e 344.00 kB) x    xDownloading ipoibtools (download size 19.00 kB) q   84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipoibtools-1.1-80.22.x86_64.rpm (installed size 55.00 kB) 51  x   xInstalling ipoibtools-1.1-80.22.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  59%  92  8  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/infiniband... q  2 49 45  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXp (download size 17.00 kB)  Additional rpm output: Updating etc/sysconfig/infiniband...   Downloading xorg-x11-libXp (download size 17.00 kB)  96%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXp-7.4-1.18.7.x86_64.rpm (installed size 39.00 kB) 45  x   xInstalling xorg-x11-libXp-7.4-1.18.7.x86_64.rpm (installed size 39.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  49 8 1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXmu (download size 48.00 kB)   x   x~) x    xDownloading xorg-x11-libXmu (download size 48.00 kB) q   33  67%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXmu-7.4-1.17.x86_64.rpm (installed size 119.00 kB) 1  x   xInstalling xorg-x11-libXmu-7.4-1.17.x86_64.rpm (installed size 119.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  53%  87%  99  100  37 7 30  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading DirectFB (download size 1.05 MB)   x   x~) x    xDownloading DirectFB (download size 1.05 MB) q   5  10  5  21  7  33  9  ±±45  50 5%  61%  7  73  9  85  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing DirectFB-1.2.3-6.32.x86_64.rpm (installed size 3.58 MB) 30  x   xInstalling DirectFB-1.2.3-6.32.x86_64.rpm (installed size 3.58 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±±41  ±46  51 6%  61%  7  72  7  83  8  93  8  100  5.78 6 5.71  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libstdc++-devel (download size 5.00 kB)   x   xDownloading libstdc++-devel (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libstdc++-devel-4.3-62.200.2.x86_64.rpm (installed size 1.00 kB 5.715  x   xInstalling libstdc++-devel-4.3-62.200.2.x86_64.rpm (installed size 1.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  75%  100  5  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc43-c++ (download size 3.22 MB)   x   x~kB) x    xDownloading gcc43-c++ (download size 3.22 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  - 1.63 MB/s (download size 3.22 MB) 60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc43-c++-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 8.6  x   xInstalling gcc43-c++-4.3.4_20091019-0.37.30.x86_64.rpm (installed size 8q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  07.1 4 57.1  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc (download size 6.00 kB)   x   x~.61 MB) x    xDownloading gcc (download size 6.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) 57.12  x   xInstalling gcc-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  26  40  55%  71%  87  100  3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cups-client (download size 118.00 kB)   x   xDownloading cups-client (download size 118.00 kB) q   13  26  40  54%  67%  81  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cups-client-1.3.9-8.46.56.1.x86_64.rpm (installed size 230.00 k  x   xInstalling cups-client-1.3.9-8.46.56.1.x86_64.rpm (installed size 230.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  20  6  33  9  ±±45  50 63%  8  79  84  93  8  100  6.95 2 6.88  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gfxboot-branding-SLES (download size 1.84 MB)   x   x~ kB) x    xDownloading gfxboot-branding-SLES (download size 1.84 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gfxboot-branding-SLES-4.1.34-0.5.44.x86_64.rpm (installed size 6.88  x   xInstalling gfxboot-branding-SLES-4.1.34-0.5.44.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  48 53%  9%  64  70  5  80  5  90  5  100  3.52 1 3.45  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libatk-1_0-0 (download size 41.00 kB)   x   x~e 3.43 MB) x    xDownloading libatk-1_0-0 (download size 41.00 kB) q   39  78%  100  5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libatk-1_0-0-1.28.0-1.4.1.x86_64.rpm (installed size 128.00 kB) 3.451  x   xInstalling libatk-1_0-0-1.28.0-1.4.1.x86_64.rpm (installed size 128.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 99%  100  40 0 33  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXprintUtil (download size 16.00 kB)   x   x~B) x    xDownloading xorg-x11-libXprintUtil (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXprintUtil-7.4-1.17.x86_64.rpm (installed size 35.0 33  x   xInstalling xorg-x11-libXprintUtil-7.4-1.17.x86_64.rpm (installed size 35q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  37 39 29  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gcc-c++ (download size 5.00 kB)   x   x~.00 kB) x    xDownloading gcc-c++ (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gcc-c++-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) 29  x   xInstalling gcc-c++-4.3-62.200.2.x86_64.rpm (installed size 0.00 B) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  3  60%  82  100  8  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading atk-lang (download size 136.00 kB)   x   xDownloading atk-lang (download size 136.00 kB) q   11  23  35  ±46  58%  70%  82  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing atk-lang-1.28.0-1.4.1.x86_64.rpm (installed size 766.00 kB)  x   xInstalling atk-lang-1.28.0-1.4.1.x86_64.rpm (installed size 766.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 8%  63%  8  73  8  83  8  94  9  100  2.62 7 2.55  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 352.55 MBx 202x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e6784b1axxx tu Downloading libibverbs-devel (download size 67.00 kB)   x   xDownloading libibverbs-devel (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libibverbs-devel-1.1.4-0.16.37.x86_64.rpm (installed size 122.0 2.5548  x   xInstalling libibverbs-devel-1.1.4-0.16.37.x86_64.rpm (installed size 122q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  23  30  63%  8  73  9  84  9  94  9  100  43  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openmpi-devel (download size 4.78 MB)   x   x~.00 kB) x    xDownloading openmpi-devel (download size 4.78 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openmpi-devel-1.8.4-1.26.x86_64.rpm (installed size 11.69 MB) 431  x   xInstalling openmpi-devel-1.8.4-1.26.x86_64.rpm (installed size 11.69 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  40.74  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 340.74 MBx 200x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x http-10.9.0.10-e6784b1axxx tu Downloading zlib-devel (download size 184.00 kB)   x   xDownloading zlib-devel (download size 184.00 kB) q   8  17  25  34  ±±±±43  52% 60%  9  78  86  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zlib-devel-1.2.7-0.14.1.x86_64.rpm (installed size 656.00 kB) 40.7437  x   xInstalling zlib-devel-1.2.7-0.14.1.x86_64.rpm (installed size 656.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  3  ±±4  55%  64%  70  8  5  9  100  10 199 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopenssl-devel (download size 2.87 MB)   x   xDownloading libopenssl-devel (download size 2.87 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopenssl-devel-0.9.8j-0.105.1.x86_64.rpm (installed size 20.3 102  x   xInstalling libopenssl-devel-0.9.8j-0.105.1.x86_64.rpm (installed size 20q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  19.78  8 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x          qq  xSizexPackagesx w Total x 319.78 MBx 198x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x v http-10.9.0.10-e6784b1axxx tu Downloading aaa_base (download size 173.00 kB)   x   x~.32 MB) x    xDownloading aaa_base (download size 173.00 kB) q   9  18  27  3  ±46  55%  64%  7  83  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing aaa_base-11-6.115.1.x86_64.rpm (installed size 355.00 kB) 19.7824  x   xInstalling aaa_base-11-6.115.1.x86_64.rpm (installed size 355.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  23  8  35  ±±±±43  50 6%  61%  6  72  7  83  90  5  100  Updating etc/sysconfig/shutdown... Updating etc/sysconfig/readonlyroot... Updating etc/passwd...unchanged Updating etc/group...unchanged Updating etc/shadow...unchanged  43  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading w3m (download size 942.00 kB)  group...unchanged hadow...unchanged   Downloading w3m (download size 942.00 kB)  5  10  5  20  5  30  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing w3m-0.5.3.git20161120-4.1.x86_64.rpm (installed size 2.17 MB) 43  x   xInstalling w3m-0.5.3.git20161120-4.1.x86_64.rpm (installed size 2.17 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  9  34  40  ±46  52% 7%  63%  8  73  80  5  90  5  100  7.26  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXpm (download size 46.00 kB)   x   xDownloading xorg-x11-libXpm (download size 46.00 kB) q   34  69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXpm-7.4-3.1.x86_64.rpm (installed size 118.00 kB) 7.26  x   xInstalling xorg-x11-libXpm-7.4-3.1.x86_64.rpm (installed size 118.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  39  93%  9  100  15  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libXv (download size 12.00 kB)   x   xDownloading xorg-x11-libXv (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libXv-7.4-1.20.1.x86_64.rpm (installed size 23.00 kB) 153  x   xInstalling xorg-x11-libXv-7.4-1.20.1.x86_64.rpm (installed size 23.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xorg-x11-libs (download size 709.00 kB)   x   x~) x    xDownloading xorg-x11-libs (download size 709.00 kB) q   6  11  8  24  9  36  ±±±±42  ±47  54%  - 367.74 kB/s (download size 709.00 kB) 60%  5  72  8  83  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xorg-x11-libs-7.4-8.26.49.1.x86_64.rpm (installed size 2.76 MB) 2  x   xInstalling xorg-x11-libs-7.4-8.26.49.1.x86_64.rpm (installed size 2.76 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  34  40  ±±45  52% 7%  62%  9  74  9  84  91  6  100  4.36  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x       x     qq  xSizexPackagesx w Total x 314.36 MBx 193x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 102.62 MBx 137x v http-10.9.0.10-e6784b1axxx tu Downloading permissions (download size 25.00 kB)   x   x~B) x    xDownloading permissions (download size 25.00 kB) q   63%  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 kB 4.36  x   xInstalling permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  30  53%  76%  97  100  permissions (download size 25.00 kB) permissions-2013.1.7-0.3.1.x86_64.rpm (installed size 100.00 kB) Additional rpm output: Updating etc/sysconfig/security...  5 6 2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading tk (download size 1.52 MB)  Additional rpm output: Updating etc/sysconfig/security...   Downloading tk (download size 1.52 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  6  81  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing tk-8.5.5-3.14.1.x86_64.rpm (installed size 3.96 MB) 2  x   xInstalling tk-8.5.5-3.14.1.x86_64.rpm (installed size 3.96 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  8  33  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100   98.56 5 0.31  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cairo (download size 458.00 kB)   x   xDownloading cairo (download size 458.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cairo-1.8.8-2.1.48.x86_64.rpm (installed size 1.39 MB) 0.312  x   xInstalling cairo-1.8.8-2.1.48.x86_64.rpm (installed size 1.39 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  22  31  6  ±±±44  53%  62%  71  80  8  97  100  7.17 4 08.9  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading wodim (download size 388.00 kB)   x   xDownloading wodim (download size 388.00 kB) q   8  13  21  30  8  ±46  54%  63%  71  9  87  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing wodim-1.1.8-3.32.3.x86_64.rpm (installed size 1.20 MB) 08.91  x   xInstalling wodim-1.1.8-3.32.3.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  32  7  ±±±±42  ±47  53%  61%  6  74  80  5  90  5  100  5.9 3 7.72  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading vlock (download size 20.00 kB)   x   xDownloading vlock (download size 20.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing vlock-1.4-52.16.x86_64.rpm (installed size 41.00 kB) 7.72  x   xInstalling vlock-1.4-52.16.x86_64.rpm (installed size 41.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  79%  95  100  3 2 68  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading util-linux-lang (download size 905.00 kB)   x   xDownloading util-linux-lang (download size 905.00 kB) q   6  11  7  22  7  32  8  ±±±±43  48 54%  9%  64  70  5  80  6  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing util-linux-lang-2.19.1-6.72.1.x86_64.rpm (installed size 4.29 M 68  x   xInstalling util-linux-lang-2.19.1-6.72.1.x86_64.rpm (installed size 4.29q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  32  7  ±±±±43  48 53%  8%  64%  70  5  80  5  90  6  100  1.64 1 3.3  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpm (download size 1.60 MB)   x   x~ MB) x    xDownloading rpm (download size 1.60 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpm-4.4.2.3-37.63.64.1.x86_64.rpm (installed size 4.27 MB) 3.30  x   xInstalling rpm-4.4.2.3-37.63.64.1.x86_64.rpm (installed size 4.27 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  32  7  ±±±±42  ±47  53%  9%  64  9  74  9  84  9  94  9  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/services... q  87.36 0 299.11  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam-modules (download size 109.00 kB)  Additional rpm output: Updating etc/sysconfig/services...   Downloading pam-modules (download size 109.00 kB)  14  29  ±±±±43  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-modules-11-1.25.26.x86_64.rpm (installed size 457.00 kB) 299.1119  x   xInstalling pam-modules-11-1.25.26.x86_64.rpm (installed size 457.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  20  5  30  5  ±±±±±41  ±46  51 6%  62%  7  72  7  82  7  92  7  100  6.92 29 8.66  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opie (download size 148.00 kB)   x   xDownloading opie (download size 148.00 kB) q   1  21  32  ±±±±43  54%  65%  7  86  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opie-2.4-662.18.1.x86_64.rpm (installed size 661.00 kB) 8.66  x   xInstalling opie-2.4-662.18.1.x86_64.rpm (installed size 661.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  9  26  33  40  ±46  57%  6  76  8  92  7  100  27 8 02  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading eject (download size 38.00 kB)   x   xDownloading eject (download size 38.00 kB) q   ±±±±±41  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing eject-2.1.0-115.19.2.1.x86_64.rpm (installed size 103.00 kB) 028  x   xInstalling eject-2.1.0-115.19.2.1.x86_64.rpm (installed size 103.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  22  8  ±±45  53%  60%  71  9  87  95  100  1 7 7.9  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ibutils (download size 1.27 MB)   x   xDownloading ibutils (download size 1.27 MB) q   5  10  5  20  5  30  6  ±±±±±41  ±46  51 7%  62%  7  73  8  83  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ibutils-1.5.7-0.15.22.x86_64.rpm (installed size 10.49 MB) 7.9  x   xInstalling ibutils-1.5.7-0.15.22.x86_64.rpm (installed size 10.49 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  75.68 6 87.43  2 8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cdrkit-cdrtools-compat (download size 6.00 kB)   x   xDownloading cdrkit-cdrtools-compat (download size 6.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cdrkit-cdrtools-compat-1.1.8-3.32.3.x86_64.rpm (installed size 87.436  x   xInstalling cdrkit-cdrtools-compat-1.1.8-3.32.3.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±42  54%  66%  78  90  10  5  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading util-linux (download size 761.00 kB)   x   x~e 0.00 B) x    xDownloading util-linux (download size 761.00 kB) q   6  11  7  23  8  34  40  ±46  51 7%  63%  70  6  82  9  95  100  9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing util-linux-2.19.1-6.72.1.x86_64.rpm (installed size 2.16 MB)  x   xInstalling util-linux-2.19.1-6.72.1.x86_64.rpm (installed size 2.16 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  8  23  30  6  ±±±±±41  ±46  51 6%  61%  6  71  6  81  6  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  3.53 4 5.27  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading satsolver-tools (download size 351.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading satsolver-tools (download size 351.00 kB)  5  10  9  28  33  ±±±±42  51 60%  9  74  83  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing satsolver-tools-0.17.9-0.5.2.x86_64.rpm (installed size 3.25 MB 5.275  x   xInstalling satsolver-tools-0.17.9-0.5.2.x86_64.rpm (installed size 3.25 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  23  8  35  40  ±46  52% 8%  64%  70  7  82  7  92  7  100  0.27 3 2.02  79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rpm-python (download size 43.00 kB)   x   x~MB) x    xDownloading rpm-python (download size 43.00 kB) q   37  75%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rpm-python-4.4.2.3-37.63.64.2.x86_64.rpm (installed size 137.00 2.024  x   xInstalling rpm-python-4.4.2.3-37.63.64.2.x86_64.rpm (installed size 137.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  48 94%  9  100  14 2 1.89  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading python-satsolver (download size 251.00 kB)   x   x~00 kB) x    xDownloading python-satsolver (download size 251.00 kB) q   6  12  9  25  31  8  ±±±44  51 7%  63%  70  6  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing python-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 749. 1.89  x   xInstalling python-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 74q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  34  ±±±±42  51 9%  68  76  85  93  9  100  69.41 1 15  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading perl-satsolver (download size 303.00 kB)   x   x~9.00 kB) x    xDownloading perl-satsolver (download size 303.00 kB) q   5  10  5  21  6  31  6  ±±±±42  ±47  52% 8%  63%  8  74  9  84  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing perl-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 944.00 15  x   xInstalling perl-satsolver-0.44.5-0.5.194.x86_64.rpm (installed size 944.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  6  23  30  7  ±±±44  50 7%  64%  71  7  84  91  8  100  8.49 0 0.23  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading module-init-tools (download size 146.00 kB)   x   x~00 kB) x    xDownloading module-init-tools (download size 146.00 kB) q   1  21  32  ±±±±43  54%  65%  76  87  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing module-init-tools-3.11.1-31.25.x86_64.rpm (installed size 488.0 0.23  x   xInstalling module-init-tools-3.11.1-31.25.x86_64.rpm (installed size 488q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  29  ±46  53%  66%  77  83  9  96  100  01 19 79.76  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pwdutils (download size 352.00 kB)   x   x~.00 kB) x    xDownloading pwdutils (download size 352.00 kB) q   9  18  27  36  ±±45  54%  9%  68  77  86  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pwdutils-3.2.15-0.17.28.x86_64.rpm (installed size 1.95 MB) 79.76  x   xInstalling pwdutils-3.2.15-0.17.28.x86_64.rpm (installed size 1.95 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  23  9  34  ±±±±±41  48 53%  8%  63%  8  73  8  84  90  6  100  6.07 8 7.81  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam-config (download size 108.00 kB)   x   xDownloading pam-config (download size 108.00 kB) q   14  29  ±±±44  59%  74  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam-config-0.79-2.8.8.x86_64.rpm (installed size 499.00 kB) 7.813  x   xInstalling pam-config-0.79-2.8.8.x86_64.rpm (installed size 499.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  34  40  ±±45  51 6%  61%  6  71  7  83  8  95  100  **** write config for pam_ssh.so (session, disabled) **** write config for pam_selinux.so (session, disabled) **** write config for pam_gnome_keyring.so (session, disabled) **** write config for pam_exec.so (session, disabled) **** write config for pam_env.so (session, disabled)  5.58 7 32  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zisofs-tools (download size 25.00 kB)  exec.so (session, disabled)env.so (session, disabled)   Downloading zisofs-tools (download size 25.00 kB)   65%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zisofs-tools-1.0.6-138.22.x86_64.rpm (installed size 53.00 kB) 32  x   xInstalling zisofs-tools-1.0.6-138.22.x86_64.rpm (installed size 53.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  78%  92  9  100  3 6 27  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pam_mount (download size 102.00 kB)   x   x~) x    xDownloading pam_mount (download size 102.00 kB) q   15  31  ±46  62%  78  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pam_mount-0.47-13.16.1.x86_64.rpm (installed size 280.00 kB) 27  x   xInstalling pam_mount-0.47-13.16.1.x86_64.rpm (installed size 280.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  31  8  4 54%  66%  7  83  90  5  100  25 5 00  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading haveged (download size 45.00 kB)   x   xDownloading haveged (download size 45.00 kB) q   35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing haveged-1.3-0.6.1.x86_64.rpm (installed size 153.00 kB) 00  x   xInstalling haveged-1.3-0.6.1.x86_64.rpm (installed size 153.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  85%  98  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  10 4 6.85  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading grub (download size 349.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading grub (download size 349.00 kB)  9  18  27  32  ±±±±±41  50 5%  64%  73  82  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing grub-0.97-162.172.1.x86_64.rpm (installed size 1.38 MB) 6.85  x   xInstalling grub-0.97-162.172.1.x86_64.rpm (installed size 1.38 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  21  9  34  ±±±±±41  49 56%  62%  7  73  8  87  92  8  100  3.72 3 5.47  69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading scout (download size 48.00 kB)   x   xDownloading scout (download size 48.00 kB) q   33  66%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing scout-0.1.0-29.9.29.x86_64.rpm (installed size 133.00 kB) 5.472  x   xInstalling scout-0.1.0-29.9.29.x86_64.rpm (installed size 133.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  8%  63%  9  74  80  8  93  8  100  59 2 34  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-sam (download size 63.00 kB)   x   xDownloading suse-sam (download size 63.00 kB) q   25  50 76%  100  80  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-sam-0.8.5-0.4.9.noarch.rpm (installed size 174.00 kB) 34  x   xInstalling suse-sam-0.8.5-0.4.9.noarch.rpm (installed size 174.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  36  73%  86  9  100  42 1 17  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mpt-firmware (download size 5.00 kB)   x   xDownloading mpt-firmware (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mpt-firmware-1.0-236.22.noarch.rpm (installed size 1.00 kB) 17  x   xInstalling mpt-firmware-1.0-236.22.noarch.rpm (installed size 1.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  36  82%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ipw-firmware (download size 344.00 kB)   x   xDownloading ipw-firmware (download size 344.00 kB) q   9  18  23  32  7  ±46  51 60%  5  74  9  88  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipw-firmware-9-1.6.4.noarch.rpm (installed size 2.55 MB)  x   xInstalling ipw-firmware-9-1.6.4.noarch.rpm (installed size 2.55 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  24  31  8  ±±45  51 6%  61%  6  71  8  83  8  95  100  0.87 09 2.62  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootcycle (download size 8.00 kB)   x   xDownloading bootcycle (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootcycle-0.3-225.18.x86_64.rpm (installed size 5.00 kB) 2.62  x   xInstalling bootcycle-0.3-225.18.x86_64.rpm (installed size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  84%  97  100  Updating etc/sysconfig/bootloader... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  8 1  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading command-not-found (download size 12.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading command-not-found (download size 12.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing command-not-found-0.1.0-29.9.29.x86_64.rpm (installed size 13.0 1  x   xInstalling command-not-found-0.1.0-29.9.29.x86_64.rpm (installed size 13q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  32  ±±±±4  56%  69%  82  99  100  5 7 0  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x       x     qq  xSizexPackagesx w Total x 272.60 MBx 163x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 60.85 MBx 107x v http-10.9.0.10-e6784b1axxx tu Downloading nfs-client (download size 198.00 kB)   x   x~.00 kB) x    xDownloading nfs-client (download size 198.00 kB) q   7  16  24  32  40  48 56%  64%  72  80  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-client-1.2.3-18.47.1.x86_64.rpm (installed size 540.00 kB) 009  x   xInstalling nfs-client-1.2.3-18.47.1.x86_64.rpm (installed size 540.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  22  9  3  51 61%  72  8  91  6  100  Updating etc/sysconfig/nfs... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  SizexPackagesx 272.07 MBx 163x x x 60.85 MBx 107x x x qu  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pango (download size 397.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading pango (download size 397.00 kB)  8  16  24  32  40  48 56%  64%  72  80  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pango-1.26.2-1.7.1.x86_64.rpm (installed size 1.48 MB) 07  x   xInstalling pango-1.26.2-1.7.1.x86_64.rpm (installed size 1.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  8  35  ±±±±42  ±47  53%  9%  64  9  74  82  90  8  100  0.59  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading deltarpm (download size 432.00 kB)   x   xDownloading deltarpm (download size 432.00 kB) q   7  14  22  9  37  ±±±44  51 9%  66  74  81  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing deltarpm-3.6.1-4.11.1.x86_64.rpm (installed size 1.48 MB) 0.59  x   xInstalling deltarpm-3.6.1-4.11.1.x86_64.rpm (installed size 1.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  8  26  34  ±±±±42  49 56%  65%  71  9  86  94  9  100  69.11  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libsnmp15 (download size 1.12 MB)   x   xDownloading libsnmp15 (download size 1.12 MB) q   5  10  5  20  5  31  6  ±±±±42  ±47  52% 7%  63%  8  73  8  84  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libsnmp15-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 3.45 MB) 69.118  x   xInstalling libsnmp15-5.4.2.1-8.12.27.1.x86_64.rpm (installed size 3.45 Mq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  7  32  7  ±±±±43  48 53%  9%  64  70  5  81  6  91  7  100  5.66 59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading udev (download size 236.00 kB)   x   x~B) x    xDownloading udev (download size 236.00 kB) q   6  13  20  7  33  40  ±47  54%  61%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing udev-147-0.110.1.x86_64.rpm (installed size 943.00 kB) 5.66  x   xInstalling udev-147-0.110.1.x86_64.rpm (installed size 943.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  9  24  9  34  9  ±±±44  49 58%  63%  72  9  86  93  8  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.74  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgcrypt11 (download size 339.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading libgcrypt11 (download size 339.00 kB)  9  14  23  8  33  ±±±±42  ±47  56%  61%  6  75  80  5  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgcrypt11-1.5.0-0.22.1.x86_64.rpm (installed size 719.00 kB) 4.747  x   xInstalling libgcrypt11-1.5.0-0.22.1.x86_64.rpm (installed size 719.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  9  28  37  ±46  55%  64%  71  8  83  8  9  100  0  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading nfs-kernel-server (download size 117.00 kB)   x   x~) x    xDownloading nfs-kernel-server (download size 117.00 kB) q   13  27  ±±±±±41  54%  68%  82  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing nfs-kernel-server-1.2.3-18.47.1.x86_64.rpm (installed size 248. 0  x   xInstalling nfs-kernel-server-1.2.3-18.47.1.x86_64.rpm (installed size 24q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  28  53%  71%  8  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  3.79  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gtk2-lang (download size 3.16 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading gtk2-lang (download size 3.16 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gtk2-lang-2.18.9-0.44.1.x86_64.rpm (installed size 19.71 MB) 3.79  x   xInstalling gtk2-lang-2.18.9-0.44.1.x86_64.rpm (installed size 19.71 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  44.08  5 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ksh (download size 1.13 MB)   x   xDownloading ksh (download size 1.13 MB) q   5  10  5 - 168.01 kB/s (download size 1.13 MB)  20  5  31  6  ±±±±42  ±47  53%  8%  63%  8  74  9  85  90  6  100  2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ksh-93u-0.37.1.x86_64.rpm (installed size 2.57 MB) 44.082  x   xInstalling ksh-93u-0.37.1.x86_64.rpm (installed size 2.57 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  23  8  33  40  ±±45  50 6%  61%  6  71  8  83  9  94  9  100  1.51  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libudev0 (download size 50.00 kB)   x   xDownloading libudev0 (download size 50.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libudev0-147-0.110.1.x86_64.rpm (installed size 55.00 kB) 1.511  x   xInstalling libudev0-147-0.110.1.x86_64.rpm (installed size 55.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  46  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autofs (download size 368.00 kB)   x   xDownloading autofs (download size 368.00 kB) q   8  17  26  34  9  ±47  52% 60%  5  73  8  87  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autofs-5.0.6-3.10.33.1.x86_64.rpm (installed size 1.65 MB) 46  x   xInstalling autofs-5.0.6-3.10.33.1.x86_64.rpm (installed size 1.65 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  13  20  6  33  9  ±±45  51 6%  63%  8  76  81  8  95  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/autofs... q  39.81  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1 (download size 248.00 kB)  Additional rpm output: Updating etc/sysconfig/autofs...   Downloading dbus-1 (download size 248.00 kB)  6  12  9  25  32  8  ±±45  51 7%  64%  70  7  83  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-1.2.10-3.33.1.x86_64.rpm (installed size 827.00 kB) 39.81  x   xInstalling dbus-1-1.2.10-3.33.1.x86_64.rpm (installed size 827.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  8  27  36  ±±±44  50 8%  66%  75  84  91  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  00  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading at (download size 53.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading at (download size 53.00 kB)  29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing at-3.1.8-1069.22.30.1.x86_64.rpm (installed size 109.00 kB) 00  x   xInstalling at-3.1.8-1069.22.30.1.x86_64.rpm (installed size 109.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  70%  87  93  8  100  Updating etc/sysconfig/atd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  8.9  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading openssh (download size 1.09 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading openssh (download size 1.09 MB)  5  10  6  21  6  31  6  ±±±±±41  ±46  52% 7%  62%  7  72  7  82  7  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing openssh-6.6p1-35.1.x86_64.rpm (installed size 5.01 MB) 8.9  x   xInstalling openssh-6.6p1-35.1.x86_64.rpm (installed size 5.01 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  7  ±±±±43  48 53%  9%  64  9  74  9  84  9  94  9  100  Updating etc/sysconfig/ssh... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  3.89 49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gtk2 (download size 2.23 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading gtk2 (download size 2.23 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gtk2-2.18.9-0.44.1.x86_64.rpm (installed size 7.00 MB) 3.890:59  x   xInstalling gtk2-2.18.9-0.44.1.x86_64.rpm (installed size 7.00 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  26  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading device-mapper (download size 323.00 kB)   x   xDownloading device-mapper (download size 323.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing device-mapper-1.02.77-0.15.14.x86_64.rpm (installed size 1.65 M 268  x   xInstalling device-mapper-1.02.77-0.15.14.x86_64.rpm (installed size 1.65q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  20  8  36  ±±±44  51 9%  66  74  81  9  96  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  5.24  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading quota (download size 233.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading quota (download size 233.00 kB)  6  13  20  7  34  ±±±±±41  48 54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing quota-3.16-51.1.0.x86_64.rpm (installed size 1.15 MB) 5.247  x   xInstalling quota-3.16-51.1.0.x86_64.rpm (installed size 1.15 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  15  21  7  33  9  ±±45  52% 8%  65%  7  80  5  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.09  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading librsvg (download size 149.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading librsvg (download size 149.00 kB)  1  21  32  ±±±±4  53%  64%  75  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing librsvg-2.26.0-2.5.1.x86_64.rpm (installed size 420.00 kB) 4.09  x   xInstalling librsvg-2.26.0-2.5.1.x86_64.rpm (installed size 420.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  27  ±±±±43  58%  65%  84  98  100  3.68  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading lvm2 (download size 936.00 kB)   x   xDownloading lvm2 (download size 936.00 kB) q   5  11  6  21  7  32  7  ±±±±42  ±47 - 445.53 kB/s (download size 936.00 kB)  52% 7%  63%  8  73  8  83  8  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing lvm2-2.02.98-0.42.3.x86_64.rpm (installed size 2.97 MB) 3.68  x   xInstalling lvm2-2.02.98-0.42.3.x86_64.rpm (installed size 2.97 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  8  34  40  ±46  51 6%  62%  9  75  81  8  93  8  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  0.71  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kpartx (download size 53.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading kpartx (download size 53.00 kB)  29  60%  9  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kpartx-0.4.9-115.1.x86_64.rpm (installed size 46.00 kB) 0.716  x   xInstalling kpartx-0.4.9-115.1.x86_64.rpm (installed size 46.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  14  97%  100  66  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading parted (download size 589.00 kB)   x   xDownloading parted (download size 589.00 kB) q   5  10  6  21  7  32  8  ±±±±43  48 54%  9%  65  70  6  81  6  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing parted-2.3-10.51.1.x86_64.rpm (installed size 2.20 MB) 66  x   xInstalling parted-2.3-10.51.1.x86_64.rpm (installed size 2.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  24  30  6  ±±±±±41  ±46  53%  8%  65%  70  5  82  8  93  9  100  18.4  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libMagickCore1 (download size 1.26 MB)   x   xDownloading libMagickCore1 (download size 1.26 MB) q   5  11  6  21  6  31  7  ±±±±42  ±47  52% 7%  63%  8  73  8  83  9  94  9  100  3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libMagickCore1-6.4.3.6-7.70.1.x86_64.rpm (installed size 5.33 M 18.45  x   xInstalling libMagickCore1-6.4.3.6-7.70.1.x86_64.rpm (installed size 5.33q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  ±±±±±41  ±46  51 6%  62%  7  72  7  82  8  94  9  100  3.13  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading multipath-tools (download size 233.00 kB)   x   x~ MB) x    xDownloading multipath-tools (download size 233.00 kB) q   6  13  20  7  34  ±±±±±41  ±47  54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing multipath-tools-0.4.9-115.1.x86_64.rpm (installed size 690.00 k 3.134  x   xInstalling multipath-tools-0.4.9-115.1.x86_64.rpm (installed size 690.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  25  34  ±±±±43  48 53%  8%  63%  8  75  87  92  7  100  2.46  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x       x     qq  xSizexPackagesx w Total x 212.46 MBx 140x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 60.85 MBx 107x v http-10.9.0.10-e6784b1axxx tu Downloading OpenIPMI (download size 874.00 kB)   x   x~ kB) x    xDownloading OpenIPMI (download size 874.00 kB) q   5  11  7  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing OpenIPMI-2.0.16-0.15.1.x86_64.rpm (installed size 3.32 MB) 2.463  x   xInstalling OpenIPMI-2.0.16-0.15.1.x86_64.rpm (installed size 3.32 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  8  33  8  ±±±±43  49 54%  60%  6  72  7  83  8  93  8  100  Updating etc/sysconfig/ipmi... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  57.53 6 09.14  39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading uuid-runtime (download size 41.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading uuid-runtime (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing uuid-runtime-2.19.1-6.72.1.x86_64.rpm (installed size 22.00 kB) 09.142  x   xInstalling uuid-runtime-2.19.1-6.72.1.x86_64.rpm (installed size 22.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  93%  8  100  1 5 2  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading puppet (download size 1.16 MB)   x   x~B) x    xDownloading puppet (download size 1.16 MB) q   5  11  6  22  7  32  8  ±±±±43  49 54%  9%  65  70  6  81  6  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing puppet-2.7.26-0.5.1.x86_64.rpm (installed size 3.40 MB) 2  x   xInstalling puppet-2.7.26-0.5.1.x86_64.rpm (installed size 3.40 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/puppet... q  4.1 4 5.7  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading netcfg (download size 246.00 kB)  Additional rpm output: Updating etc/sysconfig/puppet...   Downloading netcfg (download size 246.00 kB)  6  12  9  25  32  9  ±±45  52% 8%  65%  71  8  84  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing netcfg-11.2-0.8.1.noarch.rpm (installed size 1.24 MB) 5.71  x   xInstalling netcfg-11.2-0.8.1.noarch.rpm (installed size 1.24 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  7  32  7  ±±±±43  48 53%  8%  63%  8  73  8  83  8  93  8  100  2.87 3 4.48  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libxslt (download size 149.00 kB)   x   xDownloading libxslt (download size 149.00 kB) q   1  21  32  ±±±±43  53%  64%  75  86  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libxslt-1.1.24-19.23.1.x86_64.rpm (installed size 386.00 kB) 4.48  x   xInstalling libxslt-1.1.24-19.23.1.x86_64.rpm (installed size 386.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  22  ±±±±4  59%  75  87  9  100  50 2 10  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dirmngr (download size 219.00 kB)   x   xDownloading dirmngr (download size 219.00 kB) q   7  14  21  9  36  ±±±±43  51 8%  65%  73  80  7  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dirmngr-1.0.2-1.19.x86_64.rpm (installed size 564.00 kB) 10  x   xInstalling dirmngr-1.0.2-1.19.x86_64.rpm (installed size 564.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±±±42  53%  65%  74  9  84  91  8  100  1.95 1 3.55  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dbus-client (download size 28.00 kB)   x   xDownloading yast2-dbus-client (download size 28.00 kB) q   58%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dbus-client-2.17.0-0.1.65.x86_64.rpm (installed size 71.0 3.55  x   xInstalling yast2-dbus-client-2.17.0-0.1.65.x86_64.rpm (installed size 71q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  71%  96  100  88 0 48  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1-glib (download size 51.00 kB)   x   x~.00 kB) x    xDownloading dbus-1-glib (download size 51.00 kB) q   3  62%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-glib-0.76-34.22.1.x86_64.rpm (installed size 137.00 kB) 48  x   xInstalling dbus-1-glib-0.76-34.22.1.x86_64.rpm (installed size 137.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  93%  9  100  74  99 35  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libcryptsetup1 (download size 48.00 kB)   x   x~) x    xDownloading libcryptsetup1 (download size 48.00 kB) q   32  66%  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libcryptsetup1-1.1.3-2.10.x86_64.rpm (installed size 88.00 kB) 35  x   xInstalling libcryptsetup1-1.1.3-2.10.x86_64.rpm (installed size 88.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  73%  99  100  66 8 26  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dmraid (download size 131.00 kB)   x   x~) x    xDownloading dmraid (download size 131.00 kB) q   12  24  36  48 60%  73  85  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331.00 kB) 26  x   xInstalling dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  26  ±4  65%  70  8  6  93  8  100  dmraid (download size 131.00 kB) dmraid-1.0.0.rc16-0.15.3.x86_64.rpm (installed size 331 Additional rpm output: Updating etc/sysconfig/dmraid...  33 7 2.94  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libfprint0 (download size 85.00 kB)  Additional rpm output: Updating etc/sysconfig/dmraid...   Downloading libfprint0 (download size 85.00 kB)  18  37  56%  75%  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libfprint0-0.0.6-18.22.136.x86_64.rpm (installed size 209.00 kB 2.94  x   xInstalling libfprint0-0.0.6-18.22.136.x86_64.rpm (installed size 209.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  3  61%  9  9  100  1 6 7  29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading telnet (download size 54.00 kB)   x   x~kB) x    xDownloading telnet (download size 54.00 kB) q   29  59%  8  100  4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing telnet-1.2-134.24.1.x86_64.rpm (installed size 118.00 kB) 7  x   xInstalling telnet-1.2-134.24.1.x86_64.rpm (installed size 118.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  54%  85%  91  9  100  01 5 62  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading icmpinfo (download size 14.00 kB)   x   xDownloading icmpinfo (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing icmpinfo-1.11-691.22.x86_64.rpm (installed size 29.00 kB) 620  x   xInstalling icmpinfo-1.11-691.22.x86_64.rpm (installed size 29.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  50 7%  75%  80  95  100  0.99 4 59  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading finger (download size 19.00 kB)   x   xDownloading finger (download size 19.00 kB) q   83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing finger-1.3-104.22.x86_64.rpm (installed size 36.00 kB) 59  x   xInstalling finger-1.3-104.22.x86_64.rpm (installed size 36.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  84%  90  9  100  5 3 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ypbind (download size 38.00 kB)   x   xDownloading ypbind (download size 38.00 kB) q   ±±±±±41  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ypbind-1.22-1.17.x86_64.rpm (installed size 76.00 kB) 6  x   xInstalling ypbind-1.22-1.17.x86_64.rpm (installed size 76.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  68%  73  8  92  7  100  Updating etc/sysconfig/ypbind... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  88 2 48  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dbus-1-python (download size 126.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading dbus-1-python (download size 126.00 kB)  12  25  38  50 63%  76  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dbus-1-python-0.83.0-27.1.43.x86_64.rpm (installed size 617.00 48  x   xInstalling dbus-1-python-0.83.0-27.1.43.x86_64.rpm (installed size 617.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  2  6  31  7  ±±±±42  49 56%  61%  9  76  83  91  9  100  27 1 1.8  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cryptsetup (download size 89.00 kB)   x   x~0 kB) x    xDownloading cryptsetup (download size 89.00 kB) q   17  36  54%  72%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cryptsetup-1.1.3-2.10.x86_64.rpm (installed size 250.00 kB) 1.8  x   xInstalling cryptsetup-1.1.3-2.10.x86_64.rpm (installed size 250.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  26  3  ±±±±43  51 8%  65%  72  80  7  95  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  03 0 64  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cryptconfig (download size 83.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading cryptconfig (download size 83.00 kB)  19  38  57%  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cryptconfig-0.3-68.8.17.x86_64.rpm (installed size 369.00 kB) 64  x   xInstalling cryptconfig-0.3-68.8.17.x86_64.rpm (installed size 369.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  29  34  ±±±±43  52% 60%  5  70  8  8  96  100  49.67 89 28  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 201.28 MBx 122x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 49.67 MBx89x v http-10.9.0.10-e6784b1axxx tu Downloading ipmitool (download size 351.00 kB)   x   xDownloading ipmitool (download size 351.00 kB) q   9  18  27  36  ±±±±±41  50 9%  68  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ipmitool-1.8.15-0.29.2.x86_64.rpm (installed size 1.20 MB) 28  x   xInstalling ipmitool-1.8.15-0.29.2.x86_64.rpm (installed size 1.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  6  31  6  ±±±±±41  ±47  52% 7%  6  72  7  82  7  92  7  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  SizexPackagesx 200.08 MBx 122x x x 49.67 MBx 89x x x qu  1 5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsh (download size 40.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading rsh (download size 40.00 kB)  39  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsh-0.17-706.22.1.x86_64.rpm (installed size 91.00 kB) 0.0  x   xInstalling rsh-0.17-706.22.1.x86_64.rpm (installed size 91.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  ±±45  65%  81  7  93  9  100  199.99  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading postfix (download size 1017.00 kB)   x   xDownloading postfix (download size 1017.00 kB) q   6  11  6  22  7  33  8  ±±±44  49 55%  60%  6  71  7  82  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing postfix-2.9.4-0.28.2.x86_64.rpm (installed size 2.71 MB) 199.99  x   xInstalling postfix-2.9.4-0.28.2.x86_64.rpm (installed size 2.71 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  6  ±±±±±41  ±46  51 8%  64%  71  6  81  6  91  8  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/mail...  7.28 19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpg2-lang (download size 472.00 kB)  ~ script `munge' Updating etc/sysconfig/mail...   Downloading gpg2-lang (download size 472.00 kB)  6  13  20  7  33  40  ±47  54%  61%  7  74  81  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpg2-lang-2.0.9-25.33.41.2.x86_64.rpm (installed size 2.31 MB) 7.2849  x   xInstalling gpg2-lang-2.0.9-25.33.41.2.x86_64.rpm (installed size 2.31 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  22  8  33  8  ±±±±43  49 55%  60%  5  70  5  80  7  93  8  100  4.97  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ConsoleKit (download size 80.00 kB)   x   x~) x    xDownloading ConsoleKit (download size 80.00 kB) q   19  3  60%  8  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ConsoleKit-0.2.10-64.69.1.x86_64.rpm (installed size 295.00 kB) 4.978  x   xInstalling ConsoleKit-0.2.10-64.69.1.x86_64.rpm (installed size 295.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  32  8  ±±±44  65%  87  93  8  100  69  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpg2 (download size 1.16 MB)   x   x~B) x    xDownloading gpg2 (download size 1.16 MB) q   5  11  6  22  7  32  8  ±±±±43  49 54%  9%  64  9  74  80  5  91  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpg2-2.0.9-25.33.41.2.x86_64.rpm (installed size 3.77 MB) 69  x   xInstalling gpg2-2.0.9-25.33.41.2.x86_64.rpm (installed size 3.77 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  7  ±±±±42  ±47  52% 7%  63%  9  75  80  5  90  5  100  0.92  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suse-build-key (download size 15.00 kB)   x   xDownloading suse-build-key (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suse-build-key-1.0-907.46.1.noarch.rpm (installed size 9.00 kB) 0.927  x   xInstalling suse-build-key-1.0-907.46.1.noarch.rpm (installed size 9.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  98%  100   imported gpg: key 50A3DD1C: public key "SuSE Package Signing Key (reserve key) " imported gpg: Total number processed: 4 gpg: imported: 4 (RSA: 3)  1  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading inst-source-utils (download size 44.00 kB)  gpg: Total number processed: 4 imported: 4 (RSA: 3)   Downloading inst-source-utils (download size 44.00 kB)  36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing inst-source-utils-2013.1.25-0.20.10.noarch.rpm (installed size 1  x   xInstalling inst-source-utils-2013.1.25-0.20.10.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  8  34  69%  76  83  8  99  100  77  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 190.77 MBx 114x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 49.67 MBx89x v http-10.9.0.10-e6784b1axxx tu Downloading cron (download size 71.00 kB)   x   x~e 139.00 kB) x    xDownloading cron (download size 71.00 kB) q   22  ±±±44  67%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cron-4.1-194.211.213.1.x86_64.rpm (installed size 152.00 kB) 778  x   xInstalling cron-4.1-194.211.213.1.x86_64.rpm (installed size 152.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  30  6  67%  72  8  91  8  100  Updating etc/sysconfig/cron... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  52 8 62  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading PolicyKit (download size 108.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading PolicyKit (download size 108.00 kB)  14  29  ±±±44  59%  74  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing PolicyKit-0.9-14.43.1.x86_64.rpm (installed size 417.00 kB) 62  x   xInstalling PolicyKit-0.9-14.43.1.x86_64.rpm (installed size 417.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  8  27  33  ±±±±±41  ±46  52% 64%  7  89  9  100  11 7 2  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading gpgme (download size 109.00 kB)   x   xDownloading gpgme (download size 109.00 kB) q   14  29  ±±±±43  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing gpgme-1.1.6-25.32.1.x86_64.rpm (installed size 344.00 kB) 2  x   xInstalling gpgme-1.1.6-25.32.1.x86_64.rpm (installed size 344.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  35  ±±4  67%  73  99  100  8.78 6 89.88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading logrotate (download size 39.00 kB)   x   xDownloading logrotate (download size 39.00 kB) q   4  82%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing logrotate-3.7.7-10.30.1.x86_64.rpm (installed size 68.00 kB) 89.88  x   xInstalling logrotate-3.7.7-10.30.1.x86_64.rpm (installed size 68.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  83%  91  9  100  1 5 1  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dbus-server (download size 116.00 kB)   x   xDownloading yast2-dbus-server (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339.0 1  x   xInstalling yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  32  55%  73%  89  96  100  yast2-dbus-server-2.17.5-0.5.1.x86_64.rpm (installed size 339 ~.00 kB) Additional rpm output: Failed to open connection to system message bus: Failed to connect to so ~cket /var/run/dbus/system_bus_socket: No such file or directory  38 4 48  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pm-utils (download size 43.00 kB)  Failed to open connection to system message bus: Failed to connect to so cket /var/run/dbus/system_bus_socket: No such file or directory   Downloading pm-utils (download size 43.00 kB)   36  74%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pm-utils-0.99.4.20071229-12.14.1.x86_64.rpm (installed size 100 48  x   xInstalling pm-utils-0.99.4.20071229-12.14.1.x86_64.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  2  3  ±±±±42  ±47  55%  60%  6  85  90  5  100  2 3 3  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading PolicyKit-doc (download size 336.00 kB)   x   x~00.00 kB) x    xDownloading PolicyKit-doc (download size 336.00 kB) q   9  14  9  28  33  8  ±47  52% 7%  62%  71  6  81  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing PolicyKit-doc-0.9-14.43.1.x86_64.rpm (installed size 1.29 MB) 3  x   xInstalling PolicyKit-doc-0.9-14.43.1.x86_64.rpm (installed size 1.29 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  9  25  30  5  40  ±±45  50 5%  60%  6  72  7  83  8  94  9  100  6.99 2 8.09  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading polkit-default-privs (download size 12.00 kB)   x   xDownloading polkit-default-privs (download size 12.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing polkit-default-privs-0.1_200812111413-3.6.noarch.rpm (installed 8.09  x   xInstalling polkit-default-privs-0.1_200812111413-3.6.noarch.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  32  67%  79  8  95  100  keep_always:auth_admin_keep_always:yes setting org.freedesktop.network-manager-settings.system.modify to auth_a ~dmin_keep_always:auth_admin_keep_always:auth_admin_keep_always setting org.freedesktop.hal.device-access.removable-block to auth_admin_ ~keep_always:auth_admin_keep_always:auth_admin_keep_always  7 1 7  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libgpgme11 (download size 105.00 kB)  setting org.freedesktop.hal.device-access.removable-block to auth_admin_ ~keep_always:auth_admin_keep_always:auth_admin_keep_always   Downloading libgpgme11 (download size 105.00 kB)   15  30  ±±45  61%  76  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libgpgme11-1.1.6-25.32.1.x86_64.rpm (installed size 574.00 kB) 7  x   xInstalling libgpgme11-1.1.6-25.32.1.x86_64.rpm (installed size 574.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  33  ±±±44  55%  66%  77  88  99  100  41 0 7.51  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading xinetd (download size 125.00 kB)   x   x~) x    xDownloading xinetd (download size 125.00 kB) q   12  25  38  51 63%  76  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing xinetd-2.3.14-130.133.1.x86_64.rpm (installed size 285.00 kB) 7.51  x   xInstalling xinetd-2.3.14-130.133.1.x86_64.rpm (installed size 285.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  36  59%  72  87  93  8  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  13 79 23  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-xml (download size 50.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading yast2-xml (download size 50.00 kB)  31  63%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-xml-2.16.1-1.23.x86_64.rpm (installed size 128.00 kB) 23  x   xInstalling yast2-xml-2.16.1-1.23.x86_64.rpm (installed size 128.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 79%  94  9  100  00 8 11  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hal (download size 464.00 kB)   x   xDownloading hal (download size 464.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  75  82  9  96  100  6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hal-0.5.12-23.76.1.x86_64.rpm (installed size 1.78 MB) 117  x   xInstalling hal-0.5.12-23.76.1.x86_64.rpm (installed size 1.78 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 7%  64%  71  8  83  8  93  8  100  Updating etc/sysconfig/hal... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.23 7 5.33  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mutt (download size 1.24 MB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading mutt (download size 1.24 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mutt-1.5.17-42.39.1.x86_64.rpm (installed size 4.33 MB) 5.33  x   xInstalling mutt-1.5.17-42.39.1.x86_64.rpm (installed size 4.33 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  33  9  ±±±44  49 54%  9%  64  9  74  9  84  9  94  9  100  39.90 6 1.00  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading pcsc-lite (download size 158.00 kB)   x   xDownloading pcsc-lite (download size 158.00 kB) q   9  20  3  4  50 60%  7  8  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing pcsc-lite-1.4.102-1.37.3.x86_64.rpm (installed size 520.00 kB) 1.006  x   xInstalling pcsc-lite-1.4.102-1.37.3.x86_64.rpm (installed size 520.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  20  7  40  52% 64%  76  81  9  9  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  39 5 0.49  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ivman (download size 62.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading ivman (download size 62.00 kB)  25  51 77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ivman-0.6.14-201.31.1.x86_64.rpm (installed size 150.00 kB) 0.49  x   xInstalling ivman-0.6.14-201.31.1.x86_64.rpm (installed size 150.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  58%  80%  9  6  100  24 4 34  99 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libopensc2 (download size 325.00 kB)   x   xDownloading libopensc2 (download size 325.00 kB) q   9  14  9  24  9  34  9  ±±±44  49 54%  9%  64  73  8  83  8  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libopensc2-0.11.6-5.27.1.x86_64.rpm (installed size 838.00 kB) 34  x   xInstalling libopensc2-0.11.6-5.27.1.x86_64.rpm (installed size 838.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  15  22  30  8  ±±45  53%  61%  8  76  84  92  7  100  8.42 3 79.52  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading opensc (download size 493.00 kB)   x   x~) x    xDownloading opensc (download size 493.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing opensc-0.11.6-5.27.1.x86_64.rpm (installed size 1.83 MB) 79.525  x   xInstalling opensc-0.11.6-5.27.1.x86_64.rpm (installed size 1.83 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  9  24  9  34  40  ±46  53%  60%  7  74  9  84  9  94  9  100  6.59 2 7.70  7 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 177.70 MBx 97x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 36.59 MBx72x v http-10.9.0.10-e6784b1axxx tu Downloading ntp (download size 763.00 kB)   x   xDownloading ntp (download size 763.00 kB) q   6  11  7  22  8  34  ±±±±±41  ±47  53%  9%  64  70  6  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) 7.706  x   xInstalling ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  9  26  33  8  ±±45  51 8%  65%  70  5  80  6  92  7  100  Installing ntp-4.2.8p9-57.2.x86_64.rpm (installed size 2.67 MB) Additional rpm output: tr: write error: Broken pipe tr: write error Updating etc/sysconfig/ntp...  SizexPackagesx 175.03 MBx 97x x x 36.59 MBx 72x x x qu  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mcelog (download size 91.00 kB)  tr: write error Updating etc/sysconfig/ntp...   Downloading mcelog (download size 91.00 kB)  17  35  52% 70%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mcelog-1.0.2014.12.20-0.23.1.x86_64.rpm (installed size 217.00 5.035  x   xInstalling mcelog-1.0.2014.12.20-0.23.1.x86_64.rpm (installed size 217.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  19  31  61%  89  94  9  100  mcelog... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  4.82  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading sysconfig (download size 350.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading sysconfig (download size 350.00 kB)  9  18  27  32  ±±±±±41  50 9%  64  73  82  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing sysconfig-0.71.61-0.15.1.x86_64.rpm (installed size 781.00 kB) 4.82  x   xInstalling sysconfig-0.71.61-0.15.1.x86_64.rpm (installed size 781.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  7  22  8  33  8  ±±±±43  48 55%  61%  6  71  6  82  90  5  100  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx  Updating etc/sysconfig/network/dhcp... Updating etc/sysconfig/network/config... Removing old autogenerated device configuration files:q 05  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading libzypp (download size 2.14 MB)  Updating etc/sysconfig/network/config... Removing old autogenerated device configuration files:   Downloading libzypp (download size 2.14 MB)   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing libzypp-9.40.2-14.7.x86_64.rpm (installed size 7.27 MB) 05  x   xInstalling libzypp-9.40.2-14.7.x86_64.rpm (installed size 7.27 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  66.79  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading hwinfo (download size 609.00 kB)   x   xDownloading hwinfo (download size 609.00 kB) q   5  10  5  20  6  31  6  ±±±±42  ±47  52% 7%  63%  8  73  8  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing hwinfo-15.57-0.12.2.x86_64.rpm (installed size 2.20 MB) 66.793  x   xInstalling hwinfo-15.57-0.12.2.x86_64.rpm (installed size 2.20 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  22  8  34  9  ±±45  51 6%  62%  8  73  9  85  90  5  100  4.5  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading SuSEfirewall2 (download size 67.00 kB)   x   xDownloading SuSEfirewall2 (download size 67.00 kB) q   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing SuSEfirewall2-3.6_SVNr208-2.11.1.noarch.rpm (installed size 234 4.5  x   xInstalling SuSEfirewall2-3.6_SVNr208-2.11.1.noarch.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  38  ±47  56%  63%  8  82  99  100  ~ script `munge' insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  36  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading dhcpcd (download size 51.00 kB)  M  xinsserv: Default-Start undefined, assuming default start runlevel(s) forx    Downloading dhcpcd (download size 51.00 kB)  31  63%  95  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing dhcpcd-3.2.3-45.5.3.x86_64.rpm (installed size 94.00 kB) 36  x   xInstalling dhcpcd-3.2.3-45.5.3.x86_64.rpm (installed size 94.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  80  94  9  100  27  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading zypper (download size 1.02 MB)   x   xDownloading zypper (download size 1.02 MB) q   5  11  7  22  8  34  40  ±±45  51 7%  63%  8  74  80  6  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing zypper-1.6.334-26.3.x86_64.rpm (installed size 4.86 MB) 27  x   xInstalling zypper-1.6.334-26.3.x86_64.rpm (installed size 4.86 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  51 6%  61%  6  71  6  81  6  91  6  100  59.40 89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mkinitrd (download size 99.00 kB)   x   xDownloading mkinitrd (download size 99.00 kB) q   16  32  48 64%  81  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mkinitrd-2.4.2-100.2.x86_64.rpm (installed size 177.00 kB) 59.401  x   xInstalling mkinitrd-2.4.2-100.2.x86_64.rpm (installed size 177.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  20  5  30  5  40  ±46  53%  9%  65  73  84  90  5  100  ~ script `munge' Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  23  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading mdadm (download size 451.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading mdadm (download size 451.00 kB)  7  14  21  8  35  ±±±±42  49 56%  63%  71  8  85  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing mdadm-3.3.1-9.1.x86_64.rpm (installed size 1.21 MB) 23  x   xInstalling mdadm-3.3.1-9.1.x86_64.rpm (installed size 1.21 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  32  7  ±±±±42  ±47  52% 7%  63%  8  74  9  84  91  6  100  Updating etc/sysconfig/mdadm... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  8.01  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kdump (download size 261.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading kdump (download size 261.00 kB)  6  12  8  24  30  6  ±±±±42  49 55%  61%  7  73  9  85  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kdump-0.8.4-0.55.1.x86_64.rpm (installed size 635.00 kB) 8.01  x   xInstalling kdump-0.8.4-0.55.1.x86_64.rpm (installed size 635.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  2  3  ±4  56%  66%  7  8  92  7  100  Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ... symlinks in /lib/mkinitrd/boot ...  7.39  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kernel-default-base (download size 15.47 MB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading kernel-default-base (download size 15.47 MB)  5  10  5  20 - 2.82 MB/s (download size 15.47 MB)  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-default-base-3.0.101-94.1.x86_64.rpm (installed size 23. 7.39  x   xInstalling kernel-default-base-3.0.101-94.1.x86_64.rpm (installed size 2q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  WARNING: no dependencies for kernel module 'virtio:d00000002v00001AF4' f ~ound. Kernel Modules: mbcache jbd ext3 Features: acpi block 27677 blocks  34.31  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading kernel-default (download size 24.27 MB)  Features: acpi block 27677 blocks    Downloading kernel-default (download size 24.27 MB)  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing kernel-default-3.0.101-94.1.x86_64.rpm (installed size 84.95 MB 34.3135  x   xInstalling kernel-default-3.0.101-94.1.x86_64.rpm (installed size 84.95 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ~ms_helper ttm cirrus virtio_ring virtio virtio_blk usb-common usbcore o ~hci-hcd uhci-hcd ehci-hcd xhci-hcd hid usbhid crc-t10dif sd_mod virtio_ ~pci Features: acpi kms block usb 33537 blocks  xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 49.36 MBx 84x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 36.59 MBx72x v http-10.9.0.10-e6784b1axxx tu 91  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 49.36 MBx 84x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 36.59 MBx72x v http-10.9.0.10-e6784b1axxx tu Downloading klogd (download size 40.00 kB)  Features: acpi kms block usb 33537 blocks    Downloading klogd (download size 40.00 kB)  4  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing klogd-1.4.1-708.44.32.x86_64.rpm (installed size 59.00 kB) 49.36 MB / 00:20)   x   xInstalling klogd-1.4.1-708.44.32.x86_64.rpm (installed size 59.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  68%  79  92  8  100  4 1 1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-pkg-bindings (download size 530.00 kB)   x   xDownloading yast2-pkg-bindings (download size 530.00 kB) q   6  11  6  22  8  34  40  ±46  52% 8%  64%  70  6  82  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-pkg-bindings-2.17.61-1.40.x86_64.rpm (installed size 1.86 118  x   xInstalling yast2-pkg-bindings-2.17.61-1.40.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  6  33  40  ±47  53%  60%  7  74  80  7  94  9  100  4.68 0 7.45  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ncurses-pkg (download size 233.00 kB)   x   x~86 MB) x    xDownloading yast2-ncurses-pkg (download size 233.00 kB) q   6  13  20  7  34  ±±±±±41  48 54%  61%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ncurses-pkg-2.17.20-0.5.1.x86_64.rpm (installed size 771. 7.457  x   xInstalling yast2-ncurses-pkg-2.17.20-0.5.1.x86_64.rpm (installed size 77q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  33  ±±±±±41  49 58%  66%  74  83  91  7  100  3.93 69 6.70  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-hardware-detection (download size 97.00 kB)   x   x~1.00 kB) x    xDownloading yast2-hardware-detection (download size 97.00 kB) q   16  32  49 65%  82  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-hardware-detection-2.17.2-2.7.1.x86_64.rpm (installed siz 6.70  x   xInstalling yast2-hardware-detection-2.17.2-2.7.1.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  30  ±±45  54%  9%  65  71  83  8  93  9  100  52 8 29  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading subscription-tools (download size 47.00 kB)   x   x~ize 417.00 kB) x    xDownloading subscription-tools (download size 47.00 kB) q   33  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing subscription-tools-1.0-0.21.1.noarch.rpm (installed size 163.00 29  x   xInstalling subscription-tools-1.0-0.21.1.noarch.rpm (installed size 163.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  32  ±46  53%  9%  66  72  8  84  91  6  100  Additional rpm output: insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  36 7 13 79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suseRegister (download size 54.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading suseRegister (download size 54.00 kB)  29  58%  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suseRegister-1.4-1.38.24.noarch.rpm (installed size 213.00 kB) 13  x   xInstalling suseRegister-1.4-1.38.24.noarch.rpm (installed size 213.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  48 53%  78%  90  8  100  15 6 5.92  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading splashy (download size 1.11 MB)   x   x~) x    xDownloading splashy (download size 1.11 MB) q   5  10  5  20  6  31  6  ±±±±±41  ±47  52% 7%  62%  7  72  8  83  8  93  9  100  2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing splashy-0.3.8-106.21.x86_64.rpm (installed size 1.47 MB) 5.92  x   xInstalling splashy-0.3.8-106.21.x86_64.rpm (installed size 1.47 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  30  9  ±±±44  53%  61%  9  78  86  91  9  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ... symlinks in /lib/mkinitrd/boot ...  1.68 5 4.45  7 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading cifs-utils (download size 166.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading cifs-utils (download size 166.00 kB)  9  1  28  3  4 57%  6  7  86  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing cifs-utils-5.1-0.14.46.x86_64.rpm (installed size 85.00 kB) 4.45  x   xInstalling cifs-utils-5.1-0.14.46.x86_64.rpm (installed size 85.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  85%  99  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  59 4 36  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootsplash (download size 205.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading bootsplash (download size 205.00 kB)  7  15  23  31  9  ±46  54%  62%  70  8  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootsplash-3.3-146.24.12.x86_64.rpm (installed size 555.00 kB) 366  x   xInstalling bootsplash-3.3-146.24.12.x86_64.rpm (installed size 555.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  26  37  48 60%  71  6  81  92  7  100  Updating etc/sysconfig/bootsplash... Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  05 3 3.82  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ofed-kmp-default (download size 66.00 kB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading ofed-kmp-default (download size 66.00 kB)  23  48 72%  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-kmp-default-1.5.4.1_3.0.101_63-20.26.x86_64.rpm (installed 3.82  x   xInstalling ofed-kmp-default-1.5.4.1_3.0.101_63-20.26.x86_64.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  ±±±±±41  73%  99  100  0.86 2 63  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading rsyslog (download size 399.00 kB)   x   x~ed size 201.00 kB) x    xDownloading rsyslog (download size 399.00 kB) q   7  15  20  8  36  ±±±44  52% 60%  8  76  84  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing rsyslog-5.10.1-0.11.1.x86_64.rpm (installed size 1.50 MB) 63  x   xInstalling rsyslog-5.10.1-0.11.1.x86_64.rpm (installed size 1.50 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  9  24  9  34  9  ±±±44  51 60%  8  74  9  87  92  8  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/syslog...  29.3 1 2.1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2 (download size 682.00 kB)  ~ script `munge' Updating etc/sysconfig/syslog...   Downloading yast2 (download size 682.00 kB)  5  11  6  23  30  7  ±±±44  51 8%  65%  70  7  84  91  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-2.17.140-1.1.x86_64.rpm (installed size 3.44 MB) 2.1  x   xInstalling yast2-2.17.140-1.1.x86_64.rpm (installed size 3.44 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  6  21  6  31  6  ±±±±42  ±47  52% 7%  62%  7  72  7  83  8  93  8  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/yast2... q  5.92 0 38.69  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading suspend (download size 91.00 kB)  Additional rpm output: Updating etc/sysconfig/yast2...   Downloading suspend (download size 91.00 kB)  17  35  52% 70%  87  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing suspend-0.80.20081103-1.59.35.x86_64.rpm (installed size 350.00 38.695  x   xInstalling suspend-0.80.20081103-1.59.35.x86_64.rpm (installed size 350.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  30  ±±45  60%  78  85  9  100  Additional rpm output: Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ...  symlinks in /lib/mkinitrd/boot ...  58 59 35  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading bootsplash-branding-SLES (download size 1.19 MB)  Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ...   Downloading bootsplash-branding-SLES (download size 1.19 MB)  5  10  5  20  5  30  6  ±±±±±41  ±46  51 7%  62%  7  72  8  83  8  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed size 1 35  x   xInstalling bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  24  31  8  ±±45  50 6%  62%  8  75  82  9  95  100  Downloading bootsplash-branding-SLES (download size 1.19 MB) Installing bootsplash-branding-SLES-3.1-47.22.x86_64.rpm (installed size ~ 1.34 MB) Additional rpm output: Updating etc/sysconfig/bootsplash...  4.24 8 7.01  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading ofed (download size 28.00 kB)  Additional rpm output: Updating etc/sysconfig/bootsplash...   Downloading ofed (download size 28.00 kB)  57%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing ofed-1.5.4.1-20.26.x86_64.rpm (installed size 52.00 kB) 7.01  x   xInstalling ofed-1.5.4.1-20.26.x86_64.rpm (installed size 52.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  89%  9  100  infiniband insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  19 7 6.96 69 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-vm (download size 41.00 kB)  insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Downloading yast2-vm (download size 41.00 kB)  38  77%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-vm-2.17.17-0.7.66.x86_64.rpm (installed size 163.00 kB) 6.96  x   xInstalling yast2-vm-2.17.17-0.7.66.x86_64.rpm (installed size 163.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  31  9  52% 61%  73  80  5  91  8  100  03 6 80  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-squid (download size 109.00 kB)   x   xDownloading yast2-squid (download size 109.00 kB) q   14  29  ±±±44  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-squid-2.17.12-0.7.24.x86_64.rpm (installed size 547.00 kB 80  x   xInstalling yast2-squid-2.17.12-0.7.24.x86_64.rpm (installed size 547.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  23  9  35  40  56%  65%  74  9  86  91  6  100  3.50 5 26  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-packager (download size 362.00 kB)   x   x~kB) x    xDownloading yast2-packager (download size 362.00 kB) q   8  13  22  30  5  ±±±44  53%  61%  6  75  84  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-packager-2.17.113-1.47.x86_64.rpm (installed size 1.72 MB 26  x   xInstalling yast2-packager-2.17.113-1.47.x86_64.rpm (installed size 1.72 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  32  9  ±46  51 6%  63%  8  74  9  84  9  94  9  100  1.77 4 4.54  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mouse (download size 26.00 kB)   x   x~MB) x    xDownloading yast2-mouse (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mouse-2.16.1-19.38.x86_64.rpm (installed size 87.00 kB) 4.544  x   xInstalling yast2-mouse-2.16.1-19.38.x86_64.rpm (installed size 87.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  ±±45  60%  83  8  93  9  100  69 3 46  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap (download size 112.00 kB)   x   xDownloading yast2-ldap (download size 112.00 kB) q   14  28  ±±±±42  57%  71%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-2.17.8-0.7.61.x86_64.rpm (installed size 475.00 kB) 46  x   xInstalling yast2-ldap-2.17.8-0.7.61.x86_64.rpm (installed size 475.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  27  40  ±±45  50 5%  61%  74  8  9  96  100  22 2 3.99  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-country (download size 104.00 kB)   x   x~) x    xDownloading yast2-country (download size 104.00 kB) q   15  30  ±46  61%  77  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-country-2.17.55-0.7.1.x86_64.rpm (installed size 478.00 k 3.99  x   xInstalling yast2-country-2.17.55-0.7.1.x86_64.rpm (installed size 478.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  ±±±±42  50 9%  64  70  6  83  8  93  8  100  0.75 1 52  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-tftp-server (download size 25.00 kB)   x   x~ kB) x    xDownloading yast2-tftp-server (download size 25.00 kB) q   64%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-tftp-server-2.17.2-0.7.38.noarch.rpm (installed size 93.0 52  x   xInstalling yast2-tftp-server-2.17.2-0.7.38.noarch.rpm (installed size 93q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  29  38  59%  72  9  88  96  100  66 0 43  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sysconfig (download size 56.00 kB)   x   x~.00 kB) x    xDownloading yast2-sysconfig (download size 56.00 kB) q   28  57%  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sysconfig-2.17.0-1.38.noarch.rpm (installed size 284.00 k 43  x   xInstalling yast2-sysconfig-2.17.0-1.38.noarch.rpm (installed size 284.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  18  ±±±44  59%  66  74  82  90  5  100  39 49 16  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-support (download size 36.00 kB)   x   x~ kB) x    xDownloading yast2-support (download size 36.00 kB) q   ±±±44  89%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-support-2.17.21-1.35.noarch.rpm (installed size 150.00 kB 16  x   xInstalling yast2-support-2.17.21-1.35.noarch.rpm (installed size 150.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  25  30  7  48 55%  69%  77  82  9  94  9  100  24 8 01  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sshd (download size 28.00 kB)   x   x~kB) x    xDownloading yast2-sshd (download size 28.00 kB) q   56%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sshd-2.17.2-1.21.noarch.rpm (installed size 114.00 kB) 01  x   xInstalling yast2-sshd-2.17.2-1.21.noarch.rpm (installed size 114.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  8  2  35  ±±±±43  59%  6  78  84  90  6  100  13 7 2.90 59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-runlevel (download size 56.00 kB)   x   xDownloading yast2-runlevel (download size 56.00 kB) q   28  57%  86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-runlevel-2.17.9-0.7.3.noarch.rpm (installed size 209.00 k 2.90  x   xInstalling yast2-runlevel-2.17.9-0.7.3.noarch.rpm (installed size 209.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  ±±±44  74%  88  9  100  19.92 6 69  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-power-management (download size 34.00 kB)   x   x~ kB) x    xDownloading yast2-power-management (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-power-management-2.17.7-0.1.84.noarch.rpm (installed size 69  x   xInstalling yast2-power-management-2.17.7-0.1.84.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  34  ±±±±42  53%  62%  71  82  7  9  100  75 5 52  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-pam (download size 30.00 kB)   x   x~ze 175.00 kB) x    xDownloading yast2-pam (download size 30.00 kB) q   54%  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-pam-2.17.3-0.5.211.noarch.rpm (installed size 129.00 kB) 52  x   xInstalling yast2-pam-2.17.3-0.5.211.noarch.rpm (installed size 129.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  15  20  6  32  ±±±±±41  ±46  63%  9  74  9  90  7  100  63 4 40  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-server (download size 129.00 kB)   x   x~) x    xDownloading yast2-nfs-server (download size 129.00 kB) q   12  24  37  49 61%  74  86  99  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-server-2.17.8-0.2.1.noarch.rpm (installed size 295.00 40  x   xInstalling yast2-nfs-server-2.17.8-0.2.1.noarch.rpm (installed size 295.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  17  23  3  8  ±±±±43  54%  69%  81  9  9  100  34 3 11  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nfs-client (download size 115.00 kB)   x   x~00 kB) x    xDownloading yast2-nfs-client (download size 115.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nfs-client-2.17.19-1.32.noarch.rpm (installed size 313.00 11  x   xInstalling yast2-nfs-client-2.17.19-1.32.noarch.rpm (installed size 313.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  31  8  ±±±±43  48 55%  62%  7  72  7  88  94  9  100  03 2 1.80  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-isns (download size 32.00 kB)   x   x~00 kB) x    xDownloading yast2-isns (download size 32.00 kB) q   49 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-isns-2.17.2-0.2.2.noarch.rpm (installed size 152.00 kB) 1.80  x   xInstalling yast2-isns-2.17.2-0.2.2.noarch.rpm (installed size 152.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  25  49 59%  72  7  83  8  94  9  100  8.89 1 66  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-iscsi-server (download size 40.00 kB)   x   xDownloading yast2-iscsi-server (download size 40.00 kB) q   4  80%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-iscsi-server-2.17.11-1.58.noarch.rpm (installed size 172. 66  x   xInstalling yast2-iscsi-server-2.17.11-1.58.noarch.rpm (installed size 17q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  27  ±46  55%  62%  74  9  84  90  6  100  72 0 49  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-irda (download size 19.00 kB)   x   x~2.00 kB) x    xDownloading yast2-irda (download size 19.00 kB) q   83%  100  4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-irda-2.17.3-0.1.85.noarch.rpm (installed size 58.00 kB) 49  x   xInstalling yast2-irda-2.17.3-0.1.85.noarch.rpm (installed size 58.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  6  31  61%  7  8  92  9  100  66 39 3  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-instserver (download size 72.00 kB)   x   xDownloading yast2-instserver (download size 72.00 kB) q   22  ±±±44  67%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-instserver-2.17.7-0.2.1.noarch.rpm (installed size 289.00 3  x   xInstalling yast2-instserver-2.17.7-0.2.1.noarch.rpm (installed size 289.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  18  ±±±±43  52% 64%  70  6  81  6  99  100  38 8 15  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-firewall (download size 83.00 kB)   x   x~00 kB) x    xDownloading yast2-firewall (download size 83.00 kB) q   19  38  57%  7  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-firewall-2.17.13-0.11.11.noarch.rpm (installed size 498.0 15  x   xInstalling yast2-firewall-2.17.13-0.11.11.noarch.rpm (installed size 498q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  12  25  38  51 64%  76  81  6  93  8  100  7.89 7 0.66 49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ca-management (download size 122.00 kB)   x   x~.00 kB) x    xDownloading yast2-ca-management (download size 122.00 kB) q   12  26  39  52% 65%  78  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ca-management-2.17.27-1.1.noarch.rpm (installed size 940. 0.663  x   xInstalling yast2-ca-management-2.17.27-1.1.noarch.rpm (installed size 94q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  8  23  30  5  40  49 56%  63%  72  8  7  92  7  100  6.97 6 29.74  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-boot-server (download size 16.00 kB)   x   x~0.00 kB) x    xDownloading yast2-boot-server (download size 16.00 kB) q   98%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-boot-server-2.14.0-276.38.noarch.rpm (installed size 45.0 29.74  x   xInstalling yast2-boot-server-2.14.0-276.38.noarch.rpm (installed size 45q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  30  7  77%  82  7  94  9  100  3 5 0  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-autofs (download size 26.00 kB)   x   x~.00 kB) x    xDownloading yast2-autofs (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-autofs-2.17.1-0.1.61.noarch.rpm (installed size 98.00 kB) 0  x   xInstalling yast2-autofs-2.17.1-0.1.61.noarch.rpm (installed size 98.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  1  22  ±±±±43  49 69%  75  80  6  94  9  100  8 4 6  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-add-on-creator (download size 100.00 kB)   x   x~B) x    xDownloading yast2-add-on-creator (download size 100.00 kB) q   15  32  48 64%  80  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-add-on-creator-2.17.15-0.5.93.noarch.rpm (installed size 6  x   xInstalling yast2-add-on-creator-2.17.15-0.5.93.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  21  36  48 59%  73  9  90  5  100  dd-on-creator (download size 100.00 kB)dd-on-creator-2.17.15-0.5.93.noarch.rpm (installed siz e 547.00 kB) Additional rpm output: Updating etc/sysconfig/add-on-creator...  30 3 07  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-registration-branding-SLE (download size 12.00 kB)  Additional rpm output: Updating etc/sysconfig/add-on-creator...   Downloading yast2-registration-branding-SLE (download size 12.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-registration-branding-SLE-2.17.38-0.8.43.noarch.rpm (inst 07  x   xInstalling yast2-registration-branding-SLE-2.17.38-0.8.43.noarch.rpm (inq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  100  2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-online-update (download size 49.00 kB)   x   x~stalled size 1.00 kB) x    xDownloading yast2-online-update (download size 49.00 kB) q   32  64%  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-online-update-2.17.24-1.82.noarch.rpm (installed size 187  x   xInstalling yast2-online-update-2.17.24-1.82.noarch.rpm (installed size 1q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  33  ±±±±42  52% 9%  6  7  84  9  95  100  12 1 8.89  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-metapackage-handler (download size 44.00 kB)   x   x~87.00 kB) x    xDownloading yast2-metapackage-handler (download size 44.00 kB) q   36  73%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-metapackage-handler-0.8.7-1.34.noarch.rpm (installed size 8.89  x   xInstalling yast2-metapackage-handler-0.8.7-1.34.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  35  ±47  58%  64%  73  81  6  98  100  5.93 0 70  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-iscsi-client (download size 58.00 kB)   x   x~ze 191.00 kB) x    xDownloading yast2-iscsi-client (download size 58.00 kB) q   27 - 15.74 kB/s (download size 58.00 kB)  55%  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-iscsi-client-2.17.42-1.4.noarch.rpm (installed size 262.0 70  x   xInstalling yast2-iscsi-client-2.17.42-1.4.noarch.rpm (installed size 262q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  24  51 60%  71  9  84  9  94  9  100  67 29 44  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-nis-client (download size 232.00 kB)   x   x~.00 kB) x    xDownloading yast2-nis-client (download size 232.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  8  75  82  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nis-client-2.17.8-0.1.1.x86_64.rpm (installed size 543.00 44  x   xInstalling yast2-nis-client-2.17.8-0.1.1.x86_64.rpm (installed size 543.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  7  28  34  ±±±4  49 60%  7  74  82  9  9  100  14 8 7.91  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-fingerprint-reader (download size 62.00 kB)   x   x~00 kB) x    xDownloading yast2-fingerprint-reader (download size 62.00 kB) q   25  52% 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-fingerprint-reader-2.17.7-0.1.201.x86_64.rpm (installed s 7.91  x   xInstalling yast2-fingerprint-reader-2.17.7-0.1.201.x86_64.rpm (installedq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  19  25  30  6  ±±±±42  49 54%  62%  75  80  5  90  5  100  4.88 7 65 39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-security (download size 51.00 kB)   x   x~ size 268.00 kB) x    xDownloading yast2-security (download size 51.00 kB) q   31  62%  94  100  5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-security-2.17.16-0.7.25.noarch.rpm (installed size 232.00 65  x   xInstalling yast2-security-2.17.16-0.7.25.noarch.rpm (installed size 232.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  22  7  32  49 57%  6  73  8  83  8  93  8  100  66 6 43  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kerberos-client (download size 45.00 kB)   x   x~00 kB) x    xDownloading yast2-kerberos-client (download size 45.00 kB) q   35  71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kerberos-client-2.17.16-0.7.79.noarch.rpm (installed size 43  x   xInstalling yast2-kerberos-client-2.17.16-0.7.79.noarch.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  20  ±46  59%  70  9  85  91  7  100  4 5 2  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-registration (download size 173.00 kB)   x   x~ze 201.00 kB) x    xDownloading yast2-registration (download size 173.00 kB) q   9  18  27  36  ±4  55%  64%  73  8  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size 44 2  x   xInstalling yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  9  28  ±47  52% 66%  72  87  92  9  100  Installing yast2-registration-2.17.38-0.8.43.noarch.rpm (installed size ~449.00 kB) Additional rpm output: Updating etc/sysconfig/suse_register... Updating etc/sysconfig/automatic_online_update...  02 4 6.79  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-online-update-frontend (download size 11.00 kB)  Updating etc/sysconfig/suse_register... Updating etc/sysconfig/automatic_online_update...   Downloading yast2-online-update-frontend (download size 11.00 kB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-online-update-frontend-2.17.24-1.82.noarch.rpm (installed 6.79  x   xInstalling yast2-online-update-frontend-2.17.24-1.82.noarch.rpm (installq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  86%  100  3  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-samba-client (download size 71.00 kB)   x   x~ed size 1.00 kB) x    xDownloading yast2-samba-client (download size 71.00 kB) q   22  ±±45  67%  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-samba-client-2.17.30-0.7.1.noarch.rpm (installed size 321  x   xInstalling yast2-samba-client-2.17.30-0.7.1.noarch.rpm (installed size 3q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  34  ±±±±43  58%  64%  73  82  90  5  100  3.71 2 48  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 26.48 MBx 34x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 13.71 MBx22x v http-10.9.0.10-e6784b1axxx tu Downloading syslog-ng (download size 247.00 kB)   x   x~21.00 kB) x    xDownloading syslog-ng (download size 247.00 kB) q   6  12  9  25  32  8  ±±45  51 8%  64%  71  7  84  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing syslog-ng-2.0.9-27.34.39.2.x86_64.rpm (installed size 787.00 kB 4820  x   xInstalling syslog-ng-2.0.9-27.34.39.2.x86_64.rpm (installed size 787.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  8  27  35  ±±±±43  50 60%  6  7  81  9  97  100  insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge' Updating etc/sysconfig/syslog...  5.71  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-slp-server (download size 31.00 kB)  ~ script `munge' Updating etc/sysconfig/syslog...   Downloading yast2-slp-server (download size 31.00 kB)  51 100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-slp-server-2.17.8-7.1.noarch.rpm (installed size 128.00 k 5.7112  x   xInstalling yast2-slp-server-2.17.8-7.1.noarch.rpm (installed size 128.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  ±±±±43  50 67%  74  80  8  94  9  100  58  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-audit-laf (download size 37.00 kB)   x   x~ kB) x    xDownloading yast2-audit-laf (download size 37.00 kB) q   ±±±±43  87%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-audit-laf-2.17.11-4.3.noarch.rpm (installed size 157.00 k 58  x   xInstalling yast2-audit-laf-2.17.11-4.3.noarch.rpm (installed size 157.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  16  22  7  ±±45  55%  68%  79  84  9  96  100  43  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ntp-client (download size 84.00 kB)   x   x~ kB) x    xDownloading yast2-ntp-client (download size 84.00 kB) q   18  3  57%  76%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ntp-client-2.17.20-10.34.noarch.rpm (installed size 452.0 43  x   xInstalling yast2-ntp-client-2.17.20-10.34.noarch.rpm (installed size 452q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  22  7  39  53%  60%  7  6  83  8  97  100  4.99  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-printer (download size 142.00 kB)   x   x~.00 kB) x    xDownloading yast2-printer (download size 142.00 kB) q   11  22  33  ±±±44  56%  67%  78  90  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-printer-2.17.64-4.4.x86_64.rpm (installed size 721.00 kB) 4.99  x   xInstalling yast2-printer-2.17.64-4.4.x86_64.rpm (installed size 721.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  20  7  34  40  49 58%  65%  7  84  90  6  100  28 29 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-storage (download size 646.00 kB)   x   x~B) x    xDownloading yast2-storage (download size 646.00 kB) q   7  12  7  22  7  32  7  ±±±±42  ±47  52% 7%  64%  9  74  9  84  9  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-storage-2.17.161-5.1.x86_64.rpm (installed size 3.53 MB) 28  x   xInstalling yast2-storage-2.17.161-5.1.x86_64.rpm (installed size 3.53 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  6  31  6  ±±±±±41  ±47  53%  8%  64%  9  74  9  85  90  6  100  storage646storage161-5.1.x86_64.rpm (installed size 3.53 MB ) Additional rpm output: Updating etc/sysconfig/storage...  0.75  8 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-network (download size 380.00 kB)  Additional rpm output: Updating etc/sysconfig/storage...   Downloading yast2-network (download size 380.00 kB)  8  16  21  9  37  ±±±±42  50 8%  63%  71  80  8  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-network-2.17.210-3.3.x86_64.rpm (installed size 2.26 MB) 0.751  x   xInstalling yast2-network-2.17.210-3.3.x86_64.rpm (installed size 2.26 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±43  48 55%  61%  6  71  7  83  9  96  100  18.49  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-bootloader (download size 353.00 kB)   x   x~) x    xDownloading yast2-bootloader (download size 353.00 kB) q   8  17  26  31  40  49 58%  67%  76  85  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 MB 18.49  x   xInstalling yast2-bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  7  23  8  34  9  ±±45  50 5%  61%  7  72  7  82  7  92  7  100  bootloader (download size 353.00 kB)bootloader-2.17.99-3.3.x86_64.rpm (installed size 2.37 MB) Additional rpm output: Updating etc/sysconfig/bootloader...  6.12  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap-client (download size 109.00 kB)  Additional rpm output: Updating etc/sysconfig/bootloader...   Downloading yast2-ldap-client (download size 109.00 kB)  14  29  ±±±44  58%  73%  88  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581.0 6.120  x   xInstalling yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  1  27  38  ±±±44  55%  68%  73  87  93  9  100  Downloading yast2-ldap-client (download size 109.00 kB) Installing yast2-ldap-client-2.17.39-12.1.noarch.rpm (installed size 581 ~.00 kB) Additional rpm output: Updating etc/sysconfig/ldap...  5.55  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kdump (download size 58.00 kB)  Additional rpm output: Updating etc/sysconfig/ldap...   Downloading yast2-kdump (download size 58.00 kB)  27  55%  83%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kdump-2.17.30-8.1.x86_64.rpm (installed size 331.00 kB) 5.55  x   xInstalling yast2-kdump-2.17.30-8.1.x86_64.rpm (installed size 331.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  30  48 59%  65  74  8  98  100  22  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-users (download size 214.00 kB)   x   xDownloading yast2-users (download size 214.00 kB) q   7  14  22  9  37  ±±±44  52% 9%  67  74  82  9  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-users-2.17.56-3.6.x86_64.rpm (installed size 1.06 MB) 22  x   xInstalling yast2-users-2.17.56-3.6.x86_64.rpm (installed size 1.06 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  21  8  35  40  ±46  53%  8%  65%  73  8  83  8  93  8  100  4.16  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 14.16 MBx 23x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 13.71 MBx22x v http-10.9.0.10-e6784b1axxx tu Downloading yast2-nis-server (download size 42.00 kB)   x   xDownloading yast2-nis-server (download size 42.00 kB) q   38  76%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-nis-server-2.17.3-0.7.130.noarch.rpm (installed size 159. 4.162  x   xInstalling yast2-nis-server-2.17.3-0.7.130.noarch.rpm (installed size 15q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  7  23  9  35  67%  85  99  100  55 1 01  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-http-server (download size 111.00 kB)   x   x~9.00 kB) x    xDownloading yast2-http-server (download size 111.00 kB) q   14  28  ±±±±43  57%  71%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-http-server-2.17.17-1.99.noarch.rpm (installed size 672.0 0112  x   xInstalling yast2-http-server-2.17.17-1.99.noarch.rpm (installed size 672q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  1  29  38  ±±±44  53%  8%  65%  72  7  89  97  100  2.90 0 3.35  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-tune (download size 79.00 kB)   x   x~.00 kB) x    xDownloading yast2-tune (download size 79.00 kB) q   2  4  60%  81  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-tune-2.17.7-1.28.x86_64.rpm (installed size 449.00 kB) 3.35  x   xInstalling yast2-tune-2.17.7-1.28.x86_64.rpm (installed size 449.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  20  8  ±±±44  59%  65  70  7  82  7  95  100  46 19 2.91  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-restore (download size 65.00 kB)   x   xDownloading yast2-restore (download size 65.00 kB) q   24  49 73%  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-restore-2.17.7-0.5.1.noarch.rpm (installed size 371.00 kB 2.91  x   xInstalling yast2-restore-2.17.7-0.5.1.noarch.rpm (installed size 371.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  37  50 65%  70  8  8  99  100  09 8 55  19 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autoyast2-installation (download size 231.00 kB)   x   x~kB) x    xDownloading autoyast2-installation (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  90  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autoyast2-installation-2.17.76-1.20.noarch.rpm (installed size 55  x   xInstalling autoyast2-installation-2.17.76-1.20.noarch.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  21  6  32  7  ±±±44  49 54%  61%  7  73  81  7  93  8  100  0.66 7 1.12  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-kerberos-server (download size 46.00 kB)   x   x~e 1.43 MB) x    xDownloading yast2-kerberos-server (download size 46.00 kB) q   34  69%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-kerberos-server-2.17.8-0.6.2.noarch.rpm (installed size 2 1.12  x   xInstalling yast2-kerberos-server-2.17.8-0.6.2.noarch.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  13  34  ±±±±42  ±47  76%  81  9  94  9  100  43 6 0.88  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dns-server (download size 134.00 kB)   x   x~ 240.00 kB) x    xDownloading yast2-dns-server (download size 134.00 kB) q   11  23  35  ±47  59%  71  83  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dns-server-2.17.24-1.2.noarch.rpm (installed size 757.00 0.88  x   xInstalling yast2-dns-server-2.17.24-1.2.noarch.rpm (installed size 757.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  22  7  36  ±47  56%  64%  72  8  83  90  7  100   9.69 5 15  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-profile-manager (download size 116.00 kB)   x   x~0 kB) x    xDownloading yast2-profile-manager (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-profile-manager-2.17.2-0.7.5.x86_64.rpm (installed size 4 15  x   xInstalling yast2-profile-manager-2.17.2-0.7.5.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  8  24  3  40  ±46  52% 7%  70%  5  80  5  90  6  100  22 4  9.68  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ldap-server (download size 295.00 kB)   x   x~ 476.00 kB) x    xDownloading yast2-ldap-server (download size 295.00 kB) q   5  10  6  21  7  32  7  ±±±±43  48 54%  9%  65  70  6  81  6  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ldap-server-2.17.44-0.5.1.x86_64.rpm (installed size 1.15 9.68 MB / 00:12)   x   xInstalling yast2-ldap-server-2.17.44-0.5.1.x86_64.rpm (installed size 1.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  7  33  8  ±±±44  50 6%  63%  9  75  82  9  7  100  8.07 3 8.53  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-sudo (download size 38.00 kB)   x   x~15 MB) x    xDownloading yast2-sudo (download size 38.00 kB) q   ±±±±42  85%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-sudo-2.17.3-1.53.noarch.rpm (installed size 164.00 kB) 8.531  x   xInstalling yast2-sudo-2.17.3-1.53.noarch.rpm (installed size 164.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  14  9  2  36  58%  65%  77  82  7  92  8  100  7.91 2 37  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-samba-server (download size 75.00 kB)   x   xDownloading yast2-samba-server (download size 75.00 kB) q   21  ±±±±42  63%  85  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-samba-server-2.18.0-1.87.noarch.rpm (installed size 333.0 37  x   xInstalling yast2-samba-server-2.18.0-1.87.noarch.rpm (installed size 333q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  25  33  8  ±±±±43  50 5%  61%  6  74  80  6  91  6  100  59 1 04  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-mail (download size 134.00 kB)   x   x~.00 kB) x    xDownloading yast2-mail (download size 134.00 kB) q   11  23  35  ±47  59%  71  83  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kB) 04  x   xInstalling yast2-mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  21  8  39  ±±±44  49 61%  8  73  9  85  90  7  100  mail (download size 134.00 kB) mail-2.17.7-0.15.1.noarch.rpm (installed size 743.00 kB ) Additional rpm output: Updating etc/sysconfig/mail...  6.86 0 7.32  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-inetd (download size 44.00 kB)  Additional rpm output: Updating etc/sysconfig/mail...   Downloading yast2-inetd (download size 44.00 kB)  35  72%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-inetd-2.17.3-1.53.noarch.rpm (installed size 208.00 kB) 7.32  x   xInstalling yast2-inetd-2.17.3-1.53.noarch.rpm (installed size 208.00 kB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  24  38  ±46  78%  8  99  100  6  9 1  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-installation (download size 258.00 kB)   x   xDownloading yast2-installation (download size 258.00 kB) q   6  12  8  24  30  7  ±±±±43  49 55%  61%  8  74  80  6  92  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-installation-2.17.114-1.14.noarch.rpm (installed size 775 1  x   xInstalling yast2-installation-2.17.114-1.14.noarch.rpm (installed size 7q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  23  31  9  ±47  52% 9%  65  70  6  84  9  94  9  100  Downloading yast2-installation (download size 258.00 kB) Installing yast2-installation-2.17.114-1.14.noarch.rpm (installed size 7 ~75.00 kB) Additional rpm output: Updating etc/sysconfig/security...  5.90 8 6.36   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dhcp-server (download size 139.00 kB)  Additional rpm output: Updating etc/sysconfig/security...   Downloading yast2-dhcp-server (download size 139.00 kB)  11  22  34  ±46  57%  69%  80  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dhcp-server-2.17.6-0.4.2.21.noarch.rpm (installed size 92 6.36  x   xInstalling yast2-dhcp-server-2.17.6-0.4.2.21.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  5  20  7  35  ±±±±42  49 56%  63%  70  7  82  7  95  100  0 7 5.4  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-ft (download size 70.00 kB)   x   x~920.00 kB) x    xDownloading yast2-ftp-server (download size 70.00 kB) q   2  ±±±±43  66%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-ftp-server-2.17.9-0.5.2.noarch.rpm (installed size 534.00 5.4  x   xInstalling yast2-ftp-server-2.17.9-0.5.2.noarch.rpm (installed size 534.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  14  26  33  ±±45  53%  9%  6  76  8  98  100  4.48 6 4.94  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx w Total x 4.94 MBx 7x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.48 MBx6x v http-10.9.0.10-e6784b1axxx tu Downloading yast2-update (download size 95.00 kB)   x   x~00 kB) x    xDownloading yast2-update (download size 95.00 kB) q   14  20  37  ±±±44  49 65%  82  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-update-2.17.27-4.12.x86_64.rpm (installed size 467.00 kB) 4.9420  x   xInstalling yast2-update-2.17.27-4.12.x86_64.rpm (installed size 467.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  26  40  52% 64%  72  7  8  92  7  100  SizexPackagesx 4.94 MBx 7x x x 4.48 MBx 6x x x qqu 48 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesx Total x 4.48 MBx 6x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.48 MBx6x tu Downloading yast2-repair (download size 263.00 kB)   x   x~B) x    xDownloading yast2-repair (download size 263.00 kB) q   5  11  6  21  7  34  40  ±46  52% 9%  65  71  6  82  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-repair-2.17.13-1.98.noarch.rpm (installed size 1.78 MB) 481  x   xInstalling yast2-repair-2.17.13-1.98.noarch.rpm (installed size 1.78 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  5  30  5  ±±±±42  ±47  52% 8%  63%  8  73  9  84  9  96  100  2.71 5 SizexPackagesx Ti2.71 MBx 6x 00:x x 2.71 MBx 5x 00:qqqu 5 Time 00:00 00:00 qq ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-dirinstall (download size 24.00 kB)   x   xDownloading yast2-dirinstall (download size 24.00 kB) q   68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-dirinstall-2.17.5-0.7.83.noarch.rpm (installed size 73.00 2.71 MB)   x   xInstalling yast2-dirinstall-2.17.5-0.7.83.noarch.rpm (installed size 73.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  14  22  9  40  ±±45  69%  76  82  8  94  9  100  64 4 64 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-backup (download size 166.00 kB)   x   x~00 kB) x    xDownloading yast2-backup (download size 166.00 kB) q   8  1  27  3  ±4  56%  66%  7  85  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-backup-2.17.14-0.5.1.noarch.rpm (installed size 689.00 kB 64  x   xInstalling yast2-backup-2.17.14-0.5.1.noarch.rpm (installed size 689.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  2  9  38  ±47  55%  65%  72  8  83  8  99  100  1.96 3 1.96 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-add-on (download size 50.00 kB)   x   x~kB) x    xDownloading yast2-add-on (download size 50.00 kB) q   28  60%  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-add-on-2.17.31-0.7.49.noarch.rpm (installed size 154.00 k 1.96  x   xInstalling yast2-add-on-2.17.31-0.7.49.noarch.rpm (installed size 154.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  54%  69%  76  81  8  95  100  81 2 81 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading autoyast2 (download size 481.00 kB)   x   x~ kB) x    xDownloading autoyast2 (download size 481.00 kB) q   5  12  8  25  32  8  ±±45  52% 8%  65%  72  8  85  92  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing autoyast2-2.17.76-1.20.noarch.rpm (installed size 1.56 MB) 81  x   xInstalling autoyast2-2.17.76-1.20.noarch.rpm (installed size 1.56 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±46  52% 9%  64  9  74  81  6  91  6  100   x   xAdditional rpm output: x    xUpdating etc/sysconfig/autoinstall... q   SizexPackagesx Tim 1.81 MBx 2x00:0 x x260.00 kBx 2x00:0 tu 1 260.00 k 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Downloading yast2-wagon (download size 63.00 kB)  Additional rpm output: Updating etc/sysconfig/autoinstall...   Downloading yast2-wagon (download size 63.00 kB)  25  50 76%  100  8  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 kB 260.00 kB)  x   xInstalling yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  51 66%  73  9  84  9  94  9  100   Done.   SizexPackagesx Time  x 1x00:00 x x Done.x x00:00 qq   9   Perform Installation  lq xMedia x SizexPackagesx Time xTotal x x x00:00 xSUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109x x x xMedium 1 xDone.x x00:00 x mq lActions performed:q xUpdating etc/sysconfig/autoinstall... x x xDownloading yast2-wagon (download size 63.00 kB) xInstalling yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 mq Installing yast2-wagon-2.17.38-0.7.29.noarch.rpm (installed size 260.00 kB)  Installing Packages... (Remaining: 260.00 kB)   F10 Next 8 Back  F9 Abort  F10 Next  Finishing Basic Installation   Finishing Basic Installation                Preparing ation finish-scripts...            [Help] [Back] [Abort] [Next]9 Abort   - Copy files to installed system - Save configuration - Install boot manager - Save installation settings - Prepare system for initial boot ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  => Executing autoinstall scripts in the installation environment...  2  4 Copying files to installed system...   7 system files to the installed system...  9 Moving to installed system...   12  x   => Setting up linker cache   4  7 aving default runlevel...  9 Initializing default window manager...  21 fonts...   4 Saving file system configuration...  6 iSCSI configuration...   9 fcoe configuration...   31 Updating kernel module dependencies...  4 Copying X Window System configuration into system...  6 Saving proxy configuration...   9 the package manager configuration...  ±±±±±41 Enabling random number generator...   ±±±43 Saving system settings...   ±46  x  => bootloader configuration...  48 kdump configuration...  51 x  => Writing YaST C 3%  Saving network configuration... 6%  hardware configuration... 8%  Writing Users Configuration...  60  Executing autoinstall scripts in the installation environment... 3  Writing automatic configuration...  5  x  => 8  Checking the installed system...  70  opying log files to installed system... 3  Unmounting all mounted devices...  5  x  100    Finished.  F10 Next 8 Back  F9 Abort  F10 Next [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxFinishing the installation...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj                            [?1l>[?1l%G[ 510.036121] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-94-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Jan 26 12:20:59 UTC 2017 (c499ea8) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37755000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa01, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72192 r8192 d22016 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a01, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910912k/2097140k available (4569k kernel code, 460k absent, 185768k reserved, 7578k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008240] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.014493] Security Framework initialized [ 0.015706] AppArmor: AppArmor initialized [ 0.016170] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.020264] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.022793] Mount-cache hash table entries: 256 [ 0.024313] Initializing cgroup subsys cpuacct [ 0.025726] Initializing cgroup subsys memory [ 0.026604] Initializing cgroup subsys devices [ 0.027457] Initializing cgroup subsys freezer [ 0.028007] Initializing cgroup subsys net_cls [ 0.028866] Initializing cgroup subsys blkio [ 0.029715] Initializing cgroup subsys perf_event [ 0.030724] Disabled fast string operations [ 0.032020] mce: CPU supports 10 MCE banks [ 0.035208] ACPI: Core revision 20110413 [ 0.037613] Enabling x2apic [ 0.038318] Enabled x2apic [ 0.039179] Switched APIC routing to physical x2apic. [ 0.041119] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.042153] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.060011] kvm-clock: cpu 1, msr 0:7fd10a01, secondary cpu clock [ 0.060027] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.060056] Brought up 2 CPUs [ 0.060057] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.067648] devtmpfs: initialized [ 0.070014] print_constraints: dummy: [ 0.070014] Time: 19:12:58 Date: 04/04/17 [ 0.072153] NET: Registered protocol family 16 [ 0.073152] ACPI: bus type pci registered [ 0.074033] PCI: Using configuration type 1 for base access [ 0.076346] bio: create slab at 0 [ 0.078303] ACPI: Interpreter enabled [ 0.080008] ACPI: (supports S0 S5) [ 0.080871] ACPI: Using IOAPIC for interrupt routing [ 0.085726] ACPI: No dock devices found. [ 0.086541] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.088027] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.090371] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.092015] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.118726] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.125603] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.126951] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.128076] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.129394] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.130686] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.132179] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.133726] vgaarb: loaded [ 0.134655] PCI: Using ACPI for IRQ routing [ 0.134655] NetLabel: Initializing [ 0.136014] NetLabel: domain hash size = 128 [ 0.136850] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.137763] NetLabel: unlabeled traffic allowed by default [ 0.138742] Switching to clocksource kvm-clock [ 0.142151] AppArmor: AppArmor Filesystem Enabled [ 0.143088] pnp: PnP ACPI init [ 0.143827] ACPI: bus type pnp registered [ 0.145339] pnp: PnP ACPI: found 6 devices [ 0.146166] ACPI: ACPI bus type pnp unregistered [ 0.153261] NET: Registered protocol family 2 [ 0.154291] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.156297] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.159611] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.161140] TCP: Hash tables configured (established 262144 bind 65536) [ 0.162215] TCP reno registered [ 0.162927] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.163946] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.165193] NET: Registered protocol family 1 [ 0.166060] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.167077] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.168069] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.169220] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.170216] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.175108] pci 0000:00:01.2: PCI INT D disabled [ 0.176093] Unpacking initramfs... [ 0.310888] Freeing initrd memory: 8812k freed [ 0.325028] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.326115] Initialise module verification [ 0.327086] audit: initializing netlink socket (disabled) [ 0.328060] type=2000 audit(1491333179.328:1): initialized [ 0.346003] bounce pool size: 64 pages [ 0.346828] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.348741] VFS: Disk quotas dquot_6.5.2 [ 0.349617] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.350849] msgmni has been set to 937 [ 0.352107] alg: No test for stdrng (krng) [ 0.352937] Asymmetric key parser 'x509' registered [ 0.353905] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.355470] io scheduler noop registered [ 0.356264] io scheduler deadline registered [ 0.357180] io scheduler cfq registered (default) [ 0.358308] GHES: HEST is not enabled! [ 0.359139] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.360340] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.601228] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.637095] Non-volatile memory driver v1.3 [ 0.637942] Linux agpgart interface v0.103 [ 0.638835] Fixed MDIO Bus: probed [ 0.639625] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.641715] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.642654] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.643634] mousedev: PS/2 mouse device common for all mice [ 0.644824] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.646754] cpuidle: using governor ladder [ 0.647576] cpuidle: using governor menu [ 0.648380] EFI Variables Facility v0.08 2004-May-17 [ 0.649480] TCP cubic registered [ 0.650207] Registering the dns_resolver key type [ 0.651195] Loading module verification certificates [ 0.652138] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.653961] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.655800] registered taskstats version 1 [ 0.656809] Magic number: 13:470:242 [ 0.659599] Freeing unused kernel memory: 1360k freed [ 0.661188] Write protecting the kernel read-only data: 10240k [ 0.665222] Freeing unused kernel memory: 1556k freed [ 0.668821] Freeing unused kernel memory: 900k freed doing fast boot [ 0.710060] SCSI subsystem initialized [ 0.734261] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.742697] scsi0 : ata_piix [ 0.743530] scsi1 : ata_piix [ 0.744249] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.745367] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.912743] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.938913] vda: vda1 vda2 [ 0.940180] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.941197] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.947612] rdac: device handler registered [ 0.952428] hp_sw: device handler registered [ 0.957154] emc: device handler registered [ 0.961976] alua: device handler registered Creating device nodes with udev [ 0.965690] udev: starting version 147 [ 0.985524] usbcore: registered new interface driver usbfs [ 0.987178] usbcore: registered new interface driver hub [ 1.008227] usbcore: registered new device driver usb [ 1.023958] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.026887] uhci_hcd: USB Universal Host Controller Interface driver [ 1.029116] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.031370] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.032327] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.034956] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.036938] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.038094] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.039597] usb usb1: Product: UHCI Host Controller [ 1.040516] usb usb1: Manufacturer: Linux 3.0.101-94-default uhci_hcd [ 1.041604] usb usb1: SerialNumber: 0000:00:01.2 [ 1.042579] hub 1-0:1.0: USB hub found [ 1.043494] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:13:00 2017 [ 1.074755] [drm] Initialized drm 1.1.0 20060810 [ 1.077601] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-94-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.308363] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 59231/1310720 files, 465589/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.340794] kjournald starting. Commit interval 15 seconds [ 1.345041] Refined TSC clocksource calibration: 2593.642 MHz. [ 1.373829] EXT3-fs (vda1): using internal journal [ 1.374750] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:13:00 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.585676] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.631702] input: PC Speaker as /devices/platform/pcspkr/input/input2 [ 1.639150] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.641445] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.643064] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.647280] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3 [ 1.649585] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.652385] ACPI: Power Button [PWRF] [ 1.667003] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.671089] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.686787] rtc_cmos 00:01: RTC can wake from S4 [ 1.687856] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.688988] rtc0: alarms up to one day, 114 bytes nvram [ 1.710157] FDC 0 is a S82078B [ 1.719353] NET: Registered protocol family 10 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 1.825307] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.327661] device-mapper: uevent: version 1.0.3 [ 2.328692] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.397414] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25h[?25h[?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25h[?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'linux'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump : No kdump initial ramdisk found.Tried to locate /boot/initrd-3.0.101-94-default-kdump skipped [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle boot.kdump System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25hStarting syslog services7[?25l done8[?25h unicode_start skipped on /dev/ttyS0 [?25lProbing connected terminal... 78 Initializing virtual console...  H H H H H H H H H [?25h Found an unknown terminal on /dev/ttyS0 (80 columns x 24 lines). Starting D-Bus daemon7[?25l done8[?25h Starting acpid 7[?25l done8[?25h Loading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h Setting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h 0 *** Starting YaST2 *** (B)0[?7h[?1h=[?1h=[?1h= YaST2 - installation @ trevis-41vm7   Initializing ...                     [Help] [Back] [Abort] [Next]                                           [Next]   [Next] [Help]      F1 Help  F9 Abort  Initializing...  Preparing the 1st system configuration...  YaST2 - installation @ trevis-41vm7    Initializing...                             Preparing the 1st system configuration...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort  YaST2 - installation @ trevis-41vm7    Initializing...                             Preparing the 1st system configuration...                              [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort [Help] 8 Back  F9 Abort  F10 Next [Help] [Help] [Next] [Help][Next] [?1h=    ...                                        [Help]                                           [Next]    - Initialize the Target System - Read Installed Packages±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Loading the Package Manager... [Next] [Help] Loading the Package Manager...  =>  x   =>  x   100%     Initializing...    Preparing the 1st system configuration...  [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxInitializing repositories...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqj ing the 1st system configuratio  [?1h=    ...                                        [Help]                                           [Next]    - Load Sources - Refresh Sources - Rebuild Cache - Load Data±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Loading the Package Manager... [Next] [Help] Loading the Package Manager...  =>  x   =>  x  =>  4  25 x  =>  50 100%  x     Initializing...      Preparing the 1st system configuration... - Install required packages ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Preparing System for Automatic Installation  => Checking for required packages...  7 packages required for firewall...  15 host...   23 iscsi-client...  30 keyboard...   8 lan...   ±46 proxy...  53%  runlevel... 61%  timezone 9  users...  76  yast-language... 84  Adding found...  92  Running scripts...  100   x     9 Abort  F10 Next  8 Back  F9 Abort  F10 Next  ackage Installation     9 Abort   erform  lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx xx xx xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± 8 Back  F9 Abort  F10 Next  ackage                                9 Abort   erform  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xMediaxSizexPackagesxTimex xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj lActions performed:qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk xx xx xx xx xx mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Installing... ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing...  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  Packages...  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu Downloading desktop-data-SLED (download size 21.29 MB) Downloading desktop-data-SLED (download size 21.29 MB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sound-theme-freedesktop (download size 734.00 kB) Downloading sound-theme-freedesktop (download size 734.00 kB)  6  12  8  23  9  34  40  ±±45  51 8%  64%  9  75  82  8  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± audit (download size 227.00 kB)  Downloading audit (download size 227.00 kB)  6  14  21  8  35  ±±±±42  49 56%  63%  70  7  84  91  8  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± -libs-32bit (download size 72.00 kB) Downloading audit-libs-32bit (download size 72.00 kB)  22  ±±±44  66%  89  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse (download size 63.00 kB)   w   xDownloading fuse (download size 63.00 kB) v   25  50 75%  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines (download size 268.00 kB)  x w x   xDownloading gtk2-engines (download size 268.00 kB) v   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± keyutils-libs-32bit (download size 7.00 kB)  x w   xDownloading keyutils-libs-32bit (download size 7.00 kB) v   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libbluetooth3 (download size 92.00 kB)   x w   xDownloading libbluetooth3 (download size 92.00 kB) v   17  34  51 69%  86  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cdio7 (download size 52.00 kB)   x w   xDownloading libcdio7 (download size 52.00 kB) v   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± db-4_5-32bit (download size 480.00 kB)  x w   xDownloading libdb-4_5-32bit (download size 480.00 kB) v   6  13  9  26  33  40  ±46  53%  60%  6  73  80  6  93  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± exif (download size 194.00 kB)   x w   xDownloading libexif (download size 194.00 kB) v   8  16  24  33  ±±±±±41  49 57%  66%  74  82  91  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse2 (download size 61  x w   xDownloading libfuse2 (download size 61.00 kB) v   26  52% 79%  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± idl (download size 94.00 kB)   x w   xDownloading libidl (download size 94.00 kB) v   16  33  51 68%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jpeg-32bit (download size 67.00 kB)  x w   xDownloading libjpeg-32bit (download size 67.00 kB) v   23  ±47  71%  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pixman-1-0-32bit (download size 200.00 kB)  x x   xDownloading libpixman-1-0-32bit (download size 200.00 kB) q   7  15  23  32  40  48 56%  64%  72  80  8  96  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± roxy0 (download size 25.00 kB)   x   xDownloading libproxy0 (download size 25.00 kB) q   64%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± selinux1-32bit (download size 52.00 kB)  x   xDownloading libselinux1-32bit (download size 52.00 kB) q   3  62%  93  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± uuid1-32bit (download size 37.00 kB)   x   xDownloading libuuid1-32bit (download size 37.00 kB) q   ±±±±42  86%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcrypt-32bit (download size 34.00 kB)  x   xDownloading libxcrypt-32bit (download size 34.00 kB) q   ±46  93%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± python-argparse42  x   xDownloading python-argparse (download size 42.00 kB) q   38  76%  100  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libICE-32bit (download size 41.00 kB)  x   xDownloading xorg-x11-libICE-32bit (download size 41.00 kB) q   38  78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xau8.00 kB)   x   xDownloading xorg-x11-libXau-32bit (download size 8.00 kB) q   100%   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-theme-SLED (download size 502.00 kB)   x   xDownloading gtk2-theme-SLED (download size 502.00 kB) q   6 - 15.74 kB/s (download size 502.00 kB)  12  9  25  31  8  ±±±44  50 7%  63%  70  6  82  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_cdda0 (download size 20.00 kB)   x   xDownloading libcdio_cdda0 (download size 20.00 kB) q   79%  100   10 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-lang (download size 345.00 kB)  x   xDownloading libgphoto2-lang (download size 345.00 kB) q   9  18  23  32  7  ±46  51 60%  9  74  83  8  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± orbit2 (download size 170.00 kB)   x   xDownloading orbit2 (download size 170.00 kB) q   9  18  2  37  ±46  56%  65%  7  84  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libproxy0-config-gnome (download size 12.00 kB)  x   xDownloading libproxy0-config-gnome (download size 12.00 kB) q   100%   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libSM-32bit (download size 16.00 kB)   x   xDownloading xorg-x11-libSM-32bit (download size 16.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcb-32bit (download size 101.00 kB)  x   xDownloading xorg-x11-libxcb-32bit (download size 101.00 kB) q   15  31  ±47  63%  79  95  100  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-branding-SLED (download size 5.00 kB)   x   xDownloading gtk2-branding-SLED (download size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_paranoia0 (download size 20  x   xDownloading libcdio_paranoia0 (download size 20.00 kB) q   79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2 (download size 705.00 kB)   x   xDownloading libgphoto2 (download size 705.00 kB) q   6  11  8  24  9  34  40  ±±45  52% 9%  65  70  7  83  8  93  9  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu openobex1 (download size 34.00 kB)  x   xDownloading libopenobex1 (download size 34.00 kB) q   ±46  94%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzmadec0 (download size 13.00 kB)   x   xDownloading liblzmadec0 (download size 13.00 kB) q   100%   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± obex-data-server (download size 93.00 kB)  x   xDownloading obex-data-server (download size 93.00 kB) q   17  34  51 69%  86  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzma (download size 58.00 kB)   x   xDownloading lzma (download size 58.00 kB) q   27  55%  83%  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu ibtasn1-3 (download size 57.00 kB)  x   xDownloading libtasn1-3 (download size 57.00 kB) q   27  55%  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssh-helpers (download size 52.00 kB)  x   xDownloading openssh-helpers (download size 52.00 kB) q   3  61%  92  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± readline-doc (download size 138.00 kB)   x   xDownloading readline-doc (download size 138.00 kB) q   11  23  34  ±46  58%  69%  81  93  100  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcares2 (download size 36.00 kB)   x   xDownloading libcares2 (download size 36.00 kB) q   ±±±44  90%  10  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± om_err2-32bit (download size 41.00 kB)  x   xDownloading libcom_err2-32bit (download size 41.00 kB) q   38  77%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnutls26 (download size 283.00 kB)   x   xDownloading libgnutls26 (download size 283.00 kB) q   5  11  6  22  8  33  9  ±±45  50 6%  62%  7  73  9  84  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± expat1-32bit (download size 62.00 kB)  x   xDownloading libexpat1-32bit (download size 62.00 kB) q   25  51 78%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gcc_s56  x   xDownloading libgcc_s1-32bit (download size 56.00 kB) q   28  57%  86%  100   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dbus-1-32bit (download size 116.00 kB)   x   xDownloading dbus-1-32bit (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  82  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libavahi-common3 (download size 34.00 kB)  x   xDownloading libavahi-common3 (download size 34.00 kB) q   ±46  93%  100   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ref_array1 (download size 8.00 kB)   x   xDownloading libref_array1 (download size 8.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhash1 (download size 9.00 kB)   x   xDownloading libdhash1 (download size 9.00 kB) q   100%  20 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bz2-1-32bit (download size 34.00 kB)  x   xDownloading libbz2-1-32bit (download size 34.00 kB) q   ±46  94%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sss_idmap0 (download size 23.00 kB)   x   xDownloading libsss_idmap0 (download size 23.00 kB) q   71%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vim-data (download size 5.49 MB)   x   xDownloading vim-data (download size 5.49 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libwbclient0 (download size 223.00 kB)  x   xDownloading libwbclient0 (download size 223.00 kB) q   7  14  21  8  35  ±±±±43  50 7%  64%  71  9  86  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± augeas-lenses (download size 99  x   xDownloading augeas-lenses (download size 99.00 kB) q   16  32  48 64%  81  97  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtdb1 (download size 231.00 kB)   x   xDownloading libtdb1 (download size 231.00 kB) q   6  13  20  7  34  ±±±±±41  48 55%  62%  9  76  83  9  96  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zlib-32bit (download size 47.00 kB)  x   xDownloading zlib-32bit (download size 47.00 kB) q   33  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcollection2 (download size 21.00 kB)  x   xDownloading libcollection2 (download size 21.00 kB) q   76%  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± archive2 (download size 101.00 kB)   x   xDownloading libarchive2 (download size 101.00 kB) q   15  31  ±47  63%  79  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bash-doc (download size 394.00 kB)   x   xDownloading bash-doc (download size 394.00 kB) q   8  16  24  32  40  48 56%  65%  73  81  9  97  100  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtevent0 (download size 220.00 kB)  x   xDownloading libtevent0 (download size 220.00 kB) q   7  14  21  9  36  ±±±±43  50 8%  65%  72  80  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± perl-doc (download size 5.78 MB)   x   xDownloading perl-doc (download size 5.78 MB) q   5 - 15.74 kB/s (download size 5.78 MB)  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100   5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpath_utils1 (download size 9.00 kB)   x   xDownloading libpath_utils1 (download size 9.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zypper-log (download size 75.00 kB)   x   xDownloading zypper-log (download size 75.00 kB) q   2  ±±±±42  63%  85  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-lang (download size 729.00 kB)  x   xDownloading gconf2-lang (download size 729.00 kB) q   6  13  9  24  30  5  ±±±±±41  ±46  52% 7%  63%  8  74  9  85  90  6  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libX11-32bit (download size 575.00 kB)  x   xDownloading xorg-x11-libX11-32bit (download size 575.00 kB) q   5  11  6  22  7  33  8  ±±±44  50 5%  61%  6  72  7  83  8  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtasn1 (download size 103.00 kB)   x   xDownloading libtasn1 (download size 103.00 kB) q   15  31  ±46  62%  77  93  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± krb5-32bit (download size 497.00 kB)  x   xDownloading krb5-32bit (download size 497.00 kB) q   6  12  9  25  32  8  ±±±44  51 7%  64%  70  7  83  90  6  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libstdc++6-32bit (download size 406.00 kB)  x   xDownloading libstdc++6-32bit (download size 406.00 kB) q   7  15  23  31  9  ±47  55%  63%  70  8  86  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± avahi-glib1 (download size 10.00 kB)   x   xDownloading libavahi-glib1 (download size 10.00 kB) q   100%   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± client3 (download size 38.00 kB)  x   xDownloading libavahi-client3 (download size 38.00 kB) q   ±±±±42  84%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssl0_9_8-32bit (download size 785.00 kB)  x   xDownloading libopenssl0_9_8-32bit (download size 785.00 kB) q   5  10  6  22  8  34  40  ±46  52% 9%  65  71  7  83  9  95  100   9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cracklib-32bit (download size 16.00 kB)   x   xDownloading cracklib-32bit (download size 16.00 kB) q   99%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpng12-0-32bit (download size 75.00 kB)  x   xDownloading libpng12-0-32bit (download size 75.00 kB) q   21  ±±±±42  64%  86  100   30 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xml2-32bit (download size 535.00 kB)   x   xDownloading libxml2-32bit (download size 535.00 kB) q   5  11  7  23  9  35  ±±±±±41  ±47  53%  9%  65  71  7  83  9  95  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± freetype2-32bit (download size 255.00 kB)  x   xDownloading freetype2-32bit (download size 255.00 kB) q   6  12  8  25  31  7  ±±±±43  50 6%  62%  8  75  81  7  94  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libldb1 (download size 310.00 kB)   x   xDownloading libldb1 (download size 310.00 kB) q   5  10  5  20  5  30  6  ±±±±±41  ±46  51 6%  61%  7  72  7  82  7  92  8  100  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± smbclient0 (download size 1.69 MB)  x   xDownloading libsmbclient0 (download size 1.69 MB) q   5  10  5  20  5  30 - 458.31 kB/s (download size 1.69 MB)  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ini_config2 (download size 17.00 kB)   x   xDownloading libini_config2 (download size 17.00 kB) q   97%  100   2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2 (download size 273.00 kB)   x   xDownloading gconf2 (download size 273.00 kB) q   5  11  7  23  9  35  40  ±46  52% 8%  64%  70  6  82  7  93  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXfixes-32bit (download size 10.00 kB)  x   xDownloading xorg-x11-libXfixes-32bit (download size 10.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pm-32bit (download size 34.00 kB)   x   xDownloading xorg-x11-libXpm-32bit (download size 34.00 kB) q   ±47  94%  100   3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± render-32bit (download size 18.00 kB)  x   xDownloading xorg-x11-libXrender-32bit (download size 18.00 kB) q   89%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtiff3-32bit (download size 116.00 kB)   x   xDownloading libtiff3-32bit (download size 116.00 kB) q   13  27  ±±±±±41  55%  69%  83  97  100   4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pam-32bit (download size 174.00 kB)   x   xDownloading pam-32bit (download size 174.00 kB) q   9  18  27  36  ±±45  55%  64%  73  82  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fontconfig-32bit (download size 102.00 kB)  x   xDownloading fontconfig-32bit (download size 102.00 kB) q   15  31  ±46  62%  78  94  100  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd (download size 1.80 MB)   x   xDownloading sssd (download size 1.80 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu xorg-x11-libfontenc-32bit (download size 14.00 kB)  x   xDownloading xorg-x11-libfontenc-32bit (download size 14.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xkbfile6  x   xDownloading xorg-x11-libxkbfile-32bit (download size 64.00 kB) q   24  50 75%  100   6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xt-32bit (download size 148.00 kB)   x   xDownloading xorg-x11-libXt-32bit (download size 148.00 kB) q   1  21  32  ±±±±43  54%  64%  75  86  97  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ext-32bit (download size 26.00 kB)  x   xDownloading xorg-x11-libXext-32bit (download size 26.00 kB) q   62%  100   7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pcre-32bit (download size 98.00 kB)   x   xDownloading pcre-32bit (download size 98.00 kB) q   16  32  48 65%  81  98  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-lang (download size 301.00 kB)  x   xDownloading gnome-keyring-lang (download size 301.00 kB) q   5  10  5  21  6  31  7  ±±±±42  ±47  53%  8%  63%  9  74  9  85  90  5  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± conf2-branding-SLES (download size 101.00 kB)  x   xDownloading gconf2-branding-SLES (download size 101.00 kB) q   15  31  ±47  63%  79  95  100   8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cups-libs-32bit (download size 192.00 kB)   x   xDownloading cups-libs-32bit (download size 192.00 kB) q   8  16  24  33  ±±±±±41  49 58%  66%  75  83  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-32bit (download size 35.00 kB)   x   xDownloading sssd-32bit (download size 35.00 kB) q   ±±45  92%  100  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ConsoleKit-32bit (download size 15.00 kB)  x   xDownloading ConsoleKit-32bit (download size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXp-32bit (download size 16.00 kB)  x   xDownloading xorg-x11-libXp-32bit (download size 16.00 kB) q   100%   40 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mu-32bit (download size 47.00 kB)  x   xDownloading xorg-x11-libXmu-32bit (download size 47.00 kB) q   34  68%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring (download size 558.00 kB)   x   xDownloading gnome-keyring (download size 558.00 kB) q   5  11  7  22  8  34  40  ±±45  51 7%  63%  8  74  80  5  91  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXprintUtil-32bit (download size 16.00 kB)  x   xDownloading xorg-x11-libXprintUtil-32bit (download size 16.00 kB) q   98%  100   1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs (download size 176.00 kB)   x   xDownloading gvfs (download size 176.00 kB) q   8  1  27  36  ±±45  54%  63%  72  81  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgvfscommon0 (download size 54.00 kB)  x   xDownloading libgvfscommon0 (download size 54.00 kB) q   29  58%  8  100   ±±±±42 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-lang (download size 343.00 kB)   x   xDownloading gvfs-lang (download size 343.00 kB) q   9  18  23  32  7  ±46  51 60%  5  70  9  84  93  8  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse26.00 kB)   x   xDownloading gvfs-fuse (download size 26.00 kB) q   62%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu xorg-x11-libXv-32bit (download size 11.00 kB)  x   xDownloading xorg-x11-libXv-32bit (download size 11.00 kB) q   100%  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libglib-2_0-0-32bit (download size 294  x   xDownloading libglib-2_0-0-32bit (download size 294.00 kB) q   5  10  6  21  7  32  8  ±±±±43  48 54%  9%  65  70  6  81  7  92  7  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± soup-2_4-1 (download size 176.00 kB)   x   xDownloading libsoup-2_4-1 (download size 176.00 kB) q   8  1  27  36  ±±45  54%  63%  72  81  90  10   ±±±44 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libs-32bit (download size 668.00 kB)  x   xDownloading xorg-x11-libs-32bit (download size 668.00 kB) q   6  11  6  21  8  33  40  ±±45  52% 7%  64%  9  76  81  8  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgmodule-2_0-0-32bit (download size 39.00 kB)  x   xDownloading libgmodule-2_0-0-32bit (download size 39.00 kB) q   4  81%  100   ±±45 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± thread40  x   xDownloading libgthread-2_0-0-32bit (download size 40.00 kB) q   39  79%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± object123.00 kB)  x   xDownloading libgobject-2_0-0-32bit (download size 123.00 kB) q   12  25  39  52% 65%  78  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± io-2_0-0-32bit (download size 242.00 kB)   x   xDownloading libgio-2_0-0-32bit (download size 242.00 kB) q   6  13  9  26  33  9  ±46  52% 9%  66  72  9  85  92  9  100   ±46 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu croco-0_6-3-32bit (download size 77.00 kB)  x   xDownloading libcroco-0_6-3-32bit (download size 77.00 kB) q   2  ±±±±±41  62%  83  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-backends (download size 532.00 kB)   x   xDownloading gvfs-backends (download size 532.00 kB) q   5  11  8  24  30  6  ±±±±42  48 54%  60%  6  72  8  84  90  6  100  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cairo-32bit (download size 224.00 kB)   x   xDownloading cairo-32bit (download size 224.00 kB) q   7  14  21  8  35  ±±±±42  50 7%  64%  71  8  85  93  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libatk-1_0-0-32bit (download size 39.00 kB)  x   xDownloading libatk-1_0-0-32bit (download size 39.00 kB) q   4  81%  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gsf-1-11492  x   xDownloading libgsf-1-114-32bit (download size 92.00 kB) q   17  34  52% 69%  87  100   48 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu pango-32bit (download size 272.00 kB)   x   xDownloading pango-32bit (download size 272.00 kB) q   5 - 15.74 kB/s (download size 272.00 kB)  11  7  23  9  35  ±±±±±41  ±46  52% 8%  64%  70  6  82  8  94  9  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-32bit (download size 1.93 MB)   x   xDownloading gtk2-32bit (download size 1.93 MB) q   5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 129.93 MBx 120x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 55.39 MBx57x http-10.9.0.10-6de35489xxx tu librsvg-32bit (download size 105.00 kB)  x   xDownloading librsvg-32bit (download size 105.00 kB) q   15  30  ±±45  60%  76  91  100  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-32bit (download size 156.00 kB)  x   xDownloading gtk2-engines-32bit (download size 156.00 kB) q   1  2  3  4  51 61%  7  82  9  100  50 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Installing desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.50 (Remaining: 129.93 MB)  x   xInstalling desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  gtk2-engines-32bit (download size 156.00 kB) Installing desktop-data-SLED-11-29.22.58.noarch.rpm (installed size 24.5 ~0 MB) Additional rpm output: Updating etc/sysconfig/windowmanager...  xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 105.43 MBx 119x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 30.89 MBx56x http-10.9.0.10-6de35489xxx tu 9%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 882 05.4 Additional rpm output: Updating etc/sysconfig/windowmanager...   Installing sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 8  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  11  7  22  3  9  ±47  57%  64%  9  74  80  6  93  9  100  03 5 4.57  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± audit-1.8-0.34.26.x86_64.rpm (installed size 927.00 kB)  4.57   Installing sound-theme-freedesktop-0.2-1.17.noarch.rpm (installed size 8 ~82.00 kB) audit-1.8-0.34.26.x86_64.rpm (installed size 927.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  22  9  36  ±47  54%  63%  70  81  6  92  7  100  Updating etc/sysconfig/auditd... insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'  29.12 4 3.66  7 60  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.00 3.66 insserv: Default-Start undefined, assuming default start runlevel(s) for ~ script `munge'   Installing audit-libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.0  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±42  61%  99  100  8.98 3 51  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse-2.8.7-0.11.1.x86_64.rpm (installed size 170.00 kB)  51   Installing audit-libs-32bit-1.8-0.34.26.x86_64.rpm (installed size 152.0 ~0 kB) fuse-2.8.7-0.11.1.x86_64.rpm (installed size 170.00 kB)   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  25  63%  80  9  9  100  setting /usr/bin/wall to root:tty 2755. (wrong permissions 0755) setting /usr/bin/write to root:tty 2755. (wrong permissions 0755) setting /usr/sbin/zypp-refresh-wrapper to root:root 4755. (wrong permiss ions 0755) Finished.  81 2 35  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-2.16.1-4.26.x86_64.rpm (installed size 1.10 MB) 35 ~ions 0755) Finished.   Installing gtk2-engines-2.16.1-4.26.x86_64.rpm (installed size 1.10 MB)  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  9  25  31  6  ±±±±±41  ±46  51 61%  6  72  9  85  94  9  100  7.7 1 2.24  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± keyutils-libs-32bit-1.2-107.29.15.x86_64.rpm (installed size 10 2.24  x   xInstalling keyutils-libs-32bit-1.2-107.29.15.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  97%  100  0 0 3  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libbluetooth3-4.99-0.12.35.x86_64.rpm (installed size 203.00 kB 3  x   x~10.00 kB) x    xInstalling libbluetooth3-4.99-0.12.35.x86_64.rpm (installed size 203.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  31  54%  64%  9  9  100  5 49 0  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cdio7-0.80-6.11.x86_64.rpm (installed size 126.00 kB)  0  x   x~kB) x    xInstalling libcdio7-0.80-6.11.x86_64.rpm (installed size 126.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  51 99%  100  37 8 1.91  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± db-4_5-32bit-4.5.20-95.39.x86_64.rpm (installed size 1.21 MB 1.91  x   xInstalling libdb-4_5-32bit-4.5.20-95.39.x86_64.rpm (installed size 1.21 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  31  6  ±±±±±41  ±46  51 7%  62%  7  72  7  82  8  93  8  100  6.1 7 0.7  0 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± exif-0.6.17-2.14.1.x86_64.rpm (installed size 938.00 kB)  0.7  x   x~MB) x    xInstalling libexif-0.6.17-2.14.1.x86_64.rpm (installed size 938.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  20  5  31  7  ±±±±42  ±47  56%  66%  7  83  8  95  100  5.25 6  99.79  09 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse2-2.8.7-0.1121 99.79 MB)   x   xInstalling libfuse2-2.8.7-0.11.1.x86_64.rpm (installed size 218.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  29  58%  8  99  100  04 5 58  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± idl-0.8.13-1.1.57.x86_64.rpm (installed size 258.00 kB)  58  x   xInstalling libidl-0.8.13-1.1.57.x86_64.rpm (installed size 258.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 67%  7  92  7  100  4.79 4 32  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± jpeg-32bit-6.2.0-879.12.1.x86_64.rpm (installed size 138.00 32  x   xInstalling libjpeg-32bit-6.2.0-879.12.1.x86_64.rpm (installed size 138.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  92%  9  100  65 3 19  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pixman-1-0-32bit-0.24.4-0.15.1.x86_64.rpm (installed size 60 19  x   x~0 kB) x    xInstalling libpixman-1-0-32bit-0.24.4-0.15.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  21  3  ±±±±42  53%  63%  74  85  9  100  06 2 8.60  5 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± roxy0-0.3.1-2.6.1.x86_64.rpm (installed size 66.00 kB)  8.60  x   x~602.00 kB) x    xInstalling libproxy0-0.3.1-2.6.1.x86_64.rpm (installed size 66.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±46  56%  65%  7  84  99  100  0 1 54  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± selinux1-32bit-2.0.91-4.2.1.x86_64.rpm (installed size 115.0 54  x   xInstalling libselinux1-32bit-2.0.91-4.2.1.x86_64.rpm (installed size 115q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  56%  99%  100  3.89 0 42  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± uuid1-32bit-2.19.1-6.72.1.x86_64.rpm (installed size 18.00 k 42  x   x~.00 kB) x    xInstalling libuuid1-32bit-2.19.1-6.72.1.x86_64.rpm (installed size 18.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  7 39 1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcrypt-32bit-3.0.3-0.6.1.x86_64.rpm (installed size 131 1  x   x~ kB) x    xInstalling libxcrypt-32bit-3.0.3-0.6.1.x86_64.rpm (installed size 131.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  2  ±±±±43  59%  76  8  99  100  74 8 28  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± python-argparse-1.2.1-2.5.65.x86_64.rpm (installed size 158.00  28  x   x~ kB) x    xInstalling python-argparse-1.2.1-2.5.65.x86_64.rpm (installed size 158.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  57%  99%  100  59 7 12  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libICE-32bit-7.4-1.15.x86_64.rpm (installed size 95.00 12  x   x~0 kB) x    xInstalling xorg-x11-libICE-32bit-7.4-1.15.x86_64.rpm (installed size 95.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  67%  99  100  0 6 03  99 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xau10 03  x   x~00 kB) x    xInstalling xorg-x11-libXau-32bit-7.4-1.15.x86_64.rpm (installed size 10.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  49 5 2  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-theme-SLED-11.1-1.29.82.noarch.rpm (installed size 623.00  2  x   x~00 kB) x    xInstalling gtk2-theme-SLED-11.1-1.29.82.noarch.rpm (installed size 623.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  2  6  32  7  ±±45  51 8%  63%  8  73  8  84  91  9  100  2.88 4 7.41  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_cdda0-0.80-6.11.x86_64.rpm (installed size 31.00 kB)  7.41  x   x~0 kB) x    xInstalling libcdio_cdda0-0.80-6.11.x86_64.rpm (installed size 31.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  5 3 38  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-lang-2.4.3-3.27.1.x86_64.rpm (installed size 1.64 MB 38  x   xInstalling libgphoto2-lang-2.4.3-3.27.1.x86_64.rpm (installed size 1.64 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  7  22  7  32  7  ±±±±42  ±47  53%  61%  6  71  6  83  9  95  100  1.21 2 5.75  5 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± orbit2-2.14.17-2.7.8.x86_64.rpm (installed size 622.00 kB)  5.75  x   x~MB) x    xInstalling orbit2-2.14.17-2.7.8.x86_64.rpm (installed size 622.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  2  3  ±±±±4  53%  63%  7  80  93  9  100  0.60 1 14  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libproxy0-config-gnome-0.3.1-2.6.3.x86_64.rpm (installed size 1 14  x   xInstalling libproxy0-config-gnome-0.3.1-2.6.3.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  59 0 2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libSM-32bit-7.4-1.18.x86_64.rpm (installed size 30.00  2  x   x~ 15.00 kB) x    xInstalling xorg-x11-libSM-32bit-7.4-1.18.x86_64.rpm (installed size 30.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  6 29 09  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xcb-32bit-7.4-1.29.1.x86_64.rpm (installed size 511 09  x   x~0 kB) x    xInstalling xorg-x11-libxcb-32bit-7.4-1.29.1.x86_64.rpm (installed size 5q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  20  5  31  40  ±46  52% 60%  8  75  90  9  100  0 8 4.60  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-branding-SLED-11.1-1.26.noarch.rpm (installed size 2.00 kB 4.60  x   x~11.00 kB) x    xInstalling gtk2-branding-SLED-11.1-1.26.noarch.rpm (installed size 2.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  6  95%  100  7 59  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcdio_paranoia0-0.80-6.11.x86_64.rpm (installed size 31 59  x   x~kB) x    xInstalling libcdio_paranoia0-0.80-6.11.x86_64.rpm (installed size 31.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3 6 6  89 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gphoto2-2.4.3-3.27.1.x86_64.rpm (installed size 5.96 MB)  6  x   x~kB) x    xInstalling libgphoto2-2.4.3-3.27.1.x86_64.rpm (installed size 5.96 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  61%  6  71  7  82  8  93  8  100  14.06 5 88.60  8 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTw Total x 88.60 MBx 88x v SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x http-10.9.0.10-6de35489xxx tu openobex1-1.5-8.9.7.x86_64.rpm (installed size 86.00 k 88.60  x   xInstalling libopenobex1-1.5-8.9.7.x86_64.rpm (installed size 86.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  65%  9  100  SizexPackagesxTi88.60 MBx 88x x x 14.06 MBx 25x x x qu 51  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzmadec0-4.32.7-1.25.x86_64.rpm (installed size 15.00 kB) 51  x   xInstalling liblzmadec0-4.32.7-1.25.x86_64.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± obex-data-server-0.4.6-2.7.90.x86_64.rpm (installed size 279.00 0  x   xInstalling obex-data-server-0.4.6-2.7.90.x86_64.rpm (installed size 279.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  23  ±46  56%  62%  83  8  99  100  23  5 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± lzma-4.32.7-1.25.x86_64.rpm (installed size 133.00 kB)  23  x   x~00 kB) x    xInstalling lzma-4.32.7-1.25.x86_64.rpm (installed size 133.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  51 77%  85  92  8  100  10  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x  x           qq  xSizexPackagesxTiw Total x 88.10 MBx 84x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x v http-10.9.0.10-e6784b1axxx tu ibtasn1-3-1.5-1.34.1.x86_64.rpm (installed size 144.00 kB) 10  x   xInstalling libtasn1-3-1.5-1.34.1.x86_64.rpm (installed size 144.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±44  57%  81%  99  100  7.96  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssh-helpers-6.6p1-35.1.x86_64.rpm (installed size 66.00 kB) 7.96  x   xInstalling openssh-helpers-6.6p1-35.1.x86_64.rpm (installed size 66.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  91%  7  100  89  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± readline-doc-5.2-147.29.1.x86_64.rpm (installed size 590 89  x   x~B) x    xInstalling readline-doc-5.2-147.29.1.x86_64.rpm (installed size 590.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  1  6  27  38  49 60%  7  81  9  9  100  32  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcares2-1.7.4-7.9.1.x86_64.rpm (installed size 63.00 kB)  32  x   x~B) x    xInstalling libcares2-1.7.4-7.9.1.x86_64.rpm (installed size 63.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  25  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± om_err2-32bit-1.41.9-2.16.1.x86_64.rpm (installed size 36.0 25  x   xInstalling libcom_err2-32bit-1.41.9-2.16.1.x86_64.rpm (installed size 36q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  38  99%  100  2 79 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnutls26-2.4.1-24.39.67.1.x86_64.rpm (installed size 745.00  2  x   x~.00 kB) x    xInstalling libgnutls26-2.4.1-24.39.67.1.x86_64.rpm (installed size 745.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  34  ±±±±43  51 60%  8  77  85  94  9  100  6.49  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± expat1-32bit-2.0.1-88.41.1.x86_64.rpm (installed size 158.00 6.49  x   x~0 kB) x    xInstalling libexpat1-32bit-2.0.1-88.41.1.x86_64.rpm (installed size 158.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  4  81%  99  100  34  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gcc_s5.3.1+r233831-10.1.x86_64.rpm (installed size 1 34  x   x~00 kB) x    xInstalling libgcc_s1-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  60%  99  100  23  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dbus-1-32bit-1.2.10-3.33.1.x86_64.rpm (installed size 259.00 kB 23  x   x~ 106.00 kB) x    xInstalling dbus-1-32bit-1.2.10-3.33.1.x86_64.rpm (installed size 259.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 74%  99  100  5.98  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libavahi-common3-0.6.23-11.34.3.x86_64.rpm (installed size 51.0 5.98  x   x~kB) x    xInstalling libavahi-common3-0.6.23-11.34.3.x86_64.rpm (installed size 51q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  3  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ref_array1-0.1.1-1.9.1.x86_64.rpm (installed size 11.00 kB)  3  x   x~.00 kB) x    xInstalling libref_array1-0.1.1-1.9.1.x86_64.rpm (installed size 11.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  2  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± dhash1-0.4.2-1.9.1.x86_64.rpm (installed size 15.00 kB)  2  x   x~) x    xInstalling libdhash1-0.4.2-1.9.1.x86_64.rpm (installed size 15.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bz2-1-32bit-1.0.5-34.255.1.x86_64.rpm (installed size 65.00 1  x   xInstalling libbz2-1-32bit-1.0.5-34.255.1.x86_64.rpm (installed size 65.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  98%  100  84  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sss_idmap0-1.9.4-0.33.1.x86_64.rpm (installed size 19.00 kB) 84  x   x~0 kB) x    xInstalling libsss_idmap0-1.9.4-0.33.1.x86_64.rpm (installed size 19.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± vim-data-7.2-8.17.1.x86_64.rpm (installed size 19.53 MB)  2  x   x~B) x    xInstalling vim-data-7.2-8.17.1.x86_64.rpm (installed size 19.53 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  66.30 69 74  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libwbclient0-3.6.3-87.1.x86_64.rpm (installed size 51.00 kB) 66.30  x   xInstalling libwbclient0-3.6.3-87.1.x86_64.rpm (installed size 51.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  25  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± augeas-lenses-0.9.0-3.17.2.x86_64.rpm (installed size 293.00 kB 25  x   xInstalling augeas-lenses-0.9.0-3.17.2.x86_64.rpm (installed size 293.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  9  24  30  5  40  ±±45  50 5%  60%  5  70  9  86  91  6  100  5.96  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtdb1-3.6.3-87.1.x86_64.rpm (installed size 67.00 kB)  5.96  x   x~kB) x    xInstalling libtdb1-3.6.3-87.1.x86_64.rpm (installed size 67.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  96%  100  0  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zlib-32bit-1.2.7-0.14.1.x86_64.rpm (installed size 86.00 kB) 0  x   xInstalling zlib-32bit-1.2.7-0.14.1.x86_64.rpm (installed size 86.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  74%  99  100  81  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libcollection2-0.6.0-1.9.1.x86_64.rpm (installed size 47.00 kB) 81  x   xInstalling libcollection2-0.6.0-1.9.1.x86_64.rpm (installed size 47.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  77  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± archive2-2.5.5-9.1.x86_64.rpm (installed size 245.00 kB)  77  x   x~B) x    xInstalling libarchive2-2.5.5-9.1.x86_64.rpm (installed size 245.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  52% 74%  8  91  6  100  53  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± bash-doc-3.2-147.21.29 MB)  53  x   xInstalling bash-doc-3.2-147.29.1.x86_64.rpm (installed size 1.29 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  7  12  21  6  31  7  ±±±±42  51 6%  61%  6  71  80  5  91  9  100  4.2  2 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtevent0-3.6.3-8747.00 kB) 4.2  x   xInstalling libtevent0-3.6.3-87.1.x86_64.rpm (installed size 47.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  19  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± perl-doc-5.10.0-64.80.1.x86_64.rpm (installed size 9.78 MB) 19  x   xInstalling perl-doc-5.10.0-64.80.1.x86_64.rpm (installed size 9.78 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  54.40  0 9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpath_utils1-0.2.1-1.9.1.x86_64.rpm (installed size 15.00 kB) 54.40  x   xInstalling libpath_utils1-0.2.1-1.9.1.x86_64.rpm (installed size 15.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  39 59 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± zypper-log-1.6.334-26.3.x86_64.rpm (installed size 5.00 kB)  39  x   x~B) x    xInstalling zypper-log-1.6.334-26.3.x86_64.rpm (installed size 5.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  87%  9  100  8  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-lang-2.28.0-1.8.1.x86_64.rpm (installed size 5.14 MB) 8  x   xInstalling gconf2-lang-2.28.0-1.8.1.x86_64.rpm (installed size 5.14 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±46  51 6%  61%  7  73  8  83  8  93  8  100  49.25  7 81  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libX11-32bit-7.4-5.11.68.1.x86_64.rpm (installed size 49.25  x   xInstalling xorg-x11-libX11-32bit-7.4-5.11.68.1.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  21  6  31  6  ±±±±42  ±47  52% 7%  62%  7  73  8  83  8  93  9  100  8.04  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtasn1-1.5-1.34.1.x86_64.rpm (installed size 197.00 kB)  8.04  x   x~e 1.20 MB) x    xInstalling libtasn1-1.5-1.34.1.x86_64.rpm (installed size 197.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  31  48 61%  9  80  5  90  5  100  7.85  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± krb5-32bit-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.39 M 7.85  x   xInstalling krb5-32bit-1.6.3-133.49.112.1.x86_64.rpm (installed size 1.39q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  13  8  25  30  6  ±±±±±41  ±46  53%  8%  67%  75  80  9  97  100  6.46  4 2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libstdc++6-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed size  6.46  x   x~ MB) x    xInstalling libstdc++6-32bit-5.3.1+r233831-10.1.x86_64.rpm (installed sizq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  17  25  30  8  ±±±±43  51 6%  64%  73  81  6  94  9  100  5.01  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± avahi-glib1-0.6.23-13.34.8.x86_64.rpm (installed size 15.00 5.01  x   x~e 1.45 MB) x    xInstalling libavahi-glib1-0.6.23-13.34.8.x86_64.rpm (installed size 15.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4.99  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± client3-0.6.23-11.34.3.x86_64.rpm (installed size 68.0 4.99  x   x~0 kB) x    xInstalling libavahi-client3-0.6.23-11.34.3.x86_64.rpm (installed size 68q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  95%  100  2  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± openssl0_9_8-32bit-0.9.8j-0.105.1.x86_64.rpm (installed size 2  x   x~.00 kB) x    xInstalling libopenssl0_9_8-32bit-0.9.8j-0.105.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  23  9  35  ±±±±±41  48 54%  60%  7  73  9  86  93  9  100  2.94  0 3  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cracklib-32bit-2.8.12-56.13.1.x86_64.rpm (installed size 34.00  2.94  x   x~ze 1.98 MB) x    xInstalling cracklib-32bit-2.8.12-56.13.1.x86_64.rpm (installed size 34.0q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 49 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libpng12-0-32bit-1.2.31-5.43.1.x86_64.rpm (installed size 162.0 1  x   x~0 kB) x    xInstalling libpng12-0-32bit-1.2.31-5.43.1.x86_64.rpm (installed size 162q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  39  79%  9  100  75  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xml2-32bit-2.7.6-0.64.1.x86_64.rpm (installed size 1.32 MB)  75  x   x~.00 kB) x    xInstalling libxml2-32bit-2.7.6-0.64.1.x86_64.rpm (installed size 1.32 MBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  14  23  8  33  ±±±±42  ±47  52% 61%  6  75  80  5  94  9  100  1.43  7 4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± freetype2-32bit-2.3.7-25.44.1.x86_64.rpm (installed size 511.00 1.43  x   x~) x    xInstalling freetype2-32bit-2.3.7-25.44.1.x86_64.rpm (installed size 511.q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  25  37  50 62%  75  87  99  100  0.9  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libldb1-3.6.3-87.1.x86_64.rpm (installed size 360.00 kB)  0.9  x   x~00 kB) x    xInstalling libldb1-3.6.3-87.1.x86_64.rpm (installed size 360.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  14  22  8  34  50 68%  86  97  100  58  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± smbclient0-3.6.3-87.1.x86_64.rpm (installed size 5.74 MB) 58  x   xInstalling libsmbclient0-3.6.3-87.1.x86_64.rpm (installed size 5.74 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  6  91  6  100  34.84  4 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ini_config2-0.6.1-1.9.1.x86_64.rpm (installed size 35.00 kB) 34.84  x   xInstalling libini_config2-0.6.1-1.9.1.x86_64.rpm (installed size 35.00 kq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gconf2-2.28.0-1.8.1.x86_64.rpm (installed size 911.00 kB)  1  x   x~B) x    xInstalling gconf2-2.28.0-1.8.1.x86_64.rpm (installed size 911.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  13  8  25  32  8  ±±45  52% 9%  65  71  8  85  92  7  100   x   xAdditional rpm output: x    xMust specify some schema files to install q  3.92  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed size 3.92 Additional rpm output: Must specify some schema files to install   Installing xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed siz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pm-32bit-7.4-3.1.x86_64.rpm (installed size 67.00 0   Installing xorg-x11-libXfixes-32bit-7.4-1.20.1.x86_64.rpm (installed siz ~e 18.00 kB)pm-32bit-7.4-3.1.x86_64.rpm (installed size 67.0  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  96%  100  83  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± render-32bit-7.4-1.23.1.x86_64.rpm (installed size 83  x   x~0 kB) x    xInstalling xorg-x11-libXrender-32bit-7.4-1.23.1.x86_64.rpm (installed siq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 39 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libtiff3-32bit-3.8.2-141.168.1.x86_64.rpm (installed size 372.0 0  x   x~ze 34.00 kB) x    xInstalling libtiff3-32bit-3.8.2-141.168.1.x86_64.rpm (installed size 372q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  34  51 68%  85  97  100  44  8 7  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pam-32bit-1.1.5-0.17.2.x86_64.rpm (installed size 603.00 kB)  44  x   x~.00 kB) x    xInstalling pam-32bit-1.1.5-0.17.2.x86_64.rpm (installed size 603.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  15  21  6  32  7  ±±±±42  ±47  54%  9%  65  70  7  83  8  99  100  2.85  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fontconfig-32bit-2.6.0-10.19.1.x86_64.rpm (installed size 234.0 2.85  x   xInstalling fontconfig-32bit-2.6.0-10.19.1.x86_64.rpm (installed size 234q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  8  ±±45  73%  99  100  62  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-1.9.4-0.33.1.x86_64.rpm (installed size 7.48 MB)  62  x   x~.00 kB) x    xInstalling sssd-1.9.4-0.33.1.x86_64.rpm (installed size 7.48 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  5  30  5  40  ±±45  50 5%  60%  5  70  5  80  5  90  5  100  25.14  5 90  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 25.14 MBx 35x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 14.06 MBx25x v http-10.9.0.10-e6784b1axxx tu xorg-x11-libfontenc-32bit-7.4-1.15.x86_64.rpm (installed size 2 25.14  x   xInstalling xorg-x11-libfontenc-32bit-7.4-1.15.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 4 2  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xkbfile41 2  x   x~ 24.00 kB) x    xInstalling xorg-x11-libxkbfile-32bit-7.4-1.14.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±43  87%  99  100  3.89 3 4.97  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± Xt-32bit-7.4-1.19.8.x86_64.rpm (installed size 360. 4.97  x   x~ 147.00 kB) x    xInstalling xorg-x11-libXt-32bit-7.4-1.19.8.x86_64.rpm (installed size 36q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  17  35  53%  71%  89  9  100  54 2 62  2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ext-32bit-7.4-1.18.16.x86_64.rpm (installed size 6 62  x   x~0.00 kB) x    xInstalling xorg-x11-libXext-32bit-7.4-1.18.16.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  48 1 56  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± pcre-32bit-7.8-2.18.x86_64.rpm (installed size 241.00 kB)  56  x   x~ 63.00 kB) x    xInstalling pcre-32bit-7.8-2.18.x86_64.rpm (installed size 241.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  26  53%  79%  95  100  25 0 32  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-lang-2.28.2-1.14.30.x86_64.rpm (installed size 2. 32  x   xInstalling gnome-keyring-lang-2.28.2-1.14.30.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  6  32  7  ±±±±42  48 53%  8%  64%  9  74  9  84  90  6  100  1.09 19 2.16 29 1  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± conf2-branding-SLES-2.25-0.13.31.noarch13 2.16  x   x~2.16 MB) x    xInstalling gconf2-branding-SLES-2.25-0.13.31.noarch.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  16  22  ±±±±4  53%  99%  100  0.95 8 03  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cups-libs-32bit-1.3.9-8.46.56.1.x86_64.rpm (installed size 336. 03  x   x~138.00 kB) x    xInstalling cups-libs-32bit-1.3.9-8.46.56.1.x86_64.rpm (installed size 33q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  2  39  58%  69%  88  99  100  62 7 1.70  7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± sssd-32bit-1.9.4-0.16.1.x86_64.rpm (installed size 60.00 kB)  1.70  x   x~6.00 kB) x    xInstalling sssd-32bit-1.9.4-0.16.1.x86_64.rpm (installed size 60.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  49 99%  100  56 6 64  6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± ConsoleKit-32bit-0.2.10-64.67.1.x86_64.rpm (installed size 23.0 64  x   xInstalling ConsoleKit-32bit-0.2.10-64.67.1.x86_64.rpm (installed size 23q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  ±±±±±41  99%  100  4 5 2  5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXp-32bit-7.4-1.18.734 2  x   x~.00 kB) x    xInstalling xorg-x11-libXp-32bit-7.4-1.18.7.x86_64.rpm (installed size 34q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  1 4 59  4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± mu-32bit-7.4-1.17.x86_64.rpm (installed size 109 59  x   x~.00 kB) x    xInstalling xorg-x11-libXmu-32bit-7.4-1.17.x86_64.rpm (installed size 109q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  58%  87%  99  100  40 3 48  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gnome-keyring-2.28.2-1.14.30.x86_64.rpm (installed size 2.57 MB 48  x   x~.00 kB) x    xInstalling gnome-keyring-2.28.2-1.14.30.x86_64.rpm (installed size 2.57 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  5  20  7  33  8  ±±±±43  48 53%  60%  5  70  5  82  7  93  8  100   7.83 2 18.91  2 2  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libXprintUtil-32bit-7.4-1.17.x86_64.rpm (installed siz 18.91  x   x~MB) x    xInstalling xorg-x11-libXprintUtil-32bit-7.4-1.17.x86_64.rpm (installed sq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  0 1 88  1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-1.4.3-0.17.21.1.x86_64.rpm (installed size 614.00 kB)  88  x   x~ize 34.00 kB) x    xInstalling gvfs-1.4.3-0.17.21.1.x86_64.rpm (installed size 614.00 kB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  6  12  8  23  8  33  ±±±44  56%  67%  73  8  9  8  100  2 0 2  0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgvfscommon0-1.4.3-0.17.21.1.x86_64.rpm (installed size 128.0 2  x   xInstalling libgvfscommon0-1.4.3-0.17.21.1.x86_64.rpm (installed size 128q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  74%  99  100  07  9 15 19 3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-lang-1.4.3-0.17.21.1.x86_64.rpm (installed size 2.10 MB)  15  x   x~.00 kB) x    xInstalling gvfs-lang-1.4.3-0.17.21.1.x86_64.rpm (installed size 2.10 MB)q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  21  7  33  8  ±±±±43  48 54%  9%  65  70  5  80  6  92  8  100  4.9 8 6.0  8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± fuse32.00 kB) 6.0  x   xInstalling gvfs-fuse-1.4.3-0.17.21.1.x86_64.rpm (installed size 32.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 16.02 MBx 17x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.94 MBx7x v http-10.9.0.10-e6784b1axxx tu ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 16.02 MBx 17x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.94 MBx7x v http-10.9.0.10-e6784b1axxx tu xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18.0 2  x   x~) x    xInstalling xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  SizexPackagesxTim16.00 MBx 17x x x 4.94 MBx 7x x x qqu 6 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libglib-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed size  0 Installing gvfs-fuse-1.4.3-0.17.21.1.x86_64.rpm (installed size 32.00 kB ~) xorg-x11-libXv-32bit-7.4-1.20.1.x86_64.rpm (installed size 18 .00 kB) ng libglib-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed siz  100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  8  16  24  32  40  48 56%  64%  72  80  9  97  100  5.23 5 4  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± soup-2_4-1-2.32.2-4.15.1.x86_64.rpm (installed size 468.00 k 5.23  x   x~e 791.00 kB) x    xInstalling libsoup-2_4-1-2.32.2-4.15.1.x86_64.rpm (installed size 468.00q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  27  ±±±±±41  54%  68%  76  81  7  99  100  4.77 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xorg-x11-libs-32bit-7.4-8.26.49.1.x86_64.rpm (installed size 2. 4.77  x   x~ kB) x    xInstalling xorg-x11-libs-32bit-7.4-8.26.49.1.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  10  6  22  8  33  9  ±±45  50 7%  62%  7  72  7  82  7  92  7  100  2.45 3 5  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libgmodule-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed si 2.45  x   x~2.33 MB) x    xInstalling libgmodule-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  4 2 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± thread 4  x   x~size 14.00 kB) x    xInstalling libgthread-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  99%  100  2 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± object 2  x   x~size 18.00 kB) x    xInstalling libgobject-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  24  49 74%  99  100  17 0 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± io-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed size 6 17  x   x~size 259.00 kB) x    xInstalling libgio-2_0-0-32bit-2.22.5-0.8.26.1.x86_64.rpm (installed sizeq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  1  28  3  ±47  57%  6  76  8  95  100  1.51  9 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTiw Total x 11.51 MBx 9x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 4.94 MBx7x v http-10.9.0.10-e6784b1axxx tu croco-0_6-3-32bit-0.6.1-120.16.x86_64.rpm (installed size 23 1.51  x   x~ 668.00 kB) x    xInstalling libcroco-0_6-3-32bit-0.6.1-120.16.x86_64.rpm (installed size q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  82%  99  100  72 6 SizexPackagesxTim11.29 MBx 9x x x 4.72 MBx 6x x x qqu 8 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gvfs-backends-1.4.3-0.17.21.1.x86_64.rpm (installed size 3.06 M 29  x   x~233.00 kB) x    xInstalling gvfs-backends-1.4.3-0.17.21.1.x86_64.rpm (installed size 3.06q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  33  8  ±±±44  50 7%  63%  9  74  9  85  90  5  100  1.65 5  8.22 7 6  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± cairo-32bit-1.8.8-2.1.48.x86_64.rpm (installed size 515.00 kB)  8.22 MB)   x   x~ MB) x    xInstalling cairo-32bit-1.8.8-2.1.48.x86_64.rpm (installed size 515.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  12  24  37  49 62%  74  87  99  100  1 4 7.7 6 7 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± libatk-1_0-0-32bit-1.28.0-1.4.1.x86_64.rpm (installed size 115. 7.7  x   x~) x    xInstalling libatk-1_0-0-32bit-1.28.0-1.4.1.x86_64.rpm (installed size 11q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  56%  99%  100  04 3 61 5 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gsf-1-11414.9-2.27.x86_64.rpm (installed size 232.0 61  x   x~5.00 kB) x    xInstalling libgsf-1-114-32bit-1.14.9-2.27.x86_64.rpm (installed size 232q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  27  55%  82%  99  100  832.00 k 2 38 4 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxTw Total x 7.38 MBx 4x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 832.00 kBx2x v http-10.9.0.10-e6784b1axxx tu pango-32bit-1.26.2-1.7.1.x86_64.rpm (installed size 789.00 kB)  38  x   x~.00 kB) x    xInstalling pango-32bit-1.26.2-1.7.1.x86_64.rpm (installed size 789.00 kBq   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  9  17  25  33  ±±±±4  51 9%  6  74  8  90  5  100  6.61  3 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-32bit-2.18.9-0.445.80 MB)  6.61  x   x~) x    xInstalling gtk2-32bit-2.18.9-0.44.1.x86_64.rpm (installed size 5.80 MB) q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  5  11  6  22  7  32  7  ±±±±42  ±47  52% 7%  62%  7  72  8  83  8  93  8  100  832.00 k  2 9  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± xSizexPackagesxTime  x        x     qq  xSizexPackagesxT Total x 832.00 kBx 2x SUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109xxx Medium 1x 832.00 kBx2x tu librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 kB 832.00 kB)  x   xInstalling librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  13  ±±±±±41  65%  89  9  100  569 1 SizexPackagesxTime569.00 kBx 2x x x 569.00 kBx 1x qq 1 ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569.0 569  x   x~kB) x    xInstalling gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569q   100%  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±  11  22  9  3  50 61%  72  80  91  9  100   Done.   SizexPackagesxTime  x 1x  x x Done.x x    100   Perform Installation lqqqqk xMedia x SizexPackagesxTime x xTotal x x x  x xSUSE-Linux-Enterprise-Server-11-SP4 11.4.4-1.109x x x  x xMedium 1 xDone.x x  x x  x mqqqqj lActions performed:qqqqk x~)  x xInstalling gtk2-32bit-2.18.9-0.44.1.x86_64.rpm (installed size 5.80 MB) x xInstalling librsvg-32bit-2.26.0-2.3.1.x86_64.rpm (installed size 263.00 x x~kB)  x xInstalling gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569 q mqqqqj Installing gtk2-engines-32bit-2.16.1-4.26.x86_64.rpm (installed size 569.00 kB   Installing Packages... (Remaining: 569.00 kB)    F10 Next 8 Back  F9 Abort  F10 Next  Configuring System according to auto-install sett   Configuring System according to auto-install settings  lk  x  x  x x  x  x  x   x  xx  x x  x x  x x  x x  x x  x x  x x  x x  x x  mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj Progress  ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±± [Help] [Back] [Abort] [Next] 9 Abort   2  4  6  Configuring lan  8  Configuring users  10  2 Configuring proxy  4  6  8  20  2  4  6  8  30 Configuring firewall  2  4 Configuring host  6  8  ±±±±±40 Configuring iscsi-client  ±±±±42  ±±44  ±46  48 51 Configuring keyboard 3%  5%  7%  9%  61  3  5  7  9  71  3  5  7  Configuring runlevel 9  81  3  5  7  9  91  3  5  Configuring timezone 7  Configuring yast-language  YaST2 - installation @ trevis-41vm7    Configuring System according to auto-install settings     lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk    xConfiguring lan x    xConfiguring users x    xConfiguring proxy x    xConfiguring firewall x    xConfiguring host x    xConfiguring iscsi-client x    xConfiguring keyboard x    xConfiguring runlevel x    xConfiguring timezone x    xConfiguring yast-language x    x x    x x    x x    x x    mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj    Progress     97% ±±   [Help] [Back] [Abort] [Next]    F1 Help  F9 Abort 100  Executing Post-Scripts Finishing Configuration 10 Nex 9 Abort  F10 Next 8 Back  F9 Abort  F10 Next  Writing the sconfiguration...               ±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±0%±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±±9 Abort   14  Setting up linker cache...  28  Running SuSEconfig.gtk2  ±±±±42 Running SuSEconfig.glib2  57%  Running SuSEconfig.scpm 71%  Running SuSEconfig.permissions 85  Running SuSEconfig.groff 100  Finishing SuSEconfig Done. 10 Nex 9 Abort  F10 Next 8 Back  F9 Abort  F10 Next [?1h=lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxFinishing the installation...x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj  lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqkxxmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj                            [?1l>Shutting down network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service network . . . . . . . . .7[?25l done8[?25h unicode_stop skipped on /dev/ttyS0 INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid 7[?25l done8[?25h [?25hD-Bus already started. Not starting. [?25hRe-Starting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25hRemoving old kernel packages... /sbin/purge-kernels: multiversion.kernels not configured in /etc/zypp/zypp.conf, exiting. 7[?25l done8[?25h [?25hInitializing random number generator7[?25l done8[?25h [?25hHAL already started. Not starting. [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 27 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh ssh-keygen: generating new host keys: ED25519 Starting SSH daemon7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hExecuting AutoYaST script: /var/adm/autoinstall/init.d/chroot-scripts_temp-resolver7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/init-scripts_chef-solo7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/init-scripts_cobbler7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/load-correct-kernel7[?25l done8[?25h Executing AutoYaST script: /var/adm/autoinstall/init.d/sanitize_logs7[?25l done8[?25h insserv: script munge is broken: incomplete LSB comment. insserv: missing `Default-Start:' entry: please add even if empty. insserv: Default-Start undefined, assuming default start runlevel(s) for script `munge' [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-94-default (console). trevis-41vm7 login: INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal [2017-04-04T19:15:44+00:00] WARN: ***************************************** [2017-04-04T19:15:44+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-04T19:15:44+00:00] WARN: ***************************************** Boot logging started on /dev/ttyS0((no tty)) at Tue Apr 4 19:15:44 2017 Master Resource Control: previous runlevel: 3, switching to runlevel: 6 Shutting down acpid 7[?25l done8[?25h [?25hNot unloading kdump during runlevel changes skipped [?25hShutting down CRON daemon7[?25l done8[?25h [?25hShutting down irqbalance 7[?25l done8[?25h [?25hShutting down mcelog... 7[?25l done8[?25h [?25h[?25hShutting down Name Service Cache Daemon7[?25l done8[?25h [?25h[?25hShutting down smartd 7[?25l done8[?25h [?25h[?25h[?25hShutting down SSH daemon *with all active connections*7[?25l done8[?25h [?25hShutting down auditd 7[?25l done8[?25h [?25h[?25hShutting down haveged daemon 7[?25l done8[?25h [?25h/etc/init.d/rc3.d/K02kbd stop7[?25l done8[?25h [?25hShutting down mail service (Postfix)7[?25l done8[?25h [?25hShutting down (remotefs) network interfaces: Shutting down service (remotefs) network . . . . . . . . .7[?25l done8[?25h [?25hSaving random seed7[?25l done8[?25h [?25hShutting down HAL daemon7[?25l done8[?25h [?25hShutting down NFS client services:7[?25l done8[?25h [?25hShutting down rpcbind 7[?25l done8[?25h [?25hShutting down syslog services7[?25l done8[?25h [?25h[?25hShutting down (localfs) network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service (localfs) network . . . . . . . . .7[?25l done8[?25h [?25hUnloading HCA driver:[ OK ] [?25hShutting down D-Bus daemon7[?25l done8[?25h [?25hRunning /etc/init.d/halt.local 7[?25l done8[?25hFuse filesystem not loaded7[?25l done8[?25h Not unloading kdump during runlevel changes skipped 7[?25l done8[?25h Turning off quota 7[?25l done8[?25h 7[?25l done8[?25h Set Hardware Clock to the current System Time7[?25l done8[?25h Turning off swap files Unmounting file systems 7[?25l done8[?25hStopping udevd: 7[?25l done8[?25h Sending all processes the TERM signal...  done Sending all processes the KILL signal...  done Please stand by while rebooting the system... [ 183.636499] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-94-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Jan 26 12:20:59 UTC 2017 (c499ea8) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37755000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa01, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72192 r8192 d22016 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a01, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910912k/2097140k available (4569k kernel code, 460k absent, 185768k reserved, 7578k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013265] Security Framework initialized [ 0.014092] AppArmor: AppArmor initialized [ 0.015071] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017041] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.018690] Mount-cache hash table entries: 256 [ 0.020227] Initializing cgroup subsys cpuacct [ 0.021085] Initializing cgroup subsys memory [ 0.021943] Initializing cgroup subsys devices [ 0.022790] Initializing cgroup subsys freezer [ 0.023641] Initializing cgroup subsys net_cls [ 0.024007] Initializing cgroup subsys blkio [ 0.024850] Initializing cgroup subsys perf_event [ 0.025853] Disabled fast string operations [ 0.026697] mce: CPU supports 10 MCE banks [ 0.030380] ACPI: Core revision 20110413 [ 0.032925] Enabling x2apic [ 0.033632] Enabled x2apic [ 0.034497] Switched APIC routing to physical x2apic. [ 0.036537] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.037550] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.040002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.040002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.040002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.056011] kvm-clock: cpu 1, msr 0:7fd10a01, secondary cpu clock [ 0.056026] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.056054] Brought up 2 CPUs [ 0.056056] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.063768] devtmpfs: initialized [ 0.066041] print_constraints: dummy: [ 0.066041] Time: 19:16:31 Date: 04/04/17 [ 0.068151] NET: Registered protocol family 16 [ 0.069166] ACPI: bus type pci registered [ 0.070061] PCI: Using configuration type 1 for base access [ 0.072381] bio: create slab at 0 [ 0.074338] ACPI: Interpreter enabled [ 0.076008] ACPI: (supports S0 S5) [ 0.076891] ACPI: Using IOAPIC for interrupt routing [ 0.082062] ACPI: No dock devices found. [ 0.082902] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.084028] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.086416] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.088014] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.116076] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.122182] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.123515] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.124787] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.126102] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.127382] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.128696] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.130248] vgaarb: loaded [ 0.131167] PCI: Using ACPI for IRQ routing [ 0.132305] NetLabel: Initializing [ 0.133037] NetLabel: domain hash size = 128 [ 0.133865] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.134779] NetLabel: unlabeled traffic allowed by default [ 0.135747] Switching to clocksource kvm-clock [ 0.138113] AppArmor: AppArmor Filesystem Enabled [ 0.139087] pnp: PnP ACPI init [ 0.139808] ACPI: bus type pnp registered [ 0.141316] pnp: PnP ACPI: found 6 devices [ 0.142134] ACPI: ACPI bus type pnp unregistered [ 0.149244] NET: Registered protocol family 2 [ 0.150318] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.152272] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.155618] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.157177] TCP: Hash tables configured (established 262144 bind 65536) [ 0.158257] TCP reno registered [ 0.158966] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.159990] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.161241] NET: Registered protocol family 1 [ 0.162104] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.163104] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.164104] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.165249] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.166244] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.171100] pci 0000:00:01.2: PCI INT D disabled [ 0.172084] Unpacking initramfs... [ 0.305788] Freeing initrd memory: 8812k freed [ 0.319854] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.320926] Initialise module verification [ 0.321906] audit: initializing netlink socket (disabled) [ 0.322874] type=2000 audit(1491333392.320:1): initialized [ 0.340737] bounce pool size: 64 pages [ 0.341575] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.343464] VFS: Disk quotas dquot_6.5.2 [ 0.344319] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.345555] msgmni has been set to 937 [ 0.346711] alg: No test for stdrng (krng) [ 0.347534] Asymmetric key parser 'x509' registered [ 0.348470] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.350030] io scheduler noop registered [ 0.350849] io scheduler deadline registered [ 0.351749] io scheduler cfq registered (default) [ 0.352869] GHES: HEST is not enabled! [ 0.353712] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.354916] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.593230] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.629096] Non-volatile memory driver v1.3 [ 0.629946] Linux agpgart interface v0.103 [ 0.630837] Fixed MDIO Bus: probed [ 0.631634] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.633739] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.634664] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.635637] mousedev: PS/2 mouse device common for all mice [ 0.636834] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.638769] cpuidle: using governor ladder [ 0.639575] cpuidle: using governor menu [ 0.640378] EFI Variables Facility v0.08 2004-May-17 [ 0.641471] TCP cubic registered [ 0.642212] Registering the dns_resolver key type [ 0.643190] Loading module verification certificates [ 0.644137] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.645975] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.647817] registered taskstats version 1 [ 0.648830] Magic number: 13:23:293 [ 0.649604] tty tty26: hash matches [ 0.652374] Freeing unused kernel memory: 1360k freed [ 0.653947] Write protecting the kernel read-only data: 10240k [ 0.657978] Freeing unused kernel memory: 1556k freed [ 0.661575] Freeing unused kernel memory: 900k freed doing fast boot [ 0.701617] SCSI subsystem initialized [ 0.724512] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.732621] scsi0 : ata_piix [ 0.733447] scsi1 : ata_piix [ 0.734149] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.735231] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.905039] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.920942] vda: vda1 vda2 [ 0.922254] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.923247] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.929474] rdac: device handler registered [ 0.934014] hp_sw: device handler registered [ 0.938558] emc: device handler registered [ 0.943140] alua: device handler registered Creating device nodes with udev [ 0.946783] udev: starting version 147 [ 0.960154] usbcore: registered new interface driver usbfs [ 0.961163] usbcore: registered new interface driver hub [ 0.971568] usbcore: registered new device driver usb [ 0.986731] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.994700] uhci_hcd: USB Universal Host Controller Interface driver [ 0.995845] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.997471] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.998419] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.000066] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.001175] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.002326] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.003942] usb usb1: Product: UHCI Host Controller [ 1.005032] usb usb1: Manufacturer: Linux 3.0.101-94-default uhci_hcd [ 1.007090] usb usb1: SerialNumber: 0000:00:01.2 [ 1.008806] hub 1-0:1.0: USB hub found [ 1.009651] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:16:33 2017 [ 1.056943] [drm] Initialized drm 1.1.0 20060810 [ 1.059778] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-94-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.341031] Refined TSC clocksource calibration: 2593.642 MHz. [ 1.354307] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 80030/1310720 files, 552177/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.387731] kjournald starting. Commit interval 15 seconds [ 1.681096] EXT3-fs (vda1): using internal journal [ 1.682040] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:16:33 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.861652] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.882572] input: PC Speaker as /devices/platform/pcspkr/input/input2 [ 1.888899] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.890187] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.891195] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.904193] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.911903] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3 [ 1.913524] ACPI: Power Button [PWRF] [ 1.929977] rtc_cmos 00:01: RTC can wake from S4 [ 1.931029] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.932141] rtc0: alarms up to one day, 114 bytes nvram [ 1.940431] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.942401] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.949841] FDC 0 is a S82078B [ 1.950685] NET: Registered protocol family 10 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 2.041345] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.586833] device-mapper: uevent: version 1.0.3 [ 2.587824] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.650037] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 3.112188] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25h[?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid acpid: starting up 7[?25l done8[?25h [?25hacpid: 3 rules loaded Starting D-Bus daemon7[?25l done8[?25h [?25hStarting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25h[?25hInitializing random number generator7[?25l done8[?25h [?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 27 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-94-default (console). trevis-41vm7 login: [2017-04-04T19:16:53+00:00] WARN: ***************************************** [2017-04-04T19:16:53+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-04T19:16:53+00:00] WARN: ***************************************** [2017-04-04T19:16:58+00:00] INFO: Forking chef instance to converge... [2017-04-04T19:16:58+00:00] INFO: *** Chef 12.4.3 *** [2017-04-04T19:16:58+00:00] INFO: Chef-client pid: 3657 [2017-04-04T19:17:00+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-04T19:17:00+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-04T19:17:00+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-04T19:17:00+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-04T19:17:00+00:00] INFO: Running start handlers [2017-04-04T19:17:00+00:00] INFO: Start handlers complete. [2017-04-04T19:17:00+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-04T19:17:00+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-04T19:17:00+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-04T19:17:00+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-04T19:17:00+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-04T19:17:00+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-04T19:17:00+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170404191700.602406 [2017-04-04T19:17:00+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-04T19:17:00+00:00] INFO: Processing zypper_package[sudo] action install (sudo::default line 20) [2017-04-04T19:17:00+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-04T19:17:00+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170404191700.951556 [2017-04-04T19:17:00+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-04T19:17:00+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-04T19:17:00+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-04T19:17:01+00:00] INFO: service[nscd] stopped [2017-04-04T19:17:01+00:00] INFO: Processing zypper_package[sssd] action install (sssd::default line 20) [2017-04-04T19:17:01+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-04T19:17:01+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170404191701.574559 [2017-04-04T19:17:01+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-04T19:17:01+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-04T19:17:01+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-04T19:17:01+00:00] INFO: service[sssd] restarted [2017-04-04T19:17:01+00:00] INFO: Processing template[/etc/nsswitch.conf] action create (sssd::default line 75) [2017-04-04T19:17:01+00:00] INFO: template[/etc/nsswitch.conf] backed up to /var/chef/backup/etc/nsswitch.conf.chef-20170404191701.882161 [2017-04-04T19:17:01+00:00] INFO: template[/etc/nsswitch.conf] updated file contents /etc/nsswitch.conf [2017-04-04T19:17:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-auth-pc] action create (sssd::default line 83) [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-auth-pc] backed up to /var/chef/backup/etc/pam.d/common-auth-pc.chef-20170404191701.885523 [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-auth-pc] updated file contents /etc/pam.d/common-auth-pc [2017-04-04T19:17:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-account-pc] action create (sssd::default line 83) [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-account-pc] backed up to /var/chef/backup/etc/pam.d/common-account-pc.chef-20170404191701.888570 [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-account-pc] updated file contents /etc/pam.d/common-account-pc [2017-04-04T19:17:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-password-pc] action create (sssd::default line 83) [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-password-pc] backed up to /var/chef/backup/etc/pam.d/common-password-pc.chef-20170404191701.891475 [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-password-pc] updated file contents /etc/pam.d/common-password-pc [2017-04-04T19:17:01+00:00] INFO: Processing cookbook_file[/etc/pam.d/common-session-pc] action create (sssd::default line 83) [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-session-pc] backed up to /var/chef/backup/etc/pam.d/common-session-pc.chef-20170404191701.894469 [2017-04-04T19:17:01+00:00] INFO: cookbook_file[/etc/pam.d/common-session-pc] updated file contents /etc/pam.d/common-session-pc [2017-04-04T19:17:01+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-04T19:17:02+00:00] INFO: service[sssd] enabled [2017-04-04T19:17:02+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-04T19:17:02+00:00] INFO: Processing zypper_package[ntp] action install (ntp::default line 25) [2017-04-04T19:17:02+00:00] INFO: Processing service[ntp] action start (ntp::default line 27) [2017-04-04T19:17:02+00:00] INFO: service[ntp] started [2017-04-04T19:17:02+00:00] INFO: Processing service[ntp] action enable (ntp::default line 27) [2017-04-04T19:17:02+00:00] INFO: service[ntp] enabled [2017-04-04T19:17:02+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-04T19:17:02+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170404191702.607356 [2017-04-04T19:17:02+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-04T19:17:02+00:00] INFO: template[/etc/ntp.conf] group changed to 0 [2017-04-04T19:17:02+00:00] INFO: template[/etc/ntp.conf] mode changed to 644 [2017-04-04T19:17:02+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntp] (immediate) [2017-04-04T19:17:02+00:00] INFO: Processing service[ntp] action restart (ntp::default line 27) [2017-04-04T19:17:02+00:00] INFO: service[ntp] restarted [2017-04-04T19:17:02+00:00] INFO: Processing service[nfsserver] action nothing (nfs::default line 29) [2017-04-04T19:17:02+00:00] INFO: Processing zypper_package[nfs-client] action install (nfs::default line 35) [2017-04-04T19:17:03+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-04T19:17:03+00:00] INFO: service[rpcbind] enabled [2017-04-04T19:17:03+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-04T19:17:03+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-04T19:17:03+00:00] INFO: service[rpcbind] enabled [2017-04-04T19:17:03+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-04T19:17:03+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-04T19:17:03+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170404191703.434157 [2017-04-04T19:17:03+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-04T19:17:03+00:00] INFO: Processing zypper_package[autofs] action install (autofs::default line 17) [2017-04-04T19:17:03+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-04T19:17:03+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170404191703.952231 [2017-04-04T19:17:03+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-04T19:17:03+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-04T19:17:03+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-04T19:17:05+00:00] INFO: service[autofs] restarted [2017-04-04T19:17:05+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-04T19:17:05+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170404191705.639531 [2017-04-04T19:17:05+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-04T19:17:05+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-04T19:17:05+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-04T19:17:07+00:00] INFO: service[autofs] restarted [2017-04-04T19:17:07+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-04T19:17:07+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-04T19:17:07+00:00] INFO: service[autofs] enabled [2017-04-04T19:17:07+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-04T19:17:07+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-04T19:17:07+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-04T19:17:07+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170404191707.618520 [2017-04-04T19:17:07+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-04T19:17:07+00:00] INFO: Processing zypper_package[munge] action install (munge::default line 20) [2017-04-04T19:17:07+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-04T19:17:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] backed up to /var/chef/backup/etc/munge/munge.key.chef-20170404191707.908696 [2017-04-04T19:17:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-04T19:17:07+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-04T19:17:07+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-04T19:17:08+00:00] INFO: service[munge] restarted [2017-04-04T19:17:08+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-04T19:17:08+00:00] INFO: template[/etc/sysconfig/munge] backed up to /var/chef/backup/etc/sysconfig/munge.chef-20170404191708.020232 [2017-04-04T19:17:08+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-04T19:17:08+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-04T19:17:08+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-04T19:17:08+00:00] INFO: service[munge] restarted [2017-04-04T19:17:08+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-04T19:17:08+00:00] INFO: service[munge] enabled [2017-04-04T19:17:08+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-04T19:17:08+00:00] INFO: Processing zypper_package[mrsh-server] action install (mrsh::default line 10) [2017-04-04T19:17:08+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T19:17:09+00:00] INFO: zypper_package[mrsh-server] installed mrsh-server at 3:2.6-6.wc1 [2017-04-04T19:17:09+00:00] INFO: Processing zypper_package[mrsh] action install (mrsh::default line 11) [2017-04-04T19:17:09+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T19:17:10+00:00] INFO: zypper_package[mrsh] installed mrsh at 3:2.6-6.wc1 [2017-04-04T19:17:10+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170404191710.079948 [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-04T19:17:10+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170404191710.086386 [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-04T19:17:10+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-04T19:17:10+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-04T19:17:10+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-04T19:17:10+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:10+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-04T19:17:10+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-04T19:17:11+00:00] INFO: service[xinetd] started [2017-04-04T19:17:11+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-04T19:17:11+00:00] INFO: service[xinetd] enabled [2017-04-04T19:17:11+00:00] INFO: Processing cookbook_file[/etc/pam.d/mrsh] action create (mrsh::server line 77) [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] backed up to /var/chef/backup/etc/pam.d/mrsh.chef-20170404191711.896929 [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] updated file contents /etc/pam.d/mrsh [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrsh] mode changed to 644 [2017-04-04T19:17:11+00:00] INFO: Processing cookbook_file[/etc/pam.d/mrlogin] action create (mrsh::server line 84) [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] backed up to /var/chef/backup/etc/pam.d/mrlogin.chef-20170404191711.900881 [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] updated file contents /etc/pam.d/mrlogin [2017-04-04T19:17:11+00:00] INFO: cookbook_file[/etc/pam.d/mrlogin] mode changed to 644 [2017-04-04T19:17:11+00:00] INFO: Processing zypper_package[openssh] action install (ssh::default line 9) [2017-04-04T19:17:12+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-04T19:17:12+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-04T19:17:12+00:00] INFO: service[sshd] enabled [2017-04-04T19:17:12+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-04T19:17:12+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170404191712.396470 [2017-04-04T19:17:12+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-04T19:17:12+00:00] INFO: template[/etc/ssh/sshd_config] mode changed to 600 [2017-04-04T19:17:12+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-04T19:17:12+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-04T19:17:12+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-04T19:17:12+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-04T19:17:12+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-04T19:17:12+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-04T19:17:12+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-04T19:17:12+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-04T19:17:12+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-04T19:17:12+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-04T19:17:12+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-04T19:17:12+00:00] INFO: Processing zypper_package[rsyslog] action install (rsyslog::client line 12) [2017-04-04T19:17:12+00:00] INFO: Processing zypper_package[ed] action install (rsyslog::client line 13) [2017-04-04T19:17:12+00:00] INFO: Processing cookbook_file[/etc/sysconfig/syslog] action create (rsyslog::client line 17) [2017-04-04T19:17:12+00:00] INFO: cookbook_file[/etc/sysconfig/syslog] backed up to /var/chef/backup/etc/sysconfig/syslog.chef-20170404191712.979586 [2017-04-04T19:17:12+00:00] INFO: cookbook_file[/etc/sysconfig/syslog] updated file contents /etc/sysconfig/syslog [2017-04-04T19:17:12+00:00] INFO: Processing service[syslog] action nothing (rsyslog::client line 22) [2017-04-04T19:17:12+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-04T19:17:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:13+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-04T19:17:13+00:00] INFO: Processing zypper_package[rsh] action install (rsh::default line 9) [2017-04-04T19:17:13+00:00] INFO: Processing zypper_package[rsh-server] action install (rsh::server line 11) [2017-04-04T19:17:13+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T19:17:14+00:00] INFO: zypper_package[rsh-server] installed rsh-server at 0.17-706.22.1 [2017-04-04T19:17:14+00:00] INFO: Processing execute[enable rsh] action run (rsh::server line 22) [2017-04-04T19:17:15+00:00] INFO: execute[enable rsh] ran successfully [2017-04-04T19:17:15+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-04T19:17:15+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:15+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-04T19:17:15+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-04T19:17:15+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:15+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-04T19:17:15+00:00] INFO: Processing execute[enable rlogin] action run (rsh::server line 22) [2017-04-04T19:17:17+00:00] INFO: execute[enable rlogin] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-04T19:17:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-04T19:17:17+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-04T19:17:17+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-04T19:17:17+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-04T19:17:17+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-04T19:17:17+00:00] INFO: Processing zypper_package[openmpi] action install (openmpi::default line 23) [2017-04-04T19:17:18+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-04T19:17:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-04T19:17:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-04T19:17:18+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-04T19:17:18+00:00] INFO: Processing zypper_package[cvs] action install (test_node::default line 12) [2017-04-04T19:17:18+00:00] INFO: Processing zypper_package[bc] action install (test_node::default line 13) [2017-04-04T19:17:18+00:00] INFO: Processing zypper_package[expect] action install (test_node::default line 14) [2017-04-04T19:17:18+00:00] INFO: Processing zypper_package[make] action install (test_node::default line 15) [2017-04-04T19:17:19+00:00] INFO: Processing zypper_package[gcc] action install (test_node::default line 16) [2017-04-04T19:17:19+00:00] INFO: Processing zypper_package[sg3_utils] action install (test_node::default line 17) [2017-04-04T19:17:19+00:00] INFO: Processing zypper_package[patch] action install (test_node::default line 18) [2017-04-04T19:17:20+00:00] INFO: Processing zypper_package[kexec-tools] action install (test_node::default line 19) [2017-04-04T19:17:20+00:00] INFO: Processing zypper_package[setools-console] action install (test_node::default line 101) [2017-04-04T19:17:20+00:00] WARN: Chef::Config[:zypper_check_gpg] was not set. All packages will be installed without gpg signature checks. This is a security hazard. [2017-04-04T19:17:21+00:00] INFO: zypper_package[setools-console] installed setools-console at 3.3.6-1.4.1 [2017-04-04T19:17:21+00:00] INFO: Processing execute[enable_unsupported_modules] action run (test_node::default line 104) [2017-04-04T19:17:21+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-04T19:17:21+00:00] INFO: execute[enable_unsupported_modules] ran successfully [2017-04-04T19:17:21+00:00] INFO: Processing service[openibd] action enable (test_node::default line 120) [2017-04-04T19:17:22+00:00] INFO: service[openibd] enabled [2017-04-04T19:17:22+00:00] INFO: Processing service[openibd] action start (test_node::default line 120) [2017-04-04T19:17:22+00:00] INFO: service[openibd] started [2017-04-04T19:17:22+00:00] INFO: Processing bash[setup kernel crash dumps] action run (test_node::default line 125) [2017-04-04T19:17:22+00:00] INFO: bash[setup kernel crash dumps] ran successfully [2017-04-04T19:17:22+00:00] INFO: Processing service[boot.kdump] action restart (test_node::default line 136) [2017-04-04T19:17:36+00:00] INFO: service[boot.kdump] restarted [2017-04-04T19:17:36+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-04T19:17:36+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-04T19:17:36+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-04T19:17:36+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-04T19:17:36+00:00] INFO: group[quota_usr] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-04T19:17:36+00:00] INFO: user[quota_usr] created [2017-04-04T19:17:36+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-04T19:17:36+00:00] INFO: group[quota_2usr] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-04T19:17:36+00:00] INFO: user[quota_2usr] created [2017-04-04T19:17:36+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-04T19:17:36+00:00] INFO: group[sanityusr] created [2017-04-04T19:17:36+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-04T19:17:36+00:00] INFO: user[sanityusr] created [2017-04-04T19:17:36+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-04T19:17:36+00:00] INFO: group[sanityusr1] created [2017-04-04T19:17:36+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-04T19:17:36+00:00] INFO: user[sanityusr1] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-04T19:17:36+00:00] INFO: user[quota15_1] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-04T19:17:36+00:00] INFO: user[quota15_2] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-04T19:17:36+00:00] INFO: user[quota15_3] created [2017-04-04T19:17:36+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_4] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_5] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_6] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_7] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_8] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_9] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_10] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_11] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_12] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_13] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_14] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_15] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_16] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_17] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_18] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_19] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_20] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_21] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-04T19:17:37+00:00] INFO: user[quota15_22] created [2017-04-04T19:17:37+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_23] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_24] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_25] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_26] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_27] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_28] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_29] created [2017-04-04T19:17:38+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-04T19:17:38+00:00] INFO: user[quota15_30] created [2017-04-04T19:17:38+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-04T19:17:38+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-04T19:17:38+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-04T19:17:38+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-04T19:17:38+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-04T19:17:38+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-04T19:17:38+00:00] INFO: bash[set_hostid] ran successfully [2017-04-04T19:17:38+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-04T19:17:38+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-04T19:17:38+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-04T19:17:38+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-04T19:17:38+00:00] INFO: Processing bash[add_panic_on_oom] action run (test_node::default line 243) [2017-04-04T19:17:38+00:00] INFO: bash[add_panic_on_oom] ran successfully [2017-04-04T19:17:38+00:00] INFO: Processing bash[set_kernel_printk] action run (test_node::default line 249) [2017-04-04T19:17:38+00:00] INFO: bash[set_kernel_printk] ran successfully [2017-04-04T19:17:38+00:00] INFO: Processing execute[restart_xinetd] action run (test_node::default line 266) [2017-04-04T19:17:40+00:00] INFO: execute[restart_xinetd] ran successfully [2017-04-04T19:17:40+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-04T19:17:40+00:00] INFO: Processing zypper_package[mcelog] action install (mcelog::default line 9) [2017-04-04T19:17:40+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-04T19:17:40+00:00] INFO: user[mpiuser] created [2017-04-04T19:17:40+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 20) [2017-04-04T19:17:40+00:00] INFO: group[mpiuser] created [2017-04-04T19:17:40+00:00] INFO: Processing bash[usermod_add_mpiuser_to_mpiuser] action run (openmpi::test_node line 24) [2017-04-04T19:17:40+00:00] INFO: bash[usermod_add_mpiuser_to_mpiuser] ran successfully [2017-04-04T19:17:40+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-04T19:17:40+00:00] INFO: directory[/localhome/mpiuser] created directory /localhome/mpiuser [2017-04-04T19:17:40+00:00] INFO: directory[/localhome/mpiuser] owner changed to 532 [2017-04-04T19:17:40+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-04T19:17:40+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-04T19:17:40+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-04T19:17:40+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-04T19:17:40+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-04T19:17:40+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-04T19:17:40+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-04T19:17:40+00:00] INFO: execute[newaliases] ran successfully [2017-04-04T19:17:40+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-04T19:17:40+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-04T19:17:41+00:00] INFO: service[sshd] restarted [2017-04-04T19:17:41+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[syslog] (delayed) [2017-04-04T19:17:41+00:00] INFO: Processing service[syslog] action restart (rsyslog::client line 22) [2017-04-04T19:17:42+00:00] INFO: service[syslog] restarted [2017-04-04T19:17:42+00:00] INFO: Chef Run complete in 42.68949257 seconds [2017-04-04T19:17:42+00:00] INFO: Skipping removal of unused files from the cache [2017-04-04T19:17:42+00:00] INFO: Running report handlers [2017-04-04T19:17:42+00:00] INFO: Creating JSON run report [2017-04-04T19:17:42+00:00] INFO: Report handlers complete INIT: Switching to runlevel: 6 INIT: Sending processes the TERM signal Boot logging started on /dev/ttyS0((no tty)) at Tue Apr 4 19:34:21 2017 Master Resource Control: previous runlevel: 3, switching to runlevel: 6 Shutting down acpid 7[?25l done8[?25h [?25hShutting down automount (force) 7[?25l done8[?25h [?25hNot unloading kdump during runlevel changes skipped [?25h[?25hShutting down CRON daemon7[?25l done8[?25h [?25hShutting down irqbalance 7[?25l done8[?25h [?25hShutting down mcelog... 7[?25l done8[?25h [?25h[?25hStopping MUNGE: munged7[?25l done8[?25h [?25hShutting down Name Service Cache Daemon7[?25l done8[?25h [?25h[?25hShutting down smartd 7[?25l done8[?25h [?25h[?25h[?25hShutting down SSH daemon *with all active connections*7[?25l done8[?25h [?25hShutting down sssd 7[?25l done8[?25h [?25hShutting down xinetd: (waiting for all children to terminate) 7[?25l done8[?25h [?25hShutting down auditd 7[?25l done8[?25h [?25h[?25hShutting down haveged daemon 7[?25l done8[?25h [?25h/etc/init.d/rc3.d/K02kbd stop7[?25l done8[?25h [?25hShutting down mail service (Postfix)7[?25l done8[?25h [?25hShutting down network time protocol daemon (NTPD)7[?25l done8[?25h [?25hSaving random seed7[?25l done8[?25h [?25hShutting down (remotefs) network interfaces: Shutting down service (remotefs) network . . . . . . . . .7[?25l done8[?25h [?25hShutting down HAL daemon7[?25l done8[?25h [?25hShutting down NFS client services: rpc.statd7[?25l done8[?25h [?25hShutting down rpcbind 7[?25l done8[?25h [?25hShutting down syslog services7[?25l done8[?25h [?25h[?25hShutting down (localfs) network interfaces: eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 7[?25l done8[?25hShutting down service (localfs) network . . . . . . . . .7[?25l done8[?25h [?25hUnloading HCA driver:[ OK ] [?25hShutting down D-Bus daemon7[?25l done8[?25h [?25hRunning /etc/init.d/halt.local 7[?25l done8[?25hUnmounting fuse control filesystem7[?25l done8[?25h Not unloading kdump during runlevel changes skipped 7[?25l done8[?25h Turning off quota 7[?25l done8[?25h 7[?25l done8[?25h The System Time is in sync with Hardware Clock good Turning off swap files Unmounting file systems 7[?25l done8[?25hStopping udevd: 7[?25l done8[?25h Sending all processes the TERM signal...  done Sending all processes the KILL signal...  done Please stand by while rebooting the system... [ 1087.668173] Restarting system. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-94-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Jan 26 12:20:59 UTC 2017 (c499ea8) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37755000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa01, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72192 r8192 d22016 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a01, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910912k/2097140k available (4569k kernel code, 460k absent, 185768k reserved, 7578k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013316] Security Framework initialized [ 0.014144] AppArmor: AppArmor initialized [ 0.015120] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017044] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.018729] Mount-cache hash table entries: 256 [ 0.020219] Initializing cgroup subsys cpuacct [ 0.021090] Initializing cgroup subsys memory [ 0.021967] Initializing cgroup subsys devices [ 0.022829] Initializing cgroup subsys freezer [ 0.024006] Initializing cgroup subsys net_cls [ 0.024858] Initializing cgroup subsys blkio [ 0.025698] Initializing cgroup subsys perf_event [ 0.026690] Disabled fast string operations [ 0.027537] mce: CPU supports 10 MCE banks [ 0.030390] ACPI: Core revision 20110413 [ 0.033286] Enabling x2apic [ 0.033988] Enabled x2apic [ 0.034853] Switched APIC routing to physical x2apic. [ 0.036905] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.037940] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.040002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.040002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.040002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.056019] kvm-clock: cpu 1, msr 0:7fd10a01, secondary cpu clock [ 0.056034] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.056053] Brought up 2 CPUs [ 0.056055] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.063845] devtmpfs: initialized [ 0.066093] print_constraints: dummy: [ 0.066093] Time: 19:35:08 Date: 04/04/17 [ 0.068160] NET: Registered protocol family 16 [ 0.069153] ACPI: bus type pci registered [ 0.070068] PCI: Using configuration type 1 for base access [ 0.072475] bio: create slab at 0 [ 0.076896] ACPI: Interpreter enabled [ 0.077805] ACPI: (supports S0 S5) [ 0.078748] ACPI: Using IOAPIC for interrupt routing [ 0.083963] ACPI: No dock devices found. [ 0.084010] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.085971] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.089505] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.091189] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.119279] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.125830] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.127191] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.128469] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.129807] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.131104] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.132220] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.133816] vgaarb: loaded [ 0.134746] PCI: Using ACPI for IRQ routing [ 0.136199] NetLabel: Initializing [ 0.136944] NetLabel: domain hash size = 128 [ 0.137769] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.138684] NetLabel: unlabeled traffic allowed by default [ 0.139659] Switching to clocksource kvm-clock [ 0.142244] AppArmor: AppArmor Filesystem Enabled [ 0.143239] pnp: PnP ACPI init [ 0.144002] ACPI: bus type pnp registered [ 0.145586] pnp: PnP ACPI: found 6 devices [ 0.146422] ACPI: ACPI bus type pnp unregistered [ 0.153695] NET: Registered protocol family 2 [ 0.154802] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.156853] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.160302] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.161928] TCP: Hash tables configured (established 262144 bind 65536) [ 0.163058] TCP reno registered [ 0.163823] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.164898] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.166209] NET: Registered protocol family 1 [ 0.167112] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.168151] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.169204] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.170380] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.171416] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.176427] pci 0000:00:01.2: PCI INT D disabled [ 0.177445] Unpacking initramfs... [ 0.315430] Freeing initrd memory: 8812k freed [ 0.318555] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.319629] Initialise module verification [ 0.320596] audit: initializing netlink socket (disabled) [ 0.321575] type=2000 audit(1491334509.320:1): initialized [ 0.339545] bounce pool size: 64 pages [ 0.340388] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.342551] VFS: Disk quotas dquot_6.5.2 [ 0.343501] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.344832] msgmni has been set to 937 [ 0.346119] alg: No test for stdrng (krng) [ 0.346995] Asymmetric key parser 'x509' registered [ 0.347989] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.349644] io scheduler noop registered [ 0.350468] io scheduler deadline registered [ 0.351423] io scheduler cfq registered (default) [ 0.352643] GHES: HEST is not enabled! [ 0.353524] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.354839] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.593283] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.629198] Non-volatile memory driver v1.3 [ 0.630698] Linux agpgart interface v0.103 [ 0.632224] Fixed MDIO Bus: probed [ 0.633371] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.637573] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.639063] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.640588] mousedev: PS/2 mouse device common for all mice [ 0.641767] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.643831] cpuidle: using governor ladder [ 0.644755] cpuidle: using governor menu [ 0.645573] EFI Variables Facility v0.08 2004-May-17 [ 0.646710] TCP cubic registered [ 0.647516] Registering the dns_resolver key type [ 0.648546] Loading module verification certificates [ 0.649501] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.651417] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.653354] registered taskstats version 1 [ 0.654419] Magic number: 13:332:596 [ 0.657941] Freeing unused kernel memory: 1360k freed [ 0.659573] Write protecting the kernel read-only data: 10240k [ 0.664122] Freeing unused kernel memory: 1556k freed [ 0.669300] Freeing unused kernel memory: 900k freed doing fast boot [ 0.715441] SCSI subsystem initialized [ 0.738745] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.747139] scsi0 : ata_piix [ 0.747998] scsi1 : ata_piix [ 0.748733] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.749901] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.917210] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.947705] vda: vda1 vda2 vda4 [ 0.949016] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.950012] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.956387] rdac: device handler registered [ 0.961074] hp_sw: device handler registered [ 0.965693] emc: device handler registered [ 0.970385] alua: device handler registered Creating device nodes with udev [ 0.974150] udev: starting version 147 [ 0.990111] usbcore: registered new interface driver usbfs [ 0.991127] usbcore: registered new interface driver hub [ 1.008988] usbcore: registered new device driver usb [ 1.014187] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.024782] uhci_hcd: USB Universal Host Controller Interface driver [ 1.026008] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.027712] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.028703] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.030349] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.031544] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.033320] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.036283] usb usb1: Product: UHCI Host Controller [ 1.037771] usb usb1: Manufacturer: Linux 3.0.101-94-default uhci_hcd [ 1.038965] usb usb1: SerialNumber: 0000:00:01.2 [ 1.040227] hub 1-0:1.0: USB hub found [ 1.041452] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:35:10 2017 [ 1.086059] [drm] Initialized drm 1.1.0 20060810 [ 1.089104] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-94-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.312612] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: clean, 80231/1310720 files, 556772/5240832 blocks [ 1.337038] Refined TSC clocksource calibration: 2593.642 MHz. fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.346009] kjournald starting. Commit interval 15 seconds [ 1.346492] EXT3-fs (vda1): using internal journal [ 1.346496] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 19:35:10 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 1.841818] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 1.866135] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 1.867725] ACPI: Power Button [PWRF] [ 1.870149] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.873110] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.874397] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.875407] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 1.892649] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 1.900235] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.912824] rtc_cmos 00:01: RTC can wake from S4 [ 1.913925] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 1.915023] rtc0: alarms up to one day, 114 bytes nvram [ 1.927375] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.933548] NET: Registered protocol family 10 [ 1.934945] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 2.029446] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 2.563102] device-mapper: uevent: version 1.0.3 [ 2.564115] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 2.626385] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 2.904287] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... No volume groups found 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... [?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting acpid acpid: starting up acpid: 3 rules loaded 7[?25l done8[?25h [?25hStarting D-Bus daemon7[?25l done8[?25h [?25hStarting syslog services7[?25l done8[?25h [?25h[?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25h[?25hInitializing random number generator7[?25l done8[?25h [?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 27 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting network time protocol daemon (NTPD)7[?25l done8[?25h [?25hStarting MUNGE: munged7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting sssd 7[?25l done8[?25h [?25hStarting automount 7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hStarting INET services. (xinetd)7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-94-default (console). trevis-41vm7 login: [ 73.119438] RPC: Registered named UNIX socket transport module. [ 73.120848] RPC: Registered udp transport module. [ 73.121855] RPC: Registered tcp transport module. [ 73.122755] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 73.125724] FS-Cache: Loaded [ 73.129680] FS-Cache: Netfs 'nfs' registered for caching [ 3856.794252] SysRq : Show State [ 3856.795017] task PC stack pid father [ 3856.796027] init S ffff88007b455ad8 0 1 0 0x00000000 [ 3856.797183] ffff88007b455948 0000000000000082 ffff88007b454010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b455fd8 [ 3856.798062] ffff88007b455fd8 0000000000010b00 ffff88007b452040 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f86f7be6943>] 0x7f86f7be6942 [ 3856.798062] kthreadd S ffff88007b458080 0 2 0 0x00000000 [ 3856.798062] ffff88007b45bea0 0000000000000046 ffff88007b45a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b45bfd8 [ 3856.798062] ffff88007b45bfd8 0000000000010b00 ffff88007b458080 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] kthreadd+0x2a1/0x2b0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 [ 3856.798062] ffff88007b461ea0 0000000000000046 ffff88007b460010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b461fd8 [ 3856.798062] ffff88007b461fd8 0000000000010b00 ffff88007b45e0c0 ffff880079c0c100 [ 3856.798062] Call Trace: [ 3856.798062] [] run_ksoftirqd+0x10d/0x170 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/u:0 S 0000000000011580 0 5 2 0x00000000 [ 3856.798062] ffff88007b469e70 0000000000000046 ffff88007b468010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b469fd8 [ 3856.798062] ffff88007b469fd8 0000000000010b00 ffff88007b466140 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] migration/0 S ffff88007fc0cee8 0 6 2 0x00000000 [ 3856.798062] ffff88007b46fe00 0000000000000046 ffff88007b46e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b46ffd8 [ 3856.798062] ffff88007b46ffd8 0000000000010b00 ffff88007b46c180 ffff880079c0c100 [ 3856.798062] Call Trace: [ 3856.798062] [] cpu_stopper_thread+0x10d/0x180 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] watchdog/0 S 0000000000000000 0 7 2 0x00000000 [ 3856.798062] ffff88007b473ec0 0000000000000046 ffff88007b472010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b473fd8 [ 3856.798062] ffff88007b473fd8 0000000000010b00 ffff88007b4701c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] watchdog+0x9f/0xf0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] migration/1 S ffff88007fd0cee8 0 8 2 0x00000000 [ 3856.798062] ffff88007b477e00 0000000000000046 ffff88007b476010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b477fd8 [ 3856.798062] ffff88007b477fd8 0000000000010b00 ffff88007b474200 ffff88007b4622c0 [ 3856.798062] Call Trace: [ 3856.798062] [] cpu_stopper_thread+0x10d/0x180 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/1:0 S 0000000000011580 0 9 2 0x00000000 [ 3856.798062] ffff88007b47fe70 0000000000000046 ffff88007b47e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b47ffd8 [ 3856.798062] ffff88007b47ffd8 0000000000010b00 ffff88007b47c240 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ksoftirqd/1 S 0000000000000000 0 10 2 0x00000000 [ 3856.798062] ffff88007b4a7ea0 0000000000000046 ffff88007b4a6010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4a7fd8 [ 3856.798062] ffff88007b4a7fd8 0000000000010b00 ffff88007b4a4280 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] run_ksoftirqd+0x10d/0x170 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/0:1 S 0000000000011580 0 11 2 0x00000000 [ 3856.798062] ffff88007b4abe70 0000000000000046 ffff88007b4aa010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4abfd8 [ 3856.798062] ffff88007b4abfd8 0000000000010b00 ffff88007b4a82c0 ffff8800797b0240 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] watchdog/1 S 0000000000000000 0 12 2 0x00000000 [ 3856.798062] ffff88007b4b5ec0 0000000000000046 ffff88007b4b4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4b5fd8 [ 3856.798062] ffff88007b4b5fd8 0000000000010b00 ffff88007b4b2340 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] watchdog+0x9f/0xf0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] cpuset S ffff88007b499860 0 13 2 0x00000000 [ 3856.798062] ffff88007b4dfe60 0000000000000046 ffff88007b4de010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4dffd8 [ 3856.798062] ffff88007b4dffd8 0000000000010b00 ffff88007b4dc380 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] khelper S ffff88007b4997e0 0 14 2 0x00000000 [ 3856.798062] ffff88007b4e5e60 0000000000000046 ffff88007b4e4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4e5fd8 [ 3856.798062] ffff88007b4e5fd8 0000000000010b00 ffff88007b4e23c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] netns S ffff88007b4991e0 0 15 2 0x00000000 [ 3856.798062] ffff88007b4ede60 0000000000000046 ffff88007b4ec010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b4edfd8 [ 3856.798062] ffff88007b4edfd8 0000000000010b00 ffff88007b4ea400 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] sync_supers S 0000000000000000 0 16 2 0x00000000 [ 3856.798062] ffff88007b571ec0 0000000000000046 ffff88007b570010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b571fd8 [ 3856.798062] ffff88007b571fd8 0000000000010b00 ffff88007b56e440 ffff8800797b0240 [ 3856.798062] Call Trace: [ 3856.798062] [] bdi_sync_supers+0x40/0x60 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] bdi-default S ffffffff81a36160 0 17 2 0x00000000 [ 3856.798062] ffff88007b577d70 0000000000000046 ffff88007b576010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b577fd8 [ 3856.798062] ffff88007b577fd8 0000000000010b00 ffff88007b574480 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x1b0/0x2a0 [ 3856.798062] [] bdi_forker_thread+0x205/0x4a0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kintegrityd S ffff88007b562ce0 0 18 2 0x00000000 [ 3856.798062] ffff88007b57be60 0000000000000046 ffff88007b57a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b57bfd8 [ 3856.798062] ffff88007b57bfd8 0000000000010b00 ffff88007b5784c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kblockd S ffff88007b5621e0 0 19 2 0x00000000 [ 3856.798062] ffff88007b681e60 0000000000000046 ffff88007b680010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b681fd8 [ 3856.798062] ffff88007b681fd8 0000000000010b00 ffff88007b64e500 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] md S ffff88007b73a9e0 0 20 2 0x00000000 [ 3856.798062] ffff88007b755e60 0000000000000046 ffff88007b754010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b755fd8 [ 3856.798062] ffff88007b755fd8 0000000000010b00 ffff88007b752540 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] khungtaskd S 0000000000000000 0 21 2 0x00000000 [ 3856.798062] ffff880079537e40 0000000000000046 ffff880079536010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079537fd8 [ 3856.798062] ffff880079537fd8 0000000000010b00 ffff880079534580 ffff88007b47c240 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] watchdog+0x3a/0x50 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/1:1 S 0000000000011580 0 22 2 0x00000000 [ 3856.798062] ffff88007953de70 0000000000000046 ffff88007953c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007953dfd8 [ 3856.798062] ffff88007953dfd8 0000000000010b00 ffff88007953a5c0 ffff88007b4a4280 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kswapd0 S ffff88007ffbc068 0 23 2 0x00000000 [ 3856.798062] ffff880079563e00 0000000000000046 ffff880079562010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079563fd8 [ 3856.798062] ffff880079563fd8 0000000000010b00 ffff880079560600 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] kswapd_try_to_sleep+0x185/0x1c0 [ 3856.798062] [] kswapd+0xd0/0x230 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ksmd S ffff8800795a2640 0 24 2 0x00000000 [ 3856.798062] ffff8800795a5e60 0000000000000046 ffff8800795a4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800795a5fd8 [ 3856.798062] ffff8800795a5fd8 0000000000010b00 ffff8800795a2640 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] ksm_scan_thread+0x295/0x2e0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] khugepaged S ffff8800795a8040 0 25 2 0x00000000 [ 3856.798062] ffff8800795abdb0 0000000000000046 ffff8800795aa010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800795abfd8 [ 3856.798062] ffff8800795abfd8 0000000000010b00 ffff8800795a8040 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x1b0/0x2a0 [ 3856.798062] [] khugepaged_loop+0x1f0/0x3c0 [ 3856.798062] [] khugepaged+0x4a/0xb0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] fsnotify_mark S ffff8800795ae080 0 26 2 0x00000000 [ 3856.798062] ffff880079601e60 0000000000000046 ffff880079600010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079601fd8 [ 3856.798062] ffff880079601fd8 0000000000010b00 ffff8800795ae080 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] fsnotify_mark_destroy+0x119/0x150 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] crypto S ffff8800795994e0 0 27 2 0x00000000 [ 3856.798062] ffff88007966fe60 0000000000000046 ffff88007966e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007966ffd8 [ 3856.798062] ffff88007966ffd8 0000000000010b00 ffff88007966c0c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kthrotld S ffff880079599260 0 31 2 0x00000000 [ 3856.798062] ffff8800796fbe60 0000000000000046 ffff8800796fa010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800796fbfd8 [ 3856.798062] ffff8800796fbfd8 0000000000010b00 ffff8800796f81c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kpsmoused S ffff88007b73a360 0 32 2 0x00000000 [ 3856.798062] ffff880079769e60 0000000000000046 ffff880079768010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079769fd8 [ 3856.798062] ffff880079769fd8 0000000000010b00 ffff880079766200 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/0:2 S 0000000000011580 0 33 2 0x00000000 [ 3856.798062] ffff8800797b3e70 0000000000000046 ffff8800797b2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800797b3fd8 [ 3856.798062] ffff8800797b3fd8 0000000000010b00 ffff8800797b0240 ffff880079c0c100 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ata_sff S ffff8800797d0460 0 93 2 0x00000000 [ 3856.798062] ffff880037889e60 0000000000000046 ffff880037888010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037889fd8 [ 3856.798062] ffff880037889fd8 0000000000010b00 ffff880037892100 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] scsi_eh_0 S 0000000000000000 0 140 2 0x00000000 [ 3856.798062] ffff880037ad1eb0 0000000000000046 ffff880037ad0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037ad1fd8 [ 3856.798062] ffff880037ad1fd8 0000000000010b00 ffff88003787c640 ffff880037804300 [ 3856.798062] Call Trace: [ 3856.798062] [] scsi_error_handler+0x7e/0x1d0 [scsi_mod] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] scsi_eh_1 S 0000000000000000 0 141 2 0x00000000 [ 3856.798062] ffff880037aadeb0 0000000000000046 ffff880037aac010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037aadfd8 [ 3856.798062] ffff880037aadfd8 0000000000010b00 ffff880037804300 ffff8800797fc4c0 [ 3856.798062] Call Trace: [ 3856.798062] [] scsi_error_handler+0x7e/0x1d0 [scsi_mod] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kworker/u:1 S 0000000000011580 0 142 2 0x00000000 [ 3856.798062] ffff8800378b1e70 0000000000000046 ffff8800378b0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800378b1fd8 [ 3856.798062] ffff8800378b1fd8 0000000000010b00 ffff880037aae2c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] worker_thread+0x258/0x410 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kmpath_rdacd S ffff88003789d260 0 173 2 0x00000000 [ 3856.798062] ffff880037aa5e60 0000000000000046 ffff880037aa4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037aa5fd8 [ 3856.798062] ffff880037aa5fd8 0000000000010b00 ffff880037876600 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kmpath_aluad S ffff8800797d2460 0 195 2 0x00000000 [ 3856.798062] ffff880037a6be60 0000000000000046 ffff880037a6a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037a6bfd8 [ 3856.798062] ffff880037a6bfd8 0000000000010b00 ffff8800797f8480 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] khubd S ffff880079111e98 0 260 2 0x00000000 [ 3856.798062] ffff880079111e70 0000000000000046 ffff880079110010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079111fd8 [ 3856.798062] ffff880079111fd8 0000000000010b00 ffff8800796f4180 ffff8800797b0240 [ 3856.798062] Call Trace: [ 3856.798062] [] hub_thread+0xf6/0x1a0 [usbcore] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kjournald S ffff8800793c9e80 0 402 2 0x00000000 [ 3856.798062] ffff8800793c9e60 0000000000000046 ffff8800793c8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800793c9fd8 [ 3856.798062] ffff8800793c9fd8 0000000000010b00 ffff880037adc380 ffff88007b47c240 [ 3856.798062] Call Trace: [ 3856.798062] [] kjournald+0x20b/0x230 [jbd] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] flush-253:0 S ffff8800378d37c8 0 408 2 0x00000000 [ 3856.798062] ffff880078c7fe60 0000000000000046 ffff880078c7e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078c7ffd8 [ 3856.798062] ffff880078c7ffd8 0000000000010b00 ffff8800377663c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] bdi_writeback_thread+0x1e4/0x240 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] udevd S ffff88007af1be3c 0 496 1 0x00000000 [ 3856.798062] ffff88007af1ba28 0000000000000082 ffff88007af1a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007af1bfd8 [ 3856.798062] ffff88007af1bfd8 0000000000010b00 ffff880078c84580 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ffae014815f>] 0x7ffae014815e [ 3856.798062] udevd S ffff880079ef3ddc 0 592 496 0x00000000 [ 3856.798062] ffff880079ef39e8 0000000000000082 ffff880079ef2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079ef3fd8 [ 3856.798062] ffff880079ef3fd8 0000000000010b00 ffff880079ef0140 ffff880078c84580 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_ppoll+0x4f/0x180 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ffae0148267>] 0x7ffae0148266 [ 3856.798062] udevd S ffff8800376c7ddc 0 593 496 0x00000000 [ 3856.798062] ffff8800376c79e8 0000000000000086 ffff8800376c6010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800376c7fd8 [ 3856.798062] ffff8800376c7fd8 0000000000010b00 ffff8800376c4180 ffff8800792be040 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_ppoll+0x4f/0x180 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ffae0148267>] 0x7ffae0148266 [ 3856.798062] vballoon S ffff88007a821c78 0 624 2 0x00000000 [ 3856.798062] ffff88007a50de50 0000000000000046 ffff88007a50c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a50dfd8 [ 3856.798062] ffff88007a50dfd8 0000000000010b00 ffff880078c64400 ffff8800376bc300 [ 3856.798062] Call Trace: [ 3856.798062] [] balloon+0x20b/0x290 [virtio_balloon] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kauditd S 0000000000000000 0 689 2 0x00000000 [ 3856.798062] ffff88007a8dde80 0000000000000046 ffff88007a8dc010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a8ddfd8 [ 3856.798062] ffff88007a8ddfd8 0000000000010b00 ffff88007900a400 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] kauditd_thread+0x185/0x1a0 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] acpid S ffff88007a715e24 0 1258 1 0x00000000 [ 3856.798062] ffff88007a715a28 0000000000000086 ffff88007a714010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a715fd8 [ 3856.798062] ffff88007a715fd8 0000000000010b00 ffff880037740100 ffff88007a02e480 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f20660f515f>] 0x7f20660f515e [ 3856.798062] dbus-daemon S ffff88007abd9e3c 0 1271 1 0x00000000 [ 3856.798062] ffff88007abd9a28 0000000000000082 ffff88007abd8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007abd9fd8 [ 3856.798062] ffff88007abd9fd8 0000000000010b00 ffff88007910a2c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd79f17d15f>] 0x7fd79f17d15e [ 3856.798062] hald S ffff88007a707e7c 0 1360 1 0x00000000 [ 3856.798062] ffff88007a707a28 0000000000000082 ffff88007a706010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a707fd8 [ 3856.798062] ffff88007a707fd8 0000000000010b00 ffff8800792be040 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ff3229da196>] 0x7ff3229da195 [ 3856.798062] hald S 0000000000800000 0 1365 1 0x00000000 [ 3856.798062] ffff88007a73bcf8 0000000000000082 ffff88007a73a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a73bfd8 [ 3856.798062] ffff88007a73bfd8 0000000000010b00 ffff88007abcc540 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] pipe_wait+0x52/0x70 [ 3856.798062] [] pipe_read+0x428/0x550 [ 3856.798062] [] do_sync_read+0xd7/0x120 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ff3229d5bbd>] 0x7ff3229d5bbc [ 3856.798062] console-kit-dae S ffff88007a5cfe34 0 1363 1 0x00000000 [ 3856.798062] ffff88007a5cfa28 0000000000000082 ffff88007a5ce010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a5cffd8 [ 3856.798062] ffff88007a5cffd8 0000000000010b00 ffff880037670200 ffff8800792be040 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4b196>] 0x7fcceca4b195 [ 3856.798062] console-kit-dae S ffff88007a73dea0 0 1366 1 0x00000000 [ 3856.798062] ffff88007a73dd48 0000000000000082 ffff88007a73c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a73dfd8 [ 3856.798062] ffff88007a73dfd8 0000000000010b00 ffff88007a5f0340 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceccfd66c>] 0x7fcceccfd66b [ 3856.798062] console-kit-dae S 0000000000000000 0 1367 1 0x00000000 [ 3856.798062] ffff88007a73fcf8 0000000000000082 ffff88007a73e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a73ffd8 [ 3856.798062] ffff88007a73ffd8 0000000000010b00 ffff88007a50a4c0 ffff88007af520c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1368 1 0x00000000 [ 3856.798062] ffff88007a741cf8 0000000000000082 ffff88007a740010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a741fd8 [ 3856.798062] ffff88007a741fd8 0000000000010b00 ffff88007abca0c0 ffff88007942c0c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1369 1 0x00000000 [ 3856.798062] ffff88007a743cf8 0000000000000082 ffff88007a742010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a743fd8 [ 3856.798062] ffff88007a743fd8 0000000000010b00 ffff88007a70e140 ffff88007a704480 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1370 1 0x00000000 [ 3856.798062] ffff88007a745cf8 0000000000000082 ffff88007a744010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a745fd8 [ 3856.798062] ffff88007a745fd8 0000000000010b00 ffff880079184300 ffff880079434140 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1371 1 0x00000000 [ 3856.798062] ffff88007a747cf8 0000000000000082 ffff88007a746010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a747fd8 [ 3856.798062] ffff88007a747fd8 0000000000010b00 ffff88007a704480 ffff880037670200 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1372 1 0x00000000 [ 3856.798062] ffff88007a749cf8 0000000000000082 ffff88007a748010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a749fd8 [ 3856.798062] ffff88007a749fd8 0000000000010b00 ffff88007af520c0 ffff88007945e3c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1373 1 0x00000000 [ 3856.798062] ffff88007a74bcf8 0000000000000082 ffff88007a74a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a74bfd8 [ 3856.798062] ffff88007a74bfd8 0000000000010b00 ffff88007a992540 ffff88007a750140 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1374 1 0x00000000 [ 3856.798062] ffff88007a74fcf8 0000000000000082 ffff88007a74e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a74ffd8 [ 3856.798062] ffff88007a74ffd8 0000000000010b00 ffff88007a74c100 ffff88007946e4c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1375 1 0x00000000 [ 3856.798062] ffff88007a753cf8 0000000000000082 ffff88007a752010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a753fd8 [ 3856.798062] ffff88007a753fd8 0000000000010b00 ffff88007a750140 ffff88007a7581c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1376 1 0x00000000 [ 3856.798062] ffff88007a757cf8 0000000000000082 ffff88007a756010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a757fd8 [ 3856.798062] ffff88007a757fd8 0000000000010b00 ffff88007a754180 ffff88007947a580 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1377 1 0x00000000 [ 3856.798062] ffff88007a75bcf8 0000000000000082 ffff88007a75a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a75bfd8 [ 3856.798062] ffff88007a75bfd8 0000000000010b00 ffff88007a7581c0 ffff88007a76c300 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1378 1 0x00000000 [ 3856.798062] ffff88007a75fcf8 0000000000000082 ffff88007a75e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a75ffd8 [ 3856.798062] ffff88007a75ffd8 0000000000010b00 ffff88007a75c200 ffff88007a760240 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1379 1 0x00000000 [ 3856.798062] ffff88007a763cf8 0000000000000082 ffff88007a762010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a763fd8 [ 3856.798062] ffff88007a763fd8 0000000000010b00 ffff88007a760240 ffff88007944a280 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1380 1 0x00000000 [ 3856.798062] ffff88007a767cf8 0000000000000082 ffff88007a766010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a767fd8 [ 3856.798062] ffff88007a767fd8 0000000000010b00 ffff88007a764280 ffff88007940c500 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1381 1 0x00000000 [ 3856.798062] ffff88007a76bcf8 0000000000000082 ffff88007a76a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a76bfd8 [ 3856.798062] ffff88007a76bfd8 0000000000010b00 ffff88007a7682c0 ffff88007a76c300 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1382 1 0x00000000 [ 3856.798062] ffff88007a76fcf8 0000000000000082 ffff88007a76e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a76ffd8 [ 3856.798062] ffff88007a76ffd8 0000000000010b00 ffff88007a76c300 ffff88007a70e140 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1383 1 0x00000000 [ 3856.798062] ffff88007a773cf8 0000000000000082 ffff88007a772010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a773fd8 [ 3856.798062] ffff88007a773fd8 0000000000010b00 ffff88007a770340 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1384 1 0x00000000 [ 3856.798062] ffff88007a777cf8 0000000000000082 ffff88007a776010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a777fd8 [ 3856.798062] ffff88007a777fd8 0000000000010b00 ffff88007a774380 ffff88007abca0c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1385 1 0x00000000 [ 3856.798062] ffff88007a77bcf8 0000000000000082 ffff88007a77a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a77bfd8 [ 3856.798062] ffff88007a77bfd8 0000000000010b00 ffff88007a7783c0 ffff880079e1c140 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1386 1 0x00000000 [ 3856.798062] ffff88007a77fcf8 0000000000000082 ffff88007a77e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a77ffd8 [ 3856.798062] ffff88007a77ffd8 0000000000010b00 ffff88007a77c400 ffff880079476540 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1387 1 0x00000000 [ 3856.798062] ffff880079403cf8 0000000000000082 ffff880079402010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079403fd8 [ 3856.798062] ffff880079403fd8 0000000000010b00 ffff880079400440 ffff880079428080 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1388 1 0x00000000 [ 3856.798062] ffff880079407cf8 0000000000000082 ffff880079406010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079407fd8 [ 3856.798062] ffff880079407fd8 0000000000010b00 ffff880079404480 ffff880079e10080 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1389 1 0x00000000 [ 3856.798062] ffff88007940bcf8 0000000000000082 ffff88007940a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007940bfd8 [ 3856.798062] ffff88007940bfd8 0000000000010b00 ffff8800794084c0 ffff88007943c1c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1390 1 0x00000000 [ 3856.798062] ffff88007940fcf8 0000000000000082 ffff88007940e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007940ffd8 [ 3856.798062] ffff88007940ffd8 0000000000010b00 ffff88007940c500 ffff88007a770340 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1391 1 0x00000000 [ 3856.798062] ffff880079413cf8 0000000000000082 ffff880079412010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079413fd8 [ 3856.798062] ffff880079413fd8 0000000000010b00 ffff880079410540 ffff88007a754180 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1392 1 0x00000000 [ 3856.798062] ffff880079417cf8 0000000000000082 ffff880079416010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079417fd8 [ 3856.798062] ffff880079417fd8 0000000000010b00 ffff880079414580 ffff880079400440 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1393 1 0x00000000 [ 3856.798062] ffff88007941bcf8 0000000000000082 ffff88007941a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007941bfd8 [ 3856.798062] ffff88007941bfd8 0000000000010b00 ffff8800794185c0 ffff880079424040 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1394 1 0x00000000 [ 3856.798062] ffff88007941fcf8 0000000000000082 ffff88007941e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007941ffd8 [ 3856.798062] ffff88007941ffd8 0000000000010b00 ffff88007941c600 ffff88007a50a4c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1395 1 0x00000000 [ 3856.798062] ffff880079423cf8 0000000000000082 ffff880079422010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079423fd8 [ 3856.798062] ffff880079423fd8 0000000000010b00 ffff880079420640 ffff880079e0a040 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1396 1 0x00000000 [ 3856.798062] ffff880079427cf8 0000000000000082 ffff880079426010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079427fd8 [ 3856.798062] ffff880079427fd8 0000000000010b00 ffff880079424040 ffff880079184300 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1397 1 0x00000000 [ 3856.798062] ffff88007942bcf8 0000000000000082 ffff88007942a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007942bfd8 [ 3856.798062] ffff88007942bfd8 0000000000010b00 ffff880079428080 ffff88007a77c400 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1398 1 0x00000000 [ 3856.798062] ffff88007942fcf8 0000000000000082 ffff88007942e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007942ffd8 [ 3856.798062] ffff88007942ffd8 0000000000010b00 ffff88007942c0c0 ffff880079414580 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1399 1 0x00000000 [ 3856.798062] ffff880079433cf8 0000000000000082 ffff880079432010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079433fd8 [ 3856.798062] ffff880079433fd8 0000000000010b00 ffff880079430100 ffff880079456340 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1400 1 0x00000000 [ 3856.798062] ffff880079437cf8 0000000000000082 ffff880079436010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079437fd8 [ 3856.798062] ffff880079437fd8 0000000000010b00 ffff880079434140 ffff88007a7783c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1401 1 0x00000000 [ 3856.798062] ffff88007943bcf8 0000000000000082 ffff88007943a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007943bfd8 [ 3856.798062] ffff88007943bfd8 0000000000010b00 ffff880079438180 ffff880079420640 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1402 1 0x00000000 [ 3856.798062] ffff88007943fcf8 0000000000000082 ffff88007943e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007943ffd8 [ 3856.798062] ffff88007943ffd8 0000000000010b00 ffff88007943c1c0 ffff88007a7682c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1403 1 0x00000000 [ 3856.798062] ffff880079445cf8 0000000000000082 ffff880079444010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079445fd8 [ 3856.798062] ffff880079445fd8 0000000000010b00 ffff880079442200 ffff88007940c500 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1404 1 0x00000000 [ 3856.798062] ffff880079449cf8 0000000000000082 ffff880079448010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079449fd8 [ 3856.798062] ffff880079449fd8 0000000000010b00 ffff880079446240 ffff880037670200 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1405 1 0x00000000 [ 3856.798062] ffff88007944dcf8 0000000000000082 ffff88007944c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007944dfd8 [ 3856.798062] ffff88007944dfd8 0000000000010b00 ffff88007944a280 ffff880079462400 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1406 1 0x00000000 [ 3856.798062] ffff880079451cf8 0000000000000082 ffff880079450010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079451fd8 [ 3856.798062] ffff880079451fd8 0000000000010b00 ffff88007944e2c0 ffff88007946a480 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1407 1 0x00000000 [ 3856.798062] ffff880079455cf8 0000000000000082 ffff880079454010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079455fd8 [ 3856.798062] ffff880079455fd8 0000000000010b00 ffff880079452300 ffff88007944e2c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1408 1 0x00000000 [ 3856.798062] ffff880079459cf8 0000000000000082 ffff880079458010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079459fd8 [ 3856.798062] ffff880079459fd8 0000000000010b00 ffff880079456340 ffff8800794185c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1409 1 0x00000000 [ 3856.798062] ffff88007945dcf8 0000000000000082 ffff88007945c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007945dfd8 [ 3856.798062] ffff88007945dfd8 0000000000010b00 ffff88007945a380 ffff88007a992540 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1410 1 0x00000000 [ 3856.798062] ffff880079461cf8 0000000000000082 ffff880079460010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079461fd8 [ 3856.798062] ffff880079461fd8 0000000000010b00 ffff88007945e3c0 ffff88007a75c200 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1411 1 0x00000000 [ 3856.798062] ffff880079465cf8 0000000000000082 ffff880079464010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079465fd8 [ 3856.798062] ffff880079465fd8 0000000000010b00 ffff880079462400 ffff880079472500 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1412 1 0x00000000 [ 3856.798062] ffff880079469cf8 0000000000000082 ffff880079468010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079469fd8 [ 3856.798062] ffff880079469fd8 0000000000010b00 ffff880079466440 ffff880079e140c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1413 1 0x00000000 [ 3856.798062] ffff88007946dcf8 0000000000000082 ffff88007946c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007946dfd8 [ 3856.798062] ffff88007946dfd8 0000000000010b00 ffff88007946a480 ffff88007945a380 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1414 1 0x00000000 [ 3856.798062] ffff880079471cf8 0000000000000082 ffff880079470010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079471fd8 [ 3856.798062] ffff880079471fd8 0000000000010b00 ffff88007946e4c0 ffff88007941c600 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1415 1 0x00000000 [ 3856.798062] ffff880079475cf8 0000000000000082 ffff880079474010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079475fd8 [ 3856.798062] ffff880079475fd8 0000000000010b00 ffff880079472500 ffff880079452300 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1416 1 0x00000000 [ 3856.798062] ffff880079479cf8 0000000000000082 ffff880079478010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079479fd8 [ 3856.798062] ffff880079479fd8 0000000000010b00 ffff880079476540 ffff880079410540 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1417 1 0x00000000 [ 3856.798062] ffff88007947dcf8 0000000000000082 ffff88007947c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007947dfd8 [ 3856.798062] ffff88007947dfd8 0000000000010b00 ffff88007947a580 ffff88007a74c100 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1418 1 0x00000000 [ 3856.798062] ffff880079e01cf8 0000000000000082 ffff880079e00010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e01fd8 [ 3856.798062] ffff880079e01fd8 0000000000010b00 ffff88007947e5c0 ffff880079e06640 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1419 1 0x00000000 [ 3856.798062] ffff880079e05cf8 0000000000000082 ffff880079e04010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e05fd8 [ 3856.798062] ffff880079e05fd8 0000000000010b00 ffff880079e02600 ffff880079430100 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1420 1 0x00000000 [ 3856.798062] ffff880079e09cf8 0000000000000082 ffff880079e08010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e09fd8 [ 3856.798062] ffff880079e09fd8 0000000000010b00 ffff880079e06640 ffff880079e20180 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1421 1 0x00000000 [ 3856.798062] ffff880079e0dcf8 0000000000000082 ffff880079e0c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e0dfd8 [ 3856.798062] ffff880079e0dfd8 0000000000010b00 ffff880079e0a040 ffff880079e18100 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1422 1 0x00000000 [ 3856.798062] ffff880079e13cf8 0000000000000082 ffff880079e12010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e13fd8 [ 3856.798062] ffff880079e13fd8 0000000000010b00 ffff880079e10080 ffff880079e02600 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1423 1 0x00000000 [ 3856.798062] ffff880079e17cf8 0000000000000082 ffff880079e16010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e17fd8 [ 3856.798062] ffff880079e17fd8 0000000000010b00 ffff880079e140c0 ffff88007abca0c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1424 1 0x00000000 [ 3856.798062] ffff880079e1bcf8 0000000000000082 ffff880079e1a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e1bfd8 [ 3856.798062] ffff880079e1bfd8 0000000000010b00 ffff880079e18100 ffff88007947e5c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1425 1 0x00000000 [ 3856.798062] ffff880079e1fcf8 0000000000000082 ffff880079e1e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e1ffd8 [ 3856.798062] ffff880079e1ffd8 0000000000010b00 ffff880079e1c140 ffff880079466440 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1426 1 0x00000000 [ 3856.798062] ffff880079e23cf8 0000000000000082 ffff880079e22010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e23fd8 [ 3856.798062] ffff880079e23fd8 0000000000010b00 ffff880079e20180 ffff880079404480 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] console-kit-dae S 0000000000000000 0 1427 1 0x00000000 [ 3856.798062] ffff880079e27cf8 0000000000000082 ffff880079e26010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e27fd8 [ 3856.798062] ffff880079e27fd8 0000000000010b00 ffff880079e241c0 ffff8800794084c0 [ 3856.798062] Call Trace: [ 3856.798062] [] vt_event_wait+0xa6/0x130 [ 3856.798062] [] vt_waitactive+0x2d/0x60 [ 3856.798062] [] vt_ioctl+0x1044/0x1cf0 [ 3856.798062] [] tty_ioctl+0x377/0x9a0 [ 3856.798062] [] do_vfs_ioctl+0x8b/0x3b0 [ 3856.798062] [] sys_ioctl+0xa1/0xb0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcceca4c9a7>] 0x7fcceca4c9a6 [ 3856.798062] hald-runner S ffff88007a739e1c 0 1364 1360 0x00000000 [ 3856.798062] ffff88007a739a28 0000000000000082 ffff88007a738010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a739fd8 [ 3856.798062] ffff88007a739fd8 0000000000010b00 ffff8800792c01c0 ffff8800792be040 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f8f3066315f>] 0x7f8f3066315e [ 3856.798062] hald-addon-inpu S ffff88007abefe2c 0 1452 1364 0x00000000 [ 3856.798062] ffff88007abefa28 0000000000000082 ffff88007abee010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007abeffd8 [ 3856.798062] ffff88007abeffd8 0000000000010b00 ffff880079e78300 ffff8800792be040 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f201395a15f>] 0x7f201395a15e [ 3856.798062] hald-addon-acpi S ffff880079e29cb8 0 1472 1364 0x00000000 [ 3856.798062] ffff88007a06bb68 0000000000000086 ffff88007a06a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a06bfd8 [ 3856.798062] ffff88007a06bfd8 0000000000010b00 ffff88007a02e480 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] unix_stream_data_wait+0xab/0x110 [ 3856.798062] [] unix_stream_recvmsg+0x3f3/0x530 [ 3856.798062] [] sock_aio_read+0x121/0x130 [ 3856.798062] [] do_sync_read+0xd7/0x120 [ 3856.798062] [] vfs_read+0x125/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f66b760aba0>] 0x7f66b760ab9f [ 3856.798062] mthca_catas S ffff88007aa1fd60 0 1495 2 0x00000000 [ 3856.798062] ffff88007aa3de60 0000000000000046 ffff88007aa3c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007aa3dfd8 [ 3856.798062] ffff88007aa3dfd8 0000000000010b00 ffff88007a8d2040 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] mlx4 S ffff88007aa1f860 0 1497 2 0x00000000 [ 3856.798062] ffff880079e71e60 0000000000000046 ffff880079e70010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e71fd8 [ 3856.798062] ffff880079e71fd8 0000000000010b00 ffff88007a6ac580 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ib_mcast S ffff88007aa1f7e0 0 1499 2 0x00000000 [ 3856.798062] ffff880079e2fe60 0000000000000046 ffff880079e2e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079e2ffd8 [ 3856.798062] ffff880079e2ffd8 0000000000010b00 ffff88007a69a080 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] mlx4_ib S ffff880079e76f60 0 1500 2 0x00000000 [ 3856.798062] ffff88007a065e60 0000000000000046 ffff88007a064010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a065fd8 [ 3856.798062] ffff88007a065fd8 0000000000010b00 ffff88007a01a5c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] mlx4_ib_mcg S ffff880079e76e60 0 1501 2 0x00000000 [ 3856.798062] ffff88007a5f5e60 0000000000000046 ffff88007a5f4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a5f5fd8 [ 3856.798062] ffff88007a5f5fd8 0000000000010b00 ffff88007a036500 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] mlx5_core_wq S ffff88007a5d7760 0 1506 2 0x00000000 [ 3856.798062] ffff88007aaf5e60 0000000000000046 ffff88007aaf4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007aaf5fd8 [ 3856.798062] ffff88007aaf5fd8 0000000000010b00 ffff880079e7c340 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] qib_cq S ffff88007a406860 0 1509 2 0x00000000 [ 3856.798062] ffff88007a4f5e60 0000000000000046 ffff88007a4f4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a4f5fd8 [ 3856.798062] ffff88007a4f5fd8 0000000000010b00 ffff88007a4f2040 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] iw_cxgb3 S ffff880079e76b60 0 1513 2 0x00000000 [ 3856.798062] ffff880079f13e60 0000000000000046 ffff880079f12010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079f13fd8 [ 3856.798062] ffff880079f13fd8 0000000000010b00 ffff880079e6c580 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ib_cm S ffff88007aa1fa60 0 1519 2 0x00000000 [ 3856.798062] ffff88007aaf7e60 0000000000000046 ffff88007aaf6010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007aaf7fd8 [ 3856.798062] ffff88007aaf7fd8 0000000000010b00 ffff88007a0324c0 ffff8800792be040 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ipoib S ffff88007aa2fce0 0 1520 2 0x00000000 [ 3856.798062] ffff880079f5be60 0000000000000046 ffff880079f5a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079f5bfd8 [ 3856.798062] ffff880079f5bfd8 0000000000010b00 ffff880079e4e280 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] ib_addr S ffff88007aa2f660 0 1527 2 0x00000000 [ 3856.798062] ffff88007a4f1e60 0000000000000046 ffff88007a4f0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a4f1fd8 [ 3856.798062] ffff88007a4f1fd8 0000000000010b00 ffff88007abe0640 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] iw_cm_wq S ffff880079e76d60 0 1528 2 0x00000000 [ 3856.798062] ffff880079f65e60 0000000000000046 ffff880079f64010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079f65fd8 [ 3856.798062] ffff880079f65fd8 0000000000010b00 ffff88007a02a440 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] rdma_cm S ffff88007a5a6860 0 1529 2 0x00000000 [ 3856.798062] ffff880079fabe60 0000000000000046 ffff880079faa010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079fabfd8 [ 3856.798062] ffff880079fabfd8 0000000000010b00 ffff88007a0223c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] dhcpcd S ffff880078db1e1c 0 2476 1 0x00000000 [ 3856.798062] ffff880078db1a28 0000000000000086 ffff880078db0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078db1fd8 [ 3856.798062] ffff880078db1fd8 0000000000010b00 ffff880079fec1c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f561c55c15f>] 0x7f561c55c15e [ 3856.798062] rsyslogd S ffff880079fc5ad8 0 2865 1 0x00000000 [ 3856.798062] ffff880079fc5948 0000000000000082 ffff880079fc4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079fc5fd8 [ 3856.798062] ffff880079fc5fd8 0000000000010b00 ffff88007919c340 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d98304963>] 0x7f9d98304962 [ 3856.798062] rs:main Q:Reg S ffff880079fa5ea0 0 2866 1 0x00000000 [ 3856.798062] ffff880079fa5d48 0000000000000082 ffff880079fa4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079fa5fd8 [ 3856.798062] ffff880079fa5fd8 0000000000010b00 ffff8800376bc300 ffff8800790a00c0 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d989c166c>] 0x7f9d989c166b [ 3856.798062] rsyslogd S ffff88007a5f3ad8 0 2867 1 0x00000000 [ 3856.798062] ffff88007a5f3948 0000000000000082 ffff88007a5f2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a5f3fd8 [ 3856.798062] ffff88007a5f3fd8 0000000000010b00 ffff880078dc6300 ffff8800376bc300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d98304963>] 0x7f9d98304962 [ 3856.798062] rsyslogd S ffff88007a5bdad8 0 2868 1 0x00000000 [ 3856.798062] ffff88007a5bd948 0000000000000082 ffff88007a5bc010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a5bdfd8 [ 3856.798062] ffff88007a5bdfd8 0000000000010b00 ffff880079e4a240 ffff8800376bc300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d98304963>] 0x7f9d98304962 [ 3856.798062] rsyslogd S 00007f9d95f965d0 0 2869 1 0x00000000 [ 3856.798062] ffff880078d01e38 0000000000000082 ffff880078d00010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078d01fd8 [ 3856.798062] ffff880078d01fd8 0000000000010b00 ffff880078d98140 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] do_syslog+0x461/0x4d0 [ 3856.798062] [] kmsg_read+0x2e/0x70 [ 3856.798062] [] proc_reg_read+0x77/0xc0 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d989c47fd>] 0x7f9d989c47fc [ 3856.798062] auditd S 00000000038f9b17 0 2881 1 0x00000000 [ 3856.798062] ffff880079ff7db8 0000000000000082 ffff880079ff6010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079ff7fd8 [ 3856.798062] ffff880079ff7fd8 0000000000010b00 ffff880079fca500 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f651fb43d03>] 0x7f651fb43d02 [ 3856.798062] auditd S ffff880078d9dea0 0 2882 1 0x00000000 [ 3856.798062] ffff880078d9dd48 0000000000000082 ffff880078d9c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078d9dfd8 [ 3856.798062] ffff880078d9dfd8 0000000000010b00 ffff880078d0e640 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f652006566c>] 0x7f652006566b [ 3856.798062] audispd S ffff88007a70dea0 0 2883 2881 0x00000000 [ 3856.798062] ffff88007a70dd48 0000000000000086 ffff88007a70c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a70dfd8 [ 3856.798062] ffff88007a70dfd8 0000000000010b00 ffff88007a5be040 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fa7b390266c>] 0x7fa7b390266b [ 3856.798062] audispd S ffff880079fd9e24 0 2884 2881 0x00000000 [ 3856.798062] ffff880079fd9a28 0000000000000086 ffff880079fd8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079fd9fd8 [ 3856.798062] ffff880079fd9fd8 0000000000010b00 ffff88007a5c42c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fa7b3650196>] 0x7fa7b3650195 [ 3856.798062] haveged S 0000000000000000 0 2902 1 0x00000000 [ 3856.798062] ffff88007a071948 0000000000000086 ffff88007a070010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a071fd8 [ 3856.798062] ffff88007a071fd8 0000000000010b00 ffff880078d1a340 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f93d6536943>] 0x7f93d6536942 [ 3856.798062] rpcbind S ffff880078dc3e54 0 2914 1 0x00000000 [ 3856.798062] ffff880078dc3a28 0000000000000082 ffff880078dc2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078dc3fd8 [ 3856.798062] ffff880078dc3fd8 0000000000010b00 ffff880079f5e280 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f052528915f>] 0x7f052528915e [ 3856.798062] irqbalance S 00007fefab64c3b4 0 3002 1 0x00000000 [ 3856.798062] ffff880079fffe78 0000000000000086 ffff880079ffe010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079ffffd8 [ 3856.798062] ffff880079ffffd8 0000000000010b00 ffff880078d04500 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] do_nanosleep+0x9d/0xd0 [ 3856.798062] [] hrtimer_nanosleep+0xa8/0x140 [ 3856.798062] [] sys_nanosleep+0x67/0x80 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fefaacce540>] 0x7fefaacce53f [ 3856.798062] mcelog S ffff880078de9de4 0 3016 1 0x00000000 [ 3856.798062] ffff880078de99e8 0000000000000082 ffff880078de8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078de9fd8 [ 3856.798062] ffff880078de9fd8 0000000000010b00 ffff880078dde380 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_ppoll+0x4f/0x180 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fe586033267>] 0x7fe586033266 [ 3856.798062] sshd S ffff88007a1ddad8 0 3313 1 0x00000000 [ 3856.798062] ffff88007a1dd948 0000000000000082 ffff88007a1dc010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1ddfd8 [ 3856.798062] ffff88007a1ddfd8 0000000000010b00 ffff88007a1c40c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f01b3764943>] 0x7f01b3764942 [ 3856.798062] ntpd S ffff88007a163ad8 0 3327 1 0x00000000 [ 3856.798062] ffff88007a163948 0000000000000082 ffff88007a162010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a163fd8 [ 3856.798062] ffff88007a163fd8 0000000000010b00 ffff88007a18a440 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5a5531d943>] 0x7f5a5531d942 [ 3856.798062] ntpd S ffff880079f8c5b8 0 3328 3327 0x00000000 [ 3856.798062] ffff880079fd3b68 0000000000000086 ffff880079fd2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079fd3fd8 [ 3856.798062] ffff880079fd3fd8 0000000000010b00 ffff88007a136300 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] unix_stream_data_wait+0xab/0x110 [ 3856.798062] [] unix_stream_recvmsg+0x3f3/0x530 [ 3856.798062] [] sock_aio_read+0x121/0x130 [ 3856.798062] [] do_sync_read+0xd7/0x120 [ 3856.798062] [] vfs_read+0x125/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5a55316ba0>] 0x7f5a55316b9f [ 3856.798062] munged S ffff88007aee5de0 0 3350 1 0x00000000 [ 3856.798062] ffff88007aee5ce8 0000000000000082 ffff88007aee4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007aee5fd8 [ 3856.798062] ffff88007aee5fd8 0000000000010b00 ffff88007a1a25c0 ffff88007a130200 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] __skb_recv_datagram+0x264/0x270 [ 3856.798062] [] skb_recv_datagram+0x1c/0x30 [ 3856.798062] [] unix_accept+0x7f/0x140 [ 3856.798062] [] sys_accept4+0x177/0x260 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d55648add>] 0x7f9d55648adc [ 3856.798062] munged S ffff88007a19dea0 0 3351 1 0x00000000 [ 3856.798062] ffff88007a19dd48 0000000000000082 ffff88007a19c010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a19dfd8 [ 3856.798062] ffff88007a19dfd8 0000000000010b00 ffff88007a1cc140 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d556459fc>] 0x7f9d556459fb [ 3856.798062] munged S ffff88007a19fea0 0 3352 1 0x00000000 [ 3856.798062] ffff88007a19fd48 0000000000000082 ffff88007a19e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a19ffd8 [ 3856.798062] ffff88007a19ffd8 0000000000010b00 ffff880079fde480 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d5564566c>] 0x7f9d5564566b [ 3856.798062] munged S ffff88007a1a9ea0 0 3353 1 0x00000000 [ 3856.798062] ffff88007a1a9d48 0000000000000082 ffff88007a1a8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1a9fd8 [ 3856.798062] ffff88007a1a9fd8 0000000000010b00 ffff88007a130200 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f9d5564566c>] 0x7f9d5564566b [ 3856.798062] nscd S 0000000001c965bf 0 3366 1 0x00000000 [ 3856.798062] ffff88007aee1db8 0000000000000086 ffff88007aee0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007aee1fd8 [ 3856.798062] ffff88007aee1fd8 0000000000010b00 ffff88007a172240 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b1f46d03>] 0x7fd4b1f46d02 [ 3856.798062] nscd S ffff88007a1abea0 0 3367 1 0x00000000 [ 3856.798062] ffff88007a1abd48 0000000000000086 ffff88007a1aa010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1abfd8 [ 3856.798062] ffff88007a1abfd8 0000000000010b00 ffff880078dae240 ffff8800375e0080 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b24079fc>] 0x7fd4b24079fb [ 3856.798062] nscd S ffff88007a16bea0 0 3368 1 0x00000000 [ 3856.798062] ffff88007a16bd48 0000000000000086 ffff88007a16a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a16bfd8 [ 3856.798062] ffff88007a16bfd8 0000000000010b00 ffff88007a168380 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b24079fc>] 0x7fd4b24079fb [ 3856.798062] nscd S ffff88007a1b3ea0 0 3369 1 0x00000000 [ 3856.798062] ffff88007a1b3d48 0000000000000086 ffff88007a1b2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1b3fd8 [ 3856.798062] ffff88007a1b3fd8 0000000000010b00 ffff88007a1b03c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b24079fc>] 0x7fd4b24079fb [ 3856.798062] nscd S ffff88007a183ea0 0 3370 1 0x00000000 [ 3856.798062] ffff88007a183d48 0000000000000086 ffff88007a182010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a183fd8 [ 3856.798062] ffff88007a183fd8 0000000000010b00 ffff88007a180400 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b240766c>] 0x7fd4b240766b [ 3856.798062] nscd S ffff88007a1dbea0 0 3371 1 0x00000000 [ 3856.798062] ffff88007a1dbd48 0000000000000086 ffff88007a1da010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1dbfd8 [ 3856.798062] ffff88007a1dbfd8 0000000000010b00 ffff88007a1d8440 ffff8800375e0080 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b240766c>] 0x7fd4b240766b [ 3856.798062] nscd S ffff88007a1afea0 0 3372 1 0x00000000 [ 3856.798062] ffff88007a1afd48 0000000000000086 ffff88007a1ae010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1affd8 [ 3856.798062] ffff88007a1affd8 0000000000010b00 ffff88007a1ac480 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b240766c>] 0x7fd4b240766b [ 3856.798062] nscd S ffff88007a1ebea0 0 3373 1 0x00000000 [ 3856.798062] ffff88007a1ebd48 0000000000000086 ffff88007a1ea010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a1ebfd8 [ 3856.798062] ffff88007a1ebfd8 0000000000010b00 ffff88007a1e84c0 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fd4b240766c>] 0x7fd4b240766b [ 3856.798062] sssd S 0000000000986b87 0 3385 1 0x00000000 [ 3856.798062] ffff88007a123db8 0000000000000086 ffff88007a122010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a123fd8 [ 3856.798062] ffff88007a123fd8 0000000000010b00 ffff880078d7c240 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f71a1fc6ce3>] 0x7f71a1fc6ce2 [ 3856.798062] sssd_be S 0000000005f5e100 0 3386 3385 0x00000000 [ 3856.798062] ffff88007a223db8 0000000000000086 ffff88007a222010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a223fd8 [ 3856.798062] ffff88007a223fd8 0000000000010b00 ffff88007a14c080 ffff88007a1be2c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f8d70221ce3>] 0x7f8d70221ce2 [ 3856.798062] sssd_nss S 0000000001c9c37f 0 3387 3385 0x00000000 [ 3856.798062] ffff88007a281db8 0000000000000086 ffff88007a280010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a281fd8 [ 3856.798062] ffff88007a281fd8 0000000000010b00 ffff88007a17e3c0 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f650b663ce3>] 0x7f650b663ce2 [ 3856.798062] sssd_pam S 0000000001c9c37f 0 3388 3385 0x00000000 [ 3856.798062] ffff88007a2c1db8 0000000000000082 ffff88007a2c0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a2c1fd8 [ 3856.798062] ffff88007a2c1fd8 0000000000010b00 ffff88007a1be2c0 ffff880078d7c240 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f343d4d4ce3>] 0x7f343d4d4ce2 [ 3856.798062] sssd_ssh S 0000000001c9c37f 0 3389 3385 0x00000000 [ 3856.798062] ffff88007a301db8 0000000000000086 ffff88007a300010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a301fd8 [ 3856.798062] ffff88007a301fd8 0000000000010b00 ffff88007a108280 ffff880078d7c240 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f51f878bce3>] 0x7f51f878bce2 [ 3856.798062] automount S 00007ffe0486c79c 0 3407 1 0x00000000 [ 3856.798062] ffff88007a279dd8 0000000000000086 ffff88007a278010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a279fd8 [ 3856.798062] ffff88007a279fd8 0000000000010b00 ffff88007a5e0300 ffff88007a1c8100 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] do_sigtimedwait+0x110/0x1c0 [ 3856.798062] [] sys_rt_sigtimedwait+0x94/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d268b3527>] 0x7f5d268b3526 [ 3856.798062] automount S ffff88007a34bea0 0 3408 1 0x00000000 [ 3856.798062] ffff88007a34bd48 0000000000000086 ffff88007a34a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a34bfd8 [ 3856.798062] ffff88007a34bfd8 0000000000010b00 ffff88007a160300 ffff88007a112180 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d268af9fc>] 0x7f5d268af9fb [ 3856.798062] automount S ffff88007a36bea0 0 3409 1 0x00000000 [ 3856.798062] ffff88007a36bd48 0000000000000086 ffff88007a36a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a36bfd8 [ 3856.798062] ffff88007a36bfd8 0000000000010b00 ffff88007a112180 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] futex_wait_queue_me+0xc5/0x110 [ 3856.798062] [] futex_wait+0x117/0x250 [ 3856.798062] [] do_futex+0xb8/0x1c0 [ 3856.798062] [] sys_futex+0x82/0x170 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d268af66c>] 0x7f5d268af66b [ 3856.798062] automount S ffff88007a391e2c 0 3412 1 0x00000000 [ 3856.798062] ffff88007a391a28 0000000000000086 ffff88007a390010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a391fd8 [ 3856.798062] ffff88007a391fd8 0000000000010b00 ffff88007a156400 ffff88007a39c600 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d259e5196>] 0x7f5d259e5195 [ 3856.798062] automount S ffff88007a39fe2c 0 3415 1 0x00000000 [ 3856.798062] ffff88007a39fa28 0000000000000086 ffff88007a39e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a39ffd8 [ 3856.798062] ffff88007a39ffd8 0000000000010b00 ffff88007a39c600 ffff88007a3a0640 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d259e5196>] 0x7f5d259e5195 [ 3856.798062] automount S ffff88007a3a3e2c 0 3416 1 0x00000000 [ 3856.798062] ffff88007a3a3a28 0000000000000086 ffff88007a3a2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a3a3fd8 [ 3856.798062] ffff88007a3a3fd8 0000000000010b00 ffff88007a3a0640 ffff88007a36c580 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d259e5196>] 0x7f5d259e5195 [ 3856.798062] automount S ffff88007a3c5e2c 0 3417 1 0x00000000 [ 3856.798062] ffff88007a3c5a28 0000000000000086 ffff88007a3c4010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a3c5fd8 [ 3856.798062] ffff88007a3c5fd8 0000000000010b00 ffff88007a36c580 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_poll+0x1b2/0x2a0 [ 3856.798062] [] do_sys_poll+0x18e/0x200 [ 3856.798062] [] sys_poll+0x66/0x100 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d259e5196>] 0x7f5d259e5195 [ 3856.798062] master S 000000000365c03f 0 3498 1 0x00000000 [ 3856.798062] ffff88007a345db8 0000000000000082 ffff88007a344010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a345fd8 [ 3856.798062] ffff88007a345fd8 0000000000010b00 ffff880079c1c400 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f5d56c2cce3>] 0x7f5d56c2cce2 [ 3856.798062] pickup S 0000000005f5e0ff 0 3514 3498 0x00000000 [ 3856.798062] ffff880079c69db8 0000000000000082 ffff880079c68010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079c69fd8 [ 3856.798062] ffff880079c69fd8 0000000000010b00 ffff88007a3a65c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f2c31374ce3>] 0x7f2c31374ce2 [ 3856.798062] qmgr S 0000000005f5e100 0 3515 3498 0x00000000 [ 3856.798062] ffff880079c6bdb8 0000000000000086 ffff880079c6a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079c6bfd8 [ 3856.798062] ffff880079c6bfd8 0000000000010b00 ffff88007a3c60c0 ffff880079c1c400 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0xa9/0x140 [ 3856.798062] [] ep_poll+0x315/0x3a0 [ 3856.798062] [] sys_epoll_wait+0xd4/0xe0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f0c33527ce3>] 0x7f0c33527ce2 [ 3856.798062] cron S 0000000000000000 0 3524 1 0x00000000 [ 3856.798062] ffff880079cb9e78 0000000000000086 ffff880079cb8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079cb9fd8 [ 3856.798062] ffff880079cb9fd8 0000000000010b00 ffff88007a15c340 ffff88007a172240 [ 3856.798062] Call Trace: [ 3856.798062] [] do_nanosleep+0x9d/0xd0 [ 3856.798062] [] hrtimer_nanosleep+0xa8/0x140 [ 3856.798062] [] sys_nanosleep+0x67/0x80 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fce620d5540>] 0x7fce620d553f [ 3856.798062] xinetd S ffff88007a341ad8 0 3548 1 0x00000000 [ 3856.798062] ffff88007a341948 0000000000000082 ffff88007a340010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a341fd8 [ 3856.798062] ffff88007a341fd8 0000000000010b00 ffff88007a1c8100 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fcc9a70b943>] 0x7fcc9a70b942 [ 3856.798062] mingetty S 00007ffebc2b7448 0 7345 1 0x00000000 [ 3856.798062] ffff880079cf1d18 0000000000000086 ffff880079cf0010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079cf1fd8 [ 3856.798062] ffff880079cf1fd8 0000000000010b00 ffff880079d36580 ffff88007953a5c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fc2ee4dcba0>] 0x7fc2ee4dcb9f [ 3856.798062] mingetty S 00007fffce204068 0 7346 1 0x00000000 [ 3856.798062] ffff880079cf7d18 0000000000000086 ffff880079cf6010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079cf7fd8 [ 3856.798062] ffff880079cf7fd8 0000000000010b00 ffff8800792ea180 ffff880079d12280 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007febc45b4ba0>] 0x7febc45b4b9f [ 3856.798062] mingetty S 00007ffcc299a108 0 7347 1 0x00000000 [ 3856.798062] ffff880079c3fd18 0000000000000082 ffff880079c3e010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079c3ffd8 [ 3856.798062] ffff880079c3ffd8 0000000000010b00 ffff880037562180 ffff880079d36580 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f8ef3ab7ba0>] 0x7f8ef3ab7b9f [ 3856.798062] mingetty S 00007ffc40c89768 0 7348 1 0x00000000 [ 3856.798062] ffff880037a57d18 0000000000000082 ffff880037a56010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037a57fd8 [ 3856.798062] ffff880037a57fd8 0000000000010b00 ffff880037600040 ffff88007a3581c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fc870e87ba0>] 0x7fc870e87b9f [ 3856.798062] mingetty S 00007ffd23496c88 0 7349 1 0x00000000 [ 3856.798062] ffff88007b643d18 0000000000000082 ffff88007b642010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007b643fd8 [ 3856.798062] ffff88007b643fd8 0000000000010b00 ffff880079d12280 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f2b3bd62ba0>] 0x7f2b3bd62b9f [ 3856.798062] mingetty S 00007ffc14a5b028 0 7350 1 0x00000000 [ 3856.798062] ffff880037769d18 0000000000000086 ffff880037768010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880037769fd8 [ 3856.798062] ffff880037769fd8 0000000000010b00 ffff88007a3581c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fa655c55ba0>] 0x7fa655c55b9f [ 3856.798062] agetty S 00007ffe3637d977 0 7351 1 0x00000000 [ 3856.798062] ffff880079d15d18 0000000000000082 ffff880079d14010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079d15fd8 [ 3856.798062] ffff880079d15fd8 0000000000010b00 ffff8800374a6080 ffff88007b45e0c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] n_tty_read+0x603/0x950 [ 3856.798062] [] tty_read+0xb5/0x110 [ 3856.798062] [] vfs_read+0xc7/0x130 [ 3856.798062] [] sys_read+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fb0b2194ba0>] 0x7fb0b2194b9f [ 3856.798062] rpc.statd S ffff8800375f9ad8 0 7524 1 0x00000000 [ 3856.798062] ffff8800375f9948 0000000000000086 ffff8800375f8010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800375f9fd8 [ 3856.798062] ffff8800375f9fd8 0000000000010b00 ffff88003789a2c0 ffff8800791023c0 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007fabb6574943>] 0x7fabb6574942 [ 3856.798062] rpciod S ffff8800791b71e0 0 7533 2 0x00000000 [ 3856.798062] ffff8800375dfe60 0000000000000046 ffff8800375de010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800375dffd8 [ 3856.798062] ffff8800375dffd8 0000000000010b00 ffff88003786e440 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] nfsiod S ffff88007a3ba160 0 7534 2 0x00000000 [ 3856.798062] ffff88007a33be60 0000000000000046 ffff88007a33a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a33bfd8 [ 3856.798062] ffff88007a33bfd8 0000000000010b00 ffff88007927a3c0 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kdmflush S ffff8800790a24e0 0 7856 2 0x00000000 [ 3856.798062] ffff88007a3bde60 0000000000000046 ffff88007a3bc010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a3bdfd8 [ 3856.798062] ffff88007a3bdfd8 0000000000010b00 ffff88007a110280 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kdmflush S ffff880037b47be0 0 7917 2 0x00000000 [ 3856.798062] ffff880079215e60 0000000000000046 ffff880079214010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880079215fd8 [ 3856.798062] ffff880079215fd8 0000000000010b00 ffff880078c80040 ffff880079102600 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] kdmflush S ffff8800377dabe0 0 7978 2 0x00000000 [ 3856.798062] ffff880078c77e60 0000000000000046 ffff880078c76010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff880078c77fd8 [ 3856.798062] ffff880078c77fd8 0000000000010b00 ffff880037a065c0 ffff880079c02500 [ 3856.798062] Call Trace: [ 3856.798062] [] rescuer_thread+0x2db/0x310 [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] sshd S ffff8800793f3ad8 0 8585 3313 0x00000000 [ 3856.798062] ffff8800793f3948 0000000000000086 ffff8800793f2010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff8800793f3fd8 [ 3856.798062] ffff8800793f3fd8 0000000000010b00 ffff8800375e0080 ffffffff81a11020 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_hrtimeout_range_clock+0x125/0x140 [ 3856.798062] [] poll_schedule_timeout+0x44/0x70 [ 3856.798062] [] do_select+0x5ff/0x680 [ 3856.798062] [] core_sys_select+0x19c/0x2f0 [ 3856.798062] [] sys_select+0x51/0x110 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007f8ee02bd943>] 0x7f8ee02bd942 [ 3856.798062] sh R running task 0 8587 8585 0x00000000 [ 3856.798062] ffffffff00000018 ffff880079f15998 ffff880079f15958 ffffffff8184d07a [ 3856.798062] ffffffff8184d07c ffffffff810a43ae 00007f8ee02bd942 ffff880079f159b0 [ 3856.798062] ffffffff81262506 0000000000000000 ffff8800f9f15b47 ffffffff812627e8 [ 3856.798062] Call Trace: [ 3856.798062] [] dump_trace+0x75/0x300 [ 3856.798062] [] show_stack_log_lvl+0xef/0x150 [ 3856.798062] [] show_state_filter+0x82/0x110 [ 3856.798062] [] __handle_sysrq+0x11d/0x170 [ 3856.798062] [] write_sysrq_trigger+0x38/0x40 [ 3856.798062] [] proc_reg_write+0x77/0xc0 [ 3856.798062] [] vfs_write+0xce/0x140 [ 3856.798062] [] sys_write+0x53/0xa0 [ 3856.798062] [] system_call_fastpath+0x16/0x1b [ 3856.798062] [<00007ff8e8903c00>] 0x7ff8e8903bff [ 3856.798062] lockd S ffff880037b43840 0 8627 2 0x00000000 [ 3856.798062] ffff88007a27bd60 0000000000000046 ffff88007a27a010 0000000000010b00 [ 3856.798062] 0000000000010b00 0000000000010b00 0000000000010b00 ffff88007a27bfd8 [ 3856.798062] ffff88007a27bfd8 0000000000010b00 ffff8800375d6200 ffff88007b4ae300 [ 3856.798062] Call Trace: [ 3856.798062] [] schedule_timeout+0x205/0x2a0 [ 3856.798062] [] svc_recv+0x4e0/0x7c0 [sunrpc] [ 3856.798062] [] lockd+0x140/0x1e0 [lockd] [ 3856.798062] [] kthread+0x96/0xa0 [ 3856.798062] [] kernel_thread_helper+0x4/0x10 [ 3856.798062] Sched Debug Version: v0.10, 3.0.101-94-default #1 [ 3856.798062] ktime : 3859033.883209 [ 3856.798062] sched_clk : 5747242.761744 [ 3856.798062] cpu_clk : 3856798.062420 [ 3856.798062] jiffies : 4295857056 [ 3856.798062] sched_clock_stable : 0 [ 3856.798062] [ 3856.798062] sysctl_sched [ 3856.798062] .sysctl_sched_latency : 12.000000 [ 3856.798062] .sysctl_sched_min_granularity : 4.000000 [ 3856.798062] .sysctl_sched_wakeup_granularity : 5.000000 [ 3856.798062] .sysctl_sched_child_runs_first : 0 [ 3856.798062] .sysctl_sched_features : 15471 [ 3856.798062] .sysctl_sched_tunable_scaling : 1 (logaritmic) [ 3856.798062] [ 3856.798062] cpu#0, 2593.768 MHz [ 3856.798062] .nr_running : 1 [ 3856.798062] .load : 1024 [ 3856.798062] .nr_switches : 87269 [ 3856.798062] .nr_load_updates : 25104 [ 3856.798062] .nr_uninterruptible : -315 [ 3856.798062] .next_balance : 4295.856503 [ 3856.798062] .curr->pid : 8587 [ 3856.798062] .clock : 3856794.118108 [ 3856.798062] .cpu_load[0] : 0 [ 3856.798062] .cpu_load[1] : 576 [ 3856.798062] .cpu_load[2] : 694 [ 3856.798062] .cpu_load[3] : 712 [ 3856.798062] .cpu_load[4] : 595 [ 3856.798062] .yld_count : 3 [ 3856.798062] .sched_switch : 0 [ 3856.798062] .sched_count : 91187 [ 3856.798062] .sched_goidle : 29967 [ 3856.798062] .avg_idle : 115249 [ 3856.798062] .ttwu_count : 48984 [ 3856.798062] .ttwu_local : 44950 [ 3856.798062] [ 3856.798062] cfs_rq[0]:/ [ 3856.798062] .exec_clock : 16526.216040 [ 3856.798062] .MIN_vruntime : 0.000001 [ 3856.798062] .min_vruntime : 37573.278178 [ 3856.798062] .max_vruntime : 0.000001 [ 3856.798062] .spread : 0.000000 [ 3856.798062] .spread0 : 0.000000 [ 3856.798062] .nr_spread_over : 5196 [ 3856.798062] .nr_running : 1 [ 3856.798062] .load : 1024 [ 3856.798062] .load_avg : 0.000000 [ 3856.798062] .load_period : 0.000000 [ 3856.798062] .load_contrib : 0 [ 3856.798062] .load_tg : 0 [ 3856.798062] .tg->cfs_bandwidth.timer_active: 0 [ 3856.798062] .throttled : 0 [ 3856.798062] .throttle_count : 0 [ 3856.798062] [ 3856.798062] rt_rq[0]:/ [ 3856.798062] .rt_nr_running : 0 [ 3856.798062] .rt_throttled : 0 [ 3856.798062] .rt_time : 0.004759 [ 3856.798062] .rt_runtime : 950.000000 [ 3856.798062] [ 3856.798062] runnable tasks: [ 3856.798062] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 3856.798062] ---------------------------------------------------------------------------------------------------------- [ 3856.798062] R sh 8587 37569.118840 63 120 37569.118840 25.048284 60.076051 / [ 3856.798062] [ 3856.798062] cpu#1, 2593.768 MHz [ 3856.798062] .nr_running : 0 [ 3856.798062] .load : 0 [ 3856.798062] .nr_switches : 58548 [ 3856.798062] .nr_load_updates : 15961 [ 3856.798062] .nr_uninterruptible : 315 [ 3856.798062] .next_balance : 4295.857071 [ 3856.798062] .curr->pid : 0 [ 3856.798062] .clock : 3859097.009351 [ 3856.798062] .cpu_load[0] : 0 [ 3856.798062] .cpu_load[1] : 0 [ 3856.798062] .cpu_load[2] : 0 [ 3856.798062] .cpu_load[3] : 0 [ 3856.798062] .cpu_load[4] : 0 [ 3856.798062] .yld_count : 1 [ 3856.798062] .sched_switch : 0 [ 3856.798062] .sched_count : 62425 [ 3856.798062] .sched_goidle : 22165 [ 3856.798062] .avg_idle : 1000000 [ 3856.798062] .ttwu_count : 31509 [ 3856.798062] .ttwu_local : 27922 [ 3856.798062] [ 3856.798062] cfs_rq[1]:/ [ 3856.798062] .exec_clock : 4686.345146 [ 3856.798062] .MIN_vruntime : 0.000001 [ 3856.798062] .min_vruntime : 27582.961155 [ 3856.798062] .max_vruntime : 0.000001 [ 3856.798062] .spread : 0.000000 [ 3856.798062] .spread0 : -9990.317023 [ 3856.798062] .nr_spread_over : 3240 [ 3856.798062] .nr_running : 0 [ 3856.798062] .load : 0 [ 3856.798062] .load_avg : 0.000000 [ 3856.798062] .load_period : 0.000000 [ 3856.798062] .load_contrib : 0 [ 3856.798062] .load_tg : 0 [ 3856.798062] .tg->cfs_bandwidth.timer_active: 0 [ 3856.798062] .throttled : 0 [ 3856.798062] .throttle_count : 0 [ 3856.798062] [ 3856.798062] rt_rq[1]:/ [ 3856.798062] .rt_nr_running : 0 [ 3856.798062] .rt_throttled : 0 [ 3856.798062] .rt_time : 0.003066 [ 3856.798062] .rt_runtime : 950.000000 [ 3856.798062] [ 3856.798062] runnable tasks: [ 3856.798062] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [ 3856.798062] ---------------------------------------------------------------------------------------------------------- [ 3856.798062] Console [trevis-41vm7] disconnected from at 04-04 20:47. Console [trevis-41vm7] connected to at 04-04 20:48. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.0.101-94-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Thu Jan 26 12:20:59 UTC 2017 (c499ea8) [ 0.000000] Command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] Disabled fast string operations [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) [ 0.000000] BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000007fffd000 (usable) [ 0.000000] BIOS-e820: 000000007fffd000 - 0000000080000000 (reserved) [ 0.000000] BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved) [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] DMI 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [ffff8800000fda30] fda30 [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: 0000000000000000-000000007fffd000 [ 0.000000] nmi ring buffer: 262144 [ 0.000000] RAMDISK: 37755000 - 37ff0000 [ 0.000000] Reserving 128MB of memory at 752MB for crashkernel (System RAM: 2047MB) [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at 0000000000000000-000000007fffd000 [ 0.000000] Initmem setup node 0 0000000000000000-000000007fffd000 [ 0.000000] NODE_DATA [000000007ff96000 - 000000007ffbcfff] [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:1befa01, boot clock [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000010 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal empty [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000010 -> 0x0000009d [ 0.000000] 0: 0x00000100 -> 0x0007fffd [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:7ffbc000) [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 25 pages/cpu @ffff88007fc00000 s72192 r8192 d22016 u1048576 [ 0.000000] kvm-clock: cpu 0, msr 0:7fc10a01, primary cpu clock [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 516995 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: root=/dev/vda1 ksdevice=bootif console=tty0 console=ttyS0,115200 resume=/dev/vda2 splash=silent crashkernel=256M-:128M text showopts [ 0.000000] bootsplash: silent mode. [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 1910912k/2097140k available (4569k kernel code, 460k absent, 185768k reserved, 7578k data, 1360k init) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:262400 nr_irqs:512 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Detected 2593.768 MHz processor. [ 0.008000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=10375072) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] kdb version 4.4 by Keith Owens, Scott Lurndal. Copyright SGI, All Rights Reserved kdb_cmd[0]: defcmd archkdb "" "First line arch debugging" kdb_cmd[8]: defcmd archkdbcpu "" "archkdb with only tasks on cpus" kdb_cmd[15]: defcmd archkdbshort "" "archkdb with less detailed backtrace" kdb_cmd[22]: defcmd archkdbcommon "" "Common arch debugging" [ 0.013451] Security Framework initialized [ 0.014299] AppArmor: AppArmor initialized [ 0.015327] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.017774] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.020930] Mount-cache hash table entries: 256 [ 0.022111] Initializing cgroup subsys cpuacct [ 0.022993] Initializing cgroup subsys memory [ 0.023887] Initializing cgroup subsys devices [ 0.024007] Initializing cgroup subsys freezer [ 0.024888] Initializing cgroup subsys net_cls [ 0.025755] Initializing cgroup subsys blkio [ 0.026618] Initializing cgroup subsys perf_event [ 0.028167] Disabled fast string operations [ 0.029029] mce: CPU supports 10 MCE banks [ 0.032615] ACPI: Core revision 20110413 [ 0.035173] Enabling x2apic [ 0.035879] Enabled x2apic [ 0.036007] Switched APIC routing to physical x2apic. [ 0.038064] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.039096] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 [ 0.044002] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.044002] NMI watchdog disabled (cpu0): hardware events not enabled [ 0.044002] Booting Node 0, Processors #1 Ok. [ 0.008000] Disabled fast string operations [ 0.064016] kvm-clock: cpu 1, msr 0:7fd10a01, secondary cpu clock [ 0.064051] NMI watchdog disabled (cpu1): hardware events not enabled [ 0.064099] Brought up 2 CPUs [ 0.064101] Total of 2 processors activated (10375.07 BogoMIPS). [ 0.068952] devtmpfs: initialized [ 0.072199] print_constraints: dummy: [ 0.073092] Time: 20:48:22 Date: 04/04/17 [ 0.074141] NET: Registered protocol family 16 [ 0.075162] ACPI: bus type pci registered [ 0.076093] PCI: Using configuration type 1 for base access [ 0.077573] bio: create slab at 0 [ 0.081862] ACPI: Interpreter enabled [ 0.082664] ACPI: (supports S0 S5) [ 0.083563] ACPI: Using IOAPIC for interrupt routing [ 0.088228] ACPI: No dock devices found. [ 0.089060] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug [ 0.090829] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.093290] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.094845] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.126898] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e) [ 0.137353] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.139456] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.140899] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.142393] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.144062] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.145426] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.147008] vgaarb: loaded [ 0.148071] PCI: Using ACPI for IRQ routing [ 0.149222] NetLabel: Initializing [ 0.149980] NetLabel: domain hash size = 128 [ 0.150835] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.151774] NetLabel: unlabeled traffic allowed by default [ 0.152036] Switching to clocksource kvm-clock [ 0.154795] AppArmor: AppArmor Filesystem Enabled [ 0.155787] pnp: PnP ACPI init [ 0.156532] ACPI: bus type pnp registered [ 0.158114] pnp: PnP ACPI: found 6 devices [ 0.158939] ACPI: ACPI bus type pnp unregistered [ 0.166336] NET: Registered protocol family 2 [ 0.167439] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.169713] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.174494] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.176420] TCP: Hash tables configured (established 262144 bind 65536) [ 0.177564] TCP reno registered [ 0.178286] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.179343] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.180784] NET: Registered protocol family 1 [ 0.181751] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.182800] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.183830] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.185046] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.186075] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.190994] pci 0000:00:01.2: PCI INT D disabled [ 0.192024] Unpacking initramfs... [ 0.335404] Freeing initrd memory: 8812k freed [ 0.349969] futex hash table entries: 512 (order: 4, 65536 bytes) [ 0.351112] Initialise module verification [ 0.352127] audit: initializing netlink socket (disabled) [ 0.353140] type=2000 audit(1491338903.352:1): initialized [ 0.371084] bounce pool size: 64 pages [ 0.371904] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.374083] VFS: Disk quotas dquot_6.5.2 [ 0.374981] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.376255] msgmni has been set to 937 [ 0.377514] alg: No test for stdrng (krng) [ 0.378349] Asymmetric key parser 'x509' registered [ 0.379340] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.380935] io scheduler noop registered [ 0.381774] io scheduler deadline registered [ 0.382729] io scheduler cfq registered (default) [ 0.383902] GHES: HEST is not enabled! [ 0.384734] Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled [ 0.386010] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.625272] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.661207] Non-volatile memory driver v1.3 [ 0.662579] Linux agpgart interface v0.103 [ 0.663830] Fixed MDIO Bus: probed [ 0.664919] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.667991] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.669007] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.670060] mousedev: PS/2 mouse device common for all mice [ 0.671309] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 0.673365] cpuidle: using governor ladder [ 0.674216] cpuidle: using governor menu [ 0.675033] EFI Variables Facility v0.08 2004-May-17 [ 0.676234] TCP cubic registered [ 0.677008] Registering the dns_resolver key type [ 0.678029] Loading module verification certificates [ 0.679045] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.680967] MODSIGN: Loaded cert 'SUSE Linux Enterprise Secure Boot Signkey: 3fb077b6cebc6ff2522e1c148c57c777c788e3e7' [ 0.682951] registered taskstats version 1 [ 0.684063] Magic number: 13:197:851 [ 0.686942] Freeing unused kernel memory: 1360k freed [ 0.688553] Write protecting the kernel read-only data: 10240k [ 0.692691] Freeing unused kernel memory: 1556k freed [ 0.696289] Freeing unused kernel memory: 900k freed doing fast boot [ 0.738044] SCSI subsystem initialized [ 0.761369] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 0.769605] scsi0 : ata_piix [ 0.770499] scsi1 : ata_piix [ 0.771219] ata2: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.772353] ata3: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.940681] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 0.963094] vda: vda1 vda2 vda4 [ 0.964429] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.965443] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 [ 0.971969] rdac: device handler registered [ 0.976574] hp_sw: device handler registered [ 0.981143] emc: device handler registered [ 0.985863] alua: device handler registered Creating device nodes with udev [ 0.989608] udev: starting version 147 [ 1.032613] usbcore: registered new interface driver usbfs [ 1.033688] usbcore: registered new interface driver hub [ 1.039758] usbcore: registered new device driver usb [ 1.050655] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.054291] uhci_hcd: USB Universal Host Controller Interface driver [ 1.057510] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 [ 1.061899] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 1.062966] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 1.064565] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 1.065657] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.066813] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.068321] usb usb1: Product: UHCI Host Controller [ 1.069298] usb usb1: Manufacturer: Linux 3.0.101-94-default uhci_hcd [ 1.070424] usb usb1: SerialNumber: 0000:00:01.2 [ 1.071430] hub 1-0:1.0: USB hub found [ 1.072268] hub 1-0:1.0: 2 ports detected mount: devpts already mounted or /dev/pts busy mount: according to mtab, devpts is already mounted on /dev/pts Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 20:48:24 2017 [ 1.142116] [drm] Initialized drm 1.1.0 20060810 [ 1.144976] This driver is only used in secure boot mode as default FATAL: Error inserting cirrus (/lib/modules/3.0.101-94-default/kernel/drivers/gpu/drm/cirrus/cirrus.ko): Invalid argument Trying manual resume from /dev/vda2 Invoking userspace resume from /dev/vda2 resume: libgcrypt version: 1.5.0 Trying manual resume from /dev/vda2 [ 1.339318] PM: Starting manual resume from disk Invoking in-kernel resume from /dev/vda2 Waiting for device /dev/vda1 to appear: ok fsck from util-linux 2.19.1 [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/vda1 /dev/vda1: recovering journal [ 1.369044] Refined TSC clocksource calibration: 2593.642 MHz. /dev/vda1: clean, 80282/1310720 files, 556945/5240832 blocks fsck succeeded. Mounting root device read-write. Mounting root /dev/vda1 mount -o rw,acl,user_xattr -t ext3 /dev/vda1 /root [ 1.629267] kjournald starting. Commit interval 15 seconds [ 1.629889] EXT3-fs (vda1): using internal journal [ 1.629893] EXT3-fs (vda1): mounted filesystem with ordered data mode INIT: version 2.86 booting System Boot Control: Running /etc/init.d/boot Mounting sysfs at /sys7[?25l done8[?25h Copying static /dev content7[?25l done8[?25h Mounting devpts at /dev/pts7[?25l done8[?25h Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr 4 20:48:25 2017 Mounting debugfs at /sys/kernel/debug7[?25l done8[?25h [?25hNo efivars filesystem driver available.[?25hStarting udevd: [ 2.270293] udev: starting version 147 7[?25l done8[?25h Loading drivers, configuring devices: [ 2.287194] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 2.288811] ACPI: Power Button [PWRF] [ 2.310044] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 2.313844] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 2.315147] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.316214] 8139cp 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 [ 2.319849] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc90000904000, 00:16:3e:5a:20:3f, IRQ 11 [ 2.326581] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 2.329532] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 2.339327] NET: Registered protocol family 10 [ 2.346516] rtc_cmos 00:01: RTC can wake from S4 [ 2.347567] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 2.348702] rtc0: alarms up to one day, 114 bytes nvram [ 2.372759] FDC 0 is a S82078B 7[?25l done8[?25h [?25hActivating swap-devices in /etc/fstab... [ 2.473501] Adding 14338044k swap on /dev/vda2. Priority:-1 extents:1 across:14338044k 7[?25l done8[?25h[?25hSet System Time to the current Hardware Clock7[?25l done8[?25h [?25hActivating device mapper... [ 3.547049] device-mapper: uevent: version 1.0.3 [ 3.548071] device-mapper: ioctl: 4.25.0-ioctl (2012-07-25) initialised: dm-devel@redhat.com 7[?25l done8[?25h [?25hLoading required kernel modules 7[?25l done8[?25h[?25h[?25hWaiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 3 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h [?25hChecking file systems... fsck from util-linux 2.19.1 Checking all file systems. 7[?25l done8[?25hMounting local file systems... proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) [ 3.708292] loop: module loaded udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,mode=1777) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) nothing was mounted 7[?25l done8[?25h[?25hCheck if the profiles matches the system7[?25l done8[?25h [?25h[?25hLoading fuse module [ 4.028687] fuse init (API version 7.16) 7[?25l done8[?25h Mounting fuse control filesystem7[?25l done8[?25h [?25hCreating /var/log/boot.msg 7[?25l done8[?25h[?25hEnabling monitoring on LVM volume groups... 3 logical volume(s) in volume group "lvm-Role_MDS" monitored 7[?25l done8[?25h [?25h[?25h[?25hActivating remaining swap-devices in /etc/fstab... 7[?25l done8[?25h[?25h[?25h[?25hTurning quota on Checking quotas. This may take some time. 7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... [?25hEnabling syn flood protection done Disabling IP forwarding7[?25l done8[?25h Disabling IPv6 forwarding7[?25l done8[?25h Disabling IPv6 privacy7[?25l done8[?25h 7[?25l done8[?25h [?25hSetting up hostname 'trevis-41vm7'7[?25l done8[?25h Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h [?25hLoading kdump 7[?25l done8[?25h [?25hSystem Boot Control: The system has been set up Skipped features: boot.efivars boot.open-iscsi boot.cycle System Boot Control: Running /etc/init.d/boot.local 7[?25l done8[?25h INIT: Entering runlevel: 3 Master Resource Control: previous runlevel: N, switching to runlevel: 3 Starting syslog services7[?25l done8[?25h [?25hChecking CPU.. KVM guest detected7[?25l unused8[?25h [?25hInitializing random number generator7[?25l done8[?25h [?25hWaiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 3 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h [?25hStarting acpid 7[?25l done8[?25h [?25hStarting D-Bus daemon7[?25l done8[?25h [?25h[?25h[?25hLoading CPUFreq modules (CPUFreq not supported) Starting HAL daemon7[?25l done8[?25h [?25hLoading HCA driver and Access Layer:[ OK ] Setting up InfiniBand network interfaces: Setting up service network . . .[ done ] [?25hSetting up (localfs) network interfaces: lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 7[?25l done8[?25h eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 Starting DHCP4 client. eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com)  waiting Waiting for mandatory devices: eth0 27 eth0 device: Realtek Semiconductor Co., Ltd. RTL-8100/8101 eth0 DHCP4 client (dhcpcd) is running eth0 IP address: 10.9.5.243/16 (trevis-41vm7.trevis.hpdd.intel.com) eth0 IP address: 10.9.5.243/16 eth0 is up 7[?25l done8[?25hSetting up service (localfs) network . . . . . . . . . .7[?25l done8[?25h [?25hSetting current sysctl status from /etc/sysctl.conf7[?25l done8[?25h [?25hRe-Starting syslog services7[?25l done8[?25h [?25hStarting auditd 7[?25l done8[?25h [?25hStarting haveged daemon 7[?25l done8[?25h [?25hStarting rpcbind 7[?25l done8[?25h [?25h[?25hNot starting NFS client services - no NFS found in /etc/fstab: unused [?25hLoading console font lat9w-16.psfu -m trivial G0:loadable 7[?25l done8[?25hLoading keymap assuming iso-8859-15 euro Loading /usr/share/kbd/keymaps/i386/qwerty/us.map.gz 7[?25l done8[?25hLoading compose table winkeys shiftctrl latin1.add7[?25l done8[?25h Start Unicode mode 7[?25l done8[?25h[?25hLoading kdump 7[?25l done8[?25h [?25hStarting irqbalance 7[?25l done8[?25h [?25hStarting mcelog... 7[?25l done8[?25h [?25hSetting up (remotefs) network interfaces: Setting up service (remotefs) network . . . . . . . . . .7[?25l done8[?25h [?25h[?25hChecking for missing server keys in /etc/ssh Starting SSH daemon7[?25l done8[?25h [?25hStarting network time protocol daemon (NTPD)7[?25l done8[?25h [?25hStarting MUNGE: munged7[?25l done8[?25h [?25hStarting Name Service Cache Daemon7[?25l done8[?25h [?25hStarting sssd 7[?25l done8[?25h [?25hStarting automount 7[?25l done8[?25h [?25hStarting mail service (Postfix)7[?25l done8[?25h [?25hStarting CRON daemon7[?25l done8[?25h [?25hStarting smartd unused [?25hStarting INET services. (xinetd) Console [trevis-41vm7] departed by on pts/55 at 04-04 20:48. 7[?25l done8[?25h [?25hMaster Resource Control: Running /etc/init.d/after.local Waiting for udev to settle... Scanning for LVM volume groups... Reading all physical volumes. This may take a while... Found volume group "lvm-Role_MDS" using metadata type lvm2 Activating LVM volume groups... PARTIAL MODE. Incomplete logical volumes will be processed. 3 logical volume(s) in volume group "lvm-Role_MDS" now active 7[?25l done8[?25h 7[?25l done8[?25hMaster Resource Control: runlevel 3 has been reached Skipped services in runlevel 3: microcode.ctl nfs splash smartd Welcome to SUSE Linux Enterprise Server 11 SP4 (x86_64) - Kernel 3.0.101-94-default (console). trevis-41vm7 login: Console [trevis-41vm7] joined by on pts/44 at 04-05 14:49. Console [trevis-41vm7] disconnected from at 04-05 15:11. Console [trevis-41vm7] connected to at 04-05 15:11. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004006] pid_max: default: 32768 minimum: 301 [ 0.005033] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008214] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011876] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014313] Mount-cache hash table entries: 4096 [ 0.015299] Initializing cgroup subsys memory [ 0.017010] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.019006] Initializing cgroup subsys net_cls [ 0.020006] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.023006] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.025006] Initializing cgroup subsys net_prio [ 0.027073] Disabled fast string operations [ 0.028020] mce: CPU supports 10 MCE banks [ 0.029058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] tlb_flushall_shift: 6 [ 0.033187] Freeing SMP alternatives: 28k freed [ 0.038918] ACPI: Core revision 20130517 [ 0.040878] ACPI: All ACPI Tables successfully acquired [ 0.042027] ftrace: allocating 25811 entries in 101 pages [ 0.051280] smpboot: Max logical packages: 2 [ 0.052006] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.055263] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056005] Switched APIC routing to physical x2apic. [ 0.058495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.059003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.062027] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.064666] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.318050] Brought up 2 CPUs [ 0.318050] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.320006] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.322002] devtmpfs: initialized [ 0.325995] EVM: security.selinux [ 0.326005] EVM: security.ima [ 0.327005] EVM: security.capability [ 0.328780] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.329037] pinctrl core: initialized pinctrl subsystem [ 0.330157] NET: Registered protocol family 16 [ 0.331303] ACPI: bus type PCI registered [ 0.332007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.333096] PCI: Using configuration type 1 for base access [ 0.337087] ACPI: Added _OSI(Module Device) [ 0.338013] ACPI: Added _OSI(Processor Device) [ 0.339007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.342707] ACPI: Interpreter enabled [ 0.343013] ACPI: (supports S0 S5) [ 0.344005] ACPI: Using IOAPIC for interrupt routing [ 0.345049] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.348634] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.349011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.350011] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.351022] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352263] acpiphp: Slot [1] registered [ 0.353032] acpiphp: Slot [2] registered [ 0.354023] acpiphp: Slot [3] registered [ 0.355024] acpiphp: Slot [4] registered [ 0.356024] acpiphp: Slot [5] registered [ 0.357006] acpiphp: Slot [6] registered [ 0.358023] acpiphp: Slot [7] registered [ 0.359023] acpiphp: Slot [8] registered [ 0.360032] acpiphp: Slot [9] registered [ 0.361028] acpiphp: Slot [10] registered [ 0.362023] acpiphp: Slot [11] registered [ 0.363023] acpiphp: Slot [12] registered [ 0.364023] acpiphp: Slot [13] registered [ 0.365023] acpiphp: Slot [14] registered [ 0.366023] acpiphp: Slot [15] registered [ 0.367033] acpiphp: Slot [16] registered [ 0.368031] acpiphp: Slot [17] registered [ 0.369023] acpiphp: Slot [18] registered [ 0.370023] acpiphp: Slot [19] registered [ 0.371023] acpiphp: Slot [20] registered [ 0.372023] acpiphp: Slot [21] registered [ 0.373023] acpiphp: Slot [22] registered [ 0.374031] acpiphp: Slot [23] registered [ 0.375023] acpiphp: Slot [24] registered [ 0.376023] acpiphp: Slot [25] registered [ 0.377023] acpiphp: Slot [26] registered [ 0.378032] acpiphp: Slot [27] registered [ 0.379031] acpiphp: Slot [28] registered [ 0.380022] acpiphp: Slot [29] registered [ 0.381039] acpiphp: Slot [30] registered [ 0.382023] acpiphp: Slot [31] registered [ 0.383269] PCI host bridge to bus 0000:00 [ 0.384008] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385007] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.388007] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.390005] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394829] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.395013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.412664] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.414075] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.415749] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.417072] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.418723] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.420205] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.421281] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.422012] vgaarb: loaded [ 0.422992] vgaarb: bridge control possible 0000:00:02.0 [ 0.423223] SCSI subsystem initialized [ 0.424053] ACPI: bus type USB registered [ 0.425039] usbcore: registered new interface driver usbfs [ 0.426017] usbcore: registered new interface driver hub [ 0.427056] usbcore: registered new device driver usb [ 0.428210] PCI: Using ACPI for IRQ routing [ 0.429358] NetLabel: Initializing [ 0.430007] NetLabel: domain hash size = 128 [ 0.431005] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.432057] NetLabel: unlabeled traffic allowed by default [ 0.433161] Switched to clocksource kvm-clock [ 0.443379] pnp: PnP ACPI init [ 0.444448] ACPI: bus type PNP registered [ 0.446103] pnp: PnP ACPI: found 5 devices [ 0.447289] ACPI: bus type PNP unregistered [ 0.456994] NET: Registered protocol family 2 [ 0.458553] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.460812] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.462509] TCP: Hash tables configured (established 16384 bind 16384) [ 0.464100] TCP: reno registered [ 0.465163] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.466582] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.468167] NET: Registered protocol family 1 [ 0.469407] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.470776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.472126] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.473783] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.475438] Unpacking initramfs... [ 4.611693] Freeing initrd memory: 42360k freed [ 4.631212] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.632816] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.634545] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.635976] Initialise system trusted keyring [ 4.637199] audit: initializing netlink socket (disabled) [ 4.638517] type=2000 audit(1491405135.513:1): initialized [ 4.659334] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.660773] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.663199] zpool: loaded [ 4.664203] zbud: loaded [ 4.665392] VFS: Disk quotas dquot_6.5.2 [ 4.666559] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.668194] msgmni has been set to 3357 [ 4.669387] Key type big_key registered [ 4.671714] NET: Registered protocol family 38 [ 4.672914] Key type asymmetric registered [ 4.674071] Asymmetric key parser 'x509' registered [ 4.675326] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.677505] io scheduler noop registered [ 4.678776] io scheduler deadline registered (default) [ 4.680072] io scheduler cfq registered [ 4.681304] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.682626] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.684210] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.686340] ACPI: Power Button [PWRF] [ 4.687595] GHES: HEST is not enabled! [ 4.688760] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.711556] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.713284] Non-volatile memory driver v1.3 [ 4.714467] Linux agpgart interface v0.103 [ 4.715730] crash memory driver: version 1.1 [ 4.716966] rdac: device handler registered [ 4.718181] hp_sw: device handler registered [ 4.719363] emc: device handler registered [ 4.720509] alua: device handler registered [ 4.721709] libphy: Fixed MDIO Bus: probed [ 4.722900] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.724338] ehci-pci: EHCI PCI platform driver [ 4.725531] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.726937] ohci-pci: OHCI PCI platform driver [ 4.728142] uhci_hcd: USB Universal Host Controller Interface driver [ 4.729770] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.731085] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.733263] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.734591] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.736030] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.737559] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.739753] usb usb1: Product: UHCI Host Controller [ 4.741064] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.742569] usb usb1: SerialNumber: 0000:00:01.2 [ 4.743910] hub 1-0:1.0: USB hub found [ 4.745106] hub 1-0:1.0: 2 ports detected [ 4.746449] usbcore: registered new interface driver usbserial [ 4.747865] usbcore: registered new interface driver usbserial_generic [ 4.749393] usbserial: USB Serial support registered for generic [ 4.750831] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.753718] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.755105] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.756593] mousedev: PS/2 mouse device common for all mice [ 4.758260] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.760764] rtc_cmos 00:00: RTC can wake from S4 [ 4.761456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.761707] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.767385] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.768755] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.770256] cpuidle: using governor menu [ 4.771498] hidraw: raw HID events driver (C) Jiri Kosina [ 4.772943] usbcore: registered new interface driver usbhid [ 4.774330] usbhid: USB HID core driver [ 4.775517] drop_monitor: Initializing network drop monitor service [ 4.777028] TCP: cubic registered [ 4.778130] Initializing XFRM netlink socket [ 4.779439] NET: Registered protocol family 10 [ 4.781014] NET: Registered protocol family 17 [ 4.782561] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.783958] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.785401] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.787941] Loading compiled-in X.509 certificates [ 4.789153] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.791641] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.794653] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.797196] registered taskstats version 1 [ 4.801011] Key type trusted registered [ 4.804189] Key type encrypted registered [ 4.807382] IMA: No TPM chip found, activating TPM-bypass! [ 4.809077] rtc_cmos 00:00: setting system clock to 2017-04-05 15:12:14 UTC (1491405134) [ 4.812165] Freeing unused kernel memory: 1680k freed [ 4.818311] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.837384] systemd[1]: Detected virtualization kvm. [ 4.838702] systemd[1]: Detected architecture x86-64. [ 4.840015] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.842135] systemd[1]: No hostname configured. [ 4.843415] systemd[1]: Set hostname to . [ 4.844785] systemd[1]: Initializing machine ID from KVM UUID. [ 4.869210] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Reached target Local File Systems. [ 4.872614] systemd[1]: Reached target Local File Systems. [ 4.873961] systemd[1]: Starting Local File Systems. [ OK ] Created slice Root Slice. [ 4.875905] systemd[1]: Created slice Root Slice. [ 4.877148] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 4.878971] systemd[1]: Listening on udev Kernel Socket. [ 4.880282] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ 4.882218] systemd[1]: Listening on udev Control Socket. [ 4.883528] systemd[1]: Starting udev Control Socket. [ OK ] Created slice System Slice. [ 4.885449] systemd[1]: Created slice System Slice. [ 4.886718] systemd[1]: Starting System Slice. [ OK ] Reached target Slices. [ 4.888480] systemd[1]: Reached target Slices. [ 4.889680] systemd[1]: Starting Slices. [ OK ] Reached target Timers. [ 4.891393] systemd[1]: Reached target Timers. [ 4.892598] systemd[1]: Starting Timers. [ OK ] Listening on Journal Socket. [ 4.894390] systemd[1]: Listening on Journal Socket. [ 4.895658] systemd[1]: Starting Journal Socket. [ 4.897464] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.899718] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 4.902186] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Encrypted Volumes. [ 4.907471] systemd[1]: Reached target Encrypted Volumes. [ 4.908821] systemd[1]: Starting Encrypted Volumes. [ 4.910576] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.913138] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Swap. [ 4.916440] systemd[1]: Reached target Swap. [ 4.917651] systemd[1]: Starting Swap. [ OK ] Started Journal Service. [ 4.919702] systemd[1]: Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 4.979885] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.053481] dracut-cmdline[92]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.185923] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.190575] Loading iSCSI transport class v2.0-870. [ 5.195107] iscsi: registered transport (tcp) [ 5.228324] FDC 0 is a S82078B [ 5.233404] iBFT detected. [ 5.234455] iBFT has incorrect checksum (0x54)! [ 5.242153] md: raid0 personality registered for level 0 [ 5.245798] md: raid1 personality registered for level 1 [ 5.249169] async_tx: api initialized (async) [ 5.251202] xor: automatically using best checksumming function: [ 5.262010] avx : 22548.000 MB/sec [ 5.282010] raid6: sse2x1 gen() 7929 MB/s [ 5.300009] raid6: sse2x2 gen() 9980 MB/s [ 5.318018] raid6: sse2x4 gen() 11550 MB/s [ 5.319196] raid6: using algorithm sse2x4 gen() (11550 MB/s) [ 5.320511] raid6: using ssse3x2 recovery algorithm [ 5.326366] md: raid6 personality registered for level 6 [ 5.328403] md: raid5 personality registered for level 5 [ 5.330508] md: raid4 personality registered for level 4 [ 5.338794] md: raid10 personality registered for level 10 [ 5.343059] md: linear personality registered for level -1 [ 5.348335] device-mapper: uevent: version 1.0.3 [ 5.349642] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.363686] device-mapper: multipath: version 1.11.0 loaded [ 5.367047] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.397727] RPC: Registered named UNIX socket transport module. [ 5.399141] RPC: Registered udp transport module. [ 5.400370] RPC: Registered tcp transport module. [ 5.401590] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Reached target Sockets. Starting Device-Mapper Multipath Device Controller... Starting Show Plymouth Boot Screen... [ 5.620770] [drm] Initialized drm 1.1.0 20060810 [ 5.624207] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 5.626116] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 5.629855] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 5.632735] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.634320] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ 5.635690] 8139too: 8139too Fast Ethernet driver 0.9.28 [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 5.647474] AES CTR mode by8 optimization enabled [ OK ] Started Device-Mapper Multipath Device Controller. Starting Open-iSCSI... [ 5.654297] scsi host0: ata_piix [ 5.655673] scsi host1: ata_piix [ 5.656761] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.658234] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 5.660871] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 5.662252] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.667926] 8021q: 802.1Q VLAN Support v1.8 [ 5.672606] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ OK ] Started Open-iSCSI. Starting dracut initqueue hook... %G[ 5.704342] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.705833] [TTM] Initializing pool allocator [ 5.707049] [TTM] Initializing DMA pool allocator [ 5.708335] vda: vda1 vda2 vda4 [ 5.708686] [drm] fb mappable at 0xF0000000 [ 5.708686] [drm] vram aper at 0xF0000000 [ 5.708687] [drm] size 33554432 [ 5.708687] [drm] fb depth is 24 [ 5.708687] [drm] pitch is 3072 %G[ 5.715194] fbcon: cirrusdrmfb (fb0) is primary device [ 5.748097] alg: No test for crc32 (crc32-pclmul) [ 5.749086] Console: switching to colour frame buffer device 128x48 [ 5.760728] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 5.766037] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 5.826469] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.549971] dracut-initqueue[581]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.551152] dracut-initqueue[581]: Dload Upload Total Spent Left Speed [ 8.665321] dracut-initqueue[581]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14606 100 14606 0 0 122k 0 --:--:-- --:--:-- --:--:-- 124k [ 9.195918] dracut-initqueue[581]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.197683] dracut-initqueue[581]: Dload Upload Total Spent Left Speed [ 12.551658] dracut-initqueue[581]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 23 317M 23 74.7M 0 0 92.8M 0 0:00:03 --:--:-- 0:00:03 92.7M 52 317M 52 167M 0 0 92.9M 0 0:00:03 0:00:01 0:00:02 92.9M 82 317M 82 261M 0 0 93.0M 0 0:00:03 0:00:02 0:00:01 93.0M 100 317M 100 317M 0 0 94.5M 0 0:00:03 0:00:03 --:--:-- 94.5M [ 12.661717] loop: module loaded [ OK ] Started dracut initqueue hook. Starting dracut pre-mount hook... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. Starting dracut mount hook... [ 12.833985] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 14.907067] systemd-journald[85]: Received SIGTERM from PID 1 (n/a). [ 15.450583] type=1403 audit(1491405145.140:2): policy loaded auid=4294967295 ses=4294967295 [ 15.456255] systemd[1]: Successfully loaded SELinux policy in 266.888ms. [ 15.556237] systemd[1]: Relabelled /dev and /run in 19.340ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Stopped target Switch Root. Mounting Debug File System... Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-anaconda\x2dtmux.slice. Mounting Temporary Directory... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Stopped Journal Service. Starting Journal Service... Mounting POSIX Message Queue File System... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. [ OK ] Mounted Temporary Directory. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Mounted Huge Pages File System. [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Journal Service. Starting Rebuild Hardware Database... Starting Load/Save Random Seed... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started Apply Kernel Variables. [ OK ] Started Load/Save Random Seed. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Configure read-only root support. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting udev Wait for Complete Device Initialization... Starting Device-Mapper Multipath Device Controller... [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... %G%G[ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Local File Systems. Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... Starting Tell Plymouth To Write Out Runtime Data... [ OK ] Reached target Encrypted Volumes. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting firewalld - dynamic firewall daemon... Starting Service enabling compressing RAM with zRam... Starting Terminate Plymouth Boot Screen... Starting Dump dmesg to /var/log/dmesg... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... Starting Wait for Plymouth Boot Screen to Quit... Starting Login Service... Starting pre-anaconda logging service... %G (BStarting installer, one moment...                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (B [anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (B anaconda 21.48.22.93-1 for CentOS Linux 7 started.  * installation log files are stored in /tmp during the installation  * shell is available on TTY2  * when reporting a bug add logs from /tmp as separate text/plain attachments 15:12:32 Running pre-installation scripts 15:12:33 Not asking for VNC because of an automated install15:12:33 Not asking for VNC because text mode was explicitly asked for in kickstartStarting automated install......................Checking software selectionGenerating updated storage configurationChecking storage configuration...================================================================================ ================================================================================ Installation  1) [x] Language settings 2) [x] Time settings (English (United States)) (Etc/UTC timezone) 3) [x] Installation source 4) [x] Software selection (http://10.9.0.10/cblr/links/Ce (Custom software selected) ntOS-7.3-x86_64) 6) [x] Kdump 5) [x] Installation Destination (Kdump is enabled) (Custom partitioning selected) 8) [ ] User creation 7) [x] Network configuration (No user will be created) (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/sys/fs/selinux... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... Unmounting Configuration File System... Unmounting /mnt/sysimage/proc... Unmounting /mnt/sysimage/dev/shm... Unmounting /mnt/sysimage/run... Unmounting Temporary Directory... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... Stopping Load/Save Random Seed... [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. [ OK ] Unmounted Configuration File System. [ OK ] Failed unmounting Temporary Directory. [ OK ] Stopped Load/Save Random Seed. [ OK ] Failed unmounting /mnt/sysimage/dev/pts. [ OK ] Failed unmounting /mnt/sysimage/proc. [ OK ] Failed unmounting /mnt/sysimage/dev/shm. [ OK ] Failed unmounting /mnt/sysimage/run. Unmounting /mnt/sysimage/dev... Unmounting /mnt/sysimage/sys... [ OK ] Unmounted /mnt/sysimage/sys. [ OK ] Unmounted /mnt/sysimage/dev. Unmounting /mnt/sysimage... [ OK ] Unmounted /mnt/sysimage. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Started Restore /run/initramfs. [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 854.629337] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fde000-0x367e6fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694572k/2097140k available (6765k kernel code, 400k absent, 402168k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005028] Security Framework initialized [ 0.007008] SELinux: Initializing. [ 0.008187] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011738] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013483] Mount-cache hash table entries: 4096 [ 0.015234] Initializing cgroup subsys memory [ 0.016012] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.018005] Initializing cgroup subsys net_cls [ 0.020004] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.024005] Initializing cgroup subsys net_prio [ 0.026104] Disabled fast string operations [ 0.027019] mce: CPU supports 10 MCE banks [ 0.028056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031005] tlb_flushall_shift: 6 [ 0.032176] Freeing SMP alternatives: 28k freed [ 0.038073] ACPI: Core revision 20130517 [ 0.040148] ACPI: All ACPI Tables successfully acquired [ 0.041015] ftrace: allocating 25813 entries in 101 pages [ 0.049183] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.050005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.052003] smpboot: Max logical packages: 2 [ 0.053246] Enabling x2apic [ 0.054000] Enabled x2apic [ 0.054004] Switched APIC routing to physical x2apic. [ 0.057376] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.058003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.061024] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.063434] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.066040] Brought up 2 CPUs [ 0.066030] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.068005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.069495] devtmpfs: initialized [ 0.071861] EVM: security.selinux [ 0.072005] EVM: security.ima [ 0.073004] EVM: security.capability [ 0.074737] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.075028] pinctrl core: initialized pinctrl subsystem [ 0.076144] NET: Registered protocol family 16 [ 0.077267] ACPI: bus type PCI registered [ 0.078006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.079084] PCI: Using configuration type 1 for base access [ 0.083084] ACPI: Added _OSI(Module Device) [ 0.084013] ACPI: Added _OSI(Processor Device) [ 0.085006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.086005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.088678] ACPI: Interpreter enabled [ 0.089013] ACPI: (supports S0 S5) [ 0.090005] ACPI: Using IOAPIC for interrupt routing [ 0.091017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.094538] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.095010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.096010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.097018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.098238] acpiphp: Slot [1] registered [ 0.099030] acpiphp: Slot [2] registered [ 0.100022] acpiphp: Slot [3] registered [ 0.101022] acpiphp: Slot [4] registered [ 0.102022] acpiphp: Slot [5] registered [ 0.103022] acpiphp: Slot [6] registered [ 0.104026] acpiphp: Slot [7] registered [ 0.105021] acpiphp: Slot [8] registered [ 0.106028] acpiphp: Slot [9] registered [ 0.107022] acpiphp: Slot [10] registered [ 0.108022] acpiphp: Slot [11] registered [ 0.109021] acpiphp: Slot [12] registered [ 0.110021] acpiphp: Slot [13] registered [ 0.111022] acpiphp: Slot [14] registered [ 0.112021] acpiphp: Slot [15] registered [ 0.113028] acpiphp: Slot [16] registered [ 0.114028] acpiphp: Slot [17] registered [ 0.115022] acpiphp: Slot [18] registered [ 0.116022] acpiphp: Slot [19] registered [ 0.117022] acpiphp: Slot [20] registered [ 0.118022] acpiphp: Slot [21] registered [ 0.119021] acpiphp: Slot [22] registered [ 0.120028] acpiphp: Slot [23] registered [ 0.121022] acpiphp: Slot [24] registered [ 0.122022] acpiphp: Slot [25] registered [ 0.123022] acpiphp: Slot [26] registered [ 0.124022] acpiphp: Slot [27] registered [ 0.125028] acpiphp: Slot [28] registered [ 0.126022] acpiphp: Slot [29] registered [ 0.127022] acpiphp: Slot [30] registered [ 0.128028] acpiphp: Slot [31] registered [ 0.129260] PCI host bridge to bus 0000:00 [ 0.130007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.131006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.132006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.133006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.134005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.135971] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.136006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.137005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.138005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.139805] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.140013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.157611] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.159075] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.160746] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.162069] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.163730] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.165201] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.166267] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.167012] vgaarb: loaded [ 0.167991] vgaarb: bridge control possible 0000:00:02.0 [ 0.168211] SCSI subsystem initialized [ 0.169050] ACPI: bus type USB registered [ 0.170036] usbcore: registered new interface driver usbfs [ 0.171017] usbcore: registered new interface driver hub [ 0.172048] usbcore: registered new device driver usb [ 0.173179] PCI: Using ACPI for IRQ routing [ 0.174335] NetLabel: Initializing [ 0.175007] NetLabel: domain hash size = 128 [ 0.176004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.177039] NetLabel: unlabeled traffic allowed by default [ 0.178138] Switched to clocksource kvm-clock [ 0.186830] pnp: PnP ACPI init [ 0.187889] ACPI: bus type PNP registered [ 0.189502] pnp: PnP ACPI: found 5 devices [ 0.190667] ACPI: bus type PNP unregistered [ 0.200314] NET: Registered protocol family 2 [ 0.201829] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.204050] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.205611] TCP: Hash tables configured (established 16384 bind 16384) [ 0.207107] TCP: reno registered [ 0.208160] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.209580] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.211137] NET: Registered protocol family 1 [ 0.212361] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.213727] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.215067] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.216705] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.218350] Unpacking initramfs... [ 0.612437] Freeing initrd memory: 24612k freed [ 0.628199] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.629759] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.631584] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.633012] Initialise system trusted keyring [ 0.634256] audit: initializing netlink socket (disabled) [ 0.635578] type=2000 audit(1491406013.961:1): initialized [ 0.656360] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.657803] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.660250] zpool: loaded [ 0.661243] zbud: loaded [ 0.662440] VFS: Disk quotas dquot_6.5.2 [ 0.663617] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.665297] msgmni has been set to 3357 [ 0.666539] Key type big_key registered [ 0.669092] NET: Registered protocol family 38 [ 0.670309] Key type asymmetric registered [ 0.671460] Asymmetric key parser 'x509' registered [ 0.672727] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.674939] io scheduler noop registered [ 0.676079] io scheduler deadline registered (default) [ 0.677382] io scheduler cfq registered [ 0.678619] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.679943] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.681562] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.683703] ACPI: Power Button [PWRF] [ 0.685017] GHES: HEST is not enabled! [ 0.686223] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.708955] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.710892] Non-volatile memory driver v1.3 [ 0.712071] Linux agpgart interface v0.103 [ 0.713380] crash memory driver: version 1.1 [ 0.714636] rdac: device handler registered [ 0.715879] hp_sw: device handler registered [ 0.731624] emc: device handler registered [ 0.732796] alua: device handler registered [ 0.733998] libphy: Fixed MDIO Bus: probed [ 0.735285] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.736723] ehci-pci: EHCI PCI platform driver [ 0.737928] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.739333] ohci-pci: OHCI PCI platform driver [ 0.740537] uhci_hcd: USB Universal Host Controller Interface driver [ 0.742178] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.743619] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.745853] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.747195] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.748650] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.750193] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.752429] usb usb1: Product: UHCI Host Controller [ 0.753767] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.756126] usb usb1: SerialNumber: 0000:00:01.2 [ 0.757526] hub 1-0:1.0: USB hub found [ 0.758736] hub 1-0:1.0: 2 ports detected [ 0.760128] usbcore: registered new interface driver usbserial [ 0.761577] usbcore: registered new interface driver usbserial_generic [ 0.763156] usbserial: USB Serial support registered for generic [ 0.764653] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.767598] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.769102] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.770541] mousedev: PS/2 mouse device common for all mice [ 0.772292] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.774905] rtc_cmos 00:00: RTC can wake from S4 [ 0.775432] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.776724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.781431] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.782926] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.784361] cpuidle: using governor menu [ 0.785598] hidraw: raw HID events driver (C) Jiri Kosina [ 0.787049] usbcore: registered new interface driver usbhid [ 0.788410] usbhid: USB HID core driver [ 0.789601] drop_monitor: Initializing network drop monitor service [ 0.791118] TCP: cubic registered [ 0.792204] Initializing XFRM netlink socket [ 0.793485] NET: Registered protocol family 10 [ 0.795033] NET: Registered protocol family 17 [ 0.796474] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.797946] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.799377] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.801988] Loading compiled-in X.509 certificates [ 0.803284] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.805746] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.808758] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.811261] registered taskstats version 1 [ 0.815086] Key type trusted registered [ 0.818305] Key type encrypted registered [ 0.821502] IMA: No TPM chip found, activating TPM-bypass! [ 0.823209] rtc_cmos 00:00: setting system clock to 2017-04-05 15:26:53 UTC (1491406013) [ 0.826357] Freeing unused kernel memory: 1680k freed [ 0.832323] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.836595] systemd[1]: Detected virtualization kvm. [ 0.837993] systemd[1]: Detected architecture x86-64. [ 0.839338] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.841514] systemd[1]: Set hostname to . [ OK ] Created slice Root Slice. [ 0.874528] systemd[1]: Created slice Root Slice. [ 0.875779] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 0.877627] systemd[1]: Listening on udev Kernel Socket. [ 0.878952] systemd[1]: Starting udev Kernel Socket. [ OK ] Created slice System Slice. [ 0.880857] systemd[1]: Created slice System Slice. [ 0.882125] systemd[1]: Starting System Slice. [ OK ] Reached target Slices. [ 0.883870] systemd[1]: Reached target Slices. [ 0.885081] systemd[1]: Starting Slices. [ OK ] Reached target Timers. [ 0.886791] systemd[1]: Reached target Timers. [ 0.888007] systemd[1]: Starting Timers. [ OK ] Listening on udev Control Socket. [ 0.889787] systemd[1]: Listening on udev Control Socket. [ 0.891108] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Swap. [ 0.892929] systemd[1]: Reached target Swap. [ 0.894126] systemd[1]: Starting Swap. [ OK ] Listening on Journal Socket. [ 0.895868] systemd[1]: Listening on Journal Socket. [ 0.897139] systemd[1]: Starting Journal Socket. [ OK ] Reached target Sockets. [ 0.898923] systemd[1]: Reached target Sockets. [ 0.900145] systemd[1]: Starting Sockets. [ 0.901732] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.903993] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.907174] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.985439] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 0.987906] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ 0.990485] systemd[1]: Reached target Local File Systems. [ 0.991825] systemd[1]: Starting Local File Systems. [ OK ] Started Journal Service. [ 0.994102] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ 1.151268] FDC 0 is a S82078B [ 1.157730] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.159489] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.163069] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.165540] [drm] Initialized drm 1.1.0 20060810 [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.177032] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.179243] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.180755] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.186777] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.188117] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.189568] [TTM] Initializing pool allocator [ 1.190934] [TTM] Initializing DMA pool allocator [ 1.193158] [drm] fb mappable at 0xF0000000 [ 1.194351] [drm] vram aper at 0xF0000000 [ 1.195498] [drm] size 33554432 [ 1.196521] [drm] fb depth is 24 [ 1.197582] [drm] pitch is 3072 %G[ 1.198966] fbcon: cirrusdrmfb (fb0) is primary device [ 1.278456] vda: vda1 vda2 [ 1.285944] Console: switching to colour frame buffer device 128x48 [ 1.298391] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.304028] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 1.309266] scsi host0: ata_piix [ 1.309430] scsi host1: ata_piix [ 1.309463] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.309464] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 %G[ OK ] Found device /dev/disk/by-uuid/420143da-7f83-49b4-8a24-6cffb1d410f7. Starting File System Check on /dev/...a-7f83-49b4-8a24-6cffb1d410f7... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d...3da-7f83-49b4-8a24-6cffb1d410f7. Mounting /sysroot... [ 1.490831] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.492395] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 1.628047] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.862912] systemd-journald[95]: Received SIGTERM from PID 1 (systemd). [ 2.228441] SELinux: Disabled at runtime. [ 2.265025] type=1404 audit(1491406014.941:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.273560] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.275028] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Remote File Systems. Mounting Huge Pages File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Reached target Encrypted Volumes. Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Journal Service. [ OK ] Started udev Kernel Device Manager. [ 2.845660] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Rebuild Hardware Database... Starting Load/Save Random Seed... [ 2.862905] systemd-journald[339]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Load/Save Random Seed. [ OK ] Started Configure read-only root support. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ 3.622602] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 3.625252] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. Activating swap /dev/disk/by-uuid/d...6-1e76-4988-acad-74e281f3721c... %G[ 3.663755] AES CTR mode by8 optimization enabled %G[ 3.666120] ppdev: user-space parallel port driver [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Run[ 3.677735] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) time Data... [ 3.680846] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS Starting Mark the need to relabel after reboot... Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... [ 3.692858] alg: No test for crc32 (crc32-pclmul) [ 3.719377] intel_rapl: no valid rapl domains found in package 0 [ 3.749513] intel_rapl: no valid rapl domains found in package 0 [ 3.757418] intel_powerclamp: No package C-state available[ OK ] Activated swap /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. [ OK ] Reached target Swap. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Import network configuration from initramfs. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... Starting Create Volatile Files and Directories... [ OK ] Started Update is Completed. [ 3.773433] intel_powerclamp: No package C-state available[ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Reached target Paths. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ 4.058255] type=1305 audit(1491406016.734:3): audit_pid=451 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting Network Manager... Starting Authorization Manager... Starting OpenSSH Server Key Generation... Starting Login Service... Starting Dump dmesg to /var/log/dmesg... Starting NTP client/server... Starting Permit User Sessions... [ OK ] Reached target Timers. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. Starting Command Scheduler... Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Terminate Plymouth Boot Screen... [ OK ] Started NTP client/server. [ 5.821640] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 5.823488] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-05T15:27:09+00:00] WARN: ***************************************** [2017-04-05T15:27:09+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-05T15:27:09+00:00] WARN: ***************************************** [2017-04-05T15:27:13+00:00] INFO: Forking chef instance to converge... [2017-04-05T15:27:14+00:00] INFO: *** Chef 12.4.3 *** [2017-04-05T15:27:14+00:00] INFO: Chef-client pid: 7814 [2017-04-05T15:27:16+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-05T15:27:16+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-05T15:27:16+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-05T15:27:16+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-05T15:27:16+00:00] INFO: Running start handlers [2017-04-05T15:27:16+00:00] INFO: Start handlers complete. [2017-04-05T15:27:16+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-05T15:27:16+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-05T15:27:16+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-05T15:27:16+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-05T15:27:16+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170405152716.949000 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170405152716.951703 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170405152716.955272 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170405152716.958887 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170405152716.962520 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170405152716.966141 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-05T15:27:16+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170405152716.969694 [2017-04-05T15:27:16+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-05T15:27:16+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-05T15:27:16+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170405152716.978248 [2017-04-05T15:27:16+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-05T15:27:17+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-05T15:27:23+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-05T15:27:23+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170405152723.741746 [2017-04-05T15:27:23+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-05T15:27:23+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-05T15:27:23+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-05T15:28:02+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-05T15:28:02+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-05T15:28:02+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-05T15:28:16+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-05T15:28:16+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-05T15:28:16+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-05T15:28:16+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-05T15:28:16+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-05T15:28:16+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-05T15:28:16+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-05T15:28:17+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-05T15:28:17+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-05T15:28:17+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170405152817.343426 [2017-04-05T15:28:17+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-05T15:28:17+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-05T15:28:17+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-05T15:28:17+00:00] INFO: service[sssd] restarted [2017-04-05T15:28:17+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-05T15:28:17+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:17+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:17+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-05T15:28:17+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-05T15:28:18+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-05T15:28:18+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-05T15:28:18+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-05T15:28:18+00:00] INFO: service[ntpd] enabled [2017-04-05T15:28:18+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-05T15:28:18+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170405152818.823937 [2017-04-05T15:28:18+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-05T15:28:18+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-05T15:28:18+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-05T15:28:19+00:00] INFO: service[ntpd] restarted [2017-04-05T15:28:19+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-05T15:28:19+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170405152819.123103 [2017-04-05T15:28:19+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-05T15:28:19+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-05T15:28:19+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-05T15:28:19+00:00] INFO: service[ntpd] restarted [2017-04-05T15:28:19+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-05T15:28:19+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-05T15:28:19+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-05T15:28:19+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-05T15:28:19+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-05T15:28:19+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170405152819.933179 [2017-04-05T15:28:19+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-05T15:28:19+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-05T15:28:19+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-05T15:28:19+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170405152819.941863 [2017-04-05T15:28:19+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-05T15:28:19+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-05T15:28:19+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T15:28:20+00:00] INFO: service[autofs] restarted [2017-04-05T15:28:20+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-05T15:28:20+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170405152820.233972 [2017-04-05T15:28:20+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-05T15:28:20+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-05T15:28:20+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T15:28:20+00:00] INFO: service[autofs] restarted [2017-04-05T15:28:20+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-05T15:28:20+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-05T15:28:20+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-05T15:28:20+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T15:28:21+00:00] INFO: service[autofs] restarted [2017-04-05T15:28:21+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-05T15:28:21+00:00] INFO: service[autofs] enabled [2017-04-05T15:28:21+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-05T15:28:22+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-05T15:28:22+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-05T15:28:22+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-05T15:28:22+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-05T15:28:22+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170405152822.549663 [2017-04-05T15:28:22+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-05T15:28:22+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-05T15:28:22+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170405152822.555091 [2017-04-05T15:28:22+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-05T15:28:22+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-05T15:28:22+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-05T15:28:22+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-05T15:28:22+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-05T15:28:22+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170405152822.568023 [2017-04-05T15:28:22+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-05T15:28:22+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-05T15:28:22+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-05T15:28:24+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-05T15:28:24+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-05T15:28:24+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-05T15:28:24+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-05T15:28:25+00:00] INFO: service[munge] restarted [2017-04-05T15:28:25+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-05T15:28:25+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-05T15:28:25+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-05T15:28:25+00:00] INFO: service[munge] restarted [2017-04-05T15:28:25+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-05T15:28:25+00:00] INFO: service[munge] enabled [2017-04-05T15:28:25+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-05T15:28:26+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-05T15:28:26+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-05T15:28:29+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-05T15:28:29+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-05T15:28:29+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-05T15:28:31+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-05T15:28:31+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170405152831.897232 [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-05T15:28:31+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170405152831.902960 [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-05T15:28:31+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-05T15:28:31+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-05T15:28:31+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:31+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-05T15:28:31+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-05T15:28:31+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:31+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-05T15:28:31+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-05T15:28:32+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-05T15:28:32+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T15:28:32+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:32+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-05T15:28:32+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-05T15:28:32+00:00] INFO: service[xinetd] started [2017-04-05T15:28:32+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-05T15:28:32+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-05T15:28:33+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-05T15:28:33+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-05T15:28:33+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-05T15:28:33+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-05T15:28:33+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170405152833.733008 [2017-04-05T15:28:33+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-05T15:28:33+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-05T15:28:33+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-05T15:28:33+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-05T15:28:33+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-05T15:28:33+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-05T15:28:33+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-05T15:28:33+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-05T15:28:33+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-05T15:28:33+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-05T15:28:33+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-05T15:28:33+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-05T15:28:33+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-05T15:28:33+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-05T15:28:33+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-05T15:28:33+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-05T15:28:33+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:33+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-05T15:28:33+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-05T15:28:33+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-05T15:28:35+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-05T15:28:35+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-05T15:28:36+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-05T15:28:38+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-05T15:28:38+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-05T15:28:38+00:00] INFO: service[rsh.socket] enabled [2017-04-05T15:28:38+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-05T15:28:38+00:00] INFO: service[rsh.socket] started [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-05T15:28:38+00:00] INFO: service[rlogin.socket] enabled [2017-04-05T15:28:38+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-05T15:28:38+00:00] INFO: service[rlogin.socket] started [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T15:28:38+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T15:28:38+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T15:28:38+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-05T15:28:38+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-05T15:28:38+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-05T15:28:38+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-05T15:28:38+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-05T15:28:41+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-05T15:28:41+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-05T15:28:41+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-05T15:28:41+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-05T15:28:41+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-05T15:28:41+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-05T15:28:43+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-05T15:28:43+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-05T15:28:44+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.el7 from addon-epel7-x86_64 repository [2017-04-05T15:28:47+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.el7 [2017-04-05T15:28:47+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-05T15:28:47+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-05T15:28:47+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-05T15:28:48+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-05T15:28:48+00:00] INFO: service[rdma] enabled [2017-04-05T15:28:48+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 115.230986] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 115.240101] Loading iSCSI transport class v2.0-870. [ 115.243584] iscsi: registered transport (iser) [ 115.266184] RPC: Registered named UNIX socket transport module. [ 115.266693] RPC: Registered udp transport module. [ 115.267103] RPC: Registered tcp transport module. [ 115.267490] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 115.269947] RPC: Registered rdma transport module. [ 115.270377] RPC: Registered rdma backchannel transport module. [2017-04-05T15:28:48+00:00] INFO: service[rdma] started [2017-04-05T15:28:48+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-05T15:28:48+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-05T15:28:48+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 115.353957] FS-Cache: Loaded [ 115.362906] FS-Cache: Netfs 'nfs' registered for caching [ 115.365880] Key type dns_resolver registered [ 115.374206] NFS: Registering the id_resolver key type [ 115.374652] Key type id_resolver registered [ 115.375017] Key type id_legacy registered [2017-04-05T15:28:49+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-05T15:28:49+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-05T15:28:49+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170405152849.098777 [2017-04-05T15:28:49+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-05T15:28:49+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-05T15:28:49+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-05T15:29:25+00:00] INFO: service[kdump] restarted [2017-04-05T15:29:25+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-05T15:29:25+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-05T15:29:25+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-05T15:29:25+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-05T15:29:25+00:00] INFO: group[quota_usr] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-05T15:29:25+00:00] INFO: user[quota_usr] created [2017-04-05T15:29:25+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-05T15:29:25+00:00] INFO: group[quota_2usr] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-05T15:29:25+00:00] INFO: user[quota_2usr] created [2017-04-05T15:29:25+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-05T15:29:25+00:00] INFO: group[sanityusr] created [2017-04-05T15:29:25+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-05T15:29:25+00:00] INFO: user[sanityusr] created [2017-04-05T15:29:25+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-05T15:29:25+00:00] INFO: group[sanityusr1] created [2017-04-05T15:29:25+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-05T15:29:25+00:00] INFO: user[sanityusr1] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-05T15:29:25+00:00] INFO: user[quota15_1] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-05T15:29:25+00:00] INFO: user[quota15_2] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-05T15:29:25+00:00] INFO: user[quota15_3] created [2017-04-05T15:29:25+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_4] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_5] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_6] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_7] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_8] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_9] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_10] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_11] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_12] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_13] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_14] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_15] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_16] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_17] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_18] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_19] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_20] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_21] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_22] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_23] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_24] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_25] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_26] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_27] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_28] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_29] created [2017-04-05T15:29:26+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-05T15:29:26+00:00] INFO: user[quota15_30] created [2017-04-05T15:29:26+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-05T15:29:27+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-05T15:29:27+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-05T15:29:27+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-05T15:29:27+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-05T15:29:27+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-05T15:29:27+00:00] INFO: bash[set_hostid] ran successfully [2017-04-05T15:29:27+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-05T15:29:27+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-05T15:29:27+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-05T15:29:27+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-05T15:29:27+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-05T15:29:27+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-05T15:29:27+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-05T15:29:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-05T15:29:27+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-05T15:29:27+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-05T15:29:27+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-05T15:29:27+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-05T15:29:27+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-05T15:29:27+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-05T15:29:27+00:00] INFO: user[mpiuser] created [2017-04-05T15:29:27+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-05T15:29:27+00:00] INFO: group[mpiuser] created [2017-04-05T15:29:27+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-05T15:29:27+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-05T15:29:27+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-05T15:29:27+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-05T15:29:27+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-05T15:29:27+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-05T15:29:27+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-05T15:29:27+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-05T15:29:28+00:00] INFO: service[postfix] restarted [2017-04-05T15:29:28+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-05T15:29:28+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-05T15:29:28+00:00] INFO: execute[newaliases] ran successfully [2017-04-05T15:29:28+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-05T15:29:28+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-05T15:29:28+00:00] INFO: service[sshd] restarted [2017-04-05T15:29:28+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-05T15:29:28+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-05T15:29:28+00:00] INFO: service[rsyslog] restarted [2017-04-05T15:29:28+00:00] INFO: Chef Run complete in 132.264778393 seconds [2017-04-05T15:29:28+00:00] INFO: Skipping removal of unused files from the cache [2017-04-05T15:29:28+00:00] INFO: Running report handlers [2017-04-05T15:29:28+00:00] INFO: Creating JSON run report [2017-04-05T15:29:28+00:00] INFO: Report handlers complete [ OK ] Stopped target User and Group Name Lookups. [ OK ] Stopped System Logging Service. [ OK ] Unmounted /scratch. [ OK ] Stopped Automounts filesystems on demand. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... Stopping System Security Services Daemon... [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped target Encrypted Volumes. Stopping Load/Save Random Seed... [ OK ] Stopped target Swap. Stopping Security Auditing Service... [ OK ] Stopped Load/Save Random Seed. [ 2050.739622] type=1305 audit(1491408064.411:1160): audit_pid=0 old=451 auid=4294967295 ses=4294967295 res=1 [ 2050.740511] type=1130 audit(1491408064.412:1161): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2050.741953] type=1131 audit(1491408064.414:1162): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2050.744694] type=1130 audit(1491408064.417:1163): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK [ 2050.748243] type=1131 audit(1491408064.421:1164): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ] Stopped Security Auditing Service. [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Dire[ 2050.754916] type=1130 audit(1491408064.426:1165): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ctories... [ OK [ 2050.758848] type=1131 audit(1491408064.430:1166): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ 2050.767812] type=1130 audit(1491408064.439:1167): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2050.772034] type=1131 audit(1491408064.444:1168): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped target Local File Systems. Unmounting /run/user/0... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... [ 2050.782469] type=1130 audit(1491408064.454:1169): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. [ 2050.803543] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 2050.807860] systemd-journald[339]: Received SIGTERM from PID 1 (systemd-shutdow). Stopping Create Static Device Nodes in /dev... [ OK ] Reached target Shutdown. [ 2060.811066] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 2060.816816] systemd-shutdown[1]: Sending SIGKILL to PID 357 (lvmetad). [ 2060.820894] systemd-shutdown[1]: Unmounting file systems. [ 2060.898692] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2060.917369] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2060.919486] systemd-shutdown[1]: All filesystems unmounted. [ 2060.921638] systemd-shutdown[1]: Deactivating swaps. [ 2060.923815] systemd-shutdown[1]: All swaps deactivated. [ 2060.925918] systemd-shutdown[1]: Detaching loop devices. [ 2060.928504] systemd-shutdown[1]: All loop devices detached. [ 2060.930611] systemd-shutdown[1]: Detaching DM devices. [ 2060.932717] systemd-shutdown[1]: All DM devices detached. [ 2060.954643] systemd-shutdown[1]: Rebooting. [ 2060.957343] Restarting system. [ 2060.959109] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x35e0c000-0x36efdfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 528MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1701832k/2097140k available (6765k kernel code, 400k absent, 394908k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005028] Security Framework initialized [ 0.007008] SELinux: Initializing. [ 0.008188] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011826] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013489] Mount-cache hash table entries: 4096 [ 0.015234] Initializing cgroup subsys memory [ 0.016012] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.019004] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.025004] Initializing cgroup subsys net_prio [ 0.026105] Disabled fast string operations [ 0.027019] mce: CPU supports 10 MCE banks [ 0.028056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] tlb_flushall_shift: 6 [ 0.032176] Freeing SMP alternatives: 28k freed [ 0.038133] ACPI: Core revision 20130517 [ 0.040140] ACPI: All ACPI Tables successfully acquired [ 0.041015] ftrace: allocating 25813 entries in 101 pages [ 0.049187] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.050005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.052003] smpboot: Max logical packages: 2 [ 0.053247] Enabling x2apic [ 0.054000] Enabled x2apic [ 0.054004] Switched APIC routing to physical x2apic. [ 0.057298] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.058003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.061025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.063439] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.066040] Brought up 2 CPUs [ 0.066029] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.068005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.069494] devtmpfs: initialized [ 0.071856] EVM: security.selinux [ 0.072006] EVM: security.ima [ 0.073004] EVM: security.capability [ 0.074732] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.075029] pinctrl core: initialized pinctrl subsystem [ 0.076133] NET: Registered protocol family 16 [ 0.077266] ACPI: bus type PCI registered [ 0.078006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.079084] PCI: Using configuration type 1 for base access [ 0.083077] ACPI: Added _OSI(Module Device) [ 0.084013] ACPI: Added _OSI(Processor Device) [ 0.085006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.086005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.088656] ACPI: Interpreter enabled [ 0.089014] ACPI: (supports S0 S5) [ 0.090004] ACPI: Using IOAPIC for interrupt routing [ 0.091017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.094604] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.095010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.096009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.097018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.098237] acpiphp: Slot [1] registered [ 0.099030] acpiphp: Slot [2] registered [ 0.100025] acpiphp: Slot [3] registered [ 0.101023] acpiphp: Slot [4] registered [ 0.102022] acpiphp: Slot [5] registered [ 0.103022] acpiphp: Slot [6] registered [ 0.104022] acpiphp: Slot [7] registered [ 0.105022] acpiphp: Slot [8] registered [ 0.106029] acpiphp: Slot [9] registered [ 0.107022] acpiphp: Slot [10] registered [ 0.108022] acpiphp: Slot [11] registered [ 0.109022] acpiphp: Slot [12] registered [ 0.110009] acpiphp: Slot [13] registered [ 0.111022] acpiphp: Slot [14] registered [ 0.112021] acpiphp: Slot [15] registered [ 0.113028] acpiphp: Slot [16] registered [ 0.114028] acpiphp: Slot [17] registered [ 0.115022] acpiphp: Slot [18] registered [ 0.116022] acpiphp: Slot [19] registered [ 0.117021] acpiphp: Slot [20] registered [ 0.118021] acpiphp: Slot [21] registered [ 0.119022] acpiphp: Slot [22] registered [ 0.120028] acpiphp: Slot [23] registered [ 0.121022] acpiphp: Slot [24] registered [ 0.122022] acpiphp: Slot [25] registered [ 0.123021] acpiphp: Slot [26] registered [ 0.124021] acpiphp: Slot [27] registered [ 0.125028] acpiphp: Slot [28] registered [ 0.126022] acpiphp: Slot [29] registered [ 0.127027] acpiphp: Slot [30] registered [ 0.128028] acpiphp: Slot [31] registered [ 0.129253] PCI host bridge to bus 0000:00 [ 0.130006] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.131006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.132005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.133005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.134005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.135971] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.136006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.137005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.138005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.139807] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.140013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.157607] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.159078] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.160770] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.162073] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.163731] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.165299] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.166272] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.167013] vgaarb: loaded [ 0.168006] vgaarb: bridge control possible 0000:00:02.0 [ 0.169202] SCSI subsystem initialized [ 0.170052] ACPI: bus type USB registered [ 0.171038] usbcore: registered new interface driver usbfs [ 0.172018] usbcore: registered new interface driver hub [ 0.173050] usbcore: registered new device driver usb [ 0.174179] PCI: Using ACPI for IRQ routing [ 0.175337] NetLabel: Initializing [ 0.176008] NetLabel: domain hash size = 128 [ 0.177005] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.178040] NetLabel: unlabeled traffic allowed by default [ 0.179147] Switched to clocksource kvm-clock [ 0.188012] pnp: PnP ACPI init [ 0.189068] ACPI: bus type PNP registered [ 0.190714] pnp: PnP ACPI: found 5 devices [ 0.191886] ACPI: bus type PNP unregistered [ 0.202643] NET: Registered protocol family 2 [ 0.204231] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.206491] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.208100] TCP: Hash tables configured (established 16384 bind 16384) [ 0.209600] TCP: reno registered [ 0.210655] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.212071] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.213626] NET: Registered protocol family 1 [ 0.214834] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.216196] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.217535] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.219171] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.220808] Unpacking initramfs... [ 0.490221] Freeing initrd memory: 17352k freed [ 0.500419] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.502066] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.503992] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.505446] Initialise system trusted keyring [ 0.506675] audit: initializing netlink socket (disabled) [ 0.508032] type=2000 audit(1491408102.166:1): initialized [ 0.529087] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.530553] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.533039] zpool: loaded [ 0.534045] zbud: loaded [ 0.535299] VFS: Disk quotas dquot_6.5.2 [ 0.536508] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.538239] msgmni has been set to 3357 [ 0.539478] Key type big_key registered [ 0.542303] NET: Registered protocol family 38 [ 0.543534] Key type asymmetric registered [ 0.544700] Asymmetric key parser 'x509' registered [ 0.545983] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.548266] io scheduler noop registered [ 0.549434] io scheduler deadline registered (default) [ 0.550763] io scheduler cfq registered [ 0.552042] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.553491] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.555142] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.557315] ACPI: Power Button [PWRF] [ 0.558662] GHES: HEST is not enabled! [ 0.559932] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.582675] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.584718] Non-volatile memory driver v1.3 [ 0.585917] Linux agpgart interface v0.103 [ 0.587273] crash memory driver: version 1.1 [ 0.588558] rdac: device handler registered [ 0.589820] hp_sw: device handler registered [ 0.605667] emc: device handler registered [ 0.606841] alua: device handler registered [ 0.608051] libphy: Fixed MDIO Bus: probed [ 0.609298] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.610740] ehci-pci: EHCI PCI platform driver [ 0.611945] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.613337] ohci-pci: OHCI PCI platform driver [ 0.614537] uhci_hcd: USB Universal Host Controller Interface driver [ 0.616176] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.617552] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.619707] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.621043] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.622502] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.624013] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.626191] usb usb1: Product: UHCI Host Controller [ 0.627475] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.629748] usb usb1: SerialNumber: 0000:00:01.2 [ 0.631144] hub 1-0:1.0: USB hub found [ 0.632304] hub 1-0:1.0: 2 ports detected [ 0.633665] usbcore: registered new interface driver usbserial [ 0.635098] usbcore: registered new interface driver usbserial_generic [ 0.636609] usbserial: USB Serial support registered for generic [ 0.638073] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.641012] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.642335] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.643785] mousedev: PS/2 mouse device common for all mice [ 0.645497] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.648136] rtc_cmos 00:00: RTC can wake from S4 [ 0.648839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.652538] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.652883] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.656594] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.658082] cpuidle: using governor menu [ 0.659316] hidraw: raw HID events driver (C) Jiri Kosina [ 0.660758] usbcore: registered new interface driver usbhid [ 0.662111] usbhid: USB HID core driver [ 0.663300] drop_monitor: Initializing network drop monitor service [ 0.664835] TCP: cubic registered [ 0.665923] Initializing XFRM netlink socket [ 0.667219] NET: Registered protocol family 10 [ 0.668757] NET: Registered protocol family 17 [ 0.670255] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.672063] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.673491] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.676141] Loading compiled-in X.509 certificates [ 0.677447] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.679916] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.682889] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.685365] registered taskstats version 1 [ 0.689383] Key type trusted registered [ 0.692803] Key type encrypted registered [ 0.696179] IMA: No TPM chip found, activating TPM-bypass! [ 0.697910] rtc_cmos 00:00: setting system clock to 2017-04-05 16:01:42 UTC (1491408102) [ 0.701121] Freeing unused kernel memory: 1680k freed [ 0.707046] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.711181] systemd[1]: Detected virtualization kvm. [ 0.712495] systemd[1]: Detected architecture x86-64. [ 0.713826] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.715938] systemd[1]: Set hostname to . [ OK ] Created slice Root Slice. [ 0.750503] systemd[1]: Created slice Root Slice. [ 0.751766] systemd[1]: Starting Root Slice. [ OK ] Reached target Local File Systems. [ 0.753611] systemd[1]: Reached target Local File Systems. [ 0.754976] systemd[1]: Starting Local File Systems. [ OK ] Listening on Journal Socket. [ 0.756887] systemd[1]: Listening on Journal Socket. [ 0.758208] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Kernel Socket. [ 0.760073] systemd[1]: Listening on udev Kernel Socket. [ 0.761404] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Swap. [ 0.763221] systemd[1]: Reached target Swap. [ 0.764437] systemd[1]: Starting Swap. [ OK ] Listening on udev Control Socket. [ 0.766217] systemd[1]: Listening on udev Control Socket. [ 0.767562] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Sockets. [ 0.769400] systemd[1]: Reached target Sockets. [ 0.770637] systemd[1]: Starting Sockets. [ OK ] Reached target Timers. [ 0.772351] systemd[1]: Reached target Timers. [ 0.773588] systemd[1]: Starting Timers. [ OK ] Created slice System Slice. [ 0.775383] systemd[1]: Created slice System Slice. [ 0.776671] systemd[1]: Starting System Slice. [ 0.778503] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 0.782114] systemd[1]: Starting Journal Service... Starting Journal Service... [ OK ] Reached target Slices. [ 0.785838] systemd[1]: Reached target Slices. [ 0.787756] systemd[1]: Starting Slices. [ 0.790317] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.796322] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.802141] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ 0.807835] systemd[1]: Started Journal Service. [ OK ] Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... Starting Apply Kernel Variables... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ OK ] Mounted Configuration File System. [ OK ] Reached target System Initialization. [ 0.977380] FDC 0 is a S82078B [ 0.980965] scsi host0: ata_piix [ 0.982141] scsi host1: ata_piix [ 0.983237] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 0.984727] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 0.988742] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 0.991388] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 0.993552] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 0.994648] [drm] Initialized drm 1.1.0 20060810 [ 0.998267] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.007607] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.012045] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.014678] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.021928] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.023398] [TTM] Initializing pool allocator [ 1.024922] [TTM] Initializing DMA pool allocator [ 1.026687] [drm] fb mappable at 0xF0000000 [ 1.027862] [drm] vram aper at 0xF0000000 [ 1.029007] [drm] size 33554432 [ 1.030041] [drm] fb depth is 24 [ 1.031084] [drm] pitch is 3072 %G[ 1.032707] fbcon: cirrusdrmfb (fb0) is primary device [ 1.061210] vda: vda1 vda2 vda4 [ 1.072539] Console: switching to colour frame buffer device 128x48 [ 1.084150] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Found device /dev/disk/by-uuid/420143da-7f83-49b4-8a24-6cffb1d410f7. Starting File System Check on /dev/...a-7f83-49b4-8a24-6cffb1d410f7... %G[ 1.099031] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started File System Check on /dev/d...3da-7f83-49b4-8a24-6cffb1d410f7. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 1.177897] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.179248] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Load Kernel Modules. Stopping Load Kernel Modules... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Plymouth switch root service. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ 1.400833] systemd-journald[95]: Received SIGTERM from PID 1 (systemd). [ 1.421201] SELinux: Disabled at runtime. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 1.461026] type=1404 audit(1491408103.263:2): selinux=0 auid=4294967295 ses=4294967295 [ 1.468167] ip_tables: (C) 2000-2006 Netfilter Core Team [ 1.469614] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ 1.501051] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Stopped Switch Root. Mounting Huge Pages File System... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on LVM2 poll daemon socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Created slice User and Session Slice. Starting Read and set NIS domainname from /etc/sysconfig/network... Starting Kernel Module supporting RPCSEC_GSS... Starting Load Kernel Modules... [ OK ] Reached target Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Device-mapper event daemon FIFOs. Mounting NFSD configuration filesystem... [ OK ] Reached target Slices. [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... Mounting POSIX Message Queue File System... [ 1.560647] EXT4-fs (vda1): re-mounted. Opts: (null) [ 1.570692] RPC: Registered named UNIX socket transport module. [ 1.570692] RPC: Registered udp transport module. [ 1.570693] RPC: Registered tcp transport module. [ 1.570693] RPC: Registered tcp NFSv4.1 backchannel transport module. Starting Remount Root and Kernel File Systems... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. Mounting Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Load Kernel Modules. [ OK ] Mounted Debug File System. [ OK ] Started Remount Root and Kernel File Systems. Starting udev Coldplug all Devices... Starting Configure read-only root support... Starting Load/Save Random Seed... Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started Load/Save Random Seed. [ 1.605073] systemd-journald[356]: Received request to flush runtime journal from PID 1 [ 1.613917] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Mounted NFSD configuration filesystem. [ OK ] Started Apply Kernel Variables. [ OK ] Started Configure read-only root support. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 1.682882] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.684793] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. %G%G[ 1.715394] ppdev: user-space parallel port driver [ 1.723589] AES CTR mode by8 optimization enabled [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 253:4... [ OK ] Found device /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. [ 1.741166] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 1.745754] alg: No test for crc32 (crc32-pclmul) [ 1.749324] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 1.751217] intel_rapl: no valid rapl domains found in package 0 Activating swap /dev/disk/by-uuid/d...6-1e76-4988-acad-74e281f3721c... [ 1.764609] intel_powerclamp: No package C-state available [ OK ] Activated swap /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. [ OK ] Reached target Swap. [ 1.774180] intel_powerclamp: No package C-state available[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Preprocess NFS configuration... [ OK ] Started Preprocess NFS configuration. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... Mounting RPC Pipe File System... [ OK ] Mounted RPC Pipe File System. [ 1.872497] type=1305 audit(1491408103.674:3): audit_pid=499 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Reached target Paths. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Reached target Timers. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting GSSAPI Proxy Daemon... Starting System Security Services Daemon... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting Network Manager... Starting Authorization Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting Dump dmesg to /var/log/dmesg... Starting NTP client/server... [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Started Authorization Manager. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started NTP client/server. [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ 2.094890] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [[ 2.100487] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1  OK ] Started Hostname Service. [ 2.116575] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 2.168141] Loading iSCSI transport class v2.0-870. [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ 2.178169] iscsi: registered transport (iser) [ OK ] Started Login Service. [ 2.213596] RPC: Registered rdma transport module. [ 2.215088] RPC: Registered rdma backchannel transport module. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started LVM2 PV scan on device 253:4. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting System Logging Service... Starting MUNGE authentication service... Starting Dynamic System Tuning Daemon... Starting Xinetd A Powerful Replacement For Inetd... Starting OpenSSH server daemon... Starting Postfix Mail Transport Agent... Starting Notify NFS peers of a restart... [ OK ] Reached target Network is Online. Mounting /scratch... [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ OK ] Started MUNGE authentication service. [ 7.724306] FS-Cache: Loaded [ OK ] Started OpenSSH server daemon. [ 7.748210] FS-Cache: Netfs 'nfs' registered for caching [ 7.762558] Key type dns_resolver registered [ 7.778432] NFS: Registering the id_resolver key type [ 7.780041] Key type id_resolver registered [ 7.781530] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Permit User Sessions... Starting Crash recovery kernel arming... [ OK ] Started Permit User Sessions. Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ OK ] Started Job spooling tools. Starting Job spooling tools... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [ 257.883815] device-mapper: uevent: version 1.0.3 [ 257.885192] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 517.979932] spl: loading out-of-tree module taints kernel. [ 517.980637] spl: module verification failed: signature and/or required key missing - tainting kernel [ 517.986167] SPL: Loaded module v0.6.5.9-1 [ 517.987028] znvpair: module license 'CDDL' taints kernel. [ 517.987578] Disabling lock debugging due to kernel taint [ 518.008080] ZFS: Loaded module v0.6.5.9-1, ZFS pool version 5000, ZFS filesystem version 5 [ 518.074344] SPL: using hostid 0x61303930 [ 527.976116] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 527.979233] alg: No test for adler32 (adler32-zlib) [ 527.979782] alg: No test for crc32 (crc32-table) [ 528.740741] Lustre: Lustre: Build Version: 2.9.54_26_gaaeee33 [ 528.760740] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 528.761343] LNet: Accept all, port 7988 [ 528.784805] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 528.833889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 529.094174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 529.117795] Lustre: lustre-MDT0000: new disk, initializing [ 529.445266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.520710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 529.886574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 530.173638] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 530.705356] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 530.992431] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 531.288345] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 531.578700] Lustre: DEBUG MARKER: sync; sync; sync [ 534.398254] Lustre: lustre-MDT0000: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 536.972587] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 537.259293] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 540.127078] Lustre: MGS: Connection restored to 03cec0f4-053a-ef86-8d9b-edfc1ffa9412 (at 10.9.5.244@tcp) [ 549.472135] Lustre: lustre-MDT0000: Connection restored to 03cec0f4-053a-ef86-8d9b-edfc1ffa9412 (at 10.9.5.244@tcp) [ 549.473221] Lustre: Skipped 1 previous similar message [ 557.187101] Lustre: MGS: Connection restored to e30d5548-457d-0a7a-0f95-9e322a62a804 (at 10.9.5.237@tcp) [ 560.987397] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 561.321672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 561.513674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 561.675167] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 561.992976] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 571.692327] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 572.739882] Lustre: DEBUG MARKER: uname -n [ 573.398999] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 573.793540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 574.160879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.54_26_gaaeee33 [ 574.348758] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.54_26_gaaeee33 [ 574.527693] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 574.859607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.54_26_gaaeee33 [ 575.057639] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.54_26_gaaeee33 [ 575.587828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.54_26_gaaeee33 [ 575.852709] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.54_26_gaaeee33 [ 576.053355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: runtests ============----- Wed Apr 5 16:11:18 UTC 2017 [ 576.252952] Lustre: DEBUG MARKER: -----============= acceptance-small: runtests ============----- Wed Apr 5 16:11:18 UTC 2017 [ 576.575563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration -o sec =========================================================== 16:11:18 \(1491408678\) [ 576.754952] Lustre: DEBUG MARKER: == runtests test complete, duration -o sec =========================================================== 16:11:18 (1491408678) [ 578.206864] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 578.206864] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 578.206864] if [ $running -ne $mpts ]; then [ 578.206864] echo $(hostname) env are INSANE!; [ 578.206864] exit 1; [ 578.206864] fi [ 578.549641] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 578.549641] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 578.549641] if [ $running -ne $mpts ]; then [ 578.549641] echo $(hostname) env are INSANE!; [ 578.549641] exit 1; [ 578.549641] fi [ 579.460611] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 579.770774] Lustre: DEBUG MARKER: cat /proc/mounts [ 581.456306] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 581.789986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 581.976651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 582.145314] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 582.500303] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 583.053663] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 583.396900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test 1: All Runtests ===================================================================== 16:11:25 \(1491408685\) [ 583.586385] Lustre: DEBUG MARKER: == runtests test 1: All Runtests ===================================================================== 16:11:25 (1491408685) [ 583.801803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark touching \/mnt\/lustre at Wed Apr 5 16:11:25 UTC 2017 \(@1491408685\) [ 583.994021] Lustre: DEBUG MARKER: touching /mnt/lustre at Wed Apr 5 16:11:25 UTC 2017 (@1491408685) [ 584.198509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create an empty file \/mnt\/lustre\/hosts.7387 [ 584.380217] Lustre: DEBUG MARKER: create an empty file /mnt/lustre/hosts.7387 [ 584.614039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.7387 [ 584.805695] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.7387 [ 585.022541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing \/etc\/hosts and \/mnt\/lustre\/hosts.7387 [ 585.214531] Lustre: DEBUG MARKER: comparing /etc/hosts and /mnt/lustre/hosts.7387 [ 585.554173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.7387 to \/mnt\/lustre\/hosts.7387.ren [ 585.726159] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.7387 to /mnt/lustre/hosts.7387.ren [ 585.935369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.7387 again [ 586.120681] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.7387 again [ 586.327506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.7387 [ 586.515590] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.7387 [ 586.717393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.7387 [ 586.901034] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.7387 [ 587.096338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.7387.2 [ 587.291314] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.7387.2 [ 587.497111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.7387.2 to 123 bytes [ 587.680252] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.7387.2 to 123 bytes [ 587.938878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating \/mnt\/lustre\/d1.runtests [ 588.130531] Lustre: DEBUG MARKER: creating /mnt/lustre/d1.runtests [ 589.294651] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying 1000 files from \/etc \/bin to \/mnt\/lustre\/d1.runtests\/etc \/bin at Wed Apr 5 16:11:31 UTC 2017 [ 589.482532] Lustre: DEBUG MARKER: copying 1000 files from /etc /bin to /mnt/lustre/d1.runtests/etc /bin at Wed Apr 5 16:11:31 UTC 2017 [ 596.572785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 1000 newly copied files at Wed Apr 5 16:11:38 UTC 2017 [ 596.767050] Lustre: DEBUG MARKER: comparing 1000 newly copied files at Wed Apr 5 16:11:38 UTC 2017 [ 599.616376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark finished at Wed Apr 5 16:11:41 UTC 2017 \(16\) [ 599.804448] Lustre: DEBUG MARKER: finished at Wed Apr 5 16:11:41 UTC 2017 (16) [ 601.093528] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 601.392447] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 603.359053] LustreError: 5925:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005d874300 x1563855295629824/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 604.480300] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.485346] LustreError: Skipped 1 previous similar message [ 609.576034] Lustre: 9973:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491408705/real 1491408705] req@ffff88005d874300 x1563855295629904/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491408711 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 609.597858] Lustre: server umount lustre-MDT0000 complete [ 609.744031] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 610.043902] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 610.043902] grep -q ^lustre-mdt1/ /proc/mounts || [ 610.043902] zpool export lustre-mdt1 [ 613.904181] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 613.904181] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 613.904181] if [ $running -ne $mpts ]; then [ 613.904181] echo $(hostname) env are INSANE!; [ 613.904181] exit 1; [ 613.904181] fi [ 614.207976] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 614.207976] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 614.207976] if [ $running -ne $mpts ]; then [ 614.207976] echo $(hostname) env are INSANE!; [ 614.207976] exit 1; [ 614.207976] fi [ 615.127444] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 615.408020] Lustre: DEBUG MARKER: modprobe zfs; [ 615.408020] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 615.408020] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 615.874815] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 616.161927] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 616.349194] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 616.351571] Lustre: Skipped 3 previous similar messages [ 616.528009] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.738888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 617.029944] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 617.523077] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 617.815070] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 618.107926] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 618.406773] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 618.700575] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 621.470072] Lustre: 5923:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491408718/real 1491408718] req@ffff88005e275200 x1563855295630080/t0(0) o8->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491408723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 628.709128] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 629.044439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 629.239704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 629.406064] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 634.755063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 1000 previously copied files [ 634.943235] Lustre: DEBUG MARKER: comparing 1000 previously copied files [ 641.915470] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 642.208569] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 642.767088] LustreError: 5924:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880065fa4000 x1563855295635920/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 644.102730] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 644.105051] Lustre: Skipped 1 previous similar message [ 646.511068] LustreError: 5925:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880068950900 x1563855295635952/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 654.090209] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.094956] LustreError: Skipped 1 previous similar message [ 654.390049] Lustre: 12697:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491408750/real 1491408750] req@ffff88006892a700 x1563855295636032/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491408756 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 654.397376] Lustre: 12697:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 654.416641] Lustre: server umount lustre-MDT0000 complete [ 654.562295] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 654.865432] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 654.865432] grep -q ^lustre-mdt1/ /proc/mounts || [ 654.865432] zpool export lustre-mdt1 [ 658.750239] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 658.750239] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 658.750239] if [ $running -ne $mpts ]; then [ 658.750239] echo $(hostname) env are INSANE!; [ 658.750239] exit 1; [ 658.750239] fi [ 659.060783] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 659.060783] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 659.060783] if [ $running -ne $mpts ]; then [ 659.060783] echo $(hostname) env are INSANE!; [ 659.060783] exit 1; [ 659.060783] fi [ 659.973914] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 660.260211] Lustre: DEBUG MARKER: modprobe zfs; [ 660.260211] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 660.260211] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 660.815214] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 661.118909] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 661.315540] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 661.317894] Lustre: Skipped 8 previous similar messages [ 661.493667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.686214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 661.981606] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 662.475153] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 662.769405] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 663.061432] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 663.359838] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 663.652040] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 666.456085] Lustre: 5923:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491408763/real 1491408763] req@ffff88006892ad00 x1563855295636208/t0(0) o8->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491408768 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 673.638733] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 673.972535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 674.162913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 674.337544] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 679.702104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/d1.runtests [ 679.894136] Lustre: DEBUG MARKER: removing /mnt/lustre/d1.runtests [ 682.679512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.7387.ren to \/mnt\/lustre\/hosts.7387 [ 682.858754] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.7387.ren to /mnt/lustre/hosts.7387 [ 683.070380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.7387 [ 683.258054] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.7387 [ 683.613785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.7387 [ 683.845802] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.7387 [ 684.060257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark verifying \/mnt\/lustre\/hosts.7387.2 is 123 bytes [ 684.247898] Lustre: DEBUG MARKER: verifying /mnt/lustre/hosts.7387.2 is 123 bytes [ 684.478227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running createmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 684.666952] Lustre: DEBUG MARKER: running createmany -d /mnt/lustre/d1.runtests/d 100 [ 685.014980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running unlinkmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 685.210586] Lustre: DEBUG MARKER: running unlinkmany -d /mnt/lustre/d1.runtests/d 100 [ 685.532119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 685.735074] Lustre: DEBUG MARKER: done [ 685.913195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 686.213371] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 686.626789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 687.947610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 689.266275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 690.580303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 696.948658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Space was freed: now 3328kB, was 3328kB. [ 697.135067] Lustre: DEBUG MARKER: Space was freed: now 3328kB, was 3328kB. [ 697.338833] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 697.992999] Lustre: DEBUG MARKER: rc=0; [ 697.992999] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 697.992999] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 697.992999] echo $(hostname -s): $val; [ 697.992999] rc=$val; [ 697.992999] fi; [ 697.992999] exit $rc [ 698.673013] Lustre: DEBUG MARKER: dmesg [ 699.086095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration 123 sec ========================================================== 16:13:21 \(1491408801\) [ 699.273647] Lustre: DEBUG MARKER: == runtests test complete, duration 123 sec ========================================================== 16:13:21 (1491408801) [ 699.511933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity ============----- Wed Apr 5 16:13:21 UTC 2017 [ 699.695058] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Wed Apr 5 16:13:21 UTC 2017 [ 700.061123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration -o sec ============================================================= 16:13:22 \(1491408802\) [ 700.250682] Lustre: DEBUG MARKER: == sanity test complete, duration -o sec ============================================================= 16:13:22 (1491408802) [ 701.696857] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 701.696857] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 701.696857] if [ $running -ne $mpts ]; then [ 701.696857] echo $(hostname) env are INSANE!; [ 701.696857] exit 1; [ 701.696857] fi [ 702.037623] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 702.037623] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 702.037623] if [ $running -ne $mpts ]; then [ 702.037623] echo $(hostname) env are INSANE!; [ 702.037623] exit 1; [ 702.037623] fi [ 702.947833] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 703.245973] Lustre: DEBUG MARKER: cat /proc/mounts [ 704.920917] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 705.252410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 705.436677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 705.605778] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 705.955610] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 706.542443] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 707.053747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 76 407 42a 42b 42c 42d 45 68b 65ic 180 [ 707.240846] Lustre: DEBUG MARKER: excepting tests: 76 407 42a 42b 42c 42d 45 68b 65ic 180 [ 707.460621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o 51b [ 707.633744] Lustre: DEBUG MARKER: skipping tests SLOW=no: 24D 27m 64b 68 71 115 300o 51b [ 707.971600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0a: touch\; rm ========================================================================= 16:13:29 \(1491408809\) [ 708.162991] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ========================================================================= 16:13:29 (1491408809) [ 708.380487] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 709.045887] Lustre: DEBUG MARKER: rc=0; [ 709.045887] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 709.045887] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 709.045887] echo $(hostname -s): $val; [ 709.045887] rc=$val; [ 709.045887] fi; [ 709.045887] exit $rc [ 709.737915] Lustre: DEBUG MARKER: dmesg [ 710.176369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0b: chmod 0755 \/mnt\/lustre ============================================================ 16:13:32 \(1491408812\) [ 710.364543] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ============================================================ 16:13:32 (1491408812) [ 710.574751] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 711.251315] Lustre: DEBUG MARKER: rc=0; [ 711.251315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 711.251315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 711.251315] echo $(hostname -s): $val; [ 711.251315] rc=$val; [ 711.251315] fi; [ 711.251315] exit $rc [ 711.942955] Lustre: DEBUG MARKER: dmesg [ 712.360693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 0c: check import proc ================================================================= 16:13:34 \(1491408814\) [ 712.548703] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ================================================================= 16:13:34 (1491408814) [ 712.751332] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 713.411627] Lustre: DEBUG MARKER: rc=0; [ 713.411627] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 713.411627] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 713.411627] echo $(hostname -s): $val; [ 713.411627] rc=$val; [ 713.411627] fi; [ 713.411627] exit $rc [ 714.101286] Lustre: DEBUG MARKER: dmesg [ 714.542162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 1: mkdir\; remkdir\; rmdir ============================================================== 16:13:36 \(1491408816\) [ 714.712211] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ============================================================== 16:13:36 (1491408816) [ 714.988935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 715.656992] Lustre: DEBUG MARKER: rc=0; [ 715.656992] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 715.656992] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 715.656992] echo $(hostname -s): $val; [ 715.656992] rc=$val; [ 715.656992] fi; [ 715.656992] exit $rc [ 716.335015] Lustre: DEBUG MARKER: dmesg [ 716.763114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 2: mkdir\; touch\; rmdir\; check file ==================================================== 16:13:38 \(1491408818\) [ 716.937661] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ==================================================== 16:13:38 (1491408818) [ 717.168179] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 717.840758] Lustre: DEBUG MARKER: rc=0; [ 717.840758] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 717.840758] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 717.840758] echo $(hostname -s): $val; [ 717.840758] rc=$val; [ 717.840758] fi; [ 717.840758] exit $rc [ 718.525678] Lustre: DEBUG MARKER: dmesg [ 718.945899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 3: mkdir\; touch\; rmdir\; check dir ===================================================== 16:13:40 \(1491408820\) [ 719.133934] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ===================================================== 16:13:40 (1491408820) [ 719.360130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 720.022687] Lustre: DEBUG MARKER: rc=0; [ 720.022687] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 720.022687] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 720.022687] echo $(hostname -s): $val; [ 720.022687] rc=$val; [ 720.022687] fi; [ 720.022687] exit $rc [ 720.702658] Lustre: DEBUG MARKER: dmesg [ 721.140740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 4: mkdir\; touch dir\/file\; rmdir\; checkdir \(expect error\) ============================== 16:13:43 \(1491408823\) [ 721.332446] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ============================== 16:13:43 (1491408823) [ 721.557736] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 722.240619] Lustre: DEBUG MARKER: rc=0; [ 722.240619] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 722.240619] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 722.240619] echo $(hostname -s): $val; [ 722.240619] rc=$val; [ 722.240619] fi; [ 722.240619] exit $rc [ 722.923049] Lustre: DEBUG MARKER: dmesg [ 723.356184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 5: mkdir ...\/d5 ...\/d5\/d2\; chmod ...\/d5\/d2 ============================================ 16:13:45 \(1491408825\) [ 723.544808] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ============================================ 16:13:45 (1491408825) [ 723.837645] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 724.505721] Lustre: DEBUG MARKER: rc=0; [ 724.505721] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 724.505721] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 724.505721] echo $(hostname -s): $val; [ 724.505721] rc=$val; [ 724.505721] fi; [ 724.505721] exit $rc [ 725.204996] Lustre: DEBUG MARKER: dmesg [ 725.679579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6a: touch f6a\; chmod f6a\; runas -u 500 -g 500 chmod f6a \(should return error\) ========= 16:13:47 \(1491408827\) [ 725.872845] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ========= 16:13:47 (1491408827) [ 726.110345] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 726.776492] Lustre: DEBUG MARKER: rc=0; [ 726.776492] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 726.776492] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 726.776492] echo $(hostname -s): $val; [ 726.776492] rc=$val; [ 726.776492] fi; [ 726.776492] exit $rc [ 727.492044] Lustre: DEBUG MARKER: dmesg [ 727.947487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6c: touch f6c\; chown f6c\; runas -u 500 -g 500 chown f6c \(should return error\) ========= 16:13:49 \(1491408829\) [ 728.155475] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ========= 16:13:49 (1491408829) [ 728.387113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 729.049711] Lustre: DEBUG MARKER: rc=0; [ 729.049711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 729.049711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 729.049711] echo $(hostname -s): $val; [ 729.049711] rc=$val; [ 729.049711] fi; [ 729.049711] exit $rc [ 729.738091] Lustre: DEBUG MARKER: dmesg [ 730.178778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6e: touch f6e\; chgrp f6e\; runas -u 500 -g 500 chgrp f6e \(should return error\) ========= 16:13:52 \(1491408832\) [ 730.366793] Lustre: DEBUG MARKER: == sanity test 6e: touch f6e; chgrp f6e; runas -u 500 -g 500 chgrp f6e (should return error) ========= 16:13:52 (1491408832) [ 730.591655] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 731.271130] Lustre: DEBUG MARKER: rc=0; [ 731.271130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 731.271130] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 731.271130] echo $(hostname -s): $val; [ 731.271130] rc=$val; [ 731.271130] fi; [ 731.271130] exit $rc [ 731.963276] Lustre: DEBUG MARKER: dmesg [ 732.395612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 16:13:54 \(1491408834\) [ 732.589936] Lustre: DEBUG MARKER: == sanity test 6g: Is new dir in sgid dir inheriting group? ========================================== 16:13:54 (1491408834) [ 732.877998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 733.545464] Lustre: DEBUG MARKER: rc=0; [ 733.545464] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 733.545464] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 733.545464] echo $(hostname -s): $val; [ 733.545464] rc=$val; [ 733.545464] fi; [ 733.545464] exit $rc [ 734.236083] Lustre: DEBUG MARKER: dmesg [ 734.674978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 ...\/f6h \(should return error\) ================ 16:13:56 \(1491408836\) [ 734.861990] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../f6h (should return error) ================ 16:13:56 (1491408836) [ 735.089369] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 735.769760] Lustre: DEBUG MARKER: rc=0; [ 735.769760] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 735.769760] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 735.769760] echo $(hostname -s): $val; [ 735.769760] rc=$val; [ 735.769760] fi; [ 735.769760] exit $rc [ 736.472433] Lustre: DEBUG MARKER: dmesg [ 736.912620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7a: mkdir ...\/d7\; mcreate ...\/d7\/f\; chmod ...\/d7\/f ==================================== 16:13:58 \(1491408838\) [ 737.113281] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ==================================== 16:13:58 (1491408838) [ 737.326825] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 737.984728] Lustre: DEBUG MARKER: rc=0; [ 737.984728] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 737.984728] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 737.984728] echo $(hostname -s): $val; [ 737.984728] rc=$val; [ 737.984728] fi; [ 737.984728] exit $rc [ 738.681294] Lustre: DEBUG MARKER: dmesg [ 739.113810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 7b: mkdir ...\/d7\; mcreate d7\/f2\; echo foo \> d7\/f2 ===================================== 16:14:01 \(1491408841\) [ 739.330028] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 ===================================== 16:14:01 (1491408841) [ 739.545224] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 740.241453] Lustre: DEBUG MARKER: rc=0; [ 740.241453] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 740.241453] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 740.241453] echo $(hostname -s): $val; [ 740.241453] rc=$val; [ 740.241453] fi; [ 740.241453] exit $rc [ 740.953069] Lustre: DEBUG MARKER: dmesg [ 741.399670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 8: mkdir ...\/d8\; touch ...\/d8\/f\; chmod ...\/d8\/f ======================================= 16:14:03 \(1491408843\) [ 741.590956] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ======================================= 16:14:03 (1491408843) [ 741.832216] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 742.514539] Lustre: DEBUG MARKER: rc=0; [ 742.514539] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 742.514539] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 742.514539] echo $(hostname -s): $val; [ 742.514539] rc=$val; [ 742.514539] fi; [ 742.514539] exit $rc [ 743.252360] Lustre: DEBUG MARKER: dmesg [ 743.717888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 9: mkdir ...\/d9 ...\/d9\/d2 ...\/d9\/d2\/d3 ================================================ 16:14:05 \(1491408845\) [ 743.939725] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ================================================ 16:14:05 (1491408845) [ 744.158526] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 744.827555] Lustre: DEBUG MARKER: rc=0; [ 744.827555] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 744.827555] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 744.827555] echo $(hostname -s): $val; [ 744.827555] rc=$val; [ 744.827555] fi; [ 744.827555] exit $rc [ 745.506115] Lustre: DEBUG MARKER: dmesg [ 745.940153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 10: mkdir ...\/d10 ...\/d10\/d2\; touch ...\/d10\/d2\/f ====================================== 16:14:07 \(1491408847\) [ 746.142671] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ====================================== 16:14:07 (1491408847) [ 746.364306] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 747.042632] Lustre: DEBUG MARKER: rc=0; [ 747.042632] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 747.042632] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 747.042632] echo $(hostname -s): $val; [ 747.042632] rc=$val; [ 747.042632] fi; [ 747.042632] exit $rc [ 747.746056] Lustre: DEBUG MARKER: dmesg [ 748.167958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 11: mkdir ...\/d11 d11\/d2\; chmod ...\/d11\/d2 ============================================ 16:14:10 \(1491408850\) [ 748.363151] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ============================================ 16:14:10 (1491408850) [ 748.720291] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 749.377422] Lustre: DEBUG MARKER: rc=0; [ 749.377422] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 749.377422] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 749.377422] echo $(hostname -s): $val; [ 749.377422] rc=$val; [ 749.377422] fi; [ 749.377422] exit $rc [ 750.073880] Lustre: DEBUG MARKER: dmesg [ 750.532088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 12: touch ...\/d12\/f\; chmod ...\/d12\/f ...\/d12\/f ======================================== 16:14:12 \(1491408852\) [ 750.721102] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ======================================== 16:14:12 (1491408852) [ 750.943561] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 751.612421] Lustre: DEBUG MARKER: rc=0; [ 751.612421] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 751.612421] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 751.612421] echo $(hostname -s): $val; [ 751.612421] rc=$val; [ 751.612421] fi; [ 751.612421] exit $rc [ 752.301564] Lustre: DEBUG MARKER: dmesg [ 752.731710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 13: creat ...\/d13\/f\; dd ...\/d13\/f\; \> ...\/d13\/f ======================================== 16:14:14 \(1491408854\) [ 752.938400] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ======================================== 16:14:14 (1491408854) [ 753.156553] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 753.815266] Lustre: DEBUG MARKER: rc=0; [ 753.815266] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 753.815266] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 753.815266] echo $(hostname -s): $val; [ 753.815266] rc=$val; [ 753.815266] fi; [ 753.815266] exit $rc [ 754.526111] Lustre: DEBUG MARKER: dmesg [ 754.958897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 14: touch ...\/d14\/f\; rm ...\/d14\/f\; rm ...\/d14\/f ======================================= 16:14:16 \(1491408856\) [ 755.150258] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ======================================= 16:14:16 (1491408856) [ 755.366765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 756.045600] Lustre: DEBUG MARKER: rc=0; [ 756.045600] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 756.045600] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 756.045600] echo $(hostname -s): $val; [ 756.045600] rc=$val; [ 756.045600] fi; [ 756.045600] exit $rc [ 756.737204] Lustre: DEBUG MARKER: dmesg [ 757.157523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 15: touch ...\/d15\/f\; mv ...\/d15\/f ...\/d15\/f2 ========================================== 16:14:19 \(1491408859\) [ 757.353666] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ========================================== 16:14:19 (1491408859) [ 757.575823] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 758.247179] Lustre: DEBUG MARKER: rc=0; [ 758.247179] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 758.247179] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 758.247179] echo $(hostname -s): $val; [ 758.247179] rc=$val; [ 758.247179] fi; [ 758.247179] exit $rc [ 758.941789] Lustre: DEBUG MARKER: dmesg [ 759.371371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 16: touch ...\/d16\/f\; rm -rf ...\/d16\/f ================================================= 16:14:21 \(1491408861\) [ 759.558399] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ================================================= 16:14:21 (1491408861) [ 759.773970] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 760.445219] Lustre: DEBUG MARKER: rc=0; [ 760.445219] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 760.445219] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 760.445219] echo $(hostname -s): $val; [ 760.445219] rc=$val; [ 760.445219] fi; [ 760.445219] exit $rc [ 761.131275] Lustre: DEBUG MARKER: dmesg [ 761.571543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17a: symlinks: create, remove \(real\) ================================================== 16:14:23 \(1491408863\) [ 761.756441] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ================================================== 16:14:23 (1491408863) [ 761.986924] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 762.634065] Lustre: DEBUG MARKER: rc=0; [ 762.634065] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 762.634065] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 762.634065] echo $(hostname -s): $val; [ 762.634065] rc=$val; [ 762.634065] fi; [ 762.634065] exit $rc [ 763.331145] Lustre: DEBUG MARKER: dmesg [ 763.757449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17b: symlinks: create, remove \(dangling\) ============================================== 16:14:25 \(1491408865\) [ 763.943451] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) ============================================== 16:14:25 (1491408865) [ 764.170255] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 764.843694] Lustre: DEBUG MARKER: rc=0; [ 764.843694] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 764.843694] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 764.843694] echo $(hostname -s): $val; [ 764.843694] rc=$val; [ 764.843694] fi; [ 764.843694] exit $rc [ 765.535080] Lustre: DEBUG MARKER: dmesg [ 765.975566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17c: symlinks: open dangling \(should return error\) ==================================== 16:14:28 \(1491408868\) [ 766.173034] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ==================================== 16:14:28 (1491408868) [ 766.405476] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 767.061583] Lustre: DEBUG MARKER: rc=0; [ 767.061583] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 767.061583] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 767.061583] echo $(hostname -s): $val; [ 767.061583] rc=$val; [ 767.061583] fi; [ 767.061583] exit $rc [ 767.751119] Lustre: DEBUG MARKER: dmesg [ 768.206399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17d: symlinks: create dangling ======================================================== 16:14:30 \(1491408870\) [ 768.406225] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ======================================================== 16:14:30 (1491408870) [ 768.629378] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 769.297569] Lustre: DEBUG MARKER: rc=0; [ 769.297569] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 769.297569] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 769.297569] echo $(hostname -s): $val; [ 769.297569] rc=$val; [ 769.297569] fi; [ 769.297569] exit $rc [ 769.983367] Lustre: DEBUG MARKER: dmesg [ 770.424523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17e: symlinks: create recursive symlink \(should return error\) ========================= 16:14:32 \(1491408872\) [ 770.619927] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================= 16:14:32 (1491408872) [ 770.841136] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 771.515844] Lustre: DEBUG MARKER: rc=0; [ 771.515844] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 771.515844] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 771.515844] echo $(hostname -s): $val; [ 771.515844] rc=$val; [ 771.515844] fi; [ 771.515844] exit $rc [ 772.200156] Lustre: DEBUG MARKER: dmesg [ 772.622323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17f: symlinks: long and very long symlink name ======================================== 16:14:34 \(1491408874\) [ 772.806540] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ======================================== 16:14:34 (1491408874) [ 773.035675] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 773.686959] Lustre: DEBUG MARKER: rc=0; [ 773.686959] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 773.686959] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 773.686959] echo $(hostname -s): $val; [ 773.686959] rc=$val; [ 773.686959] fi; [ 773.686959] exit $rc [ 774.375585] Lustre: DEBUG MARKER: dmesg [ 774.796721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 16:14:36 \(1491408876\) [ 774.988173] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================== 16:14:36 (1491408876) [ 775.165379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 775.463964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 775.771308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 776.331897] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 777.004605] Lustre: DEBUG MARKER: rc=0; [ 777.004605] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 777.004605] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 777.004605] echo $(hostname -s): $val; [ 777.004605] rc=$val; [ 777.004605] fi; [ 777.004605] exit $rc [ 777.701594] Lustre: DEBUG MARKER: dmesg [ 778.138198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17h: create objects: lov_free_memmd\(\) doesn\'t lbug ==================================== 16:14:40 \(1491408880\) [ 778.338198] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ==================================== 16:14:40 (1491408880) [ 778.532694] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000141 [ 778.864532] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 779.531429] Lustre: DEBUG MARKER: rc=0; [ 779.531429] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 779.531429] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 779.531429] echo $(hostname -s): $val; [ 779.531429] rc=$val; [ 779.531429] fi; [ 779.531429] exit $rc [ 780.222125] Lustre: DEBUG MARKER: dmesg [ 780.640255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17i: don\'t panic on short symlink ===================================================== 16:14:42 \(1491408882\) [ 780.838736] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink ===================================================== 16:14:42 (1491408882) [ 781.028427] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000143 [ 781.177139] Lustre: *** cfs_fail_loc=143, val=0*** [ 781.360594] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 782.038972] Lustre: DEBUG MARKER: rc=0; [ 782.038972] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 782.038972] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 782.038972] echo $(hostname -s): $val; [ 782.038972] rc=$val; [ 782.038972] fi; [ 782.038972] exit $rc [ 782.720443] Lustre: DEBUG MARKER: dmesg [ 783.152815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 16:14:45 \(1491408885\) [ 783.353377] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled ============================================== 16:14:45 (1491408885) [ 783.755846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 784.441244] Lustre: DEBUG MARKER: rc=0; [ 784.441244] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 784.441244] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 784.441244] echo $(hostname -s): $val; [ 784.441244] rc=$val; [ 784.441244] fi; [ 784.441244] exit $rc [ 785.133635] Lustre: DEBUG MARKER: dmesg [ 785.568712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17l: Ensure lgetxattr\'s returned xattr size is consistent ============================= 16:14:47 \(1491408887\) [ 785.761128] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ============================= 16:14:47 (1491408887) [ 786.060750] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 786.747268] Lustre: DEBUG MARKER: rc=0; [ 786.747268] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 786.747268] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 786.747268] echo $(hostname -s): $val; [ 786.747268] rc=$val; [ 786.747268] fi; [ 786.747268] exit $rc [ 787.438796] Lustre: DEBUG MARKER: dmesg [ 787.873020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17m: run e2fsck against MDT which contains short\/long symlink ========================= 16:14:49 \(1491408889\) [ 788.058163] Lustre: DEBUG MARKER: == sanity test 17m: run e2fsck against MDT which contains short/long symlink ========================= 16:14:49 (1491408889) [ 788.241705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 788.539365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 788.881356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_17m only for ldiskfs MDT [ 789.067459] Lustre: DEBUG MARKER: SKIP: sanity test_17m only for ldiskfs MDT [ 789.276830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 789.944122] Lustre: DEBUG MARKER: rc=0; [ 789.944122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 789.944122] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 789.944122] echo $(hostname -s): $val; [ 789.944122] rc=$val; [ 789.944122] fi; [ 789.944122] exit $rc [ 790.629105] Lustre: DEBUG MARKER: dmesg [ 791.073881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17n: run e2fsck against master\/slave MDT which contains remote dir ==================== 16:14:53 \(1491408893\) [ 791.274171] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ==================== 16:14:53 (1491408893) [ 791.460111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 791.760975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 792.094856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_17n only for ldiskfs MDT [ 792.282186] Lustre: DEBUG MARKER: SKIP: sanity test_17n only for ldiskfs MDT [ 792.494609] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 793.154992] Lustre: DEBUG MARKER: rc=0; [ 793.154992] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 793.154992] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 793.154992] echo $(hostname -s): $val; [ 793.154992] rc=$val; [ 793.154992] fi; [ 793.154992] exit $rc [ 793.838200] Lustre: DEBUG MARKER: dmesg [ 794.288183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 17o: stat file with incompat LMA feature ============================================== 16:14:56 \(1491408896\) [ 794.464325] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature ============================================== 16:14:56 (1491408896) [ 794.642439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 794.963705] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 795.252951] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 795.404458] Lustre: Failing over lustre-MDT0000 [ 796.527804] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.016299] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.016300] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.552053] Lustre: 29676:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491408897/real 1491408897] req@ffff88005f5d5200 x1563855295654480/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491408903 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 801.558518] Lustre: 29676:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 801.573622] Lustre: server umount lustre-MDT0000 complete [ 801.721475] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 802.024492] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 802.024492] grep -q ^lustre-mdt1/ /proc/mounts || [ 802.024492] zpool export lustre-mdt1 [ 812.465050] Lustre: DEBUG MARKER: hostname [ 812.782701] Lustre: DEBUG MARKER: modprobe zfs; [ 812.782701] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 812.782701] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 813.352302] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 813.640870] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 813.826023] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 813.828452] Lustre: Skipped 8 previous similar messages [ 814.070680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.233875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 814.526017] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 815.051731] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 815.342692] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 815.639225] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 815.973361] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 818.530028] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 823.284317] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 823.799345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 823.980307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 828.387114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 828.574344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 829.873849] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x194 [ 830.029746] Lustre: *** cfs_fail_loc=194, val=0*** [ 830.032065] Lustre: 31806:0:(osd_object.c:381:osd_check_lma()) lustre-MDT0000: unsupported incompat LMA feature(s) 0x0 for fid = [0x200000bd0:0xbd:0x0] [ 830.178740] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 830.522993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 831.177135] Lustre: DEBUG MARKER: rc=0; [ 831.177135] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 831.177135] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 831.177135] echo $(hostname -s): $val; [ 831.177135] rc=$val; [ 831.177135] fi; [ 831.177135] exit $rc [ 831.866086] Lustre: DEBUG MARKER: dmesg [ 832.295178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 18: touch ...\/f \; ls ... ============================================================== 16:15:34 \(1491408934\) [ 832.483792] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ============================================================== 16:15:34 (1491408934) [ 832.705529] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 833.377320] Lustre: DEBUG MARKER: rc=0; [ 833.377320] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 833.377320] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 833.377320] echo $(hostname -s): $val; [ 833.377320] rc=$val; [ 833.377320] fi; [ 833.377320] exit $rc [ 834.071742] Lustre: DEBUG MARKER: dmesg [ 834.495091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19a: touch ...\/f19 \; ls -l ... \; rm ...\/f19 =========================================== 16:15:36 \(1491408936\) [ 834.687109] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 =========================================== 16:15:36 (1491408936) [ 834.935804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 835.621246] Lustre: DEBUG MARKER: rc=0; [ 835.621246] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 835.621246] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 835.621246] echo $(hostname -s): $val; [ 835.621246] rc=$val; [ 835.621246] fi; [ 835.621246] exit $rc [ 836.329206] Lustre: DEBUG MARKER: dmesg [ 836.772599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19b: ls -l ...\/f19 \(should return error\) ============================================== 16:15:38 \(1491408938\) [ 836.959084] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ============================================== 16:15:38 (1491408938) [ 837.169130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 837.843828] Lustre: DEBUG MARKER: rc=0; [ 837.843828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 837.843828] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 837.843828] echo $(hostname -s): $val; [ 837.843828] rc=$val; [ 837.843828] fi; [ 837.843828] exit $rc [ 838.549842] Lustre: DEBUG MARKER: dmesg [ 838.997410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19c: runas -u 500 -g 500 touch ...\/f19 \(should return error\) ========================== 16:15:41 \(1491408941\) [ 839.189371] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ========================== 16:15:41 (1491408941) [ 839.406573] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 840.066521] Lustre: DEBUG MARKER: rc=0; [ 840.066521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 840.066521] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 840.066521] echo $(hostname -s): $val; [ 840.066521] rc=$val; [ 840.066521] fi; [ 840.066521] exit $rc [ 840.770695] Lustre: DEBUG MARKER: dmesg [ 841.235449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 19d: cat ...\/f19 \(should return error\) ================================================ 16:15:43 \(1491408943\) [ 841.427578] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ================================================ 16:15:43 (1491408943) [ 841.652206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 842.334799] Lustre: DEBUG MARKER: rc=0; [ 842.334799] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 842.334799] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 842.334799] echo $(hostname -s): $val; [ 842.334799] rc=$val; [ 842.334799] fi; [ 842.334799] exit $rc [ 843.037190] Lustre: DEBUG MARKER: dmesg [ 843.452744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 20: touch ...\/f \; ls -l ... =========================================================== 16:15:45 \(1491408945\) [ 843.654173] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =========================================================== 16:15:45 (1491408945) [ 843.875315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 done [ 844.066631] Lustre: DEBUG MARKER: 1 done [ 844.279114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 done [ 844.478095] Lustre: DEBUG MARKER: 2 done [ 844.706295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 done [ 844.899395] Lustre: DEBUG MARKER: 3 done [ 845.121619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 845.800526] Lustre: DEBUG MARKER: rc=0; [ 845.800526] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 845.800526] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 845.800526] echo $(hostname -s): $val; [ 845.800526] rc=$val; [ 845.800526] fi; [ 845.800526] exit $rc [ 846.499686] Lustre: DEBUG MARKER: dmesg [ 846.925863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 21: write to dangling link ============================================================ 16:15:48 \(1491408948\) [ 847.113362] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ============================================================ 16:15:48 (1491408948) [ 847.336675] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 847.996127] Lustre: DEBUG MARKER: rc=0; [ 847.996127] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 847.996127] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 847.996127] echo $(hostname -s): $val; [ 847.996127] rc=$val; [ 847.996127] fi; [ 847.996127] exit $rc [ 848.685176] Lustre: DEBUG MARKER: dmesg [ 849.111963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 22: unpack tar archive as non-root user =============================================== 16:15:51 \(1491408951\) [ 849.307709] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user =============================================== 16:15:51 (1491408951) [ 849.659030] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 850.320970] Lustre: DEBUG MARKER: rc=0; [ 850.320970] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 850.320970] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 850.320970] echo $(hostname -s): $val; [ 850.320970] rc=$val; [ 850.320970] fi; [ 850.320970] exit $rc [ 851.015257] Lustre: DEBUG MARKER: dmesg [ 851.461099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23a: O_CREAT\|O_EXCL in subdir ========================================================= 16:15:53 \(1491408953\) [ 851.639715] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ========================================================= 16:15:53 (1491408953) [ 851.850417] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 852.506283] Lustre: DEBUG MARKER: rc=0; [ 852.506283] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 852.506283] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 852.506283] echo $(hostname -s): $val; [ 852.506283] rc=$val; [ 852.506283] fi; [ 852.506283] exit $rc [ 853.192588] Lustre: DEBUG MARKER: dmesg [ 853.630158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 23b: O_APPEND check =================================================================== 16:15:55 \(1491408955\) [ 853.818190] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check =================================================================== 16:15:55 (1491408955) [ 854.047575] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 854.711859] Lustre: DEBUG MARKER: rc=0; [ 854.711859] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 854.711859] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 854.711859] echo $(hostname -s): $val; [ 854.711859] rc=$val; [ 854.711859] fi; [ 854.711859] exit $rc [ 855.394652] Lustre: DEBUG MARKER: dmesg [ 855.835402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24a: rename file to non-existent target =============================================== 16:15:57 \(1491408957\) [ 856.028999] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target =============================================== 16:15:57 (1491408957) [ 856.251919] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 856.920687] Lustre: DEBUG MARKER: rc=0; [ 856.920687] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 856.920687] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 856.920687] echo $(hostname -s): $val; [ 856.920687] rc=$val; [ 856.920687] fi; [ 856.920687] exit $rc [ 857.604848] Lustre: DEBUG MARKER: dmesg [ 858.037420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24b: rename file to existing target =================================================== 16:16:00 \(1491408960\) [ 858.229156] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target =================================================== 16:16:00 (1491408960) [ 858.463465] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 859.144088] Lustre: DEBUG MARKER: rc=0; [ 859.144088] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 859.144088] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 859.144088] echo $(hostname -s): $val; [ 859.144088] rc=$val; [ 859.144088] fi; [ 859.144088] exit $rc [ 859.825266] Lustre: DEBUG MARKER: dmesg [ 860.290429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24c: rename directory to non-existent target ========================================== 16:16:02 \(1491408962\) [ 860.481814] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================== 16:16:02 (1491408962) [ 860.696185] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 861.356397] Lustre: DEBUG MARKER: rc=0; [ 861.356397] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 861.356397] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 861.356397] echo $(hostname -s): $val; [ 861.356397] rc=$val; [ 861.356397] fi; [ 861.356397] exit $rc [ 862.034364] Lustre: DEBUG MARKER: dmesg [ 862.470088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24d: rename directory to existing target ============================================== 16:16:04 \(1491408964\) [ 862.658936] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target ============================================== 16:16:04 (1491408964) [ 862.925663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 863.602790] Lustre: DEBUG MARKER: rc=0; [ 863.602790] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 863.602790] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 863.602790] echo $(hostname -s): $val; [ 863.602790] rc=$val; [ 863.602790] fi; [ 863.602790] exit $rc [ 864.286942] Lustre: DEBUG MARKER: dmesg [ 864.723585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24e: touch ...\/R5a\/f\; rename ...\/R5a\/f ...\/R5b\/g ====================================== 16:16:06 \(1491408966\) [ 864.918325] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ====================================== 16:16:06 (1491408966) [ 865.150965] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 865.846973] Lustre: DEBUG MARKER: rc=0; [ 865.846973] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 865.846973] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 865.846973] echo $(hostname -s): $val; [ 865.846973] rc=$val; [ 865.846973] fi; [ 865.846973] exit $rc [ 866.546422] Lustre: DEBUG MARKER: dmesg [ 866.999713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24f: touch ...\/R6a\/f R6b\/g\; mv ...\/R6a\/f ...\/R6b\/g ==================================== 16:16:09 \(1491408969\) [ 867.189326] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ==================================== 16:16:09 (1491408969) [ 867.426324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 868.102621] Lustre: DEBUG MARKER: rc=0; [ 868.102621] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 868.102621] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 868.102621] echo $(hostname -s): $val; [ 868.102621] rc=$val; [ 868.102621] fi; [ 868.102621] exit $rc [ 868.792260] Lustre: DEBUG MARKER: dmesg [ 869.224877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24g: mkdir ...\/R7{a,b}\/d\; mv ...\/R7a\/d ...\/R7b\/e ====================================== 16:16:11 \(1491408971\) [ 869.414733] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ====================================== 16:16:11 (1491408971) [ 869.638908] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 870.304866] Lustre: DEBUG MARKER: rc=0; [ 870.304866] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 870.304866] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 870.304866] echo $(hostname -s): $val; [ 870.304866] rc=$val; [ 870.304866] fi; [ 870.304866] exit $rc [ 870.994834] Lustre: DEBUG MARKER: dmesg [ 871.431806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24h: mkdir ...\/R8{a,b}\/{d,e}\; rename ...\/R8a\/d ...\/R8b\/e ============================== 16:16:13 \(1491408973\) [ 871.623771] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ============================== 16:16:13 (1491408973) [ 871.848992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 872.521868] Lustre: DEBUG MARKER: rc=0; [ 872.521868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 872.521868] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 872.521868] echo $(hostname -s): $val; [ 872.521868] rc=$val; [ 872.521868] fi; [ 872.521868] exit $rc [ 873.202698] Lustre: DEBUG MARKER: dmesg [ 873.654247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24i: rename file to dir error: touch f \; mkdir a \; rename f a ========================= 16:16:15 \(1491408975\) [ 873.844831] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================= 16:16:15 (1491408975) [ 874.065019] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 874.753379] Lustre: DEBUG MARKER: rc=0; [ 874.753379] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 874.753379] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 874.753379] echo $(hostname -s): $val; [ 874.753379] rc=$val; [ 874.753379] fi; [ 874.753379] exit $rc [ 875.434724] Lustre: DEBUG MARKER: dmesg [ 875.861030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24j: source does not exist ============================================================ 16:16:17 \(1491408977\) [ 876.049761] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ============================================================ 16:16:17 (1491408977) [ 876.401535] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 877.512675] Lustre: DEBUG MARKER: rc=0; [ 877.512675] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 877.512675] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 877.512675] echo $(hostname -s): $val; [ 877.512675] rc=$val; [ 877.512675] fi; [ 877.512675] exit $rc [ 878.304549] Lustre: DEBUG MARKER: dmesg [ 878.755957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24k: touch ...\/R11a\/f\; mv ...\/R11a\/f ...\/R11a\/d ======================================= 16:16:20 \(1491408980\) [ 878.957690] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ======================================= 16:16:20 (1491408980) [ 879.198546] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 879.919319] Lustre: DEBUG MARKER: rc=0; [ 879.919319] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 879.919319] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 879.919319] echo $(hostname -s): $val; [ 879.919319] rc=$val; [ 879.919319] fi; [ 879.919319] exit $rc [ 880.639682] Lustre: DEBUG MARKER: dmesg [ 881.097669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24l: Renaming a file to itself ======================================================== 16:16:23 \(1491408983\) [ 881.293946] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ======================================================== 16:16:23 (1491408983) [ 881.541944] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 882.224547] Lustre: DEBUG MARKER: rc=0; [ 882.224547] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 882.224547] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 882.224547] echo $(hostname -s): $val; [ 882.224547] rc=$val; [ 882.224547] fi; [ 882.224547] exit $rc [ 882.982658] Lustre: DEBUG MARKER: dmesg [ 883.388744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24m: Renaming a file to a hard link to itself ========================================= 16:16:25 \(1491408985\) [ 883.579659] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself ========================================= 16:16:25 (1491408985) [ 883.801322] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 884.480945] Lustre: DEBUG MARKER: rc=0; [ 884.480945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 884.480945] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 884.480945] echo $(hostname -s): $val; [ 884.480945] rc=$val; [ 884.480945] fi; [ 884.480945] exit $rc [ 885.178299] Lustre: DEBUG MARKER: dmesg [ 885.627988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24n: Statting the old file after renaming \(Posix rename 2\) ============================ 16:16:27 \(1491408987\) [ 885.824405] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ============================ 16:16:27 (1491408987) [ 886.047587] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 886.730350] Lustre: DEBUG MARKER: rc=0; [ 886.730350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 886.730350] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 886.730350] echo $(hostname -s): $val; [ 886.730350] rc=$val; [ 886.730350] fi; [ 886.730350] exit $rc [ 887.436281] Lustre: DEBUG MARKER: dmesg [ 887.897233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24o: rename of files during htree split =============================================== 16:16:29 \(1491408989\) [ 888.107577] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split =============================================== 16:16:29 (1491408989) [ 932.160892] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 932.840806] Lustre: DEBUG MARKER: rc=0; [ 932.840806] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 932.840806] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 932.840806] echo $(hostname -s): $val; [ 932.840806] rc=$val; [ 932.840806] fi; [ 932.840806] exit $rc [ 933.564283] Lustre: DEBUG MARKER: dmesg [ 933.997840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24p: mkdir ...\/R12{a,b}\; rename ...\/R12a ...\/R12b ===================================== 16:17:16 \(1491409036\) [ 934.188601] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ===================================== 16:17:16 (1491409036) [ 934.445491] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 935.117553] Lustre: DEBUG MARKER: rc=0; [ 935.117553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 935.117553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 935.117553] echo $(hostname -s): $val; [ 935.117553] rc=$val; [ 935.117553] fi; [ 935.117553] exit $rc [ 935.808532] Lustre: DEBUG MARKER: dmesg [ 936.262620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24q: mkdir ...\/R13{a,b}\; open R13b rename R13a R13b =================================== 16:17:18 \(1491409038\) [ 936.450238] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b =================================== 16:17:18 (1491409038) [ 936.701642] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 937.372227] Lustre: DEBUG MARKER: rc=0; [ 937.372227] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 937.372227] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 937.372227] echo $(hostname -s): $val; [ 937.372227] rc=$val; [ 937.372227] fi; [ 937.372227] exit $rc [ 938.134463] Lustre: DEBUG MARKER: dmesg [ 938.560311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24r: mkdir ...\/R14a\/b\; rename ...\/R14a ...\/R14a\/b ===================================== 16:17:20 \(1491409040\) [ 938.763020] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b ===================================== 16:17:20 (1491409040) [ 938.987500] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 939.672428] Lustre: DEBUG MARKER: rc=0; [ 939.672428] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 939.672428] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 939.672428] echo $(hostname -s): $val; [ 939.672428] rc=$val; [ 939.672428] fi; [ 939.672428] exit $rc [ 940.380175] Lustre: DEBUG MARKER: dmesg [ 940.805793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24s: mkdir ...\/R15a\/b\/c\; rename ...\/R15a ...\/R15a\/b\/c ================================= 16:17:22 \(1491409042\) [ 941.020941] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c ================================= 16:17:22 (1491409042) [ 941.240268] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 941.911131] Lustre: DEBUG MARKER: rc=0; [ 941.911131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 941.911131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 941.911131] echo $(hostname -s): $val; [ 941.911131] rc=$val; [ 941.911131] fi; [ 941.911131] exit $rc [ 942.602162] Lustre: DEBUG MARKER: dmesg [ 943.043488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24t: mkdir ...\/R16a\/b\/c\; rename ...\/R16a\/b\/c ...\/R16a ================================= 16:17:25 \(1491409045\) [ 943.233149] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a ================================= 16:17:25 (1491409045) [ 943.456078] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 944.125015] Lustre: DEBUG MARKER: rc=0; [ 944.125015] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 944.125015] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 944.125015] echo $(hostname -s): $val; [ 944.125015] rc=$val; [ 944.125015] fi; [ 944.125015] exit $rc [ 944.823820] Lustre: DEBUG MARKER: dmesg [ 945.260700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24u: create stripe file =============================================================== 16:17:27 \(1491409047\) [ 945.450466] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =============================================================== 16:17:27 (1491409047) [ 945.733382] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 946.400192] Lustre: DEBUG MARKER: rc=0; [ 946.400192] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 946.400192] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 946.400192] echo $(hostname -s): $val; [ 946.400192] rc=$val; [ 946.400192] fi; [ 946.400192] exit $rc [ 947.097049] Lustre: DEBUG MARKER: dmesg [ 947.528867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24v: list directory with large files \(handle hash collision, bug: 17560\) ============== 16:17:29 \(1491409049\) [ 947.720663] Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) ============== 16:17:29 (1491409049) [ 970.511626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 976.878850] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 977.554244] Lustre: DEBUG MARKER: rc=0; [ 977.554244] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 977.554244] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 977.554244] echo $(hostname -s): $val; [ 977.554244] rc=$val; [ 977.554244] fi; [ 977.554244] exit $rc [ 978.264152] Lustre: DEBUG MARKER: dmesg [ 978.710501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24w: Reading a file larger than 4Gb =================================================== 16:18:00 \(1491409080\) [ 978.905596] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb =================================================== 16:18:00 (1491409080) [ 979.214700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 979.907492] Lustre: DEBUG MARKER: rc=0; [ 979.907492] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 979.907492] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 979.907492] echo $(hostname -s): $val; [ 979.907492] rc=$val; [ 979.907492] fi; [ 979.907492] exit $rc [ 980.637183] Lustre: DEBUG MARKER: dmesg [ 981.092883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24x: cross MDT rename\/link ============================================================ 16:18:03 \(1491409083\) [ 981.280470] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ============================================================ 16:18:03 (1491409083) [ 981.487372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24x needs \>= 2 MDTs [ 981.679166] Lustre: DEBUG MARKER: SKIP: sanity test_24x needs >= 2 MDTs [ 981.887532] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 982.580491] Lustre: DEBUG MARKER: rc=0; [ 982.580491] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 982.580491] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 982.580491] echo $(hostname -s): $val; [ 982.580491] rc=$val; [ 982.580491] fi; [ 982.580491] exit $rc [ 983.298536] Lustre: DEBUG MARKER: dmesg [ 983.745036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24y: rename\/link on the same dir should succeed ======================================= 16:18:05 \(1491409085\) [ 983.936728] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ======================================= 16:18:05 (1491409085) [ 984.194489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24y needs \>= 2 MDTs [ 984.364733] Lustre: DEBUG MARKER: SKIP: sanity test_24y needs >= 2 MDTs [ 984.657499] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 985.329380] Lustre: DEBUG MARKER: rc=0; [ 985.329380] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 985.329380] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 985.329380] echo $(hostname -s): $val; [ 985.329380] rc=$val; [ 985.329380] fi; [ 985.329380] exit $rc [ 986.029349] Lustre: DEBUG MARKER: dmesg [ 986.477402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24A: readdir\(\) returns correct number of entries. ===================================== 16:18:08 \(1491409088\) [ 986.683713] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ===================================== 16:18:08 (1491409088) [ 998.037308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 998.701247] Lustre: DEBUG MARKER: rc=0; [ 998.701247] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 998.701247] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 998.701247] echo $(hostname -s): $val; [ 998.701247] rc=$val; [ 998.701247] fi; [ 998.701247] exit $rc [ 999.412997] Lustre: DEBUG MARKER: dmesg [ 999.866519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24B: readdir for striped dir return correct number of entries ========================= 16:18:21 \(1491409101\) [ 1000.058230] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================= 16:18:21 (1491409101) [ 1000.267248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24B needs \>= 2 MDTs [ 1000.464080] Lustre: DEBUG MARKER: SKIP: sanity test_24B needs >= 2 MDTs [ 1000.696068] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1001.379382] Lustre: DEBUG MARKER: rc=0; [ 1001.379382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1001.379382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1001.379382] echo $(hostname -s): $val; [ 1001.379382] rc=$val; [ 1001.379382] fi; [ 1001.379382] exit $rc [ 1002.071169] Lustre: DEBUG MARKER: dmesg [ 1002.520600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24C: check .. in striped dir ========================================================== 16:18:24 \(1491409104\) [ 1002.713716] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ========================================================== 16:18:24 (1491409104) [ 1002.929086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24C needs \>= 2 MDTs [ 1003.132449] Lustre: DEBUG MARKER: SKIP: sanity test_24C needs >= 2 MDTs [ 1003.352159] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1004.045488] Lustre: DEBUG MARKER: rc=0; [ 1004.045488] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1004.045488] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1004.045488] echo $(hostname -s): $val; [ 1004.045488] rc=$val; [ 1004.045488] fi; [ 1004.045488] exit $rc [ 1004.743914] Lustre: DEBUG MARKER: dmesg [ 1005.181173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24D skipping SLOW test 24D [ 1005.378682] Lustre: DEBUG MARKER: SKIP: sanity test_24D skipping SLOW test 24D [ 1005.606587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 24E: cross MDT rename\/link ============================================================ 16:18:27 \(1491409107\) [ 1005.821131] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ============================================================ 16:18:27 (1491409107) [ 1006.046357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_24E needs \>= 4 MDTs [ 1006.243248] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 1006.458723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1007.137418] Lustre: DEBUG MARKER: rc=0; [ 1007.137418] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1007.137418] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1007.137418] echo $(hostname -s): $val; [ 1007.137418] rc=$val; [ 1007.137418] fi; [ 1007.137418] exit $rc [ 1007.846374] Lustre: DEBUG MARKER: dmesg [ 1008.297489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25a: create file in symlinked directory =============================================== 16:18:30 \(1491409110\) [ 1008.476214] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory =============================================== 16:18:30 (1491409110) [ 1008.698417] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1009.382838] Lustre: DEBUG MARKER: rc=0; [ 1009.382838] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1009.382838] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1009.382838] echo $(hostname -s): $val; [ 1009.382838] rc=$val; [ 1009.382838] fi; [ 1009.382838] exit $rc [ 1010.088390] Lustre: DEBUG MARKER: dmesg [ 1010.542705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 25b: lookup file in symlinked directory =============================================== 16:18:32 \(1491409112\) [ 1010.743817] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory =============================================== 16:18:32 (1491409112) [ 1010.980731] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1011.663719] Lustre: DEBUG MARKER: rc=0; [ 1011.663719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1011.663719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1011.663719] echo $(hostname -s): $val; [ 1011.663719] rc=$val; [ 1011.663719] fi; [ 1011.663719] exit $rc [ 1012.369803] Lustre: DEBUG MARKER: dmesg [ 1012.810690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26a: multiple component symlink ======================================================= 16:18:34 \(1491409114\) [ 1013.007955] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ======================================================= 16:18:34 (1491409114) [ 1013.242744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1013.921254] Lustre: DEBUG MARKER: rc=0; [ 1013.921254] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1013.921254] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1013.921254] echo $(hostname -s): $val; [ 1013.921254] rc=$val; [ 1013.921254] fi; [ 1013.921254] exit $rc [ 1014.627847] Lustre: DEBUG MARKER: dmesg [ 1015.097628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26b: multiple component symlink at end of lookup ====================================== 16:18:37 \(1491409117\) [ 1015.297226] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ====================================== 16:18:37 (1491409117) [ 1015.533080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1016.215319] Lustre: DEBUG MARKER: rc=0; [ 1016.215319] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1016.215319] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1016.215319] echo $(hostname -s): $val; [ 1016.215319] rc=$val; [ 1016.215319] fi; [ 1016.215319] exit $rc [ 1016.936823] Lustre: DEBUG MARKER: dmesg [ 1017.388956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26c: chain of symlinks ================================================================ 16:18:39 \(1491409119\) [ 1017.583545] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ================================================================ 16:18:39 (1491409119) [ 1017.806990] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1018.486947] Lustre: DEBUG MARKER: rc=0; [ 1018.486947] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1018.486947] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1018.486947] echo $(hostname -s): $val; [ 1018.486947] rc=$val; [ 1018.486947] fi; [ 1018.486947] exit $rc [ 1019.191865] Lustre: DEBUG MARKER: dmesg [ 1019.631280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26d: create multiple component recursive symlink ====================================== 16:18:41 \(1491409121\) [ 1019.828527] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ====================================== 16:18:41 (1491409121) [ 1020.040646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1020.749466] Lustre: DEBUG MARKER: rc=0; [ 1020.749466] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1020.749466] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1020.749466] echo $(hostname -s): $val; [ 1020.749466] rc=$val; [ 1020.749466] fi; [ 1020.749466] exit $rc [ 1021.449729] Lustre: DEBUG MARKER: dmesg [ 1021.892635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26e: unlink multiple component recursive symlink ====================================== 16:18:43 \(1491409123\) [ 1022.091986] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ====================================== 16:18:43 (1491409123) [ 1022.335088] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1022.995352] Lustre: DEBUG MARKER: rc=0; [ 1022.995352] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1022.995352] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1022.995352] echo $(hostname -s): $val; [ 1022.995352] rc=$val; [ 1022.995352] fi; [ 1022.995352] exit $rc [ 1023.689266] Lustre: DEBUG MARKER: dmesg [ 1024.134047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 16:18:46 \(1491409126\) [ 1024.327541] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ================================= 16:18:46 (1491409126) [ 1024.603569] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1025.269763] Lustre: DEBUG MARKER: rc=0; [ 1025.269763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1025.269763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1025.269763] echo $(hostname -s): $val; [ 1025.269763] rc=$val; [ 1025.269763] fi; [ 1025.269763] exit $rc [ 1025.976462] Lustre: DEBUG MARKER: dmesg [ 1026.399383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27a: one stripe file ================================================================== 16:18:48 \(1491409128\) [ 1026.593781] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ================================================================== 16:18:48 (1491409128) [ 1026.831937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == test_27a: write to one stripe file ========================= [ 1027.038122] Lustre: DEBUG MARKER: == test_27a: write to one stripe file ========================= [ 1027.272326] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1027.939225] Lustre: DEBUG MARKER: rc=0; [ 1027.939225] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1027.939225] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1027.939225] echo $(hostname -s): $val; [ 1027.939225] rc=$val; [ 1027.939225] fi; [ 1027.939225] exit $rc [ 1028.637373] Lustre: DEBUG MARKER: dmesg [ 1029.103392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27b: create and write to two stripe file ============================================== 16:18:51 \(1491409131\) [ 1029.314677] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file ============================================== 16:18:51 (1491409131) [ 1029.554263] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1030.234832] Lustre: DEBUG MARKER: rc=0; [ 1030.234832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1030.234832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1030.234832] echo $(hostname -s): $val; [ 1030.234832] rc=$val; [ 1030.234832] fi; [ 1030.234832] exit $rc [ 1030.937237] Lustre: DEBUG MARKER: dmesg [ 1031.382083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27d: create file with default settings ================================================ 16:18:53 \(1491409133\) [ 1031.569680] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ================================================ 16:18:53 (1491409133) [ 1031.791145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1032.473171] Lustre: DEBUG MARKER: rc=0; [ 1032.473171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1032.473171] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1032.473171] echo $(hostname -s): $val; [ 1032.473171] rc=$val; [ 1032.473171] fi; [ 1032.473171] exit $rc [ 1033.193917] Lustre: DEBUG MARKER: dmesg [ 1033.638268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27e: setstripe existing file \(should return error\) ==================================== 16:18:55 \(1491409135\) [ 1033.834297] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ==================================== 16:18:55 (1491409135) [ 1034.001124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1034.365115] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1035.038073] Lustre: DEBUG MARKER: rc=0; [ 1035.038073] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1035.038073] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1035.038073] echo $(hostname -s): $val; [ 1035.038073] rc=$val; [ 1035.038073] fi; [ 1035.038073] exit $rc [ 1035.735871] Lustre: DEBUG MARKER: dmesg [ 1036.185776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27f: setstripe with bad stripe size \(should return error\) ============================= 16:18:58 \(1491409138\) [ 1036.393999] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ============================= 16:18:58 (1491409138) [ 1036.630549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1037.317394] Lustre: DEBUG MARKER: rc=0; [ 1037.317394] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1037.317394] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1037.317394] echo $(hostname -s): $val; [ 1037.317394] rc=$val; [ 1037.317394] fi; [ 1037.317394] exit $rc [ 1038.020224] Lustre: DEBUG MARKER: dmesg [ 1038.478974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27g: \/usr\/bin\/lfs getstripe with no objects =========================================== 16:19:00 \(1491409140\) [ 1038.681941] Lustre: DEBUG MARKER: == sanity test 27g: /usr/bin/lfs getstripe with no objects =========================================== 16:19:00 (1491409140) [ 1038.913324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1039.597681] Lustre: DEBUG MARKER: rc=0; [ 1039.597681] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1039.597681] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1039.597681] echo $(hostname -s): $val; [ 1039.597681] rc=$val; [ 1039.597681] fi; [ 1039.597681] exit $rc [ 1040.295422] Lustre: DEBUG MARKER: dmesg [ 1040.754393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27i: \/usr\/bin\/lfs getstripe with some objects ========================================= 16:19:02 \(1491409142\) [ 1040.950657] Lustre: DEBUG MARKER: == sanity test 27i: /usr/bin/lfs getstripe with some objects ========================================= 16:19:02 (1491409142) [ 1041.183723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1041.857386] Lustre: DEBUG MARKER: rc=0; [ 1041.857386] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1041.857386] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1041.857386] echo $(hostname -s): $val; [ 1041.857386] rc=$val; [ 1041.857386] fi; [ 1041.857386] exit $rc [ 1042.563526] Lustre: DEBUG MARKER: dmesg [ 1043.016964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27j: setstripe with bad stripe offset \(should return error\) =========================== 16:19:05 \(1491409145\) [ 1043.211636] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) =========================== 16:19:05 (1491409145) [ 1043.256358] LustreError: 31142:0:(lod_qos.c:1201:lod_alloc_specific()) Start index 2 not found in pool '' [ 1043.446600] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1044.122953] Lustre: DEBUG MARKER: rc=0; [ 1044.122953] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1044.122953] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1044.122953] echo $(hostname -s): $val; [ 1044.122953] rc=$val; [ 1044.122953] fi; [ 1044.122953] exit $rc [ 1044.826395] Lustre: DEBUG MARKER: dmesg [ 1045.272056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27k: limit i_blksize for broken user apps ============================================= 16:19:07 \(1491409147\) [ 1045.464235] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ============================================= 16:19:07 (1491409147) [ 1045.692613] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1046.391682] Lustre: DEBUG MARKER: rc=0; [ 1046.391682] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1046.391682] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1046.391682] echo $(hostname -s): $val; [ 1046.391682] rc=$val; [ 1046.391682] fi; [ 1046.391682] exit $rc [ 1047.092511] Lustre: DEBUG MARKER: dmesg [ 1047.540512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27l: check setstripe permissions \(should return error\) ================================ 16:19:09 \(1491409149\) [ 1047.729888] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ================================ 16:19:09 (1491409149) [ 1047.953956] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1048.618916] Lustre: DEBUG MARKER: rc=0; [ 1048.618916] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1048.618916] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1048.618916] echo $(hostname -s): $val; [ 1048.618916] rc=$val; [ 1048.618916] fi; [ 1048.618916] exit $rc [ 1049.314610] Lustre: DEBUG MARKER: dmesg [ 1049.749526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_27m skipping SLOW test 27m [ 1049.942730] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 1050.178275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27n: create file with some full OSTs ================================================== 16:19:12 \(1491409152\) [ 1050.369347] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ================================================== 16:19:12 (1491409152) [ 1051.009941] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1061.328324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1061.631974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1061.928698] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1062.222226] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1062.515762] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1062.819334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1063.119935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1063.420697] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1063.716889] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1064.472768] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1065.113374] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1075.834127] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1086.170295] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1086.862416] Lustre: DEBUG MARKER: rc=0; [ 1086.862416] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1086.862416] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1086.862416] echo $(hostname -s): $val; [ 1086.862416] rc=$val; [ 1086.862416] fi; [ 1086.862416] exit $rc [ 1087.589797] Lustre: DEBUG MARKER: dmesg [ 1088.034962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27o: create file with all full OSTs \(should error\) ==================================== 16:19:50 \(1491409190\) [ 1088.225596] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ==================================== 16:19:50 (1491409190) [ 1088.870118] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1099.199693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1099.505303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1099.804726] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1100.116893] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1100.410985] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1100.708984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1101.013120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1101.325217] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1101.634789] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1102.332677] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1102.997232] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1113.310112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1113.615481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1113.919442] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1114.223663] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1114.522677] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1114.828163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1115.135515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1115.461235] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1115.768627] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1116.286551] LustreError: 31143:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x6332:0x0]: have 0 want 1 [ 1116.435138] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1117.095915] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1127.800194] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1138.197342] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1138.898699] Lustre: DEBUG MARKER: rc=0; [ 1138.898699] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1138.898699] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1138.898699] echo $(hostname -s): $val; [ 1138.898699] rc=$val; [ 1138.898699] fi; [ 1138.898699] exit $rc [ 1139.630448] Lustre: DEBUG MARKER: dmesg [ 1140.114453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27p: append to a truncated file with some full OSTs =================================== 16:20:42 \(1491409242\) [ 1140.306857] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs =================================== 16:20:42 (1491409242) [ 1140.998805] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1151.332583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1151.636890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1151.935690] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1152.228610] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1152.521911] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1152.828381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1153.145468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1153.459443] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1153.750693] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1154.454155] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1155.113743] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1165.842756] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1176.183883] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1176.906143] Lustre: DEBUG MARKER: rc=0; [ 1176.906143] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1176.906143] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1176.906143] echo $(hostname -s): $val; [ 1176.906143] rc=$val; [ 1176.906143] fi; [ 1176.906143] exit $rc [ 1177.630930] Lustre: DEBUG MARKER: dmesg [ 1178.084422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27q: append to truncated file with all OSTs full \(should error\) ======================= 16:21:20 \(1491409280\) [ 1178.283797] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ======================= 16:21:20 (1491409280) [ 1178.937373] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1189.278030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1189.585181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1189.897889] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1190.199716] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1190.495125] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1190.799506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1191.104373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1191.408906] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1191.702268] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1192.418474] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1193.080567] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1203.402958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1203.712783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1204.013406] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1204.307069] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1204.601069] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1204.908395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1205.210713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1205.507985] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1205.806966] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1206.320269] LustreError: 31142:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x637a:0x0]: have 0 want 1 [ 1206.471506] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1207.129723] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1218.122447] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1228.466284] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1229.156632] Lustre: DEBUG MARKER: rc=0; [ 1229.156632] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1229.156632] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1229.156632] echo $(hostname -s): $val; [ 1229.156632] rc=$val; [ 1229.156632] fi; [ 1229.156632] exit $rc [ 1229.854617] Lustre: DEBUG MARKER: dmesg [ 1230.313457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27r: stripe file with some full OSTs \(shouldn\'t LBUG\) ================================= 16:22:12 \(1491409332\) [ 1230.546182] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) ================================= 16:22:12 (1491409332) [ 1231.658674] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1241.979927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1242.289258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1242.590477] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1242.903729] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1243.212948] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1243.526263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1243.847223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1244.160748] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1244.477844] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1245.185135] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1245.834962] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1256.584348] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1266.929909] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1267.628686] Lustre: DEBUG MARKER: rc=0; [ 1267.628686] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1267.628686] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1267.628686] echo $(hostname -s): $val; [ 1267.628686] rc=$val; [ 1267.628686] fi; [ 1267.628686] exit $rc [ 1268.335442] Lustre: DEBUG MARKER: dmesg [ 1268.791114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27s: lsm_xfersize overflow \(should error\) \(bug 10725\) ================================= 16:22:50 \(1491409370\) [ 1268.985355] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ================================= 16:22:50 (1491409370) [ 1269.217542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1269.896676] Lustre: DEBUG MARKER: rc=0; [ 1269.896676] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1269.896676] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1269.896676] echo $(hostname -s): $val; [ 1269.896676] rc=$val; [ 1269.896676] fi; [ 1269.896676] exit $rc [ 1270.599092] Lustre: DEBUG MARKER: dmesg [ 1271.059398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27t: check that utils parse path correctly ============================================ 16:22:53 \(1491409373\) [ 1271.264488] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ============================================ 16:22:53 (1491409373) [ 1271.490529] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1272.170579] Lustre: DEBUG MARKER: rc=0; [ 1272.170579] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1272.170579] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1272.170579] echo $(hostname -s): $val; [ 1272.170579] rc=$val; [ 1272.170579] fi; [ 1272.170579] exit $rc [ 1272.880793] Lustre: DEBUG MARKER: dmesg [ 1273.354482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27u: skip object creation on OSC w\/o objects ========================================== 16:22:55 \(1491409375\) [ 1273.559405] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================== 16:22:55 (1491409375) [ 1273.787695] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x139 [ 1273.969221] Lustre: *** cfs_fail_loc=139, val=0*** [ 1274.472422] Lustre: *** cfs_fail_loc=139, val=0*** [ 1274.474174] Lustre: Skipped 619 previous similar messages [ 1275.473682] Lustre: *** cfs_fail_loc=139, val=0*** [ 1275.475385] Lustre: Skipped 1211 previous similar messages [ 1275.763691] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 1278.650258] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1279.384062] Lustre: DEBUG MARKER: rc=0; [ 1279.384062] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1279.384062] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1279.384062] echo $(hostname -s): $val; [ 1279.384062] rc=$val; [ 1279.384062] fi; [ 1279.384062] exit $rc [ 1280.175492] Lustre: DEBUG MARKER: dmesg [ 1280.620814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27v: skip object creation on slow OST ================================================= 16:23:02 \(1491409382\) [ 1280.829993] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ================================================= 16:23:02 (1491409382) [ 1281.083806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1281.397775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1281.707341] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1282.015819] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1282.324757] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1282.637612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1282.950619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1283.254771] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1283.554301] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1284.076105] LustreError: 32618:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x6793:0x0]: have 0 want 1 [ 1284.229303] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1284.887419] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1295.202683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1295.507820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1295.839105] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1296.136330] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1296.430238] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_next_id [ 1296.736558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1297.049260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1297.354591] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1297.654189] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1298.168079] LustreError: 32618:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x6795:0x0]: have 0 want 1 [ 1298.322894] Lustre: DEBUG MARKER: lctl get_param osc.*OST*-osc-MDT0000.prealloc* [ 1298.984958] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1309.850463] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1330.991633] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1341.338646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1342.071742] Lustre: DEBUG MARKER: rc=0; [ 1342.071742] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1342.071742] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1342.071742] echo $(hostname -s): $val; [ 1342.071742] rc=$val; [ 1342.071742] fi; [ 1342.071742] exit $rc [ 1342.797744] Lustre: DEBUG MARKER: dmesg [ 1343.262643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27w: check \/usr\/bin\/lfs setstripe -S option =========================================== 16:24:05 \(1491409445\) [ 1343.458897] Lustre: DEBUG MARKER: == sanity test 27w: check /usr/bin/lfs setstripe -S option =========================================== 16:24:05 (1491409445) [ 1343.703906] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1344.434977] Lustre: DEBUG MARKER: rc=0; [ 1344.434977] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1344.434977] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1344.434977] echo $(hostname -s): $val; [ 1344.434977] rc=$val; [ 1344.434977] fi; [ 1344.434977] exit $rc [ 1345.158399] Lustre: DEBUG MARKER: dmesg [ 1345.621498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27wa: check \/usr\/bin\/lfs setstripe -c -i options ====================================== 16:24:07 \(1491409447\) [ 1345.818050] Lustre: DEBUG MARKER: == sanity test 27wa: check /usr/bin/lfs setstripe -c -i options ====================================== 16:24:07 (1491409447) [ 1346.098800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1346.802855] Lustre: DEBUG MARKER: rc=0; [ 1346.802855] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1346.802855] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1346.802855] echo $(hostname -s): $val; [ 1346.802855] rc=$val; [ 1346.802855] fi; [ 1346.802855] exit $rc [ 1347.525245] Lustre: DEBUG MARKER: dmesg [ 1347.959666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27x: create files while OST0 is degraded ============================================== 16:24:09 \(1491409449\) [ 1348.174274] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded ============================================== 16:24:09 (1491409449) [ 1348.726828] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1359.454134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1360.163320] Lustre: DEBUG MARKER: rc=0; [ 1360.163320] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1360.163320] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1360.163320] echo $(hostname -s): $val; [ 1360.163320] rc=$val; [ 1360.163320] fi; [ 1360.163320] exit $rc [ 1360.906053] Lustre: DEBUG MARKER: dmesg [ 1361.357398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 16:24:23 \(1491409463\) [ 1361.549150] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 16:24:23 (1491409463) [ 1361.772073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1362.070969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1362.370851] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1362.664511] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1362.965895] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 1363.260034] Lustre: DEBUG MARKER: lctl dl [ 1363.580137] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 1363.732140] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1364.287604] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1374.978269] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 1375.138576] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1375.150699] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1375.154737] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 1375.157178] Lustre: Skipped 8 previous similar messages [ 1375.312253] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 1385.674862] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1386.381208] Lustre: DEBUG MARKER: rc=0; [ 1386.381208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1386.381208] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1386.381208] echo $(hostname -s): $val; [ 1386.381208] rc=$val; [ 1386.381208] fi; [ 1386.381208] exit $rc [ 1387.111073] Lustre: DEBUG MARKER: dmesg [ 1387.572826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27z: check SEQ\/OID on the MDT and OST filesystems ===================================== 16:24:49 \(1491409489\) [ 1387.778052] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ===================================== 16:24:49 (1491409489) [ 1393.212932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-1 [ 1393.394675] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 1393.623992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000bd0, oid 0x67c4 ver 0x0 [ 1393.820949] Lustre: DEBUG MARKER: FID seq 0x200000bd0, oid 0x67c4 ver 0x0 [ 1394.046533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000bd0, oid 0x67c4, count: 1 [ 1394.246529] Lustre: DEBUG MARKER: LOV seq 0x200000bd0, oid 0x67c4, count: 1 [ 1394.472653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:0 oid:830\/0x33e seq:0 [ 1394.662192] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:830/0x33e seq:0 [ 1399.247876] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1399.251604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1414.373912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark check file \/mnt\/lustre\/d27z.sanity\/f27z.sanity-2 [ 1414.595389] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 1414.832104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark FID seq 0x200000bd0, oid 0x67c6 ver 0x0 [ 1415.025384] Lustre: DEBUG MARKER: FID seq 0x200000bd0, oid 0x67c6 ver 0x0 [ 1415.262849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LOV seq 0x200000bd0, oid 0x67c6, count: 2 [ 1415.459075] Lustre: DEBUG MARKER: LOV seq 0x200000bd0, oid 0x67c6, count: 2 [ 1415.701405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:0 ost:1 oid:1655\/0x677 seq:0 [ 1415.892305] Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1655/0x677 seq:0 [ 1420.239818] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1420.242298] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1434.464051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark want: stripe:1 ost:0 oid:831\/0x33f seq:0 [ 1434.657524] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:831/0x33f seq:0 [ 1439.087639] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1439.090247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1452.816455] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1453.520370] Lustre: DEBUG MARKER: rc=0; [ 1453.520370] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1453.520370] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1453.520370] echo $(hostname -s): $val; [ 1453.520370] rc=$val; [ 1453.520370] fi; [ 1453.520370] exit $rc [ 1454.247409] Lustre: DEBUG MARKER: dmesg [ 1454.713304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 16:25:56 \(1491409556\) [ 1454.906620] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ====================================== 16:25:56 (1491409556) [ 1454.937755] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1454.940206] Lustre: Skipped 4 previous similar messages [ 1456.038209] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1456.040717] Lustre: Skipped 5 previous similar messages [ 1456.237217] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1456.923642] Lustre: DEBUG MARKER: rc=0; [ 1456.923642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1456.923642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1456.923642] echo $(hostname -s): $val; [ 1456.923642] rc=$val; [ 1456.923642] fi; [ 1456.923642] exit $rc [ 1457.646543] Lustre: DEBUG MARKER: dmesg [ 1458.089853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27B: call setstripe on open unlinked file\/rename victim =============================== 16:26:00 \(1491409560\) [ 1458.275895] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim =============================== 16:26:00 (1491409560) [ 1458.519875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1459.209241] Lustre: DEBUG MARKER: rc=0; [ 1459.209241] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1459.209241] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1459.209241] echo $(hostname -s): $val; [ 1459.209241] rc=$val; [ 1459.209241] fi; [ 1459.209241] exit $rc [ 1459.937882] Lustre: DEBUG MARKER: dmesg [ 1460.400122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27C: check full striping across all OSTs ============================================== 16:26:02 \(1491409562\) [ 1460.601561] Lustre: DEBUG MARKER: == sanity test 27C: check full striping across all OSTs ============================================== 16:26:02 (1491409562) [ 1460.867167] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1461.406450] LustreError: 32747:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 1461.578137] Lustre: DEBUG MARKER: rc=0; [ 1461.578137] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1461.578137] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1461.578137] echo $(hostname -s): $val; [ 1461.578137] rc=$val; [ 1461.578137] fi; [ 1461.578137] exit $rc [ 1462.308112] Lustre: DEBUG MARKER: dmesg [ 1462.777387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27D: validate llapi_layout API ======================================================== 16:26:04 \(1491409564\) [ 1462.973376] Lustre: DEBUG MARKER: == sanity test 27D: validate llapi_layout API ======================================================== 16:26:04 (1491409564) [ 1463.175373] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1473.478855] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1473.782050] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1474.124976] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.testpool lustre-OST[0-1/1] [ 1480.429946] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1480.429946] sort -u | tr '\n' ' ' [ 1480.728995] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1480.728995] sort -u | tr '\n' ' ' [ 1484.095481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1484.822035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1485.136349] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1485.443141] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1495.751646] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 1504.064264] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1512.377583] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1512.679280] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1513.052667] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1513.754855] Lustre: DEBUG MARKER: rc=0; [ 1513.754855] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1513.754855] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1513.754855] echo $(hostname -s): $val; [ 1513.754855] rc=$val; [ 1513.754855] fi; [ 1513.754855] exit $rc [ 1514.450755] Lustre: DEBUG MARKER: dmesg [ 1514.949289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27E: check that default extended attribute size properly increases ==================== 16:26:56 \(1491409616\) [ 1515.138722] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ==================== 16:26:56 (1491409616) [ 1515.421518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1516.100049] Lustre: DEBUG MARKER: rc=0; [ 1516.100049] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1516.100049] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1516.100049] echo $(hostname -s): $val; [ 1516.100049] rc=$val; [ 1516.100049] fi; [ 1516.100049] exit $rc [ 1516.811795] Lustre: DEBUG MARKER: dmesg [ 1517.275603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 16:26:59 \(1491409619\) [ 1517.480062] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================= 16:26:59 (1491409619) [ 1517.651897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1520.133247] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 10.9.5.244@tcp failed: rc = -107 [ 1520.135772] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1530.005866] Lustre: MGS: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 1530.007933] Lustre: Skipped 6 previous similar messages [ 1539.981047] Lustre: 5923:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491409630/real 1491409630] req@ffff880062845800 x1563855295688960/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491409641 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1543.770278] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1544.481754] Lustre: DEBUG MARKER: rc=0; [ 1544.481754] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1544.481754] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1544.481754] echo $(hostname -s): $val; [ 1544.481754] rc=$val; [ 1544.481754] fi; [ 1544.481754] exit $rc [ 1545.206774] Lustre: DEBUG MARKER: dmesg [ 1545.667370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 28: create\/mknod\/mkdir with bad file types ============================================ 16:27:27 \(1491409647\) [ 1545.848021] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ============================================ 16:27:27 (1491409647) [ 1546.178873] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1546.883728] Lustre: DEBUG MARKER: rc=0; [ 1546.883728] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1546.883728] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1546.883728] echo $(hostname -s): $val; [ 1546.883728] rc=$val; [ 1546.883728] fi; [ 1546.883728] exit $rc [ 1547.623395] Lustre: DEBUG MARKER: dmesg [ 1548.089512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 29: IT_GETATTR regression ============================================================ 16:27:30 \(1491409650\) [ 1548.296456] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 16:27:30 (1491409650) [ 1549.567860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first d29 [ 1549.784794] Lustre: DEBUG MARKER: first d29 [ 1550.035089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark second d29 [ 1550.242699] Lustre: DEBUG MARKER: second d29 [ 1550.462159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 1550.654505] Lustre: DEBUG MARKER: done [ 1550.878343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1551.572823] Lustre: DEBUG MARKER: rc=0; [ 1551.572823] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1551.572823] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1551.572823] echo $(hostname -s): $val; [ 1551.572823] rc=$val; [ 1551.572823] fi; [ 1551.572823] exit $rc [ 1552.290181] Lustre: DEBUG MARKER: dmesg [ 1552.823982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30a: execute binary from Lustre \(execve\) ============================================== 16:27:34 \(1491409654\) [ 1553.036317] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ============================================== 16:27:34 (1491409654) [ 1553.282489] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1553.996411] Lustre: DEBUG MARKER: rc=0; [ 1553.996411] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1553.996411] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1553.996411] echo $(hostname -s): $val; [ 1553.996411] rc=$val; [ 1553.996411] fi; [ 1553.996411] exit $rc [ 1554.716848] Lustre: DEBUG MARKER: dmesg [ 1555.212459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30b: execute binary from Lustre as non-root =========================================== 16:27:37 \(1491409657\) [ 1555.418977] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root =========================================== 16:27:37 (1491409657) [ 1555.673512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1556.357889] Lustre: DEBUG MARKER: rc=0; [ 1556.357889] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1556.357889] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1556.357889] echo $(hostname -s): $val; [ 1556.357889] rc=$val; [ 1556.357889] fi; [ 1556.357889] exit $rc [ 1557.084239] Lustre: DEBUG MARKER: dmesg [ 1557.552625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 30c: execute binary from Lustre without read perms ==================================== 16:27:39 \(1491409659\) [ 1557.746405] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ==================================== 16:27:39 (1491409659) [ 1558.080250] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1558.798753] Lustre: DEBUG MARKER: rc=0; [ 1558.798753] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1558.798753] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1558.798753] echo $(hostname -s): $val; [ 1558.798753] rc=$val; [ 1558.798753] fi; [ 1558.798753] exit $rc [ 1559.521123] Lustre: DEBUG MARKER: dmesg [ 1559.981850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31a: open-unlink file ================================================================= 16:27:41 \(1491409661\) [ 1560.181408] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ================================================================= 16:27:41 (1491409661) [ 1560.452416] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1561.144509] Lustre: DEBUG MARKER: rc=0; [ 1561.144509] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1561.144509] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1561.144509] echo $(hostname -s): $val; [ 1561.144509] rc=$val; [ 1561.144509] fi; [ 1561.144509] exit $rc [ 1561.857030] Lustre: DEBUG MARKER: dmesg [ 1562.346253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31b: unlink file with multiple links while open ======================================= 16:27:44 \(1491409664\) [ 1562.551886] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ======================================= 16:27:44 (1491409664) [ 1562.777929] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1563.471830] Lustre: DEBUG MARKER: rc=0; [ 1563.471830] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1563.471830] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1563.471830] echo $(hostname -s): $val; [ 1563.471830] rc=$val; [ 1563.471830] fi; [ 1563.471830] exit $rc [ 1564.183636] Lustre: DEBUG MARKER: dmesg [ 1564.632348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31c: open-unlink file with multiple links ============================================= 16:27:46 \(1491409666\) [ 1564.824936] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ============================================= 16:27:46 (1491409666) [ 1565.057766] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1565.779636] Lustre: DEBUG MARKER: rc=0; [ 1565.779636] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1565.779636] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1565.779636] echo $(hostname -s): $val; [ 1565.779636] rc=$val; [ 1565.779636] fi; [ 1565.779636] exit $rc [ 1566.522450] Lustre: DEBUG MARKER: dmesg [ 1566.971142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31d: remove of open directory ========================================================= 16:27:48 \(1491409668\) [ 1567.161665] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory ========================================================= 16:27:48 (1491409668) [ 1567.692646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1568.376215] Lustre: DEBUG MARKER: rc=0; [ 1568.376215] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1568.376215] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1568.376215] echo $(hostname -s): $val; [ 1568.376215] rc=$val; [ 1568.376215] fi; [ 1568.376215] exit $rc [ 1569.103740] Lustre: DEBUG MARKER: dmesg [ 1569.596221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31e: remove of open non-empty directory =============================================== 16:27:51 \(1491409671\) [ 1569.790531] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory =============================================== 16:27:51 (1491409671) [ 1570.020437] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1570.723505] Lustre: DEBUG MARKER: rc=0; [ 1570.723505] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1570.723505] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1570.723505] echo $(hostname -s): $val; [ 1570.723505] rc=$val; [ 1570.723505] fi; [ 1570.723505] exit $rc [ 1571.437940] Lustre: DEBUG MARKER: dmesg [ 1571.903668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31f: remove of open directory with open-unlink file =================================== 16:27:53 \(1491409673\) [ 1572.107954] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file =================================== 16:27:53 (1491409673) [ 1578.380309] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1579.131214] Lustre: DEBUG MARKER: rc=0; [ 1579.131214] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1579.131214] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1579.131214] echo $(hostname -s): $val; [ 1579.131214] rc=$val; [ 1579.131214] fi; [ 1579.131214] exit $rc [ 1579.841677] Lustre: DEBUG MARKER: dmesg [ 1580.315695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31g: cross directory link============================================================== 16:28:02 \(1491409682\) [ 1580.504916] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link============================================================== 16:28:02 (1491409682) [ 1580.749968] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1581.438293] Lustre: DEBUG MARKER: rc=0; [ 1581.438293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1581.438293] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1581.438293] echo $(hostname -s): $val; [ 1581.438293] rc=$val; [ 1581.438293] fi; [ 1581.438293] exit $rc [ 1582.170937] Lustre: DEBUG MARKER: dmesg [ 1582.664811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31h: cross directory link under child================================================== 16:28:04 \(1491409684\) [ 1582.862786] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child================================================== 16:28:04 (1491409684) [ 1583.098500] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1583.780533] Lustre: DEBUG MARKER: rc=0; [ 1583.780533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1583.780533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1583.780533] echo $(hostname -s): $val; [ 1583.780533] rc=$val; [ 1583.780533] fi; [ 1583.780533] exit $rc [ 1584.512038] Lustre: DEBUG MARKER: dmesg [ 1584.982972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31i: cross directory link under parent================================================= 16:28:06 \(1491409686\) [ 1585.202673] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent================================================= 16:28:06 (1491409686) [ 1585.434217] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1586.127471] Lustre: DEBUG MARKER: rc=0; [ 1586.127471] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1586.127471] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1586.127471] echo $(hostname -s): $val; [ 1586.127471] rc=$val; [ 1586.127471] fi; [ 1586.127471] exit $rc [ 1586.873183] Lustre: DEBUG MARKER: dmesg [ 1587.347692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31j: link for directory================================================================ 16:28:09 \(1491409689\) [ 1587.539811] Lustre: DEBUG MARKER: == sanity test 31j: link for directory================================================================ 16:28:09 (1491409689) [ 1587.851978] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1588.563503] Lustre: DEBUG MARKER: rc=0; [ 1588.563503] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1588.563503] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1588.563503] echo $(hostname -s): $val; [ 1588.563503] rc=$val; [ 1588.563503] fi; [ 1588.563503] exit $rc [ 1589.298733] Lustre: DEBUG MARKER: dmesg [ 1589.767403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31k: link to file: the same, non-existing, dir========================================= 16:28:11 \(1491409691\) [ 1589.949244] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================= 16:28:11 (1491409691) [ 1590.222032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1590.913609] Lustre: DEBUG MARKER: rc=0; [ 1590.913609] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1590.913609] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1590.913609] echo $(hostname -s): $val; [ 1590.913609] rc=$val; [ 1590.913609] fi; [ 1590.913609] exit $rc [ 1591.630830] Lustre: DEBUG MARKER: dmesg [ 1592.072793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31m: link to file: the same, non-existing, dir========================================= 16:28:14 \(1491409694\) [ 1592.267483] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================= 16:28:14 (1491409694) [ 1592.514863] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1593.221490] Lustre: DEBUG MARKER: rc=0; [ 1593.221490] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1593.221490] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1593.221490] echo $(hostname -s): $val; [ 1593.221490] rc=$val; [ 1593.221490] fi; [ 1593.221490] exit $rc [ 1593.933964] Lustre: DEBUG MARKER: dmesg [ 1594.384358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31n: check link count of unlinked file ================================================ 16:28:16 \(1491409696\) [ 1594.607402] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ================================================ 16:28:16 (1491409696) [ 1594.828270] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1595.520027] Lustre: DEBUG MARKER: rc=0; [ 1595.520027] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1595.520027] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1595.520027] echo $(hostname -s): $val; [ 1595.520027] rc=$val; [ 1595.520027] fi; [ 1595.520027] exit $rc [ 1596.252223] Lustre: DEBUG MARKER: dmesg [ 1596.721627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31o: duplicate hard links with same filename ========================================== 16:28:18 \(1491409698\) [ 1596.925893] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================== 16:28:18 (1491409698) [ 1603.213629] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1603.938727] Lustre: DEBUG MARKER: rc=0; [ 1603.938727] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1603.938727] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1603.938727] echo $(hostname -s): $val; [ 1603.938727] rc=$val; [ 1603.938727] fi; [ 1603.938727] exit $rc [ 1604.685419] Lustre: DEBUG MARKER: dmesg [ 1605.162309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 31p: remove of open striped directory ================================================= 16:28:27 \(1491409707\) [ 1605.365582] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ================================================= 16:28:27 (1491409707) [ 1605.613016] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_31p needs \>= 2 MDTs [ 1605.802023] Lustre: DEBUG MARKER: SKIP: sanity test_31p needs >= 2 MDTs [ 1606.023803] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1606.732208] Lustre: DEBUG MARKER: rc=0; [ 1606.732208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1606.732208] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1606.732208] echo $(hostname -s): $val; [ 1606.732208] rc=$val; [ 1606.732208] fi; [ 1606.732208] exit $rc [ 1607.442210] Lustre: DEBUG MARKER: dmesg [ 1607.924053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32a: stat d32a\/ext2-mountpoint\/.. ===================================================== 16:28:29 \(1491409709\) [ 1608.145254] Lustre: DEBUG MARKER: == sanity test 32a: stat d32a/ext2-mountpoint/.. ===================================================== 16:28:29 (1491409709) [ 1608.506900] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1609.269528] Lustre: DEBUG MARKER: rc=0; [ 1609.269528] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1609.269528] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1609.269528] echo $(hostname -s): $val; [ 1609.269528] rc=$val; [ 1609.269528] fi; [ 1609.269528] exit $rc [ 1609.991407] Lustre: DEBUG MARKER: dmesg [ 1610.440060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32b: open d32b\/ext2-mountpoint\/.. ===================================================== 16:28:32 \(1491409712\) [ 1610.643401] Lustre: DEBUG MARKER: == sanity test 32b: open d32b/ext2-mountpoint/.. ===================================================== 16:28:32 (1491409712) [ 1610.919795] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1611.589198] Lustre: DEBUG MARKER: rc=0; [ 1611.589198] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1611.589198] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1611.589198] echo $(hostname -s): $val; [ 1611.589198] rc=$val; [ 1611.589198] fi; [ 1611.589198] exit $rc [ 1612.298704] Lustre: DEBUG MARKER: dmesg [ 1612.764614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32c: stat d32c\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 16:28:34 \(1491409714\) [ 1612.970723] Lustre: DEBUG MARKER: == sanity test 32c: stat d32c/ext2-mountpoint/../d2/test_dir ========================================= 16:28:34 (1491409714) [ 1613.259144] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1613.996774] Lustre: DEBUG MARKER: rc=0; [ 1613.996774] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1613.996774] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1613.996774] echo $(hostname -s): $val; [ 1613.996774] rc=$val; [ 1613.996774] fi; [ 1613.996774] exit $rc [ 1614.726522] Lustre: DEBUG MARKER: dmesg [ 1615.202739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32d: open d32d\/ext2-mountpoint\/..\/d2\/test_dir ========================================= 16:28:37 \(1491409717\) [ 1615.397433] Lustre: DEBUG MARKER: == sanity test 32d: open d32d/ext2-mountpoint/../d2/test_dir ========================================= 16:28:37 (1491409717) [ 1615.699439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1616.389168] Lustre: DEBUG MARKER: rc=0; [ 1616.389168] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1616.389168] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1616.389168] echo $(hostname -s): $val; [ 1616.389168] rc=$val; [ 1616.389168] fi; [ 1616.389168] exit $rc [ 1617.144693] Lustre: DEBUG MARKER: dmesg [ 1617.639481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32e: stat d32e\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 16:28:39 \(1491409719\) [ 1617.833882] Lustre: DEBUG MARKER: == sanity test 32e: stat d32e/symlink->tmp/symlink->lustre-subdir ==================================== 16:28:39 (1491409719) [ 1618.084976] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1618.763476] Lustre: DEBUG MARKER: rc=0; [ 1618.763476] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1618.763476] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1618.763476] echo $(hostname -s): $val; [ 1618.763476] rc=$val; [ 1618.763476] fi; [ 1618.763476] exit $rc [ 1619.471715] Lustre: DEBUG MARKER: dmesg [ 1619.942220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32f: open d32f\/symlink-\>tmp\/symlink-\>lustre-subdir ==================================== 16:28:41 \(1491409721\) [ 1620.173926] Lustre: DEBUG MARKER: == sanity test 32f: open d32f/symlink->tmp/symlink->lustre-subdir ==================================== 16:28:41 (1491409721) [ 1620.435567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1621.137785] Lustre: DEBUG MARKER: rc=0; [ 1621.137785] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1621.137785] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1621.137785] echo $(hostname -s): $val; [ 1621.137785] rc=$val; [ 1621.137785] fi; [ 1621.137785] exit $rc [ 1621.843956] Lustre: DEBUG MARKER: dmesg [ 1622.337122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32g: stat d32g\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 16:28:44 \(1491409724\) [ 1622.526130] Lustre: DEBUG MARKER: == sanity test 32g: stat d32g/symlink->tmp/symlink->lustre-subdir/2 ================================== 16:28:44 (1491409724) [ 1622.777168] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1623.463648] Lustre: DEBUG MARKER: rc=0; [ 1623.463648] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1623.463648] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1623.463648] echo $(hostname -s): $val; [ 1623.463648] rc=$val; [ 1623.463648] fi; [ 1623.463648] exit $rc [ 1624.169280] Lustre: DEBUG MARKER: dmesg [ 1624.635526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32h: open d32h\/symlink-\>tmp\/symlink-\>lustre-subdir\/2 ================================== 16:28:46 \(1491409726\) [ 1624.838668] Lustre: DEBUG MARKER: == sanity test 32h: open d32h/symlink->tmp/symlink->lustre-subdir/2 ================================== 16:28:46 (1491409726) [ 1625.080364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1625.775257] Lustre: DEBUG MARKER: rc=0; [ 1625.775257] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1625.775257] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1625.775257] echo $(hostname -s): $val; [ 1625.775257] rc=$val; [ 1625.775257] fi; [ 1625.775257] exit $rc [ 1626.485701] Lustre: DEBUG MARKER: dmesg [ 1626.953444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32i: stat d32i\/ext2-mountpoint\/..\/test_file =========================================== 16:28:48 \(1491409728\) [ 1627.154050] Lustre: DEBUG MARKER: == sanity test 32i: stat d32i/ext2-mountpoint/../test_file =========================================== 16:28:48 (1491409728) [ 1627.455070] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1628.134368] Lustre: DEBUG MARKER: rc=0; [ 1628.134368] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1628.134368] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1628.134368] echo $(hostname -s): $val; [ 1628.134368] rc=$val; [ 1628.134368] fi; [ 1628.134368] exit $rc [ 1628.859815] Lustre: DEBUG MARKER: dmesg [ 1629.331127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32j: open d32j\/ext2-mountpoint\/..\/test_file =========================================== 16:28:51 \(1491409731\) [ 1629.540192] Lustre: DEBUG MARKER: == sanity test 32j: open d32j/ext2-mountpoint/../test_file =========================================== 16:28:51 (1491409731) [ 1629.822880] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1630.512197] Lustre: DEBUG MARKER: rc=0; [ 1630.512197] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1630.512197] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1630.512197] echo $(hostname -s): $val; [ 1630.512197] rc=$val; [ 1630.512197] fi; [ 1630.512197] exit $rc [ 1631.215560] Lustre: DEBUG MARKER: dmesg [ 1631.724036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32k: stat d32k\/ext2-mountpoint\/..\/d2\/test_file ======================================== 16:28:53 \(1491409733\) [ 1631.911378] Lustre: DEBUG MARKER: == sanity test 32k: stat d32k/ext2-mountpoint/../d2/test_file ======================================== 16:28:53 (1491409733) [ 1632.194696] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1632.888342] Lustre: DEBUG MARKER: rc=0; [ 1632.888342] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1632.888342] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1632.888342] echo $(hostname -s): $val; [ 1632.888342] rc=$val; [ 1632.888342] fi; [ 1632.888342] exit $rc [ 1633.595640] Lustre: DEBUG MARKER: dmesg [ 1634.071730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32l: open d32l\/ext2-mountpoint\/..\/d2\/test_file ======================================== 16:28:56 \(1491409736\) [ 1634.261059] Lustre: DEBUG MARKER: == sanity test 32l: open d32l/ext2-mountpoint/../d2/test_file ======================================== 16:28:56 (1491409736) [ 1634.548768] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1635.239939] Lustre: DEBUG MARKER: rc=0; [ 1635.239939] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1635.239939] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1635.239939] echo $(hostname -s): $val; [ 1635.239939] rc=$val; [ 1635.239939] fi; [ 1635.239939] exit $rc [ 1636.039175] Lustre: DEBUG MARKER: dmesg [ 1636.483593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32m: stat d32m\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 16:28:58 \(1491409738\) [ 1636.684374] Lustre: DEBUG MARKER: == sanity test 32m: stat d32m/symlink->tmp/symlink->lustre-root ====================================== 16:28:58 (1491409738) [ 1636.936425] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1637.625213] Lustre: DEBUG MARKER: rc=0; [ 1637.625213] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1637.625213] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1637.625213] echo $(hostname -s): $val; [ 1637.625213] rc=$val; [ 1637.625213] fi; [ 1637.625213] exit $rc [ 1638.371767] Lustre: DEBUG MARKER: dmesg [ 1638.852364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32n: open d32n\/symlink-\>tmp\/symlink-\>lustre-root ====================================== 16:29:00 \(1491409740\) [ 1639.078004] Lustre: DEBUG MARKER: == sanity test 32n: open d32n/symlink->tmp/symlink->lustre-root ====================================== 16:29:00 (1491409740) [ 1639.321194] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1640.033562] Lustre: DEBUG MARKER: rc=0; [ 1640.033562] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1640.033562] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1640.033562] echo $(hostname -s): $val; [ 1640.033562] rc=$val; [ 1640.033562] fi; [ 1640.033562] exit $rc [ 1640.744977] Lustre: DEBUG MARKER: dmesg [ 1641.238343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32o: stat d32o\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 16:29:03 \(1491409743\) [ 1641.446763] Lustre: DEBUG MARKER: == sanity test 32o: stat d32o/symlink->tmp/symlink->lustre-root/ ===================================== 16:29:03 (1491409743) [ 1641.689731] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1642.429901] Lustre: DEBUG MARKER: rc=0; [ 1642.429901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1642.429901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1642.429901] echo $(hostname -s): $val; [ 1642.429901] rc=$val; [ 1642.429901] fi; [ 1642.429901] exit $rc [ 1643.146765] Lustre: DEBUG MARKER: dmesg [ 1643.629027] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32p: open d32p\/symlink-\>tmp\/symlink-\>lustre-root\/ ===================================== 16:29:05 \(1491409745\) [ 1643.838530] Lustre: DEBUG MARKER: == sanity test 32p: open d32p/symlink->tmp/symlink->lustre-root/ ===================================== 16:29:05 (1491409745) [ 1644.061608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_1 [ 1644.263233] Lustre: DEBUG MARKER: 32p_1 [ 1644.473875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_2 [ 1644.665626] Lustre: DEBUG MARKER: 32p_2 [ 1644.920967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_3 [ 1645.116206] Lustre: DEBUG MARKER: 32p_3 [ 1645.362075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_4 [ 1645.557377] Lustre: DEBUG MARKER: 32p_4 [ 1645.796535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_5 [ 1645.986184] Lustre: DEBUG MARKER: 32p_5 [ 1646.212429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_6 [ 1646.398920] Lustre: DEBUG MARKER: 32p_6 [ 1646.631737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_7 [ 1646.829216] Lustre: DEBUG MARKER: 32p_7 [ 1647.087878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_8 [ 1647.295103] Lustre: DEBUG MARKER: 32p_8 [ 1647.537330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_9 [ 1647.720683] Lustre: DEBUG MARKER: 32p_9 [ 1647.964300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 32p_10 [ 1648.148961] Lustre: DEBUG MARKER: 32p_10 [ 1648.374650] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1649.046171] Lustre: DEBUG MARKER: rc=0; [ 1649.046171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1649.046171] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1649.046171] echo $(hostname -s): $val; [ 1649.046171] rc=$val; [ 1649.046171] fi; [ 1649.046171] exit $rc [ 1649.755593] Lustre: DEBUG MARKER: dmesg [ 1650.224483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32q: stat follows mountpoints in Lustre \(should return error\) ========================= 16:29:12 \(1491409752\) [ 1650.417163] Lustre: DEBUG MARKER: == sanity test 32q: stat follows mountpoints in Lustre (should return error) ========================= 16:29:12 (1491409752) [ 1650.942027] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1651.653710] Lustre: DEBUG MARKER: rc=0; [ 1651.653710] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1651.653710] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1651.653710] echo $(hostname -s): $val; [ 1651.653710] rc=$val; [ 1651.653710] fi; [ 1651.653710] exit $rc [ 1652.376122] Lustre: DEBUG MARKER: dmesg [ 1652.856838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 32r: opendir follows mountpoints in Lustre \(should return error\) ====================== 16:29:14 \(1491409754\) [ 1653.059653] Lustre: DEBUG MARKER: == sanity test 32r: opendir follows mountpoints in Lustre (should return error) ====================== 16:29:14 (1491409754) [ 1653.334205] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1654.039924] Lustre: DEBUG MARKER: rc=0; [ 1654.039924] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1654.039924] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1654.039924] echo $(hostname -s): $val; [ 1654.039924] rc=$val; [ 1654.039924] fi; [ 1654.039924] exit $rc [ 1654.750882] Lustre: DEBUG MARKER: dmesg [ 1655.265504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33aa: write file with mode 444 \(should return error\) ================================== 16:29:17 \(1491409757\) [ 1655.471039] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ================================== 16:29:17 (1491409757) [ 1655.729555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_1 [ 1655.938207] Lustre: DEBUG MARKER: 33_1 [ 1656.174817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 33_2 [ 1656.383787] Lustre: DEBUG MARKER: 33_2 [ 1656.595749] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1657.285413] Lustre: DEBUG MARKER: rc=0; [ 1657.285413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1657.285413] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1657.285413] echo $(hostname -s): $val; [ 1657.285413] rc=$val; [ 1657.285413] fi; [ 1657.285413] exit $rc [ 1657.999302] Lustre: DEBUG MARKER: dmesg [ 1658.463145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33a: test open file\(mode=0444\) with O_RDWR \(should return error\) ====================== 16:29:20 \(1491409760\) [ 1658.661807] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ====================== 16:29:20 (1491409760) [ 1659.012596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1659.726695] Lustre: DEBUG MARKER: rc=0; [ 1659.726695] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1659.726695] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1659.726695] echo $(hostname -s): $val; [ 1659.726695] rc=$val; [ 1659.726695] fi; [ 1659.726695] exit $rc [ 1660.480872] Lustre: DEBUG MARKER: dmesg [ 1660.972731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33b: test open file with malformed flags \(No panic\) =================================== 16:29:22 \(1491409762\) [ 1661.162869] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) =================================== 16:29:22 (1491409762) [ 1663.284025] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1664.017843] Lustre: DEBUG MARKER: rc=0; [ 1664.017843] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1664.017843] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1664.017843] echo $(hostname -s): $val; [ 1664.017843] rc=$val; [ 1664.017843] fi; [ 1664.017843] exit $rc [ 1664.806898] Lustre: DEBUG MARKER: dmesg [ 1665.283433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33c: test llobdstat and write_bytes =================================================== 16:29:27 \(1491409767\) [ 1665.486702] Lustre: DEBUG MARKER: == sanity test 33c: test llobdstat and write_bytes =================================================== 16:29:27 (1491409767) [ 1666.101769] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1666.809833] Lustre: DEBUG MARKER: rc=0; [ 1666.809833] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1666.809833] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1666.809833] echo $(hostname -s): $val; [ 1666.809833] rc=$val; [ 1666.809833] fi; [ 1666.809833] exit $rc [ 1667.522401] Lustre: DEBUG MARKER: dmesg [ 1667.979189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 16:29:29 \(1491409769\) [ 1668.171880] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ===================== 16:29:29 (1491409769) [ 1668.413631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33d needs \>= 2 MDTs [ 1668.622893] Lustre: DEBUG MARKER: SKIP: sanity test_33d needs >= 2 MDTs [ 1668.837893] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1669.522370] Lustre: DEBUG MARKER: rc=0; [ 1669.522370] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1669.522370] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1669.522370] echo $(hostname -s): $val; [ 1669.522370] rc=$val; [ 1669.522370] fi; [ 1669.522370] exit $rc [ 1670.225836] Lustre: DEBUG MARKER: dmesg [ 1670.702366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33e: mkdir and striped directory should have same mode ================================ 16:29:32 \(1491409772\) [ 1670.897958] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ================================ 16:29:32 (1491409772) [ 1671.133677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33e needs \>= 2 MDTs [ 1671.329767] Lustre: DEBUG MARKER: SKIP: sanity test_33e needs >= 2 MDTs [ 1671.556725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1672.264363] Lustre: DEBUG MARKER: rc=0; [ 1672.264363] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1672.264363] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1672.264363] echo $(hostname -s): $val; [ 1672.264363] rc=$val; [ 1672.264363] fi; [ 1672.264363] exit $rc [ 1673.053198] Lustre: DEBUG MARKER: dmesg [ 1673.537059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 16:29:35 \(1491409775\) [ 1673.744033] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ================== 16:29:35 (1491409775) [ 1673.963919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_33f needs \>= 2 MDTs [ 1674.162511] Lustre: DEBUG MARKER: SKIP: sanity test_33f needs >= 2 MDTs [ 1674.377767] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1675.066126] Lustre: DEBUG MARKER: rc=0; [ 1675.066126] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1675.066126] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1675.066126] echo $(hostname -s): $val; [ 1675.066126] rc=$val; [ 1675.066126] fi; [ 1675.066126] exit $rc [ 1675.777661] Lustre: DEBUG MARKER: dmesg [ 1676.255461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 33g: nonroot user create already existing root created file =========================== 16:29:38 \(1491409778\) [ 1676.454501] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file =========================== 16:29:38 (1491409778) [ 1676.668640] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1677.372274] Lustre: DEBUG MARKER: rc=0; [ 1677.372274] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1677.372274] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1677.372274] echo $(hostname -s): $val; [ 1677.372274] rc=$val; [ 1677.372274] fi; [ 1677.372274] exit $rc [ 1678.075622] Lustre: DEBUG MARKER: dmesg [ 1678.540378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34a: truncate file that has not been opened =========================================== 16:29:40 \(1491409780\) [ 1678.760151] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened =========================================== 16:29:40 (1491409780) [ 1678.976353] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1679.689421] Lustre: DEBUG MARKER: rc=0; [ 1679.689421] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1679.689421] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1679.689421] echo $(hostname -s): $val; [ 1679.689421] rc=$val; [ 1679.689421] fi; [ 1679.689421] exit $rc [ 1680.428106] Lustre: DEBUG MARKER: dmesg [ 1680.911420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34b: O_RDONLY opening file doesn\'t create objects ===================================== 16:29:42 \(1491409782\) [ 1681.113479] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects ===================================== 16:29:42 (1491409782) [ 1681.347168] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1682.044158] Lustre: DEBUG MARKER: rc=0; [ 1682.044158] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1682.044158] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1682.044158] echo $(hostname -s): $val; [ 1682.044158] rc=$val; [ 1682.044158] fi; [ 1682.044158] exit $rc [ 1682.751694] Lustre: DEBUG MARKER: dmesg [ 1683.240244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34c: O_RDWR opening file-with-size works ============================================== 16:29:45 \(1491409785\) [ 1683.457969] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ============================================== 16:29:45 (1491409785) [ 1683.696062] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1684.383484] Lustre: DEBUG MARKER: rc=0; [ 1684.383484] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1684.383484] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1684.383484] echo $(hostname -s): $val; [ 1684.383484] rc=$val; [ 1684.383484] fi; [ 1684.383484] exit $rc [ 1685.180063] Lustre: DEBUG MARKER: dmesg [ 1685.685565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34d: write to sparse file ============================================================= 16:29:47 \(1491409787\) [ 1685.874172] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ============================================================= 16:29:47 (1491409787) [ 1686.115186] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1686.819618] Lustre: DEBUG MARKER: rc=0; [ 1686.819618] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1686.819618] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1686.819618] echo $(hostname -s): $val; [ 1686.819618] rc=$val; [ 1686.819618] fi; [ 1686.819618] exit $rc [ 1687.543117] Lustre: DEBUG MARKER: dmesg [ 1688.018429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34e: create objects, some with size and some without ================================== 16:29:50 \(1491409790\) [ 1688.211171] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ================================== 16:29:50 (1491409790) [ 1688.445251] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1689.154606] Lustre: DEBUG MARKER: rc=0; [ 1689.154606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1689.154606] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1689.154606] echo $(hostname -s): $val; [ 1689.154606] rc=$val; [ 1689.154606] fi; [ 1689.154606] exit $rc [ 1689.860077] Lustre: DEBUG MARKER: dmesg [ 1690.328323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34f: read from a file with no objects until EOF ======================================= 16:29:52 \(1491409792\) [ 1690.533445] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ======================================= 16:29:52 (1491409792) [ 1690.802921] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1691.505525] Lustre: DEBUG MARKER: rc=0; [ 1691.505525] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1691.505525] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1691.505525] echo $(hostname -s): $val; [ 1691.505525] rc=$val; [ 1691.505525] fi; [ 1691.505525] exit $rc [ 1692.248086] Lustre: DEBUG MARKER: dmesg [ 1692.718300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34g: truncate long file =============================================================== 16:29:54 \(1491409794\) [ 1692.926981] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file =============================================================== 16:29:54 (1491409794) [ 1693.221336] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1693.964930] Lustre: DEBUG MARKER: rc=0; [ 1693.964930] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1693.964930] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1693.964930] echo $(hostname -s): $val; [ 1693.964930] rc=$val; [ 1693.964930] fi; [ 1693.964930] exit $rc [ 1694.710247] Lustre: DEBUG MARKER: dmesg [ 1695.181263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 34h: ftruncate file under grouplock should not block ================================== 16:29:57 \(1491409797\) [ 1695.385947] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ================================== 16:29:57 (1491409797) [ 1698.066236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1698.791887] Lustre: DEBUG MARKER: rc=0; [ 1698.791887] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1698.791887] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1698.791887] echo $(hostname -s): $val; [ 1698.791887] rc=$val; [ 1698.791887] fi; [ 1698.791887] exit $rc [ 1699.513501] Lustre: DEBUG MARKER: dmesg [ 1699.990473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 35a: exec file with mode 444 \(should return and not leak\) ============================= 16:30:01 \(1491409801\) [ 1700.184815] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ============================= 16:30:01 (1491409801) [ 1700.519247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1701.254775] Lustre: DEBUG MARKER: rc=0; [ 1701.254775] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1701.254775] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1701.254775] echo $(hostname -s): $val; [ 1701.254775] rc=$val; [ 1701.254775] fi; [ 1701.254775] exit $rc [ 1701.972389] Lustre: DEBUG MARKER: dmesg [ 1702.437127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36a: MDS utime check \(mknod, utime\) =================================================== 16:30:04 \(1491409804\) [ 1702.640019] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) =================================================== 16:30:04 (1491409804) [ 1702.947361] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1703.661720] Lustre: DEBUG MARKER: rc=0; [ 1703.661720] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1703.661720] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1703.661720] echo $(hostname -s): $val; [ 1703.661720] rc=$val; [ 1703.661720] fi; [ 1703.661720] exit $rc [ 1704.388157] Lustre: DEBUG MARKER: dmesg [ 1704.915802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36b: OST utime check \(open, utime\) ==================================================== 16:30:06 \(1491409806\) [ 1705.114428] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ==================================================== 16:30:06 (1491409806) [ 1705.349510] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1706.064498] Lustre: DEBUG MARKER: rc=0; [ 1706.064498] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1706.064498] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1706.064498] echo $(hostname -s): $val; [ 1706.064498] rc=$val; [ 1706.064498] fi; [ 1706.064498] exit $rc [ 1706.777710] Lustre: DEBUG MARKER: dmesg [ 1707.249136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36c: non-root MDS utime check \(mknod, utime\) ========================================== 16:30:09 \(1491409809\) [ 1707.442223] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================== 16:30:09 (1491409809) [ 1707.777907] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1708.479391] Lustre: DEBUG MARKER: rc=0; [ 1708.479391] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1708.479391] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1708.479391] echo $(hostname -s): $val; [ 1708.479391] rc=$val; [ 1708.479391] fi; [ 1708.479391] exit $rc [ 1709.207526] Lustre: DEBUG MARKER: dmesg [ 1709.668659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36d: non-root OST utime check \(open, utime\) =========================================== 16:30:11 \(1491409811\) [ 1709.867615] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) =========================================== 16:30:11 (1491409811) [ 1710.099530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1710.810878] Lustre: DEBUG MARKER: rc=0; [ 1710.810878] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1710.810878] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1710.810878] echo $(hostname -s): $val; [ 1710.810878] rc=$val; [ 1710.810878] fi; [ 1710.810878] exit $rc [ 1711.510997] Lustre: DEBUG MARKER: dmesg [ 1711.973539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36e: utime on non-owned file \(should return error\) ==================================== 16:30:13 \(1491409813\) [ 1712.167095] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ==================================== 16:30:13 (1491409813) [ 1712.406711] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1713.082713] Lustre: DEBUG MARKER: rc=0; [ 1713.082713] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1713.082713] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1713.082713] echo $(hostname -s): $val; [ 1713.082713] rc=$val; [ 1713.082713] fi; [ 1713.082713] exit $rc [ 1713.798784] Lustre: DEBUG MARKER: dmesg [ 1714.318750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36f: utime on file racing with OST BRW write ========================================== 16:30:16 \(1491409816\) [ 1714.522200] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ========================================== 16:30:16 (1491409816) [ 1715.792292] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1716.492561] Lustre: DEBUG MARKER: rc=0; [ 1716.492561] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1716.492561] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1716.492561] echo $(hostname -s): $val; [ 1716.492561] rc=$val; [ 1716.492561] fi; [ 1716.492561] exit $rc [ 1717.200831] Lustre: DEBUG MARKER: dmesg [ 1717.659086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36g: filter mod data cache expiry ===================================================== 16:30:19 \(1491409819\) [ 1717.847138] Lustre: DEBUG MARKER: == sanity test 36g: filter mod data cache expiry ===================================================== 16:30:19 (1491409819) [ 1761.173968] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1761.888556] Lustre: DEBUG MARKER: rc=0; [ 1761.888556] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1761.888556] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1761.888556] echo $(hostname -s): $val; [ 1761.888556] rc=$val; [ 1761.888556] fi; [ 1761.888556] exit $rc [ 1762.600133] Lustre: DEBUG MARKER: dmesg [ 1763.104454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36h: utime on file racing with OST BRW write ========================================== 16:31:05 \(1491409865\) [ 1763.289956] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ========================================== 16:31:05 (1491409865) [ 1764.541300] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1765.268984] Lustre: DEBUG MARKER: rc=0; [ 1765.268984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1765.268984] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1765.268984] echo $(hostname -s): $val; [ 1765.268984] rc=$val; [ 1765.268984] fi; [ 1765.268984] exit $rc [ 1766.004555] Lustre: DEBUG MARKER: dmesg [ 1766.488234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 36i: change mtime on striped directory ================================================ 16:31:08 \(1491409868\) [ 1766.692338] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ================================================ 16:31:08 (1491409868) [ 1766.928640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_36i needs \>= 2 MDTs [ 1767.143317] Lustre: DEBUG MARKER: SKIP: sanity test_36i needs >= 2 MDTs [ 1767.348137] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1768.062481] Lustre: DEBUG MARKER: rc=0; [ 1768.062481] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1768.062481] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1768.062481] echo $(hostname -s): $val; [ 1768.062481] rc=$val; [ 1768.062481] fi; [ 1768.062481] exit $rc [ 1768.842422] Lustre: DEBUG MARKER: dmesg [ 1769.305575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 16:31:11 \(1491409871\) [ 1769.547977] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ======================= 16:31:11 (1491409871) [ 1769.773043] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1770.461692] Lustre: DEBUG MARKER: rc=0; [ 1770.461692] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1770.461692] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1770.461692] echo $(hostname -s): $val; [ 1770.461692] rc=$val; [ 1770.461692] fi; [ 1770.461692] exit $rc [ 1771.179974] Lustre: DEBUG MARKER: dmesg [ 1771.671792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39a: mtime changed on create ========================================================== 16:31:13 \(1491409873\) [ 1771.880572] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ========================================================== 16:31:13 (1491409873) [ 1774.115255] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1774.829419] Lustre: DEBUG MARKER: rc=0; [ 1774.829419] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1774.829419] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1774.829419] echo $(hostname -s): $val; [ 1774.829419] rc=$val; [ 1774.829419] fi; [ 1774.829419] exit $rc [ 1775.568311] Lustre: DEBUG MARKER: dmesg [ 1776.044595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 16:31:18 \(1491409878\) [ 1776.252543] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ====================================== 16:31:18 (1491409878) [ 1777.596366] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1778.309189] Lustre: DEBUG MARKER: rc=0; [ 1778.309189] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1778.309189] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1778.309189] echo $(hostname -s): $val; [ 1778.309189] rc=$val; [ 1778.309189] fi; [ 1778.309189] exit $rc [ 1779.041072] Lustre: DEBUG MARKER: dmesg [ 1779.541045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39c: mtime change on rename =========================================================== 16:31:21 \(1491409881\) [ 1779.746317] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename =========================================================== 16:31:21 (1491409881) [ 1782.002498] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1782.715842] Lustre: DEBUG MARKER: rc=0; [ 1782.715842] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1782.715842] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1782.715842] echo $(hostname -s): $val; [ 1782.715842] rc=$val; [ 1782.715842] fi; [ 1782.715842] exit $rc [ 1783.438904] Lustre: DEBUG MARKER: dmesg [ 1783.937888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39d: create, utime, stat ============================================================== 16:31:25 \(1491409885\) [ 1784.174898] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ============================================================== 16:31:25 (1491409885) [ 1784.398707] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1785.095504] Lustre: DEBUG MARKER: rc=0; [ 1785.095504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1785.095504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1785.095504] echo $(hostname -s): $val; [ 1785.095504] rc=$val; [ 1785.095504] fi; [ 1785.095504] exit $rc [ 1785.831445] Lustre: DEBUG MARKER: dmesg [ 1786.298889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39e: create, stat, utime, stat ======================================================== 16:31:28 \(1491409888\) [ 1786.493843] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ======================================================== 16:31:28 (1491409888) [ 1786.745682] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1787.450479] Lustre: DEBUG MARKER: rc=0; [ 1787.450479] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1787.450479] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1787.450479] echo $(hostname -s): $val; [ 1787.450479] rc=$val; [ 1787.450479] fi; [ 1787.450479] exit $rc [ 1788.192371] Lustre: DEBUG MARKER: dmesg [ 1788.679004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39f: create, stat, sleep, utime, stat ================================================= 16:31:30 \(1491409890\) [ 1788.874655] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat ================================================= 16:31:30 (1491409890) [ 1791.107305] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1791.814607] Lustre: DEBUG MARKER: rc=0; [ 1791.814607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1791.814607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1791.814607] echo $(hostname -s): $val; [ 1791.814607] rc=$val; [ 1791.814607] fi; [ 1791.814607] exit $rc [ 1792.521452] Lustre: DEBUG MARKER: dmesg [ 1793.018383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39g: write, chmod, stat =============================================================== 16:31:35 \(1491409895\) [ 1793.218657] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat =============================================================== 16:31:35 (1491409895) [ 1795.476751] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1796.183090] Lustre: DEBUG MARKER: rc=0; [ 1796.183090] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1796.183090] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1796.183090] echo $(hostname -s): $val; [ 1796.183090] rc=$val; [ 1796.183090] fi; [ 1796.183090] exit $rc [ 1796.894511] Lustre: DEBUG MARKER: dmesg [ 1797.381795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39h: write, utime within one second, stat ============================================= 16:31:39 \(1491409899\) [ 1797.592442] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ============================================= 16:31:39 (1491409899) [ 1798.843156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1799.575624] Lustre: DEBUG MARKER: rc=0; [ 1799.575624] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1799.575624] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1799.575624] echo $(hostname -s): $val; [ 1799.575624] rc=$val; [ 1799.575624] fi; [ 1799.575624] exit $rc [ 1800.290421] Lustre: DEBUG MARKER: dmesg [ 1800.745956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39i: write, rename, stat ============================================================== 16:31:42 \(1491409902\) [ 1800.937115] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ============================================================== 16:31:42 (1491409902) [ 1802.190724] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1802.876102] Lustre: DEBUG MARKER: rc=0; [ 1802.876102] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1802.876102] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1802.876102] echo $(hostname -s): $val; [ 1802.876102] rc=$val; [ 1802.876102] fi; [ 1802.876102] exit $rc [ 1803.614921] Lustre: DEBUG MARKER: dmesg [ 1804.093183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39j: write, rename, close, stat ======================================================= 16:31:46 \(1491409906\) [ 1804.295632] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ======================================================= 16:31:46 (1491409906) [ 1804.473744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 1804.822221] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 1805.266365] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 1807.836268] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 1808.518785] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 1808.941586] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1809.639076] Lustre: DEBUG MARKER: rc=0; [ 1809.639076] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1809.639076] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1809.639076] echo $(hostname -s): $val; [ 1809.639076] rc=$val; [ 1809.639076] fi; [ 1809.639076] exit $rc [ 1810.350208] Lustre: DEBUG MARKER: dmesg [ 1810.809568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39k: write, utime, close, stat ======================================================== 16:31:52 \(1491409912\) [ 1811.009941] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ======================================================== 16:31:52 (1491409912) [ 1812.352641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1813.069386] Lustre: DEBUG MARKER: rc=0; [ 1813.069386] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1813.069386] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1813.069386] echo $(hostname -s): $val; [ 1813.069386] rc=$val; [ 1813.069386] fi; [ 1813.069386] exit $rc [ 1813.821309] Lustre: DEBUG MARKER: dmesg [ 1814.307097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39l: directory atime update =========================================================== 16:31:56 \(1491409916\) [ 1814.510310] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update =========================================================== 16:31:56 (1491409916) [ 1814.713516] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 1815.030232] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=2 [ 1818.348445] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1821.670637] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1822.010551] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1822.712772] Lustre: DEBUG MARKER: rc=0; [ 1822.712772] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1822.712772] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1822.712772] echo $(hostname -s): $val; [ 1822.712772] rc=$val; [ 1822.712772] fi; [ 1822.712772] exit $rc [ 1823.430428] Lustre: DEBUG MARKER: dmesg [ 1823.910110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39m: test atime and mtime before 1970 ================================================= 16:32:05 \(1491409925\) [ 1824.107287] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ================================================= 16:32:05 (1491409925) [ 1826.375950] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1827.053353] Lustre: DEBUG MARKER: rc=0; [ 1827.053353] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1827.053353] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1827.053353] echo $(hostname -s): $val; [ 1827.053353] rc=$val; [ 1827.053353] fi; [ 1827.053353] exit $rc [ 1827.762526] Lustre: DEBUG MARKER: dmesg [ 1828.242396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39n: check that O_NOATIME is honored ================================================== 16:32:10 \(1491409930\) [ 1828.441201] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ================================================== 16:32:10 (1491409930) [ 1828.640064] Lustre: DEBUG MARKER: lctl get_param -n mdd.*MDT0000*.atime_diff [ 1828.932662] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=1 [ 1839.264940] Lustre: DEBUG MARKER: lctl set_param -n mdd.*MDT0000*.atime_diff=60 [ 1839.615270] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1840.478469] Lustre: DEBUG MARKER: rc=0; [ 1840.478469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1840.478469] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1840.478469] echo $(hostname -s): $val; [ 1840.478469] rc=$val; [ 1840.478469] fi; [ 1840.478469] exit $rc [ 1841.215158] Lustre: DEBUG MARKER: dmesg [ 1841.703475] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39o: directory cached attributes updated after create ================================= 16:32:23 \(1491409943\) [ 1841.902300] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ================================= 16:32:23 (1491409943) [ 1842.133702] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1842.840239] Lustre: DEBUG MARKER: rc=0; [ 1842.840239] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1842.840239] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1842.840239] echo $(hostname -s): $val; [ 1842.840239] rc=$val; [ 1842.840239] fi; [ 1842.840239] exit $rc [ 1843.567373] Lustre: DEBUG MARKER: dmesg [ 1844.039798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39p: remote directory cached attributes updated after create ========================== 16:32:26 \(1491409946\) [ 1844.237383] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ========================== 16:32:26 (1491409946) [ 1844.460525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_39p needs \>= 2 MDTs [ 1844.667794] Lustre: DEBUG MARKER: SKIP: sanity test_39p needs >= 2 MDTs [ 1844.887678] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1845.631415] Lustre: DEBUG MARKER: rc=0; [ 1845.631415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1845.631415] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1845.631415] echo $(hostname -s): $val; [ 1845.631415] rc=$val; [ 1845.631415] fi; [ 1845.631415] exit $rc [ 1846.364584] Lustre: DEBUG MARKER: dmesg [ 1846.884540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 39q: close won\'t zero out atime ======================================================= 16:32:28 \(1491409948\) [ 1847.073381] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime ======================================================= 16:32:28 (1491409948) [ 1847.317547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1848.026284] Lustre: DEBUG MARKER: rc=0; [ 1848.026284] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1848.026284] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1848.026284] echo $(hostname -s): $val; [ 1848.026284] rc=$val; [ 1848.026284] fi; [ 1848.026284] exit $rc [ 1848.749492] Lustre: DEBUG MARKER: dmesg [ 1849.235515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 40: failed open\(O_TRUNC\) doesn\'t truncate ============================================= 16:32:31 \(1491409951\) [ 1849.433440] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ============================================= 16:32:31 (1491409951) [ 1849.680784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1850.378534] Lustre: DEBUG MARKER: rc=0; [ 1850.378534] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1850.378534] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1850.378534] echo $(hostname -s): $val; [ 1850.378534] rc=$val; [ 1850.378534] fi; [ 1850.378534] exit $rc [ 1851.099888] Lustre: DEBUG MARKER: dmesg [ 1851.591800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 41: test small file write + fstat ===================================================== 16:32:33 \(1491409953\) [ 1851.785233] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat ===================================================== 16:32:33 (1491409953) [ 1853.094403] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1853.972644] Lustre: DEBUG MARKER: rc=0; [ 1853.972644] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1853.972644] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1853.972644] echo $(hostname -s): $val; [ 1853.972644] rc=$val; [ 1853.972644] fi; [ 1853.972644] exit $rc [ 1854.680377] Lustre: DEBUG MARKER: dmesg [ 1855.141829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 1855.332842] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 1855.572988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 1855.766475] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 1855.989916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 1856.199062] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 1856.436036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 1856.631829] Lustre: DEBUG MARKER: SKIP: sanity test_42d skipping ALWAYS excluded test 42d [ 1856.881427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 16:32:38 \(1491409958\) [ 1857.067601] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ================================= 16:32:38 (1491409958) [ 1874.102610] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1874.802644] Lustre: DEBUG MARKER: rc=0; [ 1874.802644] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1874.802644] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1874.802644] echo $(hostname -s): $val; [ 1874.802644] rc=$val; [ 1874.802644] fi; [ 1874.802644] exit $rc [ 1875.522469] Lustre: DEBUG MARKER: dmesg [ 1875.998049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 16:32:58 \(1491409978\) [ 1876.194067] Lustre: DEBUG MARKER: == sanity test 43A: execution of file opened for write should return -ETXTBSY ======================== 16:32:58 (1491409978) [ 1877.443376] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1878.357296] Lustre: DEBUG MARKER: rc=0; [ 1878.357296] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1878.357296] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1878.357296] echo $(hostname -s): $val; [ 1878.357296] rc=$val; [ 1878.357296] fi; [ 1878.357296] exit $rc [ 1879.094246] Lustre: DEBUG MARKER: dmesg [ 1879.574496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43a: open\(RDWR\) of file being executed should return -ETXTBSY ========================= 16:33:01 \(1491409981\) [ 1879.797315] Lustre: DEBUG MARKER: == sanity test 43a: open(RDWR) of file being executed should return -ETXTBSY ========================= 16:33:01 (1491409981) [ 1880.063619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1880.766593] Lustre: DEBUG MARKER: rc=0; [ 1880.766593] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1880.766593] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1880.766593] echo $(hostname -s): $val; [ 1880.766593] rc=$val; [ 1880.766593] fi; [ 1880.766593] exit $rc [ 1881.504390] Lustre: DEBUG MARKER: dmesg [ 1881.995085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 16:33:04 \(1491409984\) [ 1882.209316] Lustre: DEBUG MARKER: == sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 16:33:04 (1491409984) [ 1882.442297] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1883.149832] Lustre: DEBUG MARKER: rc=0; [ 1883.149832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1883.149832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1883.149832] echo $(hostname -s): $val; [ 1883.149832] rc=$val; [ 1883.149832] fi; [ 1883.149832] exit $rc [ 1883.871070] Lustre: DEBUG MARKER: dmesg [ 1884.424219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 43c: md5sum of copy into lustre======================================================== 16:33:06 \(1491409986\) [ 1884.644273] Lustre: DEBUG MARKER: == sanity test 43c: md5sum of copy into lustre======================================================== 16:33:06 (1491409986) [ 1884.893034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1885.592339] Lustre: DEBUG MARKER: rc=0; [ 1885.592339] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1885.592339] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1885.592339] echo $(hostname -s): $val; [ 1885.592339] rc=$val; [ 1885.592339] fi; [ 1885.592339] exit $rc [ 1886.320573] Lustre: DEBUG MARKER: dmesg [ 1886.807290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44A: zero length read from a sparse stripe ============================================ 16:33:08 \(1491409988\) [ 1887.029200] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ============================================ 16:33:08 (1491409988) [ 1887.247923] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1887.953411] Lustre: DEBUG MARKER: rc=0; [ 1887.953411] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1887.953411] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1887.953411] echo $(hostname -s): $val; [ 1887.953411] rc=$val; [ 1887.953411] fi; [ 1887.953411] exit $rc [ 1888.676177] Lustre: DEBUG MARKER: dmesg [ 1889.188080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 44a: test sparse pwrite =============================================================== 16:33:11 \(1491409991\) [ 1889.381044] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite =============================================================== 16:33:11 (1491409991) [ 1889.902641] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1890.637389] Lustre: DEBUG MARKER: rc=0; [ 1890.637389] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1890.637389] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1890.637389] echo $(hostname -s): $val; [ 1890.637389] rc=$val; [ 1890.637389] fi; [ 1890.637389] exit $rc [ 1891.384282] Lustre: DEBUG MARKER: dmesg [ 1891.858377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 1892.617658] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping ALWAYS excluded test 45 [ 1892.878814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 46: dirtying a previously written page ================================================ 16:33:14 \(1491409994\) [ 1893.074598] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ================================================ 16:33:14 (1491409994) [ 1893.764510] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1894.450984] Lustre: DEBUG MARKER: rc=0; [ 1894.450984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1894.450984] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1894.450984] echo $(hostname -s): $val; [ 1894.450984] rc=$val; [ 1894.450984] fi; [ 1894.450984] exit $rc [ 1895.205690] Lustre: DEBUG MARKER: dmesg [ 1895.684750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48a: Access renamed working dir \(should return errors\)================================= 16:33:17 \(1491409997\) [ 1895.881023] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)================================= 16:33:17 (1491409997) [ 1896.069538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1896.472319] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1897.202305] Lustre: DEBUG MARKER: rc=0; [ 1897.202305] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1897.202305] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1897.202305] echo $(hostname -s): $val; [ 1897.202305] rc=$val; [ 1897.202305] fi; [ 1897.202305] exit $rc [ 1897.940235] Lustre: DEBUG MARKER: dmesg [ 1899.816075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48b: Access removed working dir \(should return errors\)================================= 16:33:21 \(1491410001\) [ 1900.040389] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)================================= 16:33:21 (1491410001) [ 1900.277095] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1900.991878] Lustre: DEBUG MARKER: rc=0; [ 1900.991878] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1900.991878] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1900.991878] echo $(hostname -s): $val; [ 1900.991878] rc=$val; [ 1900.991878] fi; [ 1900.991878] exit $rc [ 1901.740343] Lustre: DEBUG MARKER: dmesg [ 1902.249653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48c: Access removed working subdir \(should return errors\) ============================= 16:33:24 \(1491410004\) [ 1902.432644] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ============================= 16:33:24 (1491410004) [ 1902.709074] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1903.461917] Lustre: DEBUG MARKER: rc=0; [ 1903.461917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1903.461917] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1903.461917] echo $(hostname -s): $val; [ 1903.461917] rc=$val; [ 1903.461917] fi; [ 1903.461917] exit $rc [ 1904.196287] Lustre: DEBUG MARKER: dmesg [ 1904.738587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48d: Access removed parent subdir \(should return errors\) ============================== 16:33:26 \(1491410006\) [ 1904.981025] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ============================== 16:33:26 (1491410006) [ 1905.294491] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1906.002036] Lustre: DEBUG MARKER: rc=0; [ 1906.002036] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1906.002036] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1906.002036] echo $(hostname -s): $val; [ 1906.002036] rc=$val; [ 1906.002036] fi; [ 1906.002036] exit $rc [ 1906.722657] Lustre: DEBUG MARKER: dmesg [ 1907.180530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 48e: Access to recreated parent subdir \(should return errors\) ========================= 16:33:29 \(1491410009\) [ 1907.396465] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================= 16:33:29 (1491410009) [ 1907.656339] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1908.390460] Lustre: DEBUG MARKER: rc=0; [ 1908.390460] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1908.390460] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1908.390460] echo $(hostname -s): $val; [ 1908.390460] rc=$val; [ 1908.390460] fi; [ 1908.390460] exit $rc [ 1909.121213] Lustre: DEBUG MARKER: dmesg [ 1909.602272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 49: Change max_pages_per_rpc won\'t break osc extent =================================== 16:33:31 \(1491410011\) [ 1909.843357] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent =================================== 16:33:31 (1491410011) [ 1912.235608] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1912.942915] Lustre: DEBUG MARKER: rc=0; [ 1912.942915] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1912.942915] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1912.942915] echo $(hostname -s): $val; [ 1912.942915] rc=$val; [ 1912.942915] fi; [ 1912.942915] exit $rc [ 1913.681990] Lustre: DEBUG MARKER: dmesg [ 1914.180551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 50: special situations: \/proc symlinks =============================================== 16:33:36 \(1491410016\) [ 1914.384731] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks =============================================== 16:33:36 (1491410016) [ 1914.610373] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1915.291642] Lustre: DEBUG MARKER: rc=0; [ 1915.291642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1915.291642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1915.291642] echo $(hostname -s): $val; [ 1915.291642] rc=$val; [ 1915.291642] fi; [ 1915.291642] exit $rc [ 1916.024066] Lustre: DEBUG MARKER: dmesg [ 1916.488592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51a: special situations: split htree with empty entry ================================= 16:33:38 \(1491410018\) [ 1916.694479] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ================================= 16:33:38 (1491410018) [ 1917.213066] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1917.926280] Lustre: DEBUG MARKER: rc=0; [ 1917.926280] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1917.926280] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1917.926280] echo $(hostname -s): $val; [ 1917.926280] rc=$val; [ 1917.926280] fi; [ 1917.926280] exit $rc [ 1918.645513] Lustre: DEBUG MARKER: dmesg [ 1919.158303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51b skipping SLOW test 51b [ 1919.356667] Lustre: DEBUG MARKER: SKIP: sanity test_51b skipping SLOW test 51b [ 1919.615400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51d: check object distribution ======================================================== 16:33:41 \(1491410021\) [ 1919.819268] Lustre: DEBUG MARKER: == sanity test 51d: check object distribution ======================================================== 16:33:41 (1491410021) [ 1920.051658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51d skipping test with few OSTs [ 1920.252379] Lustre: DEBUG MARKER: SKIP: sanity test_51d skipping test with few OSTs [ 1920.483902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1921.184573] Lustre: DEBUG MARKER: rc=0; [ 1921.184573] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1921.184573] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1921.184573] echo $(hostname -s): $val; [ 1921.184573] rc=$val; [ 1921.184573] fi; [ 1921.184573] exit $rc [ 1921.917826] Lustre: DEBUG MARKER: dmesg [ 1922.398155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 51e: check file nlink limit =========================================================== 16:33:44 \(1491410024\) [ 1922.601759] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =========================================================== 16:33:44 (1491410024) [ 1922.893733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_51e Only applicable to ldiskfs-based MDTs [ 1923.094629] Lustre: DEBUG MARKER: SKIP: sanity test_51e Only applicable to ldiskfs-based MDTs [ 1923.383151] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1924.113232] Lustre: DEBUG MARKER: rc=0; [ 1924.113232] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1924.113232] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1924.113232] echo $(hostname -s): $val; [ 1924.113232] rc=$val; [ 1924.113232] fi; [ 1924.113232] exit $rc [ 1924.841635] Lustre: DEBUG MARKER: dmesg [ 1925.309140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52a: append-only flag test \(should return errors\) ===================================== 16:33:47 \(1491410027\) [ 1925.514402] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ===================================== 16:33:47 (1491410027) [ 1925.840376] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1926.597728] Lustre: DEBUG MARKER: rc=0; [ 1926.597728] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1926.597728] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1926.597728] echo $(hostname -s): $val; [ 1926.597728] rc=$val; [ 1926.597728] fi; [ 1926.597728] exit $rc [ 1927.357935] Lustre: DEBUG MARKER: dmesg [ 1927.869441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 52b: immutable flag test \(should return errors\) ======================================= 16:33:49 \(1491410029\) [ 1928.100989] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ======================================= 16:33:49 (1491410029) [ 1928.351439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1929.088618] Lustre: DEBUG MARKER: rc=0; [ 1929.088618] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1929.088618] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1929.088618] echo $(hostname -s): $val; [ 1929.088618] rc=$val; [ 1929.088618] fi; [ 1929.088618] exit $rc [ 1929.830083] Lustre: DEBUG MARKER: dmesg [ 1930.291287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 16:33:52 \(1491410032\) [ 1930.493583] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ==================================== 16:33:52 (1491410032) [ 1930.711706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1931.048731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1931.365897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1931.686546] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 1932.001897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.*OST*-osc-MDT0000.prealloc_last_id [ 1932.326470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_seq [ 1932.702874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 1933.467226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_seq [ 1933.762407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-OST0001-osc-MDT0000.prealloc_last_id [ 1934.446178] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1935.139769] Lustre: DEBUG MARKER: rc=0; [ 1935.139769] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1935.139769] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1935.139769] echo $(hostname -s): $val; [ 1935.139769] rc=$val; [ 1935.139769] fi; [ 1935.139769] exit $rc [ 1935.908066] Lustre: DEBUG MARKER: dmesg [ 1936.352986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54a: unix domain socket test ========================================================== 16:33:58 \(1491410038\) [ 1936.550932] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ========================================================== 16:33:58 (1491410038) [ 1936.953953] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1937.622834] Lustre: DEBUG MARKER: rc=0; [ 1937.622834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1937.622834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1937.622834] echo $(hostname -s): $val; [ 1937.622834] rc=$val; [ 1937.622834] fi; [ 1937.622834] exit $rc [ 1938.339309] Lustre: DEBUG MARKER: dmesg [ 1938.804312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54b: char device works in lustre ====================================================== 16:34:00 \(1491410040\) [ 1939.010356] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ====================================================== 16:34:00 (1491410040) [ 1939.321418] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1940.025469] Lustre: DEBUG MARKER: rc=0; [ 1940.025469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1940.025469] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1940.025469] echo $(hostname -s): $val; [ 1940.025469] rc=$val; [ 1940.025469] fi; [ 1940.025469] exit $rc [ 1940.744591] Lustre: DEBUG MARKER: dmesg [ 1941.245686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54c: block device works in lustre ===================================================== 16:34:03 \(1491410043\) [ 1941.451088] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre ===================================================== 16:34:03 (1491410043) [ 1948.399909] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1949.124260] Lustre: DEBUG MARKER: rc=0; [ 1949.124260] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1949.124260] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1949.124260] echo $(hostname -s): $val; [ 1949.124260] rc=$val; [ 1949.124260] fi; [ 1949.124260] exit $rc [ 1949.840320] Lustre: DEBUG MARKER: dmesg [ 1950.311070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54d: fifo device works in lustre ====================================================== 16:34:12 \(1491410052\) [ 1950.533987] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ====================================================== 16:34:12 (1491410052) [ 1950.746814] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1951.466674] Lustre: DEBUG MARKER: rc=0; [ 1951.466674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1951.466674] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1951.466674] echo $(hostname -s): $val; [ 1951.466674] rc=$val; [ 1951.466674] fi; [ 1951.466674] exit $rc [ 1952.203305] Lustre: DEBUG MARKER: dmesg [ 1952.665521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 54e: console\/tty device works in lustre =============================================== 16:34:14 \(1491410054\) [ 1952.871720] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre =============================================== 16:34:14 (1491410054) [ 1953.111495] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1953.808024] Lustre: DEBUG MARKER: rc=0; [ 1953.808024] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1953.808024] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1953.808024] echo $(hostname -s): $val; [ 1953.808024] rc=$val; [ 1953.808024] fi; [ 1953.808024] exit $rc [ 1954.568869] Lustre: DEBUG MARKER: dmesg [ 1955.060817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56a: check \/usr\/bin\/lfs getstripe ===================================================== 16:34:17 \(1491410057\) [ 1955.247621] Lustre: DEBUG MARKER: == sanity test 56a: check /usr/bin/lfs getstripe ===================================================== 16:34:17 (1491410057) [ 1955.272298] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1955.274433] Lustre: Skipped 5 previous similar messages [ 1955.624938] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1956.337945] Lustre: DEBUG MARKER: rc=0; [ 1956.337945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1956.337945] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1956.337945] echo $(hostname -s): $val; [ 1956.337945] rc=$val; [ 1956.337945] fi; [ 1956.337945] exit $rc [ 1957.061871] Lustre: DEBUG MARKER: dmesg [ 1957.549984] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56b: check \/usr\/bin\/lfs getdirstripe ================================================== 16:34:19 \(1491410059\) [ 1957.754276] Lustre: DEBUG MARKER: == sanity test 56b: check /usr/bin/lfs getdirstripe ================================================== 16:34:19 (1491410059) [ 1958.041534] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1958.708728] Lustre: DEBUG MARKER: rc=0; [ 1958.708728] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1958.708728] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1958.708728] echo $(hostname -s): $val; [ 1958.708728] rc=$val; [ 1958.708728] fi; [ 1958.708728] exit $rc [ 1959.455900] Lustre: DEBUG MARKER: dmesg [ 1959.934694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56g: check lfs find -name ============================================================= 16:34:21 \(1491410061\) [ 1960.112128] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ============================================================= 16:34:21 (1491410061) [ 1960.146310] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 1960.148495] Lustre: Skipped 5 previous similar messages [ 1960.453162] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1961.180674] Lustre: DEBUG MARKER: rc=0; [ 1961.180674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1961.180674] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1961.180674] echo $(hostname -s): $val; [ 1961.180674] rc=$val; [ 1961.180674] fi; [ 1961.180674] exit $rc [ 1961.926054] Lustre: DEBUG MARKER: dmesg [ 1962.409729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56h: check lfs find ! -name =========================================================== 16:34:24 \(1491410064\) [ 1962.645195] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =========================================================== 16:34:24 (1491410064) [ 1962.965470] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1963.488488] LustreError: 7966:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 1963.661870] Lustre: DEBUG MARKER: rc=0; [ 1963.661870] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1963.661870] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1963.661870] echo $(hostname -s): $val; [ 1963.661870] rc=$val; [ 1963.661870] fi; [ 1963.661870] exit $rc [ 1964.396249] Lustre: DEBUG MARKER: dmesg [ 1964.911718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56i: check \'lfs find -ost UUID\' skips directories ===================================== 16:34:26 \(1491410066\) [ 1965.116334] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ===================================== 16:34:26 (1491410066) [ 1965.344912] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1966.044701] Lustre: DEBUG MARKER: rc=0; [ 1966.044701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1966.044701] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1966.044701] echo $(hostname -s): $val; [ 1966.044701] rc=$val; [ 1966.044701] fi; [ 1966.044701] exit $rc [ 1966.759165] Lustre: DEBUG MARKER: dmesg [ 1967.245128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56j: check lfs find -type d =========================================================== 16:34:29 \(1491410069\) [ 1967.441057] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =========================================================== 16:34:29 (1491410069) [ 1967.784947] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1968.471172] Lustre: DEBUG MARKER: rc=0; [ 1968.471172] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1968.471172] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1968.471172] echo $(hostname -s): $val; [ 1968.471172] rc=$val; [ 1968.471172] fi; [ 1968.471172] exit $rc [ 1969.175510] Lustre: DEBUG MARKER: dmesg [ 1969.633664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56k: check lfs find -type f =========================================================== 16:34:31 \(1491410071\) [ 1969.822130] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =========================================================== 16:34:31 (1491410071) [ 1970.180396] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1970.886945] Lustre: DEBUG MARKER: rc=0; [ 1970.886945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1970.886945] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1970.886945] echo $(hostname -s): $val; [ 1970.886945] rc=$val; [ 1970.886945] fi; [ 1970.886945] exit $rc [ 1971.605635] Lustre: DEBUG MARKER: dmesg [ 1972.104933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56l: check lfs find -type b =========================================================== 16:34:34 \(1491410074\) [ 1972.273442] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =========================================================== 16:34:34 (1491410074) [ 1972.623935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1973.333986] Lustre: DEBUG MARKER: rc=0; [ 1973.333986] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1973.333986] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1973.333986] echo $(hostname -s): $val; [ 1973.333986] rc=$val; [ 1973.333986] fi; [ 1973.333986] exit $rc [ 1974.051452] Lustre: DEBUG MARKER: dmesg [ 1974.503428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56m: check lfs find -type c =========================================================== 16:34:36 \(1491410076\) [ 1974.706145] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =========================================================== 16:34:36 (1491410076) [ 1975.050283] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1975.766781] Lustre: DEBUG MARKER: rc=0; [ 1975.766781] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1975.766781] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1975.766781] echo $(hostname -s): $val; [ 1975.766781] rc=$val; [ 1975.766781] fi; [ 1975.766781] exit $rc [ 1976.487926] Lustre: DEBUG MARKER: dmesg [ 1976.948734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56n: check lfs find -type l =========================================================== 16:34:38 \(1491410078\) [ 1977.138223] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =========================================================== 16:34:38 (1491410078) [ 1977.480965] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1978.163699] Lustre: DEBUG MARKER: rc=0; [ 1978.163699] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1978.163699] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1978.163699] echo $(hostname -s): $val; [ 1978.163699] rc=$val; [ 1978.163699] fi; [ 1978.163699] exit $rc [ 1978.881334] Lustre: DEBUG MARKER: dmesg [ 1979.327053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56o: check lfs find -mtime for old files ============================================== 16:34:41 \(1491410081\) [ 1979.543676] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files ============================================== 16:34:41 (1491410081) [ 1979.960091] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1980.658974] Lustre: DEBUG MARKER: rc=0; [ 1980.658974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1980.658974] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1980.658974] echo $(hostname -s): $val; [ 1980.658974] rc=$val; [ 1980.658974] fi; [ 1980.658974] exit $rc [ 1981.375885] Lustre: DEBUG MARKER: dmesg [ 1981.859818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56p: check lfs find -uid and ! -uid =================================================== 16:34:43 \(1491410083\) [ 1982.052684] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid =================================================== 16:34:43 (1491410083) [ 1982.396710] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1983.112632] Lustre: DEBUG MARKER: rc=0; [ 1983.112632] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1983.112632] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1983.112632] echo $(hostname -s): $val; [ 1983.112632] rc=$val; [ 1983.112632] fi; [ 1983.112632] exit $rc [ 1983.843304] Lustre: DEBUG MARKER: dmesg [ 1984.312835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56q: check lfs find -gid and ! -gid =================================================== 16:34:46 \(1491410086\) [ 1984.514451] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid =================================================== 16:34:46 (1491410086) [ 1984.865535] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1985.570236] Lustre: DEBUG MARKER: rc=0; [ 1985.570236] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1985.570236] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1985.570236] echo $(hostname -s): $val; [ 1985.570236] rc=$val; [ 1985.570236] fi; [ 1985.570236] exit $rc [ 1986.297765] Lustre: DEBUG MARKER: dmesg [ 1986.768229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56r: check lfs find -size works ======================================================= 16:34:48 \(1491410088\) [ 1986.966155] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works ======================================================= 16:34:48 (1491410088) [ 1987.586696] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1988.278191] Lustre: DEBUG MARKER: rc=0; [ 1988.278191] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1988.278191] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1988.278191] echo $(hostname -s): $val; [ 1988.278191] rc=$val; [ 1988.278191] fi; [ 1988.278191] exit $rc [ 1988.996080] Lustre: DEBUG MARKER: dmesg [ 1989.476210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56s: check lfs find -stripe-count works =============================================== 16:34:51 \(1491410091\) [ 1989.662670] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works =============================================== 16:34:51 (1491410091) [ 1990.081764] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1990.771780] Lustre: DEBUG MARKER: rc=0; [ 1990.771780] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1990.771780] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1990.771780] echo $(hostname -s): $val; [ 1990.771780] rc=$val; [ 1990.771780] fi; [ 1990.771780] exit $rc [ 1991.506075] Lustre: DEBUG MARKER: dmesg [ 1991.981061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56t: check lfs find -stripe-size works ================================================ 16:34:53 \(1491410093\) [ 1992.193411] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ================================================ 16:34:53 (1491410093) [ 1992.675967] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1993.364858] Lustre: DEBUG MARKER: rc=0; [ 1993.364858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1993.364858] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1993.364858] echo $(hostname -s): $val; [ 1993.364858] rc=$val; [ 1993.364858] fi; [ 1993.364858] exit $rc [ 1994.077648] Lustre: DEBUG MARKER: dmesg [ 1994.556041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56u: check lfs find -stripe-index works =============================================== 16:34:56 \(1491410096\) [ 1994.761519] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works =============================================== 16:34:56 (1491410096) [ 1995.217539] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1995.932735] Lustre: DEBUG MARKER: rc=0; [ 1995.932735] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1995.932735] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1995.932735] echo $(hostname -s): $val; [ 1995.932735] rc=$val; [ 1995.932735] fi; [ 1995.932735] exit $rc [ 1996.687715] Lustre: DEBUG MARKER: dmesg [ 1997.135564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56v: check \'lfs find -mdt match with lfs getstripe -M\' ================================ 16:34:59 \(1491410099\) [ 1997.342082] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -mdt match with lfs getstripe -M' ================================ 16:34:59 (1491410099) [ 1997.766552] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1998.457518] Lustre: DEBUG MARKER: rc=0; [ 1998.457518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1998.457518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1998.457518] echo $(hostname -s): $val; [ 1998.457518] rc=$val; [ 1998.457518] fi; [ 1998.457518] exit $rc [ 1999.181977] Lustre: DEBUG MARKER: dmesg [ 1999.640461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 16:35:01 \(1491410101\) [ 1999.841485] Lustre: DEBUG MARKER: == sanity test 56w: check lfs_migrate -c stripe_count works ========================================== 16:35:01 (1491410101) [ 2043.588433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2074.249129] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2074.994254] Lustre: DEBUG MARKER: rc=0; [ 2074.994254] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2074.994254] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2074.994254] echo $(hostname -s): $val; [ 2074.994254] rc=$val; [ 2074.994254] fi; [ 2074.994254] exit $rc [ 2075.736912] Lustre: DEBUG MARKER: dmesg [ 2076.281351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56x: lfs migration support ============================================================ 16:36:18 \(1491410178\) [ 2076.493647] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ============================================================ 16:36:18 (1491410178) [ 2077.934701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2078.668762] Lustre: DEBUG MARKER: rc=0; [ 2078.668762] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2078.668762] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2078.668762] echo $(hostname -s): $val; [ 2078.668762] rc=$val; [ 2078.668762] fi; [ 2078.668762] exit $rc [ 2079.401774] Lustre: DEBUG MARKER: dmesg [ 2080.005281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56xa: lfs migration --block support =================================================== 16:36:22 \(1491410182\) [ 2080.257707] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support =================================================== 16:36:22 (1491410182) [ 2080.749115] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2081.463696] Lustre: DEBUG MARKER: rc=0; [ 2081.463696] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2081.463696] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2081.463696] echo $(hostname -s): $val; [ 2081.463696] rc=$val; [ 2081.463696] fi; [ 2081.463696] exit $rc [ 2082.221260] Lustre: DEBUG MARKER: dmesg [ 2082.699276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56y: lfs find -L raid0\|released ======================================================= 16:36:24 \(1491410184\) [ 2082.926885] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released ======================================================= 16:36:24 (1491410184) [ 2083.112558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2083.528208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2084.231623] Lustre: DEBUG MARKER: rc=0; [ 2084.231623] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2084.231623] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2084.231623] echo $(hostname -s): $val; [ 2084.231623] rc=$val; [ 2084.231623] fi; [ 2084.231623] exit $rc [ 2085.012705] Lustre: DEBUG MARKER: dmesg [ 2085.522426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56z: lfs find should continue after an error ========================================== 16:36:27 \(1491410187\) [ 2085.724497] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================== 16:36:27 (1491410187) [ 2087.196823] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2087.918030] Lustre: DEBUG MARKER: rc=0; [ 2087.918030] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2087.918030] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2087.918030] echo $(hostname -s): $val; [ 2087.918030] rc=$val; [ 2087.918030] fi; [ 2087.918030] exit $rc [ 2088.638894] Lustre: DEBUG MARKER: dmesg [ 2089.123850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 56aa: lfs find --size under striped dir =============================================== 16:36:31 \(1491410191\) [ 2089.327229] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir =============================================== 16:36:31 (1491410191) [ 2089.585680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_56aa needs \>= 2 MDTs [ 2089.765057] Lustre: DEBUG MARKER: SKIP: sanity test_56aa needs >= 2 MDTs [ 2089.989111] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2090.689256] Lustre: DEBUG MARKER: rc=0; [ 2090.689256] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2090.689256] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2090.689256] echo $(hostname -s): $val; [ 2090.689256] rc=$val; [ 2090.689256] fi; [ 2090.689256] exit $rc [ 2091.410260] Lustre: DEBUG MARKER: dmesg [ 2091.917441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57a: verify MDS filesystem created with large inodes ================================== 16:36:33 \(1491410193\) [ 2092.120498] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ================================== 16:36:33 (1491410193) [ 2092.353189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_57a Only applicable to ldiskfs-based MDTs [ 2092.555915] Lustre: DEBUG MARKER: SKIP: sanity test_57a Only applicable to ldiskfs-based MDTs [ 2092.782281] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2093.480352] Lustre: DEBUG MARKER: rc=0; [ 2093.480352] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2093.480352] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2093.480352] echo $(hostname -s): $val; [ 2093.480352] rc=$val; [ 2093.480352] fi; [ 2093.480352] exit $rc [ 2094.215554] Lustre: DEBUG MARKER: dmesg [ 2094.713732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 16:36:36 \(1491410196\) [ 2094.913091] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes =================================== 16:36:36 (1491410196) [ 2095.180659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_57b Only applicable to ldiskfs-based MDTs [ 2095.437875] Lustre: DEBUG MARKER: SKIP: sanity test_57b Only applicable to ldiskfs-based MDTs [ 2095.638765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2096.426058] Lustre: DEBUG MARKER: rc=0; [ 2096.426058] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2096.426058] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2096.426058] echo $(hostname -s): $val; [ 2096.426058] rc=$val; [ 2096.426058] fi; [ 2096.426058] exit $rc [ 2097.163212] Lustre: DEBUG MARKER: dmesg [ 2097.744221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 58: verify cross-platform wire constants ============================================== 16:36:39 \(1491410199\) [ 2097.943591] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ============================================== 16:36:39 (1491410199) [ 2098.188831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2098.882925] Lustre: DEBUG MARKER: rc=0; [ 2098.882925] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2098.882925] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2098.882925] echo $(hostname -s): $val; [ 2098.882925] rc=$val; [ 2098.882925] fi; [ 2098.882925] exit $rc [ 2099.596464] Lustre: DEBUG MARKER: dmesg [ 2100.115896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 59: verify cancellation of llog records async ========================================= 16:36:42 \(1491410202\) [ 2100.352392] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async ========================================= 16:36:42 (1491410202) [ 2101.306777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2101.612727] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 2102.037175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2103.357302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2104.676670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2105.990430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2112.362112] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2113.109614] Lustre: DEBUG MARKER: rc=0; [ 2113.109614] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2113.109614] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2113.109614] echo $(hostname -s): $val; [ 2113.109614] rc=$val; [ 2113.109614] fi; [ 2113.109614] exit $rc [ 2113.820537] Lustre: DEBUG MARKER: dmesg [ 2114.293814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 16:36:56 \(1491410216\) [ 2114.568077] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ============================ 16:36:56 (1491410216) [ 2114.745670] Lustre: DEBUG MARKER: ! which run-llog.sh &> /dev/null [ 2115.100681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_60 run 3386 - from kernel mode [ 2115.300862] Lustre: DEBUG MARKER: test_60 run 3386 - from kernel mode [ 2115.465908] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=warning; /usr/sbin/lctl dk > /dev/null [ 2115.893125] Lustre: DEBUG MARKER: sh run-llog.sh [ 2116.368640] Lustre: 16451:0:(llog_test.c:2082:llog_test_setup()) Setup llog-test device over MGS device [ 2116.370791] Lustre: 16451:0:(llog_test.c:111:llog_test_1()) 1a: create a log with name: 9af8dfc1 [ 2116.373226] Lustre: 16451:0:(llog_test.c:128:llog_test_1()) 1b: close newly-created log [ 2116.375130] Lustre: 16451:0:(llog_test.c:159:llog_test_2()) 2a: re-open a log with name: 9af8dfc1 [ 2116.377323] Lustre: 16451:0:(llog_test.c:179:llog_test_2()) 2b: create a log without specified NAME & LOGID [ 2116.379540] Lustre: 16451:0:(llog_test.c:197:llog_test_2()) 2b: write 1 llog records, check llh_count [ 2116.381865] Lustre: 16451:0:(llog_test.c:210:llog_test_2()) 2c: re-open the log by LOGID and verify llh_count [ 2116.384695] Lustre: 16451:0:(llog_test.c:257:llog_test_2()) 2d: destroy this log [ 2116.386622] Lustre: 16451:0:(llog_test.c:411:llog_test_3()) 3a: write 1023 fixed-size llog records [ 2116.410326] Lustre: 16451:0:(llog_test.c:377:llog_test3_process()) test3: processing records from index 501 to the end [ 2116.419859] Lustre: 16451:0:(llog_test.c:386:llog_test3_process()) test3: total 525 records processed with 0 paddings [ 2116.422056] Lustre: 16451:0:(llog_test.c:467:llog_test_3()) 3b: write 566 variable size llog records [ 2116.444851] Lustre: 16451:0:(llog_test.c:536:llog_test_3()) 3c: write records with variable size until BITMAP_SIZE, return -ENOSPC [ 2118.229895] Lustre: 16451:0:(llog_test.c:559:llog_test_3()) 3c: wrote 63962 more records before end of llog is reached [ 2118.232368] Lustre: 16451:0:(llog_test.c:588:llog_test_4()) 4a: create a catalog log with name: 9af8dfc2 [ 2118.234621] Lustre: 16451:0:(llog_test.c:603:llog_test_4()) 4b: write 1 record into the catalog [ 2118.237300] Lustre: 16451:0:(llog_test.c:618:llog_test_4()) 4c: cancel 1 log record [ 2118.239442] Lustre: 16451:0:(llog_test.c:630:llog_test_4()) 4d: write 65536 more log records [ 2120.719812] Lustre: 16451:0:(llog_test.c:646:llog_test_4()) 4e: add 5 large records, one record per block [ 2120.722690] Lustre: 16451:0:(llog_test.c:666:llog_test_4()) 4f: put newly-created catalog [ 2120.724647] Lustre: 16451:0:(llog_test.c:764:llog_test_5()) 5a: re-open catalog by id [ 2120.726542] Lustre: 16451:0:(llog_test.c:777:llog_test_5()) 5b: print the catalog entries.. we expect 2 [ 2120.728821] Lustre: 16457:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf:0x0] in log [0xa:0xf:0x0] [ 2120.730935] Lustre: 16451:0:(llog_test.c:789:llog_test_5()) 5c: Cancel 65536 records, see one log zapped [ 2127.169937] Lustre: 16451:0:(llog_test.c:797:llog_test_5()) 5c: print the catalog entries.. we expect 1 [ 2127.172239] Lustre: 16458:0:(llog_test.c:695:cat_print_cb()) seeing record at index 2 - [0x1:0x10:0x0] in log [0xa:0xf:0x0] [ 2127.174418] Lustre: 16458:0:(llog_test.c:695:cat_print_cb()) Skipped 1 previous similar message [ 2127.176485] Lustre: 16451:0:(llog_test.c:809:llog_test_5()) 5d: add 1 record to the log with many canceled empty pages [ 2127.178870] Lustre: 16451:0:(llog_test.c:817:llog_test_5()) 5e: print plain log entries.. expect 6 [ 2127.181264] Lustre: 16451:0:(llog_test.c:829:llog_test_5()) 5f: print plain log entries reversely.. expect 6 [ 2127.184312] Lustre: 16451:0:(llog_test.c:843:llog_test_5()) 5g: close re-opened catalog [ 2127.186240] Lustre: 16451:0:(llog_test.c:873:llog_test_6()) 6a: re-open log 9af8dfc1 using client API [ 2127.188386] Lustre: MGS: non-config logname received: 9af8dfc1 [ 2127.190427] Lustre: 16451:0:(llog_test.c:905:llog_test_6()) 6b: process log 9af8dfc1 using client API [ 2127.279212] Lustre: 16451:0:(llog_test.c:909:llog_test_6()) 6b: processed 63962 records [ 2127.281508] Lustre: 16451:0:(llog_test.c:916:llog_test_6()) 6c: process log 9af8dfc1 reversely using client API [ 2133.324195] Lustre: 16451:0:(llog_test.c:920:llog_test_6()) 6c: processed 63962 records [ 2133.326606] Lustre: 16451:0:(llog_test.c:1067:llog_test_7()) 7a: test llog_logid_rec [ 2155.165116] Lustre: 16451:0:(llog_test.c:1078:llog_test_7()) 7b: test llog_unlink64_rec [ 2223.019073] Lustre: 16451:0:(llog_test.c:1089:llog_test_7()) 7c: test llog_setattr64_rec [ 2226.864533] Lustre: 16451:0:(llog_test.c:1100:llog_test_7()) 7d: test llog_size_change_rec [ 2244.532776] Lustre: 16451:0:(llog_test.c:1111:llog_test_7()) 7e: test llog_changelog_rec [ 2245.901390] Lustre: 16451:0:(llog_test.c:1017:llog_test_7_sub()) 7_sub: records are not aligned, written 64071 from 64767 [ 2332.485354] Lustre: 16451:0:(llog_test.c:1123:llog_test_7()) 7f: test llog_changelog_user_rec [ 2333.769320] Lustre: 16451:0:(llog_test.c:1017:llog_test_7_sub()) 7_sub: records are not aligned, written 64452 from 64767 [ 2335.402762] Lustre: 16451:0:(llog_test.c:1134:llog_test_7()) 7g: test llog_gen_rec [ 2411.018171] Lustre: 16451:0:(llog_test.c:1229:llog_test_8()) 8a: fill the first plain llog [ 2411.025551] Lustre: 16451:0:(llog_test.c:1262:llog_test_8()) 8a: pin llog [0x1:0x1a:0x0] [ 2411.027415] Lustre: 16451:0:(llog_test.c:1272:llog_test_8()) 8b: fill the second plain llog [ 2411.033980] Lustre: 16451:0:(llog_test.c:1293:llog_test_8()) 8b: second llog [0x1:0x1c:0x0] [ 2411.035867] Lustre: 16451:0:(llog_test.c:1303:llog_test_8()) 8c: drop two records from the first plain llog [ 2411.037852] Lustre: 16451:0:(llog_test.c:1306:llog_test_8()) 8d: count survived records [ 2411.040215] LustreError: 16451:0:(llog.c:616:llog_process_thread()) Local llog found corrupted [ 2411.042377] Lustre: 16451:0:(llog_test.c:1333:llog_test_8()) 8d: close re-opened catalog [ 2411.044221] Lustre: 16451:0:(llog_test.c:1395:llog_test_9()) 9a: test llog_logid_rec [ 2411.046278] Lustre: 16451:0:(llog_test.c:1379:llog_test_9_sub()) 9_sub: record type 1064553b in log 0x1:0x1e:0x0 [ 2411.048280] Lustre: 16451:0:(llog_test.c:1406:llog_test_9()) 9b: test llog_obd_cfg_rec [ 2411.050302] Lustre: 16451:0:(llog_test.c:1417:llog_test_9()) 9c: test llog_changelog_rec [ 2411.052260] Lustre: 16451:0:(llog_test.c:1429:llog_test_9()) 9d: test llog_changelog_user_rec [ 2411.054258] Lustre: 16451:0:(llog_test.c:1466:llog_test_10()) 10a: create a catalog log with name: 9af8dfc3 [ 2411.134228] Lustre: 16451:0:(llog_test.c:1494:llog_test_10()) 10b: write 65536 log records [ 2413.662269] Lustre: 16451:0:(llog_test.c:1518:llog_test_10()) 10c: write 131072 more log records [ 2420.815709] Lustre: 16451:0:(llog_test.c:1546:llog_test_10()) 10c: write 65536 more log records [ 2424.216589] Lustre: 16451:0:(llog_cat.c:89:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2424.416343] Lustre: 16451:0:(llog_test.c:1571:llog_test_10()) 10c: wrote 61704 records then 3832 failed with ENOSPC [ 2424.418651] Lustre: 16451:0:(llog_test.c:1588:llog_test_10()) 10d: Cancel 65536 records, see one log zapped [ 2453.981241] Lustre: 16451:0:(llog_test.c:1600:llog_test_10()) 10d: print the catalog entries.. we expect 3 [ 2453.983368] Lustre: 16487:0:(llog_test.c:695:cat_print_cb()) seeing record at index 2 - [0x1:0x23:0x0] in log [0xa:0x10:0x0] [ 2454.056702] Lustre: 16451:0:(llog_test.c:1628:llog_test_10()) 10e: write 65536 more log records [ 2457.072090] Lustre: 16451:0:(llog_cat.c:89:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2457.074162] Lustre: 16451:0:(llog_cat.c:89:llog_cat_new_log()) Skipped 3831 previous similar messages [ 2457.111407] Lustre: 16451:0:(llog_test.c:1653:llog_test_10()) 10e: wrote 64578 records then 958 failed with ENOSPC [ 2457.113390] Lustre: 16451:0:(llog_test.c:1655:llog_test_10()) 10e: print the catalog entries.. we expect 4 [ 2457.115551] Lustre: 16488:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2457.117655] Lustre: 16488:0:(llog_test.c:695:cat_print_cb()) Skipped 2 previous similar messages [ 2457.119637] Lustre: 16451:0:(llog_test.c:1691:llog_test_10()) 10e: catalog successfully wrap around, last_idx 1, first 1 [ 2457.282469] Lustre: 16451:0:(llog_test.c:1704:llog_test_10()) 10f: Cancel 65536 records, see one log zapped [ 2457.285481] Lustre: 16451:0:(llog_cat.c:869:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2486.761524] Lustre: 16451:0:(llog_test.c:1716:llog_test_10()) 10f: print the catalog entries.. we expect 3 [ 2486.763697] Lustre: 16490:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2486.765839] Lustre: 16490:0:(llog_test.c:695:cat_print_cb()) Skipped 3 previous similar messages [ 2486.868175] Lustre: 16451:0:(llog_test.c:1744:llog_test_10()) 10f: write 65536 more log records [ 2489.862169] Lustre: 16451:0:(llog_cat.c:89:llog_cat_new_log()) MGS: there are no more free slots in catalog [ 2489.864314] Lustre: 16451:0:(llog_cat.c:89:llog_cat_new_log()) Skipped 957 previous similar messages [ 2489.901490] Lustre: 16451:0:(llog_test.c:1769:llog_test_10()) 10f: wrote 64578 records then 958 failed with ENOSPC [ 2490.071670] Lustre: 16451:0:(llog_test.c:1812:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2490.073815] Lustre: 16451:0:(llog_cat.c:869:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2519.436258] Lustre: 16451:0:(llog_test.c:1824:llog_test_10()) 10g: print the catalog entries.. we expect 3 [ 2519.438516] Lustre: 16493:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2519.440749] Lustre: 16493:0:(llog_test.c:695:cat_print_cb()) Skipped 2 previous similar messages [ 2519.526240] Lustre: 16451:0:(llog_test.c:1851:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2519.529456] Lustre: 16451:0:(llog_cat.c:869:llog_cat_process_or_fork()) catlog 0x10:10 crosses index zero [ 2549.061970] Lustre: 16451:0:(llog_test.c:1863:llog_test_10()) 10g: print the catalog entries.. we expect 2 [ 2549.064283] Lustre: 16496:0:(llog_test.c:695:cat_print_cb()) seeing record at index 1 - [0x1:0xf1e:0x0] in log [0xa:0x10:0x0] [ 2549.066538] Lustre: 16496:0:(llog_test.c:695:cat_print_cb()) Skipped 2 previous similar messages [ 2549.165180] Lustre: 16451:0:(llog_test.c:1898:llog_test_10()) 10g: Cancel 65536 records, see one log zapped [ 2581.500541] Lustre: 16451:0:(llog_test.c:1910:llog_test_10()) 10g: print the catalog entries.. we expect 1 [ 2581.502888] Lustre: 16498:0:(llog_test.c:695:cat_print_cb()) seeing record at index 2 - [0x1:0x12dd:0x0] in log [0xa:0x10:0x0] [ 2581.506568] Lustre: 16498:0:(llog_test.c:695:cat_print_cb()) Skipped 1 previous similar message [ 2581.508997] Lustre: 16451:0:(llog_test.c:1935:llog_test_10()) 10g: llh_cat_idx has also successfully wrapped! [ 2581.511205] Lustre: 16451:0:(llog_test.c:1941:llog_test_10()) 10: put newly-created catalog [ 2581.674410] Lustre: DEBUG MARKER: /usr/sbin/lctl dk [ 2582.032144] Lustre: DEBUG MARKER: which llog_reader 2> /dev/null [ 2582.333561] Lustre: DEBUG MARKER: ls -d /sbin/llog_reader [ 2582.731064] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2583.023220] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2583.170420] Lustre: Failing over lustre-MDT0000 [ 2584.965535] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.969950] LustreError: Skipped 1 previous similar message [ 2586.272117] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.276608] LustreError: Skipped 1 previous similar message [ 2589.289049] Lustre: 16868:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491410685/real 1491410685] req@ffff88005b99a400 x1563855295764416/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491410691 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2589.295835] Lustre: 16868:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2589.314483] Lustre: server umount lustre-MDT0000 complete [ 2589.467463] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2589.784990] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2589.784990] grep -q ^lustre-mdt1/ /proc/mounts || [ 2589.784990] zpool export lustre-mdt1 [ 2590.308182] Lustre: DEBUG MARKER: modprobe zfs; [ 2590.308182] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2590.308182] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2590.898901] Lustre: DEBUG MARKER: zfs get -H -o value canmount lustre-mdt1/mdt1 [ 2591.192388] Lustre: DEBUG MARKER: zfs get -H -o value mountpoint lustre-mdt1/mdt1 [ 2591.484482] Lustre: DEBUG MARKER: zfs set canmount=noauto lustre-mdt1/mdt1 [ 2591.865546] Lustre: DEBUG MARKER: zfs set mountpoint=legacy lustre-mdt1/mdt1 [ 2592.324795] Lustre: DEBUG MARKER: mount -t zfs lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2592.627822] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2592.917851] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2593.206780] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1e:0x0 [ 2593.501743] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1f:0x0 [ 2593.797245] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x1f:0x0 [ 2594.091445] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x20:0x0 [ 2594.382616] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x20:0x0 [ 2594.677330] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x21:0x0 [ 2594.967529] Lustre: DEBUG MARKER: /sbin/llog_reader /mnt/lustre-mds1/oi.1/0x1:0x21:0x0 [ 2595.293427] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2595.581481] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2596.020497] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2596.342709] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2596.342709] grep -q ^lustre-mdt1/ /proc/mounts || [ 2596.342709] zpool export lustre-mdt1 [ 2596.798239] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2597.120671] Lustre: DEBUG MARKER: modprobe zfs; [ 2597.120671] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2597.120671] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2597.726065] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2598.013691] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2598.199126] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 2598.201158] Lustre: Skipped 4 previous similar messages [ 2598.304687] LustreError: 20729:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2598.370432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.533084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2598.827188] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2599.318663] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2599.610782] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2599.910089] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2600.245276] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2600.965264] Lustre: DEBUG MARKER: rc=0; [ 2600.965264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2600.965264] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2600.965264] echo $(hostname -s): $val; [ 2600.965264] rc=$val; [ 2600.965264] fi; [ 2600.965264] exit $rc [ 2601.669946] Lustre: DEBUG MARKER: dmesg [ 2602.153418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60b: limit repeated messages from CERROR\/CWARN ======================================== 16:45:04 \(1491410704\) [ 2602.362810] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ======================================== 16:45:04 (1491410704) [ 2607.271309] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2608.408225] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2608.653795] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2609.332942] Lustre: DEBUG MARKER: rc=0; [ 2609.332942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2609.332942] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2609.332942] echo $(hostname -s): $val; [ 2609.332942] rc=$val; [ 2609.332942] fi; [ 2609.332942] exit $rc [ 2610.030606] Lustre: DEBUG MARKER: dmesg [ 2610.517469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60c: unlink file when mds full ======================================================== 16:45:12 \(1491410712\) [ 2610.708540] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ======================================================== 16:45:12 (1491410712) [ 2627.553757] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2628.238504] Lustre: DEBUG MARKER: rc=0; [ 2628.238504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2628.238504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2628.238504] echo $(hostname -s): $val; [ 2628.238504] rc=$val; [ 2628.238504] fi; [ 2628.238504] exit $rc [ 2629.020251] Lustre: DEBUG MARKER: dmesg [ 2629.569722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60d: test printk console message masking ============================================== 16:45:31 \(1491410731\) [ 2629.763072] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking ============================================== 16:45:31 (1491410731) [ 2630.025429] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2630.699622] Lustre: DEBUG MARKER: rc=0; [ 2630.699622] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2630.699622] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2630.699622] echo $(hostname -s): $val; [ 2630.699622] rc=$val; [ 2630.699622] fi; [ 2630.699622] exit $rc [ 2631.403494] Lustre: DEBUG MARKER: dmesg [ 2631.920783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 60e: no space while new llog is being created ========================================= 16:45:33 \(1491410733\) [ 2632.139695] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================= 16:45:33 (1491410733) [ 2632.375239] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x15b [ 2632.537241] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2632.539547] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2632.731715] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2633.428588] Lustre: DEBUG MARKER: rc=0; [ 2633.428588] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2633.428588] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2633.428588] echo $(hostname -s): $val; [ 2633.428588] rc=$val; [ 2633.428588] fi; [ 2633.428588] exit $rc [ 2634.159090] Lustre: DEBUG MARKER: dmesg [ 2634.682545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 61: mmap\(\) writes don\'t make sync hang ================================================ 16:45:36 \(1491410736\) [ 2634.884055] Lustre: DEBUG MARKER: == sanity test 61: mmap() writes don't make sync hang ================================================ 16:45:36 (1491410736) [ 2635.356610] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2636.039942] Lustre: DEBUG MARKER: rc=0; [ 2636.039942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2636.039942] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2636.039942] echo $(hostname -s): $val; [ 2636.039942] rc=$val; [ 2636.039942] fi; [ 2636.039942] exit $rc [ 2636.731964] Lustre: DEBUG MARKER: dmesg [ 2637.206401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 16:45:39 \(1491410739\) [ 2637.394410] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ====================================== 16:45:39 (1491410739) [ 2697.671581] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2698.397152] Lustre: DEBUG MARKER: rc=0; [ 2698.397152] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2698.397152] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2698.397152] echo $(hostname -s): $val; [ 2698.397152] rc=$val; [ 2698.397152] fi; [ 2698.397152] exit $rc [ 2699.140217] Lustre: DEBUG MARKER: dmesg [ 2699.680142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 63b: async write errors should be returned to fsync =================================== 16:46:41 \(1491410801\) [ 2699.882685] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync =================================== 16:46:41 (1491410801) [ 2700.086372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2702.960293] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2703.389798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2704.094380] Lustre: DEBUG MARKER: rc=0; [ 2704.094380] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2704.094380] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2704.094380] echo $(hostname -s): $val; [ 2704.094380] rc=$val; [ 2704.094380] fi; [ 2704.094380] exit $rc [ 2704.811720] Lustre: DEBUG MARKER: dmesg [ 2705.303298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64a: verify filter grant calculations \(in kernel\) ===================================== 16:46:47 \(1491410807\) [ 2705.513042] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) ===================================== 16:46:47 (1491410807) [ 2705.739564] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2706.444090] Lustre: DEBUG MARKER: rc=0; [ 2706.444090] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2706.444090] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2706.444090] echo $(hostname -s): $val; [ 2706.444090] rc=$val; [ 2706.444090] fi; [ 2706.444090] exit $rc [ 2707.157275] Lustre: DEBUG MARKER: dmesg [ 2707.643187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_64b skipping SLOW test 64b [ 2707.845908] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 2708.090519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 64c: verify grant shrink ========================------ =============================== 16:46:50 \(1491410810\) [ 2708.307960] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ========================------ =============================== 16:46:50 (1491410810) [ 2708.523323] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2709.243118] Lustre: DEBUG MARKER: rc=0; [ 2709.243118] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2709.243118] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2709.243118] echo $(hostname -s): $val; [ 2709.243118] rc=$val; [ 2709.243118] fi; [ 2709.243118] exit $rc [ 2709.957327] Lustre: DEBUG MARKER: dmesg [ 2710.456533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65a: directory with no stripe info ==================================================== 16:46:52 \(1491410812\) [ 2710.658837] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ==================================================== 16:46:52 (1491410812) [ 2711.259571] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2711.932580] Lustre: DEBUG MARKER: rc=0; [ 2711.932580] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2711.932580] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2711.932580] echo $(hostname -s): $val; [ 2711.932580] rc=$val; [ 2711.932580] fi; [ 2711.932580] exit $rc [ 2712.644585] Lustre: DEBUG MARKER: dmesg [ 2713.115381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 16:46:55 \(1491410815\) [ 2713.316297] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 =================================== 16:46:55 (1491410815) [ 2713.555927] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2714.227690] Lustre: DEBUG MARKER: rc=0; [ 2714.227690] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2714.227690] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2714.227690] echo $(hostname -s): $val; [ 2714.227690] rc=$val; [ 2714.227690] fi; [ 2714.227690] exit $rc [ 2714.933564] Lustre: DEBUG MARKER: dmesg [ 2715.407966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 =================================== 16:46:57 \(1491410817\) [ 2715.602027] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 =================================== 16:46:57 (1491410817) [ 2715.857699] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2716.558836] Lustre: DEBUG MARKER: rc=0; [ 2716.558836] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2716.558836] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2716.558836] echo $(hostname -s): $val; [ 2716.558836] rc=$val; [ 2716.558836] fi; [ 2716.558836] exit $rc [ 2717.271810] Lustre: DEBUG MARKER: dmesg [ 2717.802993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 16:46:59 \(1491410819\) [ 2718.015509] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count =============================== 16:46:59 (1491410819) [ 2718.292937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2718.984575] Lustre: DEBUG MARKER: rc=0; [ 2718.984575] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2718.984575] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2718.984575] echo $(hostname -s): $val; [ 2718.984575] rc=$val; [ 2718.984575] fi; [ 2718.984575] exit $rc [ 2719.703883] Lustre: DEBUG MARKER: dmesg [ 2720.182426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65e: directory setstripe defaults ===================================================== 16:47:02 \(1491410822\) [ 2720.385738] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ===================================================== 16:47:02 (1491410822) [ 2720.662290] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2721.397954] Lustre: DEBUG MARKER: rc=0; [ 2721.397954] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2721.397954] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2721.397954] echo $(hostname -s): $val; [ 2721.397954] rc=$val; [ 2721.397954] fi; [ 2721.397954] exit $rc [ 2722.112557] Lustre: DEBUG MARKER: dmesg [ 2722.582422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65f: dir setstripe permission \(should return error\) =================================== 16:47:04 \(1491410824\) [ 2722.785305] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) =================================== 16:47:04 (1491410824) [ 2723.094706] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2723.778603] Lustre: DEBUG MARKER: rc=0; [ 2723.778603] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2723.778603] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2723.778603] echo $(hostname -s): $val; [ 2723.778603] rc=$val; [ 2723.778603] fi; [ 2723.778603] exit $rc [ 2724.479862] Lustre: DEBUG MARKER: dmesg [ 2724.998360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65g: directory setstripe -d =========================================================== 16:47:07 \(1491410827\) [ 2725.230123] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =========================================================== 16:47:07 (1491410827) [ 2725.481181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2726.167137] Lustre: DEBUG MARKER: rc=0; [ 2726.167137] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2726.167137] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2726.167137] echo $(hostname -s): $val; [ 2726.167137] rc=$val; [ 2726.167137] fi; [ 2726.167137] exit $rc [ 2726.873018] Lustre: DEBUG MARKER: dmesg [ 2727.344034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65h: directory stripe info inherit ==================================================== 16:47:09 \(1491410829\) [ 2727.533681] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ==================================================== 16:47:09 (1491410829) [ 2727.784072] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2728.457459] Lustre: DEBUG MARKER: rc=0; [ 2728.457459] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2728.457459] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2728.457459] echo $(hostname -s): $val; [ 2728.457459] rc=$val; [ 2728.457459] fi; [ 2728.457459] exit $rc [ 2729.157104] Lustre: DEBUG MARKER: dmesg [ 2729.621072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65i: set non-default striping on root directory \(bug 6367\)============================= 16:47:11 \(1491410831\) [ 2729.815133] Lustre: DEBUG MARKER: == sanity test 65i: set non-default striping on root directory (bug 6367)============================= 16:47:11 (1491410831) [ 2729.875489] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2729.877608] Lustre: Skipped 11 previous similar messages [ 2730.078488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2730.757811] Lustre: DEBUG MARKER: rc=0; [ 2730.757811] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2730.757811] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2730.757811] echo $(hostname -s): $val; [ 2730.757811] rc=$val; [ 2730.757811] fi; [ 2730.757811] exit $rc [ 2731.465124] Lustre: DEBUG MARKER: dmesg [ 2731.929058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ia: getstripe on -1 default directory striping ====================================== 16:47:13 \(1491410833\) [ 2732.114415] Lustre: DEBUG MARKER: == sanity test 65ia: getstripe on -1 default directory striping ====================================== 16:47:13 (1491410833) [ 2732.828592] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2733.493387] Lustre: DEBUG MARKER: rc=0; [ 2733.493387] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2733.493387] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2733.493387] echo $(hostname -s): $val; [ 2733.493387] rc=$val; [ 2733.493387] fi; [ 2733.493387] exit $rc [ 2734.208735] Lustre: DEBUG MARKER: dmesg [ 2734.674940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 16:47:16 \(1491410836\) [ 2734.862275] Lustre: DEBUG MARKER: == sanity test 65ib: getstripe -v on -1 default directory striping =================================== 16:47:16 (1491410836) [ 2735.432621] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2736.102738] Lustre: DEBUG MARKER: rc=0; [ 2736.102738] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2736.102738] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2736.102738] echo $(hostname -s): $val; [ 2736.102738] rc=$val; [ 2736.102738] fi; [ 2736.102738] exit $rc [ 2736.809265] Lustre: DEBUG MARKER: dmesg [ 2737.272876] LustreError: 27688:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripecount': rc = -34 [ 2737.287243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_65ic skipping ALWAYS excluded test 65ic [ 2737.497510] Lustre: DEBUG MARKER: SKIP: sanity test_65ic skipping ALWAYS excluded test 65ic [ 2737.745393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65j: set default striping on root directory \(bug 6367\)================================= 16:47:19 \(1491410839\) [ 2737.944242] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)================================= 16:47:19 (1491410839) [ 2739.049409] Lustre: Modifying parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 2739.051798] Lustre: Skipped 5 previous similar messages [ 2739.248146] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2739.935382] Lustre: DEBUG MARKER: rc=0; [ 2739.935382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2739.935382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2739.935382] echo $(hostname -s): $val; [ 2739.935382] rc=$val; [ 2739.935382] fi; [ 2739.935382] exit $rc [ 2740.642978] Lustre: DEBUG MARKER: dmesg [ 2741.129721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 16:47:23 \(1491410843\) [ 2741.321573] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ==================== 16:47:23 (1491410843) [ 2741.513326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2741.818208] Lustre: DEBUG MARKER: lctl dl [ 2742.111679] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 2742.258917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2742.263283] Lustre: Skipped 1 previous similar message [ 2742.272491] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 2742.274754] Lustre: Skipped 8 previous similar messages [ 2742.422371] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 2742.726369] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2744.604461] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 deactivate [ 2744.760240] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2744.915211] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000*MDT0000.create_count [ 2745.224074] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000*MDT0000.max_create_count [ 2745.522041] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.max_create_count=0 [ 2745.566054] LustreError: 28741:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 2745.570042] LustreError: 28741:0:(obd_config.c:1393:class_process_proc_param()) Skipped 1 previous similar message [ 2747.517458] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.max_create_count=20000 [ 2747.813504] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0000*MDT0000.create_count=32 [ 2748.106862] Lustre: DEBUG MARKER: lctl --device %lustre-OST0000-osc-MDT0000 activate [ 2748.253787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2748.258499] Lustre: Skipped 1 previous similar message [ 2748.265949] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.427079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2748.735272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2749.038783] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2749.334215] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 2749.630085] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2750.427462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.430547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.630196] Lustre: DEBUG MARKER: osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.637747] Lustre: DEBUG MARKER: osc.*OST*-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.806771] Lustre: DEBUG MARKER: lctl get_param -n lov.*md*.target_obd [ 2752.771604] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 2752.919347] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2753.069554] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0001*MDT0000.create_count [ 2753.365819] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0001*MDT0000.max_create_count [ 2753.659845] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.max_create_count=0 [ 2755.731620] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.max_create_count=20000 [ 2756.024756] Lustre: DEBUG MARKER: lctl set_param -n osp.lustre-OST0001*MDT0000.create_count=32 [ 2756.318528] Lustre: DEBUG MARKER: lctl --device %lustre-OST0001-osc-MDT0000 activate [ 2756.465067] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2756.472483] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2756.693776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2757.003652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2757.311406] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2757.611023] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 2757.908981] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2758.439449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2758.439450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2758.632529] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2758.632778] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2758.852439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2759.548946] Lustre: DEBUG MARKER: rc=0; [ 2759.548946] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2759.548946] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2759.548946] echo $(hostname -s): $val; [ 2759.548946] rc=$val; [ 2759.548946] fi; [ 2759.548946] exit $rc [ 2760.254801] Lustre: DEBUG MARKER: dmesg [ 2760.770090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65l: lfs find on -1 stripe dir ======================================================== 16:47:42 \(1491410862\) [ 2760.949988] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ======================================================== 16:47:42 (1491410862) [ 2761.187343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2761.869341] Lustre: DEBUG MARKER: rc=0; [ 2761.869341] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2761.869341] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2761.869341] echo $(hostname -s): $val; [ 2761.869341] rc=$val; [ 2761.869341] fi; [ 2761.869341] exit $rc [ 2762.580710] Lustre: DEBUG MARKER: dmesg [ 2763.048675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 65m: normal user can\'t set filesystem default stripe ================================== 16:47:45 \(1491410865\) [ 2763.237945] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ================================== 16:47:45 (1491410865) [ 2763.457727] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2764.147086] Lustre: DEBUG MARKER: rc=0; [ 2764.147086] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2764.147086] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2764.147086] echo $(hostname -s): $val; [ 2764.147086] rc=$val; [ 2764.147086] fi; [ 2764.147086] exit $rc [ 2764.847293] Lustre: DEBUG MARKER: dmesg [ 2765.374863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 66: update inode blocks count on client =============================================== 16:47:47 \(1491410867\) [ 2765.563094] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client =============================================== 16:47:47 (1491410867) [ 2765.823492] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 2766.716413] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 2767.467942] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2768.135187] Lustre: DEBUG MARKER: rc=0; [ 2768.135187] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2768.135187] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2768.135187] echo $(hostname -s): $val; [ 2768.135187] rc=$val; [ 2768.135187] fi; [ 2768.135187] exit $rc [ 2768.840850] Lustre: DEBUG MARKER: dmesg [ 2769.306474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 69: verify oa2dentry return -ENOENT doesn\'t LBUG ====================================== 16:47:51 \(1491410871\) [ 2769.523301] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ====================================== 16:47:51 (1491410871) [ 2772.541364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2773.240751] Lustre: DEBUG MARKER: rc=0; [ 2773.240751] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2773.240751] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2773.240751] echo $(hostname -s): $val; [ 2773.240751] rc=$val; [ 2773.240751] fi; [ 2773.240751] exit $rc [ 2773.939722] Lustre: DEBUG MARKER: dmesg [ 2774.413816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_71 skipping SLOW test 71 [ 2774.607018] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 2774.845737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72a: Test that remove suid works properly \(bug5695\) =================================== 16:47:56 \(1491410876\) [ 2775.037397] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) =================================== 16:47:56 (1491410876) [ 2775.421297] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2776.092621] Lustre: DEBUG MARKER: rc=0; [ 2776.092621] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2776.092621] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2776.092621] echo $(hostname -s): $val; [ 2776.092621] rc=$val; [ 2776.092621] fi; [ 2776.092621] exit $rc [ 2776.787298] Lustre: DEBUG MARKER: dmesg [ 2777.269707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 72b: Test that we keep mode setting if without file data changed \(bug 24226\) ========== 16:47:59 \(1491410879\) [ 2777.454414] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========== 16:47:59 (1491410879) [ 2778.225975] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2778.925984] Lustre: DEBUG MARKER: rc=0; [ 2778.925984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2778.925984] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2778.925984] echo $(hostname -s): $val; [ 2778.925984] rc=$val; [ 2778.925984] fi; [ 2778.925984] exit $rc [ 2779.623760] Lustre: DEBUG MARKER: dmesg [ 2780.090674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 73: multiple MDC requests \(should not deadlock\) ======================================= 16:48:02 \(1491410882\) [ 2780.284659] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ======================================= 16:48:02 (1491410882) [ 2806.557527] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2807.249729] Lustre: DEBUG MARKER: rc=0; [ 2807.249729] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2807.249729] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2807.249729] echo $(hostname -s): $val; [ 2807.249729] rc=$val; [ 2807.249729] fi; [ 2807.249729] exit $rc [ 2807.973718] Lustre: DEBUG MARKER: dmesg [ 2808.556376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74a: ldlm_enqueue freed-export error path, ls \(shouldn\'t LBUG\) ======================== 16:48:30 \(1491410910\) [ 2808.747549] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ======================== 16:48:30 (1491410910) [ 2808.971720] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2809.654557] Lustre: DEBUG MARKER: rc=0; [ 2809.654557] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2809.654557] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2809.654557] echo $(hostname -s): $val; [ 2809.654557] rc=$val; [ 2809.654557] fi; [ 2809.654557] exit $rc [ 2810.361915] Lustre: DEBUG MARKER: dmesg [ 2810.835791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74b: ldlm_enqueue freed-export error path, touch \(shouldn\'t LBUG\) ===================== 16:48:32 \(1491410912\) [ 2811.023078] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ===================== 16:48:32 (1491410912) [ 2811.259487] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2811.929006] Lustre: DEBUG MARKER: rc=0; [ 2811.929006] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2811.929006] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2811.929006] echo $(hostname -s): $val; [ 2811.929006] rc=$val; [ 2811.929006] fi; [ 2811.929006] exit $rc [ 2812.656617] Lustre: DEBUG MARKER: dmesg [ 2813.135285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 74c: ldlm_lock_create error path, \(shouldn\'t LBUG\) ==================================== 16:48:35 \(1491410915\) [ 2813.315492] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ==================================== 16:48:35 (1491410915) [ 2813.527414] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2814.199468] Lustre: DEBUG MARKER: rc=0; [ 2814.199468] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2814.199468] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2814.199468] echo $(hostname -s): $val; [ 2814.199468] rc=$val; [ 2814.199468] fi; [ 2814.199468] exit $rc [ 2814.917293] Lustre: DEBUG MARKER: dmesg [ 2815.391208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 2815.588443] Lustre: DEBUG MARKER: SKIP: sanity test_76 skipping ALWAYS excluded test 76 [ 2815.826696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77a: normal checksum read\/write operation ============================================= 16:48:37 \(1491410917\) [ 2816.015250] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ============================================= 16:48:37 (1491410917) [ 2817.036119] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2817.727687] Lustre: DEBUG MARKER: rc=0; [ 2817.727687] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2817.727687] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2817.727687] echo $(hostname -s): $val; [ 2817.727687] rc=$val; [ 2817.727687] fi; [ 2817.727687] exit $rc [ 2818.460871] Lustre: DEBUG MARKER: dmesg [ 2818.961958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77b: checksum error on client write, read ============================================= 16:48:40 \(1491410920\) [ 2819.167088] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ============================================= 16:48:40 (1491410920) [ 2820.633111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 2820.829335] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2822.446153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 2822.644873] Lustre: DEBUG MARKER: set checksum type to adler [ 2824.264202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 2824.454297] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2825.967665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 2826.160123] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2826.377638] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2827.049115] Lustre: DEBUG MARKER: rc=0; [ 2827.049115] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2827.049115] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2827.049115] echo $(hostname -s): $val; [ 2827.049115] rc=$val; [ 2827.049115] fi; [ 2827.049115] exit $rc [ 2827.783131] Lustre: DEBUG MARKER: dmesg [ 2828.284989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77d: checksum error on OST direct write, read ========================================= 16:48:50 \(1491410930\) [ 2828.480995] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================= 16:48:50 (1491410930) [ 2831.162111] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2831.842812] Lustre: DEBUG MARKER: rc=0; [ 2831.842812] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2831.842812] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2831.842812] echo $(hostname -s): $val; [ 2831.842812] rc=$val; [ 2831.842812] fi; [ 2831.842812] exit $rc [ 2832.553315] Lustre: DEBUG MARKER: dmesg [ 2833.045863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77f: repeat checksum error on write \(expect error\) ==================================== 16:48:55 \(1491410935\) [ 2833.244726] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ==================================== 16:48:55 (1491410935) [ 2833.491368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32 [ 2833.681704] Lustre: DEBUG MARKER: set checksum type to crc32 [ 2883.846929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to adler [ 2884.045656] Lustre: DEBUG MARKER: set checksum type to adler [ 2933.735282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 2933.918703] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2983.038630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark set checksum type to crc32c [ 2983.226513] Lustre: DEBUG MARKER: set checksum type to crc32c [ 2983.433589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2984.115413] Lustre: DEBUG MARKER: rc=0; [ 2984.115413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2984.115413] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2984.115413] echo $(hostname -s): $val; [ 2984.115413] rc=$val; [ 2984.115413] fi; [ 2984.115413] exit $rc [ 2984.831595] Lustre: DEBUG MARKER: dmesg [ 2985.311171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77g: checksum error on OST write, read ================================================ 16:51:27 \(1491411087\) [ 2985.501378] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ================================================ 16:51:27 (1491411087) [ 2989.829781] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2990.501138] Lustre: DEBUG MARKER: rc=0; [ 2990.501138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2990.501138] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2990.501138] echo $(hostname -s): $val; [ 2990.501138] rc=$val; [ 2990.501138] fi; [ 2990.501138] exit $rc [ 2991.201746] Lustre: DEBUG MARKER: dmesg [ 2991.680856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 77j: client only supporting ADLER32 =================================================== 16:51:33 \(1491411093\) [ 2991.874910] Lustre: DEBUG MARKER: == sanity test 77j: client only supporting ADLER32 =================================================== 16:51:33 (1491411093) [ 2993.693663] Lustre: MGS: Connection restored to e30d5548-457d-0a7a-0f95-9e322a62a804 (at 10.9.5.237@tcp) [ 2993.695887] Lustre: Skipped 3 previous similar messages [ 2994.111566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff88004e0a1800.ost_server_uuid in FULL state after 0 sec [ 2994.305238] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88004e0a1800.ost_server_uuid in FULL state after 0 sec [ 2994.693102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff88004e0a1800.ost_server_uuid in FULL state after 0 sec [ 2994.882818] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88004e0a1800.ost_server_uuid in FULL state after 0 sec [ 2995.301546] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2995.994113] Lustre: DEBUG MARKER: rc=0; [ 2995.994113] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2995.994113] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2995.994113] echo $(hostname -s): $val; [ 2995.994113] rc=$val; [ 2995.994113] fi; [ 2995.994113] exit $rc [ 2996.701340] Lustre: DEBUG MARKER: dmesg [ 2997.170952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 78: handle large O_DIRECT writes correctly ============================================ 16:51:39 \(1491411099\) [ 2997.362688] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ============================================ 16:51:39 (1491411099) [ 3003.937981] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3004.620523] Lustre: DEBUG MARKER: rc=0; [ 3004.620523] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3004.620523] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3004.620523] echo $(hostname -s): $val; [ 3004.620523] rc=$val; [ 3004.620523] fi; [ 3004.620523] exit $rc [ 3005.321892] Lustre: DEBUG MARKER: dmesg [ 3005.812229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 79: df report consistency check ======================================================= 16:51:47 \(1491411107\) [ 3006.005381] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ======================================================= 16:51:47 (1491411107) [ 3006.190731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3006.488894] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3006.870255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3008.199481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3009.526669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3010.839760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3012.167149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3018.568527] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3019.244165] Lustre: DEBUG MARKER: rc=0; [ 3019.244165] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3019.244165] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3019.244165] echo $(hostname -s): $val; [ 3019.244165] rc=$val; [ 3019.244165] fi; [ 3019.244165] exit $rc [ 3019.966338] Lustre: DEBUG MARKER: dmesg [ 3020.438828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 80: Page eviction is equally fast at high offsets too ================================ 16:52:02 \(1491411122\) [ 3020.635515] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ================================ 16:52:02 (1491411122) [ 3022.285800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3022.965641] Lustre: DEBUG MARKER: rc=0; [ 3022.965641] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3022.965641] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3022.965641] echo $(hostname -s): $val; [ 3022.965641] rc=$val; [ 3022.965641] fi; [ 3022.965641] exit $rc [ 3023.655100] Lustre: DEBUG MARKER: dmesg [ 3024.131620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 16:52:06 \(1491411126\) [ 3024.313158] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================== 16:52:06 (1491411126) [ 3025.173055] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3025.868908] Lustre: DEBUG MARKER: rc=0; [ 3025.868908] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3025.868908] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3025.868908] echo $(hostname -s): $val; [ 3025.868908] rc=$val; [ 3025.868908] fi; [ 3025.868908] exit $rc [ 3026.583376] Lustre: DEBUG MARKER: dmesg [ 3027.071562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 16:52:09 \(1491411129\) [ 3027.262828] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 16:52:09 (1491411129) [ 3027.988724] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3028.686704] Lustre: DEBUG MARKER: rc=0; [ 3028.686704] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3028.686704] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3028.686704] echo $(hostname -s): $val; [ 3028.686704] rc=$val; [ 3028.686704] fi; [ 3028.686704] exit $rc [ 3029.388080] Lustre: DEBUG MARKER: dmesg [ 3029.871803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 82: Basic grouplock test ============================================================== 16:52:11 \(1491411131\) [ 3030.064397] Lustre: DEBUG MARKER: == sanity test 82: Basic grouplock test ============================================================== 16:52:11 (1491411131) [ 3032.541626] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3033.224407] Lustre: DEBUG MARKER: rc=0; [ 3033.224407] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3033.224407] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3033.224407] echo $(hostname -s): $val; [ 3033.224407] rc=$val; [ 3033.224407] fi; [ 3033.224407] exit $rc [ 3033.927618] Lustre: DEBUG MARKER: dmesg [ 3034.394339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99a: cvs init ========================================================================= 16:52:16 \(1491411136\) [ 3034.594110] Lustre: DEBUG MARKER: == sanity test 99a: cvs init ========================================================================= 16:52:16 (1491411136) [ 3035.473578] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3036.158734] Lustre: DEBUG MARKER: rc=0; [ 3036.158734] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3036.158734] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3036.158734] echo $(hostname -s): $val; [ 3036.158734] rc=$val; [ 3036.158734] fi; [ 3036.158734] exit $rc [ 3036.842852] Lustre: DEBUG MARKER: dmesg [ 3037.354788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99b: cvs import ======================================================================= 16:52:19 \(1491411139\) [ 3037.549248] Lustre: DEBUG MARKER: == sanity test 99b: cvs import ======================================================================= 16:52:19 (1491411139) [ 3037.930596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3038.606401] Lustre: DEBUG MARKER: rc=0; [ 3038.606401] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3038.606401] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3038.606401] echo $(hostname -s): $val; [ 3038.606401] rc=$val; [ 3038.606401] fi; [ 3038.606401] exit $rc [ 3039.308613] Lustre: DEBUG MARKER: dmesg [ 3040.046935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99c: cvs checkout ===================================================================== 16:52:22 \(1491411142\) [ 3040.247089] Lustre: DEBUG MARKER: == sanity test 99c: cvs checkout ===================================================================== 16:52:22 (1491411142) [ 3040.992074] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3041.656277] Lustre: DEBUG MARKER: rc=0; [ 3041.656277] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3041.656277] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3041.656277] echo $(hostname -s): $val; [ 3041.656277] rc=$val; [ 3041.656277] fi; [ 3041.656277] exit $rc [ 3042.352495] Lustre: DEBUG MARKER: dmesg [ 3042.819502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99d: cvs add ========================================================================== 16:52:24 \(1491411144\) [ 3043.018174] Lustre: DEBUG MARKER: == sanity test 99d: cvs add ========================================================================== 16:52:24 (1491411144) [ 3043.266568] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3043.948197] Lustre: DEBUG MARKER: rc=0; [ 3043.948197] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3043.948197] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3043.948197] echo $(hostname -s): $val; [ 3043.948197] rc=$val; [ 3043.948197] fi; [ 3043.948197] exit $rc [ 3044.644983] Lustre: DEBUG MARKER: dmesg [ 3045.105735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99e: cvs update ======================================================================= 16:52:27 \(1491411147\) [ 3045.300125] Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 16:52:27 (1491411147) [ 3045.563018] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3046.257093] Lustre: DEBUG MARKER: rc=0; [ 3046.257093] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3046.257093] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3046.257093] echo $(hostname -s): $val; [ 3046.257093] rc=$val; [ 3046.257093] fi; [ 3046.257093] exit $rc [ 3046.968350] Lustre: DEBUG MARKER: dmesg [ 3047.460055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 99f: cvs commit ======================================================================= 16:52:29 \(1491411149\) [ 3047.647327] Lustre: DEBUG MARKER: == sanity test 99f: cvs commit ======================================================================= 16:52:29 (1491411149) [ 3049.033437] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3049.737007] Lustre: DEBUG MARKER: rc=0; [ 3049.737007] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3049.737007] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3049.737007] echo $(hostname -s): $val; [ 3049.737007] rc=$val; [ 3049.737007] fi; [ 3049.737007] exit $rc [ 3050.438121] Lustre: DEBUG MARKER: dmesg [ 3050.897565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 100: check local port using privileged port =========================================== 16:52:32 \(1491411152\) [ 3051.121735] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port =========================================== 16:52:32 (1491411152) [ 3051.870463] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3052.556356] Lustre: DEBUG MARKER: rc=0; [ 3052.556356] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3052.556356] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3052.556356] echo $(hostname -s): $val; [ 3052.556356] rc=$val; [ 3052.556356] fi; [ 3052.556356] exit $rc [ 3053.277883] Lustre: DEBUG MARKER: dmesg [ 3053.839822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101a: check read-ahead for random reads =============================================== 16:52:35 \(1491411155\) [ 3054.034594] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads =============================================== 16:52:35 (1491411155) [ 3079.242988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3079.932935] Lustre: DEBUG MARKER: rc=0; [ 3079.932935] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3079.932935] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3079.932935] echo $(hostname -s): $val; [ 3079.932935] rc=$val; [ 3079.932935] fi; [ 3079.932935] exit $rc [ 3080.640962] Lustre: DEBUG MARKER: dmesg [ 3081.173118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101b: check stride-io mode read-ahead ================================================= 16:53:03 \(1491411183\) [ 3081.368182] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead ================================================= 16:53:03 (1491411183) [ 3087.739923] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3088.439048] Lustre: DEBUG MARKER: rc=0; [ 3088.439048] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3088.439048] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3088.439048] echo $(hostname -s): $val; [ 3088.439048] rc=$val; [ 3088.439048] fi; [ 3088.439048] exit $rc [ 3089.142872] Lustre: DEBUG MARKER: dmesg [ 3089.618508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101c: check stripe_size aligned read-ahead ============================================ 16:53:11 \(1491411191\) [ 3089.807221] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ============================================ 16:53:11 (1491411191) [ 3097.285082] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3097.987503] Lustre: DEBUG MARKER: rc=0; [ 3097.987503] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3097.987503] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3097.987503] echo $(hostname -s): $val; [ 3097.987503] rc=$val; [ 3097.987503] fi; [ 3097.987503] exit $rc [ 3098.718471] Lustre: DEBUG MARKER: dmesg [ 3099.214604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101d: file read with and without read-ahead enabled =================================== 16:53:21 \(1491411201\) [ 3099.412208] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================================== 16:53:21 (1491411201) [ 3239.596707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3239.896385] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3240.936028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3242.248274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3243.570667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3244.889323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3251.253193] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3251.944709] Lustre: DEBUG MARKER: rc=0; [ 3251.944709] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3251.944709] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3251.944709] echo $(hostname -s): $val; [ 3251.944709] rc=$val; [ 3251.944709] fi; [ 3251.944709] exit $rc [ 3252.653199] Lustre: DEBUG MARKER: dmesg [ 3253.176700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101e: check read-ahead for small read\(1k\) for small files\(500k\) ======================= 16:55:55 \(1491411355\) [ 3253.371060] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ======================= 16:55:55 (1491411355) [ 3261.133507] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3261.809931] Lustre: DEBUG MARKER: rc=0; [ 3261.809931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3261.809931] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3261.809931] echo $(hostname -s): $val; [ 3261.809931] rc=$val; [ 3261.809931] fi; [ 3261.809931] exit $rc [ 3262.517403] Lustre: DEBUG MARKER: dmesg [ 3262.996643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101f: check mmap read performance ===================================================== 16:56:05 \(1491411365\) [ 3263.195531] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ===================================================== 16:56:05 (1491411365) [ 3276.073448] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3276.764860] Lustre: DEBUG MARKER: rc=0; [ 3276.764860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3276.764860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3276.764860] echo $(hostname -s): $val; [ 3276.764860] rc=$val; [ 3276.764860] fi; [ 3276.764860] exit $rc [ 3277.466628] Lustre: DEBUG MARKER: dmesg [ 3277.941107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101g: Big bulk\(4\/16 MiB\) readahead ==================================================== 16:56:19 \(1491411379\) [ 3278.128293] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ==================================================== 16:56:19 (1491411379) [ 3280.385039] Lustre: MGS: Connection restored to e30d5548-457d-0a7a-0f95-9e322a62a804 (at 10.9.5.237@tcp) [ 3280.387131] Lustre: Skipped 3 previous similar messages [ 3295.155274] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3295.837825] Lustre: DEBUG MARKER: rc=0; [ 3295.837825] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3295.837825] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3295.837825] echo $(hostname -s): $val; [ 3295.837825] rc=$val; [ 3295.837825] fi; [ 3295.837825] exit $rc [ 3296.535455] Lustre: DEBUG MARKER: dmesg [ 3297.020574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102a: user xattr test ================================================================= 16:56:39 \(1491411399\) [ 3297.212830] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ================================================================= 16:56:39 (1491411399) [ 3297.613035] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3298.297343] Lustre: DEBUG MARKER: rc=0; [ 3298.297343] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3298.297343] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3298.297343] echo $(hostname -s): $val; [ 3298.297343] rc=$val; [ 3298.297343] fi; [ 3298.297343] exit $rc [ 3299.021820] Lustre: DEBUG MARKER: dmesg [ 3299.587912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102b: getfattr\/setfattr for trusted.lov EAs =========================================== 16:56:41 \(1491411401\) [ 3299.799836] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs =========================================== 16:56:41 (1491411401) [ 3300.030710] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3300.691851] Lustre: DEBUG MARKER: rc=0; [ 3300.691851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3300.691851] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3300.691851] echo $(hostname -s): $val; [ 3300.691851] rc=$val; [ 3300.691851] fi; [ 3300.691851] exit $rc [ 3301.404826] Lustre: DEBUG MARKER: dmesg [ 3301.888034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102c: non-root getfattr\/setfattr for lustre.lov EAs =================================== 16:56:43 \(1491411403\) [ 3302.081778] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs =================================== 16:56:43 (1491411403) [ 3302.416274] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3303.083132] Lustre: DEBUG MARKER: rc=0; [ 3303.083132] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3303.083132] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3303.083132] echo $(hostname -s): $val; [ 3303.083132] rc=$val; [ 3303.083132] fi; [ 3303.083132] exit $rc [ 3303.781445] Lustre: DEBUG MARKER: dmesg [ 3304.277564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 16:56:46 \(1491411406\) [ 3304.472722] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ============================== 16:56:46 (1491411406) [ 3304.703986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102d lustre-aware tar is not installed [ 3304.896618] Lustre: DEBUG MARKER: SKIP: sanity test_102d lustre-aware tar is not installed [ 3305.108851] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3305.790259] Lustre: DEBUG MARKER: rc=0; [ 3305.790259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3305.790259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3305.790259] echo $(hostname -s): $val; [ 3305.790259] rc=$val; [ 3305.790259] fi; [ 3305.790259] exit $rc [ 3306.491288] Lustre: DEBUG MARKER: dmesg [ 3306.984523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102f: tar copy files, not keep osts =================================================== 16:56:48 \(1491411408\) [ 3307.184667] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts =================================================== 16:56:48 (1491411408) [ 3307.416763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102f lustre-aware tar is not installed [ 3307.614898] Lustre: DEBUG MARKER: SKIP: sanity test_102f lustre-aware tar is not installed [ 3307.855556] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3308.520032] Lustre: DEBUG MARKER: rc=0; [ 3308.520032] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3308.520032] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3308.520032] echo $(hostname -s): $val; [ 3308.520032] rc=$val; [ 3308.520032] fi; [ 3308.520032] exit $rc [ 3309.251032] Lustre: DEBUG MARKER: dmesg [ 3309.786871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102h: grow xattr from inside inode to external block ================================== 16:56:51 \(1491411411\) [ 3309.964731] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ================================== 16:56:51 (1491411411) [ 3310.202415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102h.sanity [ 3310.396606] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 3310.626152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3310.807612] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 3311.042980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102h.sanity [ 3311.244776] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 3311.479081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trusted.big still valid after growing trusted.sml [ 3311.666547] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3311.876439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3312.563426] Lustre: DEBUG MARKER: rc=0; [ 3312.563426] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3312.563426] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3312.563426] echo $(hostname -s): $val; [ 3312.563426] rc=$val; [ 3312.563426] fi; [ 3312.563426] exit $rc [ 3313.274715] Lustre: DEBUG MARKER: dmesg [ 3313.756681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102ha: grow xattr from inside inode to external inode ================================= 16:56:55 \(1491411415\) [ 3313.945039] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ================================= 16:56:55 (1491411415) [ 3314.204354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.big on \/mnt\/lustre\/f102ha.sanity [ 3314.386557] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 3314.621330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark save trusted.sml on \/mnt\/lustre\/f102ha.sanity [ 3314.809107] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 3315.042569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark grow trusted.sml on \/mnt\/lustre\/f102ha.sanity [ 3315.230629] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 3315.467244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trusted.big still valid after growing trusted.sml [ 3315.655874] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 3315.871674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3316.552328] Lustre: DEBUG MARKER: rc=0; [ 3316.552328] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3316.552328] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3316.552328] echo $(hostname -s): $val; [ 3316.552328] rc=$val; [ 3316.552328] fi; [ 3316.552328] exit $rc [ 3317.251778] Lustre: DEBUG MARKER: dmesg [ 3317.729701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102i: lgetxattr test on symbolic link ================================================= 16:56:59 \(1491411419\) [ 3317.928142] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ================================================= 16:56:59 (1491411419) [ 3318.157912] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3318.837432] Lustre: DEBUG MARKER: rc=0; [ 3318.837432] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3318.837432] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3318.837432] echo $(hostname -s): $val; [ 3318.837432] rc=$val; [ 3318.837432] fi; [ 3318.837432] exit $rc [ 3319.536574] Lustre: DEBUG MARKER: dmesg [ 3320.067586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 16:57:02 \(1491411422\) [ 3320.263296] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ==================== 16:57:02 (1491411422) [ 3320.488597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_102j lustre-aware tar is not installed [ 3320.683702] Lustre: DEBUG MARKER: SKIP: sanity test_102j lustre-aware tar is not installed [ 3320.907549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3321.595234] Lustre: DEBUG MARKER: rc=0; [ 3321.595234] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3321.595234] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3321.595234] echo $(hostname -s): $val; [ 3321.595234] rc=$val; [ 3321.595234] fi; [ 3321.595234] exit $rc [ 3322.295764] Lustre: DEBUG MARKER: dmesg [ 3322.773240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102k: setfattr without parameter of value shouldn\'t cause a crash ===================== 16:57:04 \(1491411424\) [ 3322.970721] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ===================== 16:57:04 (1491411424) [ 3323.238124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3323.928398] Lustre: DEBUG MARKER: rc=0; [ 3323.928398] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3323.928398] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3323.928398] echo $(hostname -s): $val; [ 3323.928398] rc=$val; [ 3323.928398] fi; [ 3323.928398] exit $rc [ 3324.631619] Lustre: DEBUG MARKER: dmesg [ 3325.105763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102l: listxattr size test ============================================================= 16:57:07 \(1491411427\) [ 3325.305854] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================= 16:57:07 (1491411427) [ 3325.539708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3326.223798] Lustre: DEBUG MARKER: rc=0; [ 3326.223798] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3326.223798] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3326.223798] echo $(hostname -s): $val; [ 3326.223798] rc=$val; [ 3326.223798] fi; [ 3326.223798] exit $rc [ 3326.935412] Lustre: DEBUG MARKER: dmesg [ 3327.415530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 16:57:09 \(1491411429\) [ 3327.606093] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ========================================= 16:57:09 (1491411429) [ 3327.880124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3328.576502] Lustre: DEBUG MARKER: rc=0; [ 3328.576502] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3328.576502] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3328.576502] echo $(hostname -s): $val; [ 3328.576502] rc=$val; [ 3328.576502] fi; [ 3328.576502] exit $rc [ 3329.273220] Lustre: DEBUG MARKER: dmesg [ 3329.765941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 16:57:11 \(1491411431\) [ 3329.966545] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ============================= 16:57:11 (1491411431) [ 3330.136524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3330.448704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3330.800045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3331.169105] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3331.848252] Lustre: DEBUG MARKER: rc=0; [ 3331.848252] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3331.848252] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3331.848252] echo $(hostname -s): $val; [ 3331.848252] rc=$val; [ 3331.848252] fi; [ 3331.848252] exit $rc [ 3332.561542] Lustre: DEBUG MARKER: dmesg [ 3333.036654] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102p: check setxattr\(2\) correctly fails without permission ============================ 16:57:15 \(1491411435\) [ 3333.235492] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ============================ 16:57:15 (1491411435) [ 3333.408108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3333.813902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3334.489245] Lustre: DEBUG MARKER: rc=0; [ 3334.489245] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3334.489245] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3334.489245] echo $(hostname -s): $val; [ 3334.489245] rc=$val; [ 3334.489245] fi; [ 3334.489245] exit $rc [ 3335.209698] Lustre: DEBUG MARKER: dmesg [ 3335.687426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 16:57:17 \(1491411437\) [ 3335.918239] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ======================= 16:57:17 (1491411437) [ 3336.100404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3336.503157] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3337.185134] Lustre: DEBUG MARKER: rc=0; [ 3337.185134] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3337.185134] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3337.185134] echo $(hostname -s): $val; [ 3337.185134] rc=$val; [ 3337.185134] fi; [ 3337.185134] exit $rc [ 3337.890315] Lustre: DEBUG MARKER: dmesg [ 3338.374216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 102r: set EAs with empty values ======================================================= 16:57:20 \(1491411440\) [ 3338.561226] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values ======================================================= 16:57:20 (1491411440) [ 3338.734838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3339.128871] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3339.806815] Lustre: DEBUG MARKER: rc=0; [ 3339.806815] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3339.806815] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3339.806815] echo $(hostname -s): $val; [ 3339.806815] rc=$val; [ 3339.806815] fi; [ 3339.806815] exit $rc [ 3340.512413] Lustre: DEBUG MARKER: dmesg [ 3341.005124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103a: acl test ======================================================================== 16:57:23 \(1491411443\) [ 3341.196279] Lustre: DEBUG MARKER: == sanity test 103a: acl test ======================================================================== 16:57:23 (1491411443) [ 3341.442611] Lustre: DEBUG MARKER: gpasswd -a daemon bin [ 3341.923335] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [ 3342.221128] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [ 3342.513440] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall /usr/sbin/l_getidentity [ 3342.804210] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [ 3347.267696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3347.584653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3347.889989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3353.484542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3354.218666] Lustre: DEBUG MARKER: rc=0; [ 3354.218666] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3354.218666] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3354.218666] echo $(hostname -s): $val; [ 3354.218666] rc=$val; [ 3354.218666] fi; [ 3354.218666] exit $rc [ 3354.928932] Lustre: DEBUG MARKER: dmesg [ 3355.415437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103b: MDS mount option \'noacl\' ======================================================== 16:57:37 \(1491411457\) [ 3355.615936] Lustre: DEBUG MARKER: == sanity test 103b: MDS mount option 'noacl' ======================================================== 16:57:37 (1491411457) [ 3355.823307] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3356.118535] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3356.266986] Lustre: Failing over lustre-MDT0000 [ 3357.016986] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.020937] LustreError: Skipped 1 previous similar message [ 3358.285165] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.289243] LustreError: Skipped 1 previous similar message [ 3362.461039] Lustre: 18615:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491411458/real 1491411458] req@ffff880067b10300 x1563855295913152/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491411464 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3362.481503] Lustre: server umount lustre-MDT0000 complete [ 3362.627459] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3362.933190] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3362.933190] grep -q ^lustre-mdt1/ /proc/mounts || [ 3362.933190] zpool export lustre-mdt1 [ 3363.405898] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3363.699964] Lustre: DEBUG MARKER: modprobe zfs; [ 3363.699964] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3363.699964] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 3364.324569] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 3364.616022] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 3364.932332] LustreError: 20099:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3365.141759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.289199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3365.307350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3365.602495] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3366.098869] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3366.391152] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3366.686835] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3366.994318] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 3367.291560] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3371.088184] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3371.261896] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3371.562408] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3371.719588] Lustre: Failing over lustre-MDT0000 [ 3375.297289] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.840055] Lustre: 20887:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491411473/real 1491411473] req@ffff88005e03d800 x1563855295930048/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491411479 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3377.861029] Lustre: server umount lustre-MDT0000 complete [ 3378.013349] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3378.330297] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3378.330297] grep -q ^lustre-mdt1/ /proc/mounts || [ 3378.330297] zpool export lustre-mdt1 [ 3378.743105] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3379.048120] Lustre: DEBUG MARKER: modprobe zfs; [ 3379.048120] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3379.048120] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 3379.674970] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 3379.993630] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 3380.336065] LustreError: 22368:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 3380.383684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.547530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3380.844165] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3381.357529] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3381.653499] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3381.950383] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3382.255113] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 3382.552239] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3382.913981] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3383.629898] Lustre: DEBUG MARKER: rc=0; [ 3383.629898] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3383.629898] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3383.629898] echo $(hostname -s): $val; [ 3383.629898] rc=$val; [ 3383.629898] fi; [ 3383.629898] exit $rc [ 3384.363490] Lustre: DEBUG MARKER: dmesg [ 3384.537477] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3384.871780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 103c: \'cp -rp\' won\'t set empty acl ==================================================== 16:58:06 \(1491411486\) [ 3385.018396] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3385.129900] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ==================================================== 16:58:06 (1491411486) [ 3385.451498] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3386.170119] Lustre: DEBUG MARKER: rc=0; [ 3386.170119] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3386.170119] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3386.170119] echo $(hostname -s): $val; [ 3386.170119] rc=$val; [ 3386.170119] fi; [ 3386.170119] exit $rc [ 3386.889546] Lustre: DEBUG MARKER: dmesg [ 3387.401423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104a: lfs df [-ih] [path] test ======================================================== 16:58:09 \(1491411489\) [ 3387.601144] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test ======================================================== 16:58:09 (1491411489) [ 3388.040131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880067df6800.ost_server_uuid in FULL state after 0 sec [ 3388.227593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880067df6800.ost_server_uuid in FULL state after 0 sec [ 3388.449427] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3389.145216] Lustre: DEBUG MARKER: rc=0; [ 3389.145216] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3389.145216] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3389.145216] echo $(hostname -s): $val; [ 3389.145216] rc=$val; [ 3389.145216] fi; [ 3389.145216] exit $rc [ 3389.845932] Lustre: DEBUG MARKER: dmesg [ 3390.332634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 16:58:12 \(1491411492\) [ 3390.533274] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ====================================== 16:58:12 (1491411492) [ 3390.764113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3391.459330] Lustre: DEBUG MARKER: rc=0; [ 3391.459330] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3391.459330] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3391.459330] echo $(hostname -s): $val; [ 3391.459330] rc=$val; [ 3391.459330] fi; [ 3391.459330] exit $rc [ 3392.157376] Lustre: DEBUG MARKER: dmesg [ 3392.701782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105a: flock when mounted without -o flock test ======================================== 16:58:14 \(1491411494\) [ 3392.909747] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ======================================== 16:58:14 (1491411494) [ 3393.174339] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3393.869623] Lustre: DEBUG MARKER: rc=0; [ 3393.869623] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3393.869623] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3393.869623] echo $(hostname -s): $val; [ 3393.869623] rc=$val; [ 3393.869623] fi; [ 3393.869623] exit $rc [ 3394.581985] Lustre: DEBUG MARKER: dmesg [ 3395.060050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105b: fcntl when mounted without -o flock test ======================================== 16:58:17 \(1491411497\) [ 3395.256756] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ======================================== 16:58:17 (1491411497) [ 3395.482704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3396.154252] Lustre: DEBUG MARKER: rc=0; [ 3396.154252] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3396.154252] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3396.154252] echo $(hostname -s): $val; [ 3396.154252] rc=$val; [ 3396.154252] fi; [ 3396.154252] exit $rc [ 3396.858795] Lustre: DEBUG MARKER: dmesg [ 3397.353743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105c: lockf when mounted without -o flock test ======================================== 16:58:19 \(1491411499\) [ 3397.541336] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ======================================== 16:58:19 (1491411499) [ 3397.774972] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3398.444918] Lustre: DEBUG MARKER: rc=0; [ 3398.444918] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3398.444918] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3398.444918] echo $(hostname -s): $val; [ 3398.444918] rc=$val; [ 3398.444918] fi; [ 3398.444918] exit $rc [ 3399.141916] Lustre: DEBUG MARKER: dmesg [ 3399.620253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105d: flock race \(should not freeze\) ================================================== 16:58:21 \(1491411501\) [ 3399.810474] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================== 16:58:21 (1491411501) [ 3410.051298] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3410.742267] Lustre: DEBUG MARKER: rc=0; [ 3410.742267] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3410.742267] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3410.742267] echo $(hostname -s): $val; [ 3410.742267] rc=$val; [ 3410.742267] fi; [ 3410.742267] exit $rc [ 3411.435087] Lustre: DEBUG MARKER: dmesg [ 3411.927345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 105e: Two conflicting flocks from same process ======================================== 16:58:33 \(1491411513\) [ 3412.118075] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ======================================== 16:58:33 (1491411513) [ 3412.343831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3413.018421] Lustre: DEBUG MARKER: rc=0; [ 3413.018421] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3413.018421] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3413.018421] echo $(hostname -s): $val; [ 3413.018421] rc=$val; [ 3413.018421] fi; [ 3413.018421] exit $rc [ 3413.729933] Lustre: DEBUG MARKER: dmesg [ 3414.209124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 16:58:36 \(1491411516\) [ 3414.404697] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ================================ 16:58:36 (1491411516) [ 3414.652757] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3415.352280] Lustre: DEBUG MARKER: rc=0; [ 3415.352280] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3415.352280] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3415.352280] echo $(hostname -s): $val; [ 3415.352280] rc=$val; [ 3415.352280] fi; [ 3415.352280] exit $rc [ 3416.055839] Lustre: DEBUG MARKER: dmesg [ 3416.563161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 107: Coredump on SIG ================================================================== 16:58:38 \(1491411518\) [ 3416.752319] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ================================================================== 16:58:38 (1491411518) [ 3418.016226] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3418.704504] Lustre: DEBUG MARKER: rc=0; [ 3418.704504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3418.704504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3418.704504] echo $(hostname -s): $val; [ 3418.704504] rc=$val; [ 3418.704504] fi; [ 3418.704504] exit $rc [ 3419.419264] Lustre: DEBUG MARKER: dmesg [ 3419.933236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 110: filename length checking ========================================================= 16:58:41 \(1491411521\) [ 3420.120551] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ========================================================= 16:58:41 (1491411521) [ 3420.402769] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3421.094910] Lustre: DEBUG MARKER: rc=0; [ 3421.094910] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3421.094910] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3421.094910] echo $(hostname -s): $val; [ 3421.094910] rc=$val; [ 3421.094910] fi; [ 3421.094910] exit $rc [ 3421.804340] Lustre: DEBUG MARKER: dmesg [ 3422.288660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_115 skipping SLOW test 115 [ 3422.494766] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 3422.727871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116a: stripe QOS: free space balance ================================================== 16:58:44 \(1491411524\) [ 3422.919452] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ================================================== 16:58:44 (1491411524) [ 3423.106127] Lustre: DEBUG MARKER: lctl get_param -n lo*.*-mdtlov.qos_prio_free [ 3423.411603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3423.716239] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3424.157800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3425.479734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3426.841741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3428.166266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3429.481985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3430.801400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3437.138838] Lustre: DEBUG MARKER: lctl get_param -n *.*MDT0000-mdtlov.qos_threshold_rr [ 3550.402334] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3560.709862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3573.504868] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 3583.812228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3591.863300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3592.159199] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3592.654659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3594.012953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3595.329168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3596.643739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3597.964890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3599.286829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3600.614213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3601.931644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3608.321928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3609.076550] Lustre: DEBUG MARKER: rc=0; [ 3609.076550] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3609.076550] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3609.076550] echo $(hostname -s): $val; [ 3609.076550] rc=$val; [ 3609.076550] fi; [ 3609.076550] exit $rc [ 3609.815874] Lustre: DEBUG MARKER: dmesg [ 3610.394810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 116b: QoS shouldn\'t LBUG if not enough OSTs found on the 2nd pass ===================== 17:01:52 \(1491411712\) [ 3610.664070] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ===================== 17:01:52 (1491411712) [ 3610.847455] Lustre: DEBUG MARKER: lctl get_param -n lo*.lustre-MDT0000-mdtlov.qos_threshold_rr [ 3611.159570] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=0 [ 3611.457728] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x147 [ 3611.612491] Lustre: *** cfs_fail_loc=147, val=0*** [ 3611.786944] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3612.150534] Lustre: DEBUG MARKER: lctl set_param lo*.lustre-MDT0000-mdtlov.qos_threshold_rr=17% [ 3612.492544] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3613.187279] Lustre: DEBUG MARKER: rc=0; [ 3613.187279] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3613.187279] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3613.187279] echo $(hostname -s): $val; [ 3613.187279] rc=$val; [ 3613.187279] fi; [ 3613.187279] exit $rc [ 3613.890778] Lustre: DEBUG MARKER: dmesg [ 3614.377957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 117: verify osd extend ================================================================ 17:01:56 \(1491411716\) [ 3614.556413] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ================================================================ 17:01:56 (1491411716) [ 3614.812897] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3615.475642] Lustre: DEBUG MARKER: rc=0; [ 3615.475642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3615.475642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3615.475642] echo $(hostname -s): $val; [ 3615.475642] rc=$val; [ 3615.475642] fi; [ 3615.475642] exit $rc [ 3616.165606] Lustre: DEBUG MARKER: dmesg [ 3616.640581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118a: verify O_SYNC works ============================================================= 17:01:58 \(1491411718\) [ 3616.833760] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ============================================================= 17:01:58 (1491411718) [ 3617.429881] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3618.095916] Lustre: DEBUG MARKER: rc=0; [ 3618.095916] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3618.095916] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3618.095916] echo $(hostname -s): $val; [ 3618.095916] rc=$val; [ 3618.095916] fi; [ 3618.095916] exit $rc [ 3618.842263] Lustre: DEBUG MARKER: dmesg [ 3619.330374] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 17:02:01 \(1491411721\) [ 3619.574654] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ============================================== 17:02:01 (1491411721) [ 3621.060357] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3621.733955] Lustre: DEBUG MARKER: rc=0; [ 3621.733955] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3621.733955] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3621.733955] echo $(hostname -s): $val; [ 3621.733955] rc=$val; [ 3621.733955] fi; [ 3621.733955] exit $rc [ 3622.458768] Lustre: DEBUG MARKER: dmesg [ 3622.921613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 17:02:04 \(1491411724\) [ 3623.113633] Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============================= 17:02:04 (1491411724) [ 3628.302164] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3628.974209] Lustre: DEBUG MARKER: rc=0; [ 3628.974209] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3628.974209] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3628.974209] echo $(hostname -s): $val; [ 3628.974209] rc=$val; [ 3628.974209] fi; [ 3628.974209] exit $rc [ 3629.705717] Lustre: DEBUG MARKER: dmesg [ 3630.190210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 17:02:12 \(1491411732\) [ 3630.378634] Lustre: DEBUG MARKER: == sanity test 118d: Fsync validation inject a delay of the bulk ===================================== 17:02:12 (1491411732) [ 3636.606130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3637.277442] Lustre: DEBUG MARKER: rc=0; [ 3637.277442] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3637.277442] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3637.277442] echo $(hostname -s): $val; [ 3637.277442] rc=$val; [ 3637.277442] fi; [ 3637.277442] exit $rc [ 3637.983542] Lustre: DEBUG MARKER: dmesg [ 3638.460854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 17:02:20 \(1491411740\) [ 3638.645297] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error =========================================== 17:02:20 (1491411740) [ 3639.194802] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3639.862673] Lustre: DEBUG MARKER: rc=0; [ 3639.862673] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3639.862673] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3639.862673] echo $(hostname -s): $val; [ 3639.862673] rc=$val; [ 3639.862673] fi; [ 3639.862673] exit $rc [ 3640.552040] Lustre: DEBUG MARKER: dmesg [ 3641.044073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118g: Don\'t stay in wait if we got local -ENOMEM ===================================== 17:02:23 \(1491411743\) [ 3641.226526] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ===================================== 17:02:23 (1491411743) [ 3641.795777] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3642.460155] Lustre: DEBUG MARKER: rc=0; [ 3642.460155] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3642.460155] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3642.460155] echo $(hostname -s): $val; [ 3642.460155] rc=$val; [ 3642.460155] fi; [ 3642.460155] exit $rc [ 3643.237635] Lustre: DEBUG MARKER: dmesg [ 3643.706836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118h: Verify timeout in handling recoverables errors ================================= 17:02:25 \(1491411745\) [ 3643.907725] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ================================= 17:02:25 (1491411745) [ 3654.670594] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3655.334608] Lustre: DEBUG MARKER: rc=0; [ 3655.334608] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3655.334608] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3655.334608] echo $(hostname -s): $val; [ 3655.334608] rc=$val; [ 3655.334608] fi; [ 3655.334608] exit $rc [ 3656.049921] Lustre: DEBUG MARKER: dmesg [ 3656.529588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118i: Fix error before timeout in recoverable error ================================== 17:02:38 \(1491411758\) [ 3656.730404] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ================================== 17:02:38 (1491411758) [ 3663.304610] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3663.972917] Lustre: DEBUG MARKER: rc=0; [ 3663.972917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3663.972917] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3663.972917] echo $(hostname -s): $val; [ 3663.972917] rc=$val; [ 3663.972917] fi; [ 3663.972917] exit $rc [ 3664.651955] Lustre: DEBUG MARKER: dmesg [ 3665.205706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118j: Simulate unrecoverable OST side error =========================================== 17:02:47 \(1491411767\) [ 3665.402710] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error =========================================== 17:02:47 (1491411767) [ 3666.734737] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3667.410919] Lustre: DEBUG MARKER: rc=0; [ 3667.410919] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3667.410919] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3667.410919] echo $(hostname -s): $val; [ 3667.410919] rc=$val; [ 3667.410919] fi; [ 3667.410919] exit $rc [ 3668.127326] Lustre: DEBUG MARKER: dmesg [ 3668.618343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 17:02:50 \(1491411770\) [ 3668.816811] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling ========================================== 17:02:50 (1491411770) [ 3674.866150] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3676.069104] Lustre: DEBUG MARKER: rc=0; [ 3676.069104] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3676.069104] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3676.069104] echo $(hostname -s): $val; [ 3676.069104] rc=$val; [ 3676.069104] fi; [ 3676.069104] exit $rc [ 3677.195571] Lustre: DEBUG MARKER: dmesg [ 3677.676439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118l: fsync dir ======================================================================= 17:02:59 \(1491411779\) [ 3677.880812] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir ======================================================================= 17:02:59 (1491411779) [ 3678.173475] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3678.920570] Lustre: DEBUG MARKER: rc=0; [ 3678.920570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3678.920570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3678.920570] echo $(hostname -s): $val; [ 3678.920570] rc=$val; [ 3678.920570] fi; [ 3678.920570] exit $rc [ 3679.656346] Lustre: DEBUG MARKER: dmesg [ 3680.157816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 118m: fdatasync dir =================================================================== 17:03:02 \(1491411782\) [ 3680.356858] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir =================================================================== 17:03:02 (1491411782) [ 3680.701803] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3681.379985] Lustre: DEBUG MARKER: rc=0; [ 3681.379985] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3681.379985] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3681.379985] echo $(hostname -s): $val; [ 3681.379985] rc=$val; [ 3681.379985] fi; [ 3681.379985] exit $rc [ 3682.088605] Lustre: DEBUG MARKER: dmesg [ 3682.624250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119a: Short directIO read must return actual read amount ============================== 17:03:04 \(1491411784\) [ 3682.815419] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ============================== 17:03:04 (1491411784) [ 3683.183000] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3683.877896] Lustre: DEBUG MARKER: rc=0; [ 3683.877896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3683.877896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3683.877896] echo $(hostname -s): $val; [ 3683.877896] rc=$val; [ 3683.877896] fi; [ 3683.877896] exit $rc [ 3684.574205] Lustre: DEBUG MARKER: dmesg [ 3685.053053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119b: Sparse directIO read must return actual read amount ============================= 17:03:07 \(1491411787\) [ 3685.247472] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ============================= 17:03:07 (1491411787) [ 3685.615935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3686.281981] Lustre: DEBUG MARKER: rc=0; [ 3686.281981] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3686.281981] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3686.281981] echo $(hostname -s): $val; [ 3686.281981] rc=$val; [ 3686.281981] fi; [ 3686.281981] exit $rc [ 3686.976287] Lustre: DEBUG MARKER: dmesg [ 3687.470301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119c: Testing for direct read hitting hole ============================================ 17:03:09 \(1491411789\) [ 3687.663474] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ============================================ 17:03:09 (1491411789) [ 3688.059438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3688.728711] Lustre: DEBUG MARKER: rc=0; [ 3688.728711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3688.728711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3688.728711] echo $(hostname -s): $val; [ 3688.728711] rc=$val; [ 3688.728711] fi; [ 3688.728711] exit $rc [ 3689.424366] Lustre: DEBUG MARKER: dmesg [ 3689.913142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 17:03:11 \(1491411791\) [ 3690.099858] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ================= 17:03:11 (1491411791) [ 3691.565983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark the DIO writes have completed, now wait for the reads \(should not block very long\) [ 3691.757850] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 3693.983018] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3694.662719] Lustre: DEBUG MARKER: rc=0; [ 3694.662719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3694.662719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3694.662719] echo $(hostname -s): $val; [ 3694.662719] rc=$val; [ 3694.662719] fi; [ 3694.662719] exit $rc [ 3695.367030] Lustre: DEBUG MARKER: dmesg [ 3695.867682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 17:03:17 \(1491411797\) [ 3696.061660] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test =================================================== 17:03:17 (1491411797) [ 3696.281363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3696.575174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3696.911806] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3697.580979] Lustre: DEBUG MARKER: rc=0; [ 3697.580979] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3697.580979] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3697.580979] echo $(hostname -s): $val; [ 3697.580979] rc=$val; [ 3697.580979] fi; [ 3697.580979] exit $rc [ 3698.264918] Lustre: DEBUG MARKER: dmesg [ 3698.784260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120b: Early Lock Cancel: create test ================================================== 17:03:20 \(1491411800\) [ 3699.008200] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ================================================== 17:03:20 (1491411800) [ 3699.227434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3699.539815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3699.907807] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3700.578634] Lustre: DEBUG MARKER: rc=0; [ 3700.578634] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3700.578634] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3700.578634] echo $(hostname -s): $val; [ 3700.578634] rc=$val; [ 3700.578634] fi; [ 3700.578634] exit $rc [ 3701.275559] Lustre: DEBUG MARKER: dmesg [ 3701.762556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120c: Early Lock Cancel: link test ==================================================== 17:03:23 \(1491411803\) [ 3701.957246] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ==================================================== 17:03:23 (1491411803) [ 3702.202867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3702.503402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3702.853957] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3703.548397] Lustre: DEBUG MARKER: rc=0; [ 3703.548397] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3703.548397] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3703.548397] echo $(hostname -s): $val; [ 3703.548397] rc=$val; [ 3703.548397] fi; [ 3703.548397] exit $rc [ 3704.244226] Lustre: DEBUG MARKER: dmesg [ 3704.731620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120d: Early Lock Cancel: setattr test ================================================= 17:03:26 \(1491411806\) [ 3704.929791] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ================================================= 17:03:26 (1491411806) [ 3705.149306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3705.455231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3705.814794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3706.482492] Lustre: DEBUG MARKER: rc=0; [ 3706.482492] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3706.482492] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3706.482492] echo $(hostname -s): $val; [ 3706.482492] rc=$val; [ 3706.482492] fi; [ 3706.482492] exit $rc [ 3707.174881] Lustre: DEBUG MARKER: dmesg [ 3707.650423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120e: Early Lock Cancel: unlink test ================================================== 17:03:29 \(1491411809\) [ 3707.855469] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ================================================== 17:03:29 (1491411809) [ 3710.115110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3715.439426] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3715.803570] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3716.465995] Lustre: DEBUG MARKER: rc=0; [ 3716.465995] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3716.465995] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3716.465995] echo $(hostname -s): $val; [ 3716.465995] rc=$val; [ 3716.465995] fi; [ 3716.465995] exit $rc [ 3717.156736] Lustre: DEBUG MARKER: dmesg [ 3717.731360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120f: Early Lock Cancel: rename test ================================================== 17:03:39 \(1491411819\) [ 3717.923410] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ================================================== 17:03:39 (1491411819) [ 3720.184463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3725.494192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3725.839912] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3726.511512] Lustre: DEBUG MARKER: rc=0; [ 3726.511512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3726.511512] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3726.511512] echo $(hostname -s): $val; [ 3726.511512] rc=$val; [ 3726.511512] fi; [ 3726.511512] exit $rc [ 3727.209125] Lustre: DEBUG MARKER: dmesg [ 3727.691355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 120g: Early Lock Cancel: performance test ============================================= 17:03:49 \(1491411829\) [ 3727.876042] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ============================================= 17:03:49 (1491411829) [ 3728.095340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3740.045056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3751.035578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats [ 3753.390512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3754.051849] Lustre: DEBUG MARKER: rc=0; [ 3754.051849] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3754.051849] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3754.051849] echo $(hostname -s): $val; [ 3754.051849] rc=$val; [ 3754.051849] fi; [ 3754.051849] exit $rc [ 3754.734448] Lustre: DEBUG MARKER: dmesg [ 3755.216663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 121: read cancel race ================================================================= 17:04:17 \(1491411857\) [ 3755.404467] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ================================================================= 17:04:17 (1491411857) [ 3755.661564] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3756.359161] Lustre: DEBUG MARKER: rc=0; [ 3756.359161] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3756.359161] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3756.359161] echo $(hostname -s): $val; [ 3756.359161] rc=$val; [ 3756.359161] fi; [ 3756.359161] exit $rc [ 3757.069164] Lustre: DEBUG MARKER: dmesg [ 3757.551837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123a: verify statahead work =========================================================== 17:04:19 \(1491411859\) [ 3757.748073] Lustre: DEBUG MARKER: == sanity test 123a: verify statahead work =========================================================== 17:04:19 (1491411859) [ 3758.213054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files without statahead: 0 sec [ 3758.393225] Lustre: DEBUG MARKER: ls 100 files without statahead: 0 sec [ 3758.685383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 100 files with statahead: 0 sec [ 3758.872246] Lustre: DEBUG MARKER: ls 100 files with statahead: 0 sec [ 3761.268153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files without statahead: 1 sec [ 3761.470089] Lustre: DEBUG MARKER: ls 1000 files without statahead: 1 sec [ 3762.111520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 1000 files with statahead: 1 sec [ 3762.310753] Lustre: DEBUG MARKER: ls 1000 files with statahead: 1 sec [ 3784.159416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files without statahead: 11 sec [ 3784.349184] Lustre: DEBUG MARKER: ls 10000 files without statahead: 11 sec [ 3787.749914] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls 10000 files with statahead: 3 sec [ 3787.943688] Lustre: DEBUG MARKER: ls 10000 files with statahead: 3 sec [ 3788.179249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 3788.370053] Lustre: DEBUG MARKER: ls done [ 3795.703637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm -r \/mnt\/lustre\/d123a.sanity\/: 7 seconds [ 3795.977145] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123a.sanity/: 7 seconds [ 3796.225719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark rm done [ 3796.417028] Lustre: DEBUG MARKER: rm done [ 3796.718221] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3797.494081] Lustre: DEBUG MARKER: rc=0; [ 3797.494081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3797.494081] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3797.494081] echo $(hostname -s): $val; [ 3797.494081] rc=$val; [ 3797.494081] fi; [ 3797.494081] exit $rc [ 3798.335809] Lustre: DEBUG MARKER: dmesg [ 3799.016201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 123b: not panic with network error in statahead enqueue \(bug 15027\) =================== 17:05:00 \(1491411900\) [ 3799.217216] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) =================== 17:05:00 (1491411900) [ 3801.761607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls done [ 3801.971616] Lustre: DEBUG MARKER: ls done [ 3803.025307] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3803.723896] Lustre: DEBUG MARKER: rc=0; [ 3803.723896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3803.723896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3803.723896] echo $(hostname -s): $val; [ 3803.723896] rc=$val; [ 3803.723896] fi; [ 3803.723896] exit $rc [ 3804.457999] Lustre: DEBUG MARKER: dmesg [ 3804.919569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124a: lru resize ====================================================================== 17:05:06 \(1491411906\) [ 3805.118885] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ====================================================================== 17:05:06 (1491411906) [ 3805.353721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create 2000 files at \/mnt\/lustre\/d124a.sanity [ 3805.538133] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 3808.696452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880067df6800 [ 3808.909880] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff880067df6800 [ 3809.107758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880067df6800 [ 3809.294470] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff880067df6800 [ 3809.540077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LRU=2003 [ 3809.749664] Lustre: DEBUG MARKER: LRU=2003 [ 3809.960195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LIMIT=83999 [ 3810.150072] Lustre: DEBUG MARKER: LIMIT=83999 [ 3810.365873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark LVF=50323 [ 3810.555084] Lustre: DEBUG MARKER: LVF=50323 [ 3810.780940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OLD_LVF=1 [ 3811.000883] Lustre: DEBUG MARKER: OLD_LVF=1 [ 3811.210226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Sleep 50 sec [ 3811.410729] Lustre: DEBUG MARKER: Sleep 50 sec [ 3861.674178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Dropped 697 locks in 50s [ 3861.876516] Lustre: DEBUG MARKER: Dropped 697 locks in 50s [ 3862.084842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark unlink 2000 files at \/mnt\/lustre\/d124a.sanity [ 3862.276322] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 3864.218334] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3864.912592] Lustre: DEBUG MARKER: rc=0; [ 3864.912592] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3864.912592] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3864.912592] echo $(hostname -s): $val; [ 3864.912592] rc=$val; [ 3864.912592] fi; [ 3864.912592] exit $rc [ 3865.638937] Lustre: DEBUG MARKER: dmesg [ 3866.158026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124b: lru resize \(performance test\) =================================================== 17:06:08 \(1491411968\) [ 3866.365097] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) =================================================== 17:06:08 (1491411968) [ 3871.554537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/disable_lru_resize 3 times [ 3871.748327] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 3876.804918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 5 seconds [ 3877.047701] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 3877.282657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 200 [ 3877.493619] Lustre: DEBUG MARKER: lru_size = 200 [ 3887.359780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark doing ls -la \/mnt\/lustre\/d124b.sanity\/enable_lru_resize 3 times [ 3887.555793] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 3892.256660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark ls -la time: 5 seconds [ 3892.449483] Lustre: DEBUG MARKER: ls -la time: 5 seconds [ 3892.669917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru_size = 4006 [ 3892.863048] Lustre: DEBUG MARKER: lru_size = 4006 [ 3893.079228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark lru resize performs the same with no lru resize [ 3893.266368] Lustre: DEBUG MARKER: lru resize performs the same with no lru resize [ 3896.336106] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3897.236685] Lustre: DEBUG MARKER: rc=0; [ 3897.236685] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3897.236685] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3897.236685] echo $(hostname -s): $val; [ 3897.236685] rc=$val; [ 3897.236685] fi; [ 3897.236685] exit $rc [ 3897.943089] Lustre: DEBUG MARKER: dmesg [ 3898.431798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 124c: LRUR cancel very aged locks ===================================================== 17:06:40 \(1491412000\) [ 3898.624148] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ===================================================== 17:06:40 (1491412000) [ 3919.137616] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3919.821601] Lustre: DEBUG MARKER: rc=0; [ 3919.821601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3919.821601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3919.821601] echo $(hostname -s): $val; [ 3919.821601] rc=$val; [ 3919.821601] fi; [ 3919.821601] exit $rc [ 3920.508149] Lustre: DEBUG MARKER: dmesg [ 3921.013579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 125: don\'t return EPROTO when a dir has a non-default striping and ACLs =============== 17:07:03 \(1491412023\) [ 3921.201280] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs =============== 17:07:03 (1491412023) [ 3921.559651] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3922.235041] Lustre: DEBUG MARKER: rc=0; [ 3922.235041] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3922.235041] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3922.235041] echo $(hostname -s): $val; [ 3922.235041] rc=$val; [ 3922.235041] fi; [ 3922.235041] exit $rc [ 3922.944508] Lustre: DEBUG MARKER: dmesg [ 3923.444381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 126: check that the fsgid provided by the client is taken into account ================ 17:07:05 \(1491412025\) [ 3923.638510] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ================ 17:07:05 (1491412025) [ 3923.857077] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3924.540962] Lustre: DEBUG MARKER: rc=0; [ 3924.540962] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3924.540962] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3924.540962] echo $(hostname -s): $val; [ 3924.540962] rc=$val; [ 3924.540962] fi; [ 3924.540962] exit $rc [ 3925.238753] Lustre: DEBUG MARKER: dmesg [ 3925.715585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127a: verify the client stats are sane ================================================ 17:07:07 \(1491412027\) [ 3925.911391] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ================================================ 17:07:07 (1491412027) [ 3926.231038] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3926.894930] Lustre: DEBUG MARKER: rc=0; [ 3926.894930] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3926.894930] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3926.894930] echo $(hostname -s): $val; [ 3926.894930] rc=$val; [ 3926.894930] fi; [ 3926.894930] exit $rc [ 3927.588146] Lustre: DEBUG MARKER: dmesg [ 3928.061980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 127b: verify the llite client stats are sane ========================================== 17:07:10 \(1491412030\) [ 3928.254307] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================== 17:07:10 (1491412030) [ 3928.507421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3929.182529] Lustre: DEBUG MARKER: rc=0; [ 3929.182529] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3929.182529] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3929.182529] echo $(hostname -s): $val; [ 3929.182529] rc=$val; [ 3929.182529] fi; [ 3929.182529] exit $rc [ 3929.865240] Lustre: DEBUG MARKER: dmesg [ 3930.361296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 128: interactive lfs for 2 consecutive find\'s ========================================= 17:07:12 \(1491412032\) [ 3930.551257] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================= 17:07:12 (1491412032) [ 3930.848512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3931.536369] Lustre: DEBUG MARKER: rc=0; [ 3931.536369] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3931.536369] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3931.536369] echo $(hostname -s): $val; [ 3931.536369] rc=$val; [ 3931.536369] fi; [ 3931.536369] exit $rc [ 3932.237677] Lustre: DEBUG MARKER: dmesg [ 3932.728150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 129: test directory size limit ======================================================== 17:07:14 \(1491412034\) [ 3932.917494] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ======================================================== 17:07:14 (1491412034) [ 3933.164394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_129 Only applicable to ldiskfs-based MDTs [ 3933.340457] Lustre: DEBUG MARKER: SKIP: sanity test_129 Only applicable to ldiskfs-based MDTs [ 3933.565887] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3934.238361] Lustre: DEBUG MARKER: rc=0; [ 3934.238361] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3934.238361] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3934.238361] echo $(hostname -s): $val; [ 3934.238361] rc=$val; [ 3934.238361] fi; [ 3934.238361] exit $rc [ 3934.966625] Lustre: DEBUG MARKER: dmesg [ 3935.452045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130a: FIEMAP \(1-stripe file\) ========================================================== 17:07:17 \(1491412037\) [ 3935.660201] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ========================================================== 17:07:17 (1491412037) [ 3936.223289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130a ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 3936.416232] Lustre: DEBUG MARKER: SKIP: sanity test_130a ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 3936.634454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3937.349278] Lustre: DEBUG MARKER: rc=0; [ 3937.349278] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3937.349278] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3937.349278] echo $(hostname -s): $val; [ 3937.349278] rc=$val; [ 3937.349278] fi; [ 3937.349278] exit $rc [ 3938.052097] Lustre: DEBUG MARKER: dmesg [ 3938.535159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130b: FIEMAP \(2-stripe file\) ========================================================== 17:07:20 \(1491412040\) [ 3938.721601] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ========================================================== 17:07:20 (1491412040) [ 3938.962132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130b ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 3939.147869] Lustre: DEBUG MARKER: SKIP: sanity test_130b ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 3939.369279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3940.043815] Lustre: DEBUG MARKER: rc=0; [ 3940.043815] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3940.043815] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3940.043815] echo $(hostname -s): $val; [ 3940.043815] rc=$val; [ 3940.043815] fi; [ 3940.043815] exit $rc [ 3940.735959] Lustre: DEBUG MARKER: dmesg [ 3941.249509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130c: FIEMAP \(2-stripe file with hole\) ================================================ 17:07:23 \(1491412043\) [ 3941.452117] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ================================================ 17:07:23 (1491412043) [ 3941.690338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130c ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 3941.886333] Lustre: DEBUG MARKER: SKIP: sanity test_130c ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 3942.110671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3942.784540] Lustre: DEBUG MARKER: rc=0; [ 3942.784540] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3942.784540] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3942.784540] echo $(hostname -s): $val; [ 3942.784540] rc=$val; [ 3942.784540] fi; [ 3942.784540] exit $rc [ 3943.534153] Lustre: DEBUG MARKER: dmesg [ 3944.068254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130d: FIEMAP \(N-stripe file\) ========================================================== 17:07:26 \(1491412046\) [ 3944.267524] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ========================================================== 17:07:26 (1491412046) [ 3944.479868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [ 3944.669664] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping FIEMAP on N-stripe file test [ 3944.896400] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3945.620264] Lustre: DEBUG MARKER: rc=0; [ 3945.620264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3945.620264] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3945.620264] echo $(hostname -s): $val; [ 3945.620264] rc=$val; [ 3945.620264] fi; [ 3945.620264] exit $rc [ 3946.336466] Lustre: DEBUG MARKER: dmesg [ 3946.873033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 130e: FIEMAP \(test continuation FIEMAP calls\) ========================================= 17:07:28 \(1491412048\) [ 3947.075749] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================= 17:07:28 (1491412048) [ 3947.305146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_130e ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 3947.511116] Lustre: DEBUG MARKER: SKIP: sanity test_130e ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 3947.734223] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3948.416805] Lustre: DEBUG MARKER: rc=0; [ 3948.416805] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3948.416805] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3948.416805] echo $(hostname -s): $val; [ 3948.416805] rc=$val; [ 3948.416805] fi; [ 3948.416805] exit $rc [ 3949.115498] Lustre: DEBUG MARKER: dmesg [ 3949.711754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131a: test iov\'s crossing stripe boundary for writev\/readv ============================ 17:07:31 \(1491412051\) [ 3949.901193] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ============================ 17:07:31 (1491412051) [ 3950.215278] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3950.887533] Lustre: DEBUG MARKER: rc=0; [ 3950.887533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3950.887533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3950.887533] echo $(hostname -s): $val; [ 3950.887533] rc=$val; [ 3950.887533] fi; [ 3950.887533] exit $rc [ 3951.584453] Lustre: DEBUG MARKER: dmesg [ 3952.076236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131b: test append writev ============================================================== 17:07:34 \(1491412054\) [ 3952.267141] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ============================================================== 17:07:34 (1491412054) [ 3952.552666] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3953.216716] Lustre: DEBUG MARKER: rc=0; [ 3953.216716] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3953.216716] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3953.216716] echo $(hostname -s): $val; [ 3953.216716] rc=$val; [ 3953.216716] fi; [ 3953.216716] exit $rc [ 3953.907799] Lustre: DEBUG MARKER: dmesg [ 3954.390373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131c: test read\/write on file w\/o objects ============================================= 17:07:36 \(1491412056\) [ 3954.576198] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ============================================= 17:07:36 (1491412056) [ 3954.852985] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3955.517882] Lustre: DEBUG MARKER: rc=0; [ 3955.517882] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3955.517882] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3955.517882] echo $(hostname -s): $val; [ 3955.517882] rc=$val; [ 3955.517882] fi; [ 3955.517882] exit $rc [ 3956.283069] Lustre: DEBUG MARKER: dmesg [ 3956.765101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131d: test short read ================================================================= 17:07:38 \(1491412058\) [ 3956.961075] Lustre: DEBUG MARKER: == sanity test 131d: test short read ================================================================= 17:07:38 (1491412058) [ 3957.204891] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3957.875790] Lustre: DEBUG MARKER: rc=0; [ 3957.875790] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3957.875790] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3957.875790] echo $(hostname -s): $val; [ 3957.875790] rc=$val; [ 3957.875790] fi; [ 3957.875790] exit $rc [ 3958.576934] Lustre: DEBUG MARKER: dmesg [ 3959.109125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 131e: test read hitting hole ========================================================== 17:07:41 \(1491412061\) [ 3959.296880] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ========================================================== 17:07:41 (1491412061) [ 3959.611712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3960.289986] Lustre: DEBUG MARKER: rc=0; [ 3960.289986] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3960.289986] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3960.289986] echo $(hostname -s): $val; [ 3960.289986] rc=$val; [ 3960.289986] fi; [ 3960.289986] exit $rc [ 3960.999309] Lustre: DEBUG MARKER: dmesg [ 3961.507928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133a: Verifying MDT stats ============================================================= 17:07:43 \(1491412063\) [ 3961.738531] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ============================================================= 17:07:43 (1491412063) [ 3961.940305] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 3962.229361] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 3962.849274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3963.154458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3963.456858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3963.757232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3964.057696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3964.354743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3964.653490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3964.946609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3965.251487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3965.547945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3965.904478] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3966.578149] Lustre: DEBUG MARKER: rc=0; [ 3966.578149] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3966.578149] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3966.578149] echo $(hostname -s): $val; [ 3966.578149] rc=$val; [ 3966.578149] fi; [ 3966.578149] exit $rc [ 3967.281687] Lustre: DEBUG MARKER: dmesg [ 3967.772846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133b: Verifying extra MDT stats ======================================================= 17:07:49 \(1491412069\) [ 3967.978109] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ======================================================= 17:07:49 (1491412069) [ 3968.208288] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 3968.841145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3969.137827] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 3969.442606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3969.753738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3970.065672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.md_stats [ 3970.429542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3971.121460] Lustre: DEBUG MARKER: rc=0; [ 3971.121460] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3971.121460] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3971.121460] echo $(hostname -s): $val; [ 3971.121460] rc=$val; [ 3971.121460] fi; [ 3971.121460] exit $rc [ 3971.813897] Lustre: DEBUG MARKER: dmesg [ 3972.338357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133c: Verifying OST stats ============================================================= 17:07:54 \(1491412074\) [ 3972.535501] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ============================================================= 17:07:54 (1491412074) [ 3972.842138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3979.154020] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.md_stats=clear [ 3981.044946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3981.338455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 3981.789446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3983.109270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3984.421235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3985.734585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3987.048478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 3993.861183] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3994.568114] Lustre: DEBUG MARKER: rc=0; [ 3994.568114] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3994.568114] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3994.568114] echo $(hostname -s): $val; [ 3994.568114] rc=$val; [ 3994.568114] fi; [ 3994.568114] exit $rc [ 3995.247047] Lustre: DEBUG MARKER: dmesg [ 3995.811979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133d: Verifying rename_stats ========================================================== 17:08:17 \(1491412097\) [ 3996.006200] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ========================================================== 17:08:17 (1491412097) [ 3996.276046] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mdt.*.rename_stats [ 3996.582047] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 3997.505475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3997.809268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3998.117339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3998.424061] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.rename_stats=clear [ 3998.741922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3999.034947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3999.339396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 3999.643811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.rename_stats [ 4000.562504] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4001.297752] Lustre: DEBUG MARKER: rc=0; [ 4001.297752] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4001.297752] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4001.297752] echo $(hostname -s): $val; [ 4001.297752] rc=$val; [ 4001.297752] fi; [ 4001.297752] exit $rc [ 4002.020108] Lustre: DEBUG MARKER: dmesg [ 4002.515089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133e: Verifying OST {read,write}_bytes nid stats ====================================== 17:08:24 \(1491412104\) [ 4002.751676] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats ====================================== 17:08:24 (1491412104) [ 4004.528642] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4005.220151] Lustre: DEBUG MARKER: rc=0; [ 4005.220151] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4005.220151] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4005.220151] echo $(hostname -s): $val; [ 4005.220151] rc=$val; [ 4005.220151] fi; [ 4005.220151] exit $rc [ 4005.926333] Lustre: DEBUG MARKER: dmesg [ 4006.456137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133f: Check for LBUGs\/Oopses\/unreadable files in \/proc ================================ 17:08:28 \(1491412108\) [ 4006.648722] Lustre: DEBUG MARKER: == sanity test 133f: Check for LBUGs/Oopses/unreadable files in /proc ================================ 17:08:28 (1491412108) [ 4007.312081] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ ! -name req_history -exec cat {} \; [ 4008.160453] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ ! -name req_history -type f -exec cat {} \; [ 4010.501522] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4011.189925] Lustre: DEBUG MARKER: rc=0; [ 4011.189925] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4011.189925] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4011.189925] echo $(hostname -s): $val; [ 4011.189925] rc=$val; [ 4011.189925] fi; [ 4011.189925] exit $rc [ 4011.897047] Lustre: DEBUG MARKER: dmesg [ 4012.378646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 133g: Check for Oopses on bad io area writes\/reads in \/proc =========================== 17:08:34 \(1491412114\) [ 4012.570675] Lustre: DEBUG MARKER: == sanity test 133g: Check for Oopses on bad io area writes/reads in /proc =========================== 17:08:34 (1491412114) [ 4013.124172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4013.777513] Lustre: DEBUG MARKER: find /proc/fs/lustre/ /proc/sys/lnet/ /sys/fs/lustre/ /sys/kernel/debug/lustre/ -type f -not -name force_lbug -not -name changelog_mask -exec badarea_io {} \; [ 4014.081418] Lustre: 19193:0:(mdt_coordinator.c:1789:mdt_hsm_policy_seq_write()) lustre-MDT0000: '' is unknown, supported policies are: [ 4014.123230] LustreError: 19241:0:(mdt_coordinator.c:1941:mdt_hsm_cdt_control_seq_write()) lustre-MDT0000: Valid coordinator control commands are: enabled shutdown disabled purge help [ 4014.129243] Lustre: 19243:0:(lprocfs_status.c:2376:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash due to bad address, rc = -14 [ 4014.133554] Lustre: 19243:0:(lprocfs_status.c:2372:lprocfs_wr_root_squash()) lustre-MDT0000: failed to set root_squash to "", needs uid:gid format, rc = -22 [ 4014.138958] LustreError: 19244:0:(genops.c:1524:obd_export_evict_by_uuid()) lustre-MDT0000: can't disconnect : no exports found [ 4014.145132] LustreError: 19246:0:(mdt_lproc.c:366:lprocfs_identity_info_seq_write()) lustre-MDT0000: invalid data count = 5, size = 1048 [ 4014.149748] LustreError: 19246:0:(mdt_lproc.c:383:lprocfs_identity_info_seq_write()) lustre-MDT0000: MDS identity downcall bad params [ 4014.155304] Lustre: 19247:0:(lprocfs_status.c:2444:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids to "", bad address rc = -14 [ 4014.160114] Lustre: 19247:0:(lprocfs_status.c:2448:lprocfs_wr_nosquash_nids()) lustre-MDT0000: failed to set nosquash_nids due to string too long rc = -22 [ 4014.168964] LustreError: 19251:0:(mdt_lproc.c:298:mdt_identity_upcall_seq_write()) lustre-MDT0000: identity upcall too long [ 4014.199395] LustreError: 19284:0:(mgs_nids.c:793:lprocfs_wr_ir_state()) Unable to process command: (-22) [ 4014.307361] LustreError: 19405:0:(lproc_fid.c:177:lprocfs_server_fid_width_seq_write()) ctl-lustre-MDT0000: invalid FID sequence width: rc = -14 [ 4014.360514] LustreError: 19462:0:(ldlm_resource.c:105:seq_watermark_write()) Failed to set lock_limit_mb, rc = -14. [ 4014.389430] LustreError: 19474:0:(nodemap_handler.c:1037:nodemap_create()) cannot add nodemap: '': rc = -22 [ 4014.503692] Lustre: 19510:0:(libcfs_string.c:127:cfs_str2mask()) unknown mask ''. [ 4014.503692] mask usage: [+|-] ... [ 4014.510469] Lustre: 0 MB is too small for debug buffer size, setting it to 2 MB. [ 4014.538391] LustreError: 19543:0:(ldlm_resource.c:354:lru_size_store()) lru_size: invalid value written [ 4015.759136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4016.050642] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4016.202229] Lustre: Failing over lustre-MDT0000 [ 4019.053437] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.058954] LustreError: Skipped 2 previous similar messages [ 4019.976143] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.241044] Lustre: 19813:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412118/real 1491412118] req@ffff88006209a100 x1563855296483408/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491412124 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4022.263494] Lustre: server umount lustre-MDT0000 complete [ 4022.409819] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4022.717140] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4022.717140] grep -q ^lustre-mdt1/ /proc/mounts || [ 4022.717140] zpool export lustre-mdt1 [ 4033.277662] Lustre: DEBUG MARKER: hostname [ 4033.627811] Lustre: DEBUG MARKER: modprobe zfs; [ 4033.627811] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4033.627811] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4034.224129] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4034.515275] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4034.708028] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 4034.710788] Lustre: Skipped 18 previous similar messages [ 4034.897583] LustreError: 21273:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4034.965625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.128595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4035.417543] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4035.911298] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4036.197653] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4036.485118] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4036.783843] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4041.975948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4042.620413] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4043.618865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4043.624377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4043.810793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4043.841992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4046.173111] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4046.480683] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 4047.647070] LustreError: 5925:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006209b600 x1563855296484144/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 4047.653430] LustreError: 5925:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 4051.069819] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.074825] LustreError: Skipped 1 previous similar message [ 4054.669032] Lustre: 22093:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412150/real 1491412150] req@ffff88005a667300 x1563855296484240/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491412156 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4054.689626] Lustre: server umount lustre-MDT0000 complete [ 4054.837166] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4055.145285] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4055.145285] grep -q ^lustre-mdt1/ /proc/mounts || [ 4055.145285] zpool export lustre-mdt1 [ 4073.761227] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 4073.761227] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 4073.761227] if [ $running -ne $mpts ]; then [ 4073.761227] echo $(hostname) env are INSANE!; [ 4073.761227] exit 1; [ 4073.761227] fi [ 4074.084861] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 4074.084861] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 4074.084861] if [ $running -ne $mpts ]; then [ 4074.084861] echo $(hostname) env are INSANE!; [ 4074.084861] exit 1; [ 4074.084861] fi [ 4075.051922] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4075.346458] Lustre: DEBUG MARKER: modprobe zfs; [ 4075.346458] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4075.346458] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4075.981991] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4076.302381] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4076.654637] LustreError: 23700:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4076.716773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.982077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4077.274298] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4077.776765] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4078.086808] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4078.378695] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4078.677735] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4078.973969] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4081.661110] Lustre: 5923:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412178/real 1491412178] req@ffff88005a665500 x1563855296484416/t0(0) o8->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491412183 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4089.003990] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 4089.345655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 4089.536459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4089.716866] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 4095.091701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4095.759120] Lustre: DEBUG MARKER: rc=0; [ 4095.759120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4095.759120] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4095.759120] echo $(hostname -s): $val; [ 4095.759120] rc=$val; [ 4095.759120] fi; [ 4095.759120] exit $rc [ 4096.446599] Lustre: DEBUG MARKER: dmesg [ 4096.932936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 17:09:58 \(1491412198\) [ 4097.130722] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ====================== 17:09:58 (1491412198) [ 4097.312421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4098.854135] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x327 [ 4099.138387] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4099.285051] Lustre: *** cfs_fail_loc=327, val=0*** [ 4109.487704] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4109.776718] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4111.265210] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4111.932607] Lustre: DEBUG MARKER: rc=0; [ 4111.932607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4111.932607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4111.932607] echo $(hostname -s): $val; [ 4111.932607] rc=$val; [ 4111.932607] fi; [ 4111.932607] exit $rc [ 4112.748114] Lustre: DEBUG MARKER: dmesg [ 4113.223642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 17:10:15 \(1491412215\) [ 4113.412686] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================= 17:10:15 (1491412215) [ 4113.597623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4113.917285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n ldlm.lock_reclaim_threshold_mb [ 4114.209989] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=0 [ 4114.506841] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x328 [ 4114.796715] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=500 [ 4114.952882] Lustre: *** cfs_fail_loc=328, val=0*** [ 4114.954637] Lustre: Skipped 512 previous similar messages [ 4121.664787] Lustre: *** cfs_fail_loc=328, val=0*** [ 4121.666548] Lustre: Skipped 501 previous similar messages [ 4127.667837] Lustre: *** cfs_fail_loc=328, val=0*** [ 4133.669701] Lustre: *** cfs_fail_loc=328, val=0*** [ 4133.671279] Lustre: Skipped 1 previous similar message [ 4135.111477] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4135.414783] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 [ 4135.717699] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.lock_reclaim_threshold_mb=336m [ 4140.863758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4141.545348] Lustre: DEBUG MARKER: rc=0; [ 4141.545348] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4141.545348] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4141.545348] echo $(hostname -s): $val; [ 4141.545348] rc=$val; [ 4141.545348] fi; [ 4141.545348] exit $rc [ 4142.234748] Lustre: DEBUG MARKER: dmesg [ 4142.758220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 140: Check reasonable stack depth \(shouldn\'t LBUG\) ==================================== 17:10:44 \(1491412244\) [ 4142.948560] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 17:10:44 (1491412244) [ 4144.366786] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4145.037259] Lustre: DEBUG MARKER: rc=0; [ 4145.037259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4145.037259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4145.037259] echo $(hostname -s): $val; [ 4145.037259] rc=$val; [ 4145.037259] fi; [ 4145.037259] exit $rc [ 4145.724646] Lustre: DEBUG MARKER: dmesg [ 4146.217922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 150: truncate\/append tests ============================================================ 17:10:48 \(1491412248\) [ 4146.412488] Lustre: DEBUG MARKER: == sanity test 150: truncate/append tests ============================================================ 17:10:48 (1491412248) [ 4147.157442] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4147.813351] Lustre: DEBUG MARKER: rc=0; [ 4147.813351] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4147.813351] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4147.813351] echo $(hostname -s): $val; [ 4147.813351] rc=$val; [ 4147.813351] fi; [ 4147.813351] exit $rc [ 4148.616817] Lustre: DEBUG MARKER: dmesg [ 4149.124974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 151: test cache on oss and controls =================================================== 17:10:51 \(1491412251\) [ 4149.326856] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls =================================================== 17:10:51 (1491412251) [ 4150.004547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4150.688470] Lustre: DEBUG MARKER: rc=0; [ 4150.688470] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4150.688470] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4150.688470] echo $(hostname -s): $val; [ 4150.688470] rc=$val; [ 4150.688470] fi; [ 4150.688470] exit $rc [ 4151.388720] Lustre: DEBUG MARKER: dmesg [ 4151.889257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 152: test read\/write with enomem ====================================================== 17:10:53 \(1491412253\) [ 4152.076522] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================== 17:10:53 (1491412253) [ 4152.368604] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4153.030511] Lustre: DEBUG MARKER: rc=0; [ 4153.030511] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4153.030511] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4153.030511] echo $(hostname -s): $val; [ 4153.030511] rc=$val; [ 4153.030511] fi; [ 4153.030511] exit $rc [ 4153.739204] Lustre: DEBUG MARKER: dmesg [ 4154.228899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 153: test if fdatasync does not crash ================================================= 17:10:56 \(1491412256\) [ 4154.414529] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================= 17:10:56 (1491412256) [ 4154.808112] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4155.460414] Lustre: DEBUG MARKER: rc=0; [ 4155.460414] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4155.460414] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4155.460414] echo $(hostname -s): $val; [ 4155.460414] rc=$val; [ 4155.460414] fi; [ 4155.460414] exit $rc [ 4156.155300] Lustre: DEBUG MARKER: dmesg [ 4156.648830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 17:10:58 \(1491412258\) [ 4156.836733] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================== 17:10:58 (1491412258) [ 4157.015858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4157.362404] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4158.029675] Lustre: DEBUG MARKER: rc=0; [ 4158.029675] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4158.029675] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4158.029675] echo $(hostname -s): $val; [ 4158.029675] rc=$val; [ 4158.029675] fi; [ 4158.029675] exit $rc [ 4158.727111] Lustre: DEBUG MARKER: dmesg [ 4159.204409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154B: verify the ll_decode_linkea tool ================================================ 17:11:01 \(1491412261\) [ 4159.395527] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ================================================ 17:11:01 (1491412261) [ 4159.562604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4159.978096] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4160.649974] Lustre: DEBUG MARKER: rc=0; [ 4160.649974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4160.649974] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4160.649974] echo $(hostname -s): $val; [ 4160.649974] rc=$val; [ 4160.649974] fi; [ 4160.649974] exit $rc [ 4161.348446] Lustre: DEBUG MARKER: dmesg [ 4161.831718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154a: Open-by-FID ===================================================================== 17:11:03 \(1491412263\) [ 4162.016983] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ===================================================================== 17:11:03 (1491412263) [ 4162.188563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4162.425264] LustreError: 23682:0:(fld_handler.c:256:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 4162.456136] Lustre: 23682:0:(mdd_device.c:582:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [f154a.sanity] in .lustre/fid, FID format should be [0x200000400:0x1:0x0] [ 4162.478498] Lustre: 23682:0:(mdd_device.c:592:obf_lookup()) lustre-MDD0000: Trying to lookup invalid FID [0x1:0x2:0x0] in .lustre/fid, sequence should be >= 0x200000400 or within [0xc,0xffffffff]. [ 4162.634865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4163.067971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4163.448187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4164.123477] Lustre: DEBUG MARKER: rc=0; [ 4164.123477] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4164.123477] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4164.123477] echo $(hostname -s): $val; [ 4164.123477] rc=$val; [ 4164.123477] fi; [ 4164.123477] exit $rc [ 4164.814911] Lustre: DEBUG MARKER: dmesg [ 4165.306548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154b: Open-by-FID for remote directory ================================================ 17:11:07 \(1491412267\) [ 4165.490620] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ================================================ 17:11:07 (1491412267) [ 4165.658393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4166.002997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_154b needs \>= 2 MDTs [ 4166.197214] Lustre: DEBUG MARKER: SKIP: sanity test_154b needs >= 2 MDTs [ 4166.402679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4167.061366] Lustre: DEBUG MARKER: rc=0; [ 4167.061366] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4167.061366] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4167.061366] echo $(hostname -s): $val; [ 4167.061366] rc=$val; [ 4167.061366] fi; [ 4167.061366] exit $rc [ 4167.756069] Lustre: DEBUG MARKER: dmesg [ 4168.311811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 17:11:10 \(1491412270\) [ 4168.503930] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ==================================== 17:11:10 (1491412270) [ 4168.666557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4169.042594] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4169.722554] Lustre: DEBUG MARKER: rc=0; [ 4169.722554] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4169.722554] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4169.722554] echo $(hostname -s): $val; [ 4169.722554] rc=$val; [ 4169.722554] fi; [ 4169.722554] exit $rc [ 4170.419238] Lustre: DEBUG MARKER: dmesg [ 4170.919750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154d: Verify open file fid ============================================================ 17:11:12 \(1491412272\) [ 4171.109444] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ============================================================ 17:11:12 (1491412272) [ 4171.305809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4171.648831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.exports.'10\.9\.5\.237@tcp'.open_files [ 4172.021150] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4172.695961] Lustre: DEBUG MARKER: rc=0; [ 4172.695961] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4172.695961] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4172.695961] echo $(hostname -s): $val; [ 4172.695961] rc=$val; [ 4172.695961] fi; [ 4172.695961] exit $rc [ 4173.418825] Lustre: DEBUG MARKER: dmesg [ 4173.907028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154e: .lustre is not returned by readdir ============================================== 17:11:15 \(1491412275\) [ 4174.103404] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir ============================================== 17:11:15 (1491412275) [ 4174.275913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4174.621029] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4175.291858] Lustre: DEBUG MARKER: rc=0; [ 4175.291858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4175.291858] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4175.291858] echo $(hostname -s): $val; [ 4175.291858] rc=$val; [ 4175.291858] fi; [ 4175.291858] exit $rc [ 4175.978985] Lustre: DEBUG MARKER: dmesg [ 4176.475628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154f: get parent fids by reading link ea ============================================== 17:11:18 \(1491412278\) [ 4176.675193] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea ============================================== 17:11:18 (1491412278) [ 4177.012151] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4177.678097] Lustre: DEBUG MARKER: rc=0; [ 4177.678097] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4177.678097] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4177.678097] echo $(hostname -s): $val; [ 4177.678097] rc=$val; [ 4177.678097] fi; [ 4177.678097] exit $rc [ 4178.372610] Lustre: DEBUG MARKER: dmesg [ 4178.934239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 154g: various llapi FID tests ========================================================= 17:11:20 \(1491412280\) [ 4179.124507] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ========================================================= 17:11:20 (1491412280) [ 4179.296261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4211.680286] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4212.364303] Lustre: DEBUG MARKER: rc=0; [ 4212.364303] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4212.364303] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4212.364303] echo $(hostname -s): $val; [ 4212.364303] rc=$val; [ 4212.364303] fi; [ 4212.364303] exit $rc [ 4213.060875] Lustre: DEBUG MARKER: dmesg [ 4213.550549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 17:11:55 \(1491412315\) [ 4213.747952] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ===================== 17:11:55 (1491412315) [ 4216.069974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4216.752292] Lustre: DEBUG MARKER: rc=0; [ 4216.752292] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4216.752292] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4216.752292] echo $(hostname -s): $val; [ 4216.752292] rc=$val; [ 4216.752292] fi; [ 4216.752292] exit $rc [ 4217.493990] Lustre: DEBUG MARKER: dmesg [ 4217.981029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 17:11:59 \(1491412319\) [ 4218.182073] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ==================== 17:11:59 (1491412319) [ 4220.614468] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4221.364857] Lustre: DEBUG MARKER: rc=0; [ 4221.364857] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4221.364857] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4221.364857] echo $(hostname -s): $val; [ 4221.364857] rc=$val; [ 4221.364857] fi; [ 4221.364857] exit $rc [ 4222.075780] Lustre: DEBUG MARKER: dmesg [ 4222.582685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 17:12:04 \(1491412324\) [ 4222.770195] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ==================== 17:12:04 (1491412324) [ 4225.115367] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4225.788749] Lustre: DEBUG MARKER: rc=0; [ 4225.788749] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4225.788749] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4225.788749] echo $(hostname -s): $val; [ 4225.788749] rc=$val; [ 4225.788749] fi; [ 4225.788749] exit $rc [ 4226.488441] Lustre: DEBUG MARKER: dmesg [ 4226.969887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 17:12:08 \(1491412328\) [ 4227.163587] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off =================== 17:12:08 (1491412328) [ 4229.471570] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4230.181870] Lustre: DEBUG MARKER: rc=0; [ 4230.181870] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4230.181870] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4230.181870] echo $(hostname -s): $val; [ 4230.181870] rc=$val; [ 4230.181870] fi; [ 4230.181870] exit $rc [ 4230.886721] Lustre: DEBUG MARKER: dmesg [ 4231.374638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 17:12:13 \(1491412333\) [ 4231.569025] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ======================= 17:12:13 (1491412333) [ 4233.849835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4234.142580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4234.456874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4235.778211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4244.516604] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4245.204143] Lustre: DEBUG MARKER: rc=0; [ 4245.204143] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4245.204143] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4245.204143] echo $(hostname -s): $val; [ 4245.204143] rc=$val; [ 4245.204143] fi; [ 4245.204143] exit $rc [ 4245.907358] Lustre: DEBUG MARKER: dmesg [ 4246.413489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 17:12:28 \(1491412348\) [ 4246.599377] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ====================== 17:12:28 (1491412348) [ 4248.873781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4249.167981] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4249.545355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4250.889236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4259.588133] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4260.272198] Lustre: DEBUG MARKER: rc=0; [ 4260.272198] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4260.272198] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4260.272198] echo $(hostname -s): $val; [ 4260.272198] rc=$val; [ 4260.272198] fi; [ 4260.272198] exit $rc [ 4260.972150] Lustre: DEBUG MARKER: dmesg [ 4261.563960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 17:12:43 \(1491412363\) [ 4261.770922] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ====================== 17:12:43 (1491412363) [ 4264.200372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4264.516668] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4264.857946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4266.189589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4274.845676] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4275.509565] Lustre: DEBUG MARKER: rc=0; [ 4275.509565] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4275.509565] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4275.509565] echo $(hostname -s): $val; [ 4275.509565] rc=$val; [ 4275.509565] fi; [ 4275.509565] exit $rc [ 4276.223406] Lustre: DEBUG MARKER: dmesg [ 4276.696495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 17:12:58 \(1491412378\) [ 4276.888303] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ===================== 17:12:58 (1491412378) [ 4279.165854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4279.458598] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4279.766686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4281.095590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4289.812940] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4290.492399] Lustre: DEBUG MARKER: rc=0; [ 4290.492399] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4290.492399] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4290.492399] echo $(hostname -s): $val; [ 4290.492399] rc=$val; [ 4290.492399] fi; [ 4290.492399] exit $rc [ 4291.180237] Lustre: DEBUG MARKER: dmesg [ 4291.668792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 156: Verification of tunables ========================================================= 17:13:13 \(1491412393\) [ 4291.860706] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ========================================================= 17:13:13 (1491412393) [ 4292.446766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_156 LU-1956\/LU-2261: stats not implemented on OSD ZFS [ 4292.637937] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [ 4292.867128] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4293.549396] Lustre: DEBUG MARKER: rc=0; [ 4293.549396] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4293.549396] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4293.549396] echo $(hostname -s): $val; [ 4293.549396] rc=$val; [ 4293.549396] fi; [ 4293.549396] exit $rc [ 4294.232422] Lustre: DEBUG MARKER: dmesg [ 4294.739604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160a: changelog sanity ================================================================ 17:13:16 \(1491412396\) [ 4294.956361] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ================================================================ 17:13:16 (1491412396) [ 4295.147378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4295.443477] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4295.586952] Lustre: lustre-MDD0000: changelog on [ 4295.733843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4296.065000] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4296.359892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-MKDIR [ 4296.663611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4296.953955] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-CLOSE [ 4297.251491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4297.547523] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+MKDIR [ 4297.877098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4298.184898] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+CLOSE [ 4298.528724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4298.828521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4299.139441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4299.450003] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4299.757629] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4300.061363] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4300.208533] Lustre: Failing over lustre-MDT0000 [ 4301.875433] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.323041] Lustre: 3491:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412402/real 1491412402] req@ffff880060955e00 x1563855296523152/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491412408 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4306.328978] Lustre: 3491:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4306.345727] Lustre: server umount lustre-MDT0000 complete [ 4306.495855] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4306.799876] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4306.799876] grep -q ^lustre-mdt1/ /proc/mounts || [ 4306.799876] zpool export lustre-mdt1 [ 4307.241887] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 4307.539619] Lustre: DEBUG MARKER: modprobe zfs; [ 4307.539619] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4307.539619] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4308.141731] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4308.435352] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4308.774873] LustreError: 5026:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4308.833584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.841931] Lustre: lustre-MDD0000: changelog on [ 4308.878078] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4309.003191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4309.292151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4309.781169] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4310.068410] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4310.356737] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4310.655982] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4310.958156] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4311.254372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4311.542992] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl1 [ 4311.687585] Lustre: lustre-MDD0000: changelog off [ 4311.837149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4312.138494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4312.487286] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4313.171679] Lustre: DEBUG MARKER: rc=0; [ 4313.171679] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4313.171679] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4313.171679] echo $(hostname -s): $val; [ 4313.171679] rc=$val; [ 4313.171679] fi; [ 4313.171679] exit $rc [ 4313.858855] Lustre: DEBUG MARKER: dmesg [ 4313.944097] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4314.362477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160b: Verify that very long rename doesn\'t crash in changelog ========================= 17:13:36 \(1491412416\) [ 4314.555756] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================= 17:13:36 (1491412416) [ 4314.757411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4315.058889] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4315.202494] Lustre: lustre-MDD0000: changelog on [ 4315.350566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users [ 4315.665565] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl2 [ 4315.812085] Lustre: lustre-MDD0000: changelog off [ 4316.001435] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4316.672822] Lustre: DEBUG MARKER: rc=0; [ 4316.672822] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4316.672822] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4316.672822] echo $(hostname -s): $val; [ 4316.672822] rc=$val; [ 4316.672822] fi; [ 4316.672822] exit $rc [ 4317.379945] Lustre: DEBUG MARKER: dmesg [ 4317.893996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160c: verify that changelog log catch the truncate event ============================== 17:13:39 \(1491412419\) [ 4318.088695] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ============================== 17:13:39 (1491412419) [ 4318.283399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4318.589966] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4318.741690] Lustre: lustre-MDD0000: changelog on [ 4318.908578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4319.213130] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=-TRUNC [ 4319.546666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask [ 4319.868819] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+TRUNC [ 4320.181088] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl3 [ 4320.327323] Lustre: lustre-MDD0000: changelog off [ 4320.519727] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4321.221659] Lustre: DEBUG MARKER: rc=0; [ 4321.221659] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4321.221659] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4321.221659] echo $(hostname -s): $val; [ 4321.221659] rc=$val; [ 4321.221659] fi; [ 4321.221659] exit $rc [ 4321.940344] Lustre: DEBUG MARKER: dmesg [ 4322.431021] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160d: verify that changelog log catch the migrate event =============================== 17:13:44 \(1491412424\) [ 4322.622520] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event =============================== 17:13:44 (1491412424) [ 4322.853288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_160d needs \>= 2 MDTs [ 4323.051335] Lustre: DEBUG MARKER: SKIP: sanity test_160d needs >= 2 MDTs [ 4323.255384] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4323.918383] Lustre: DEBUG MARKER: rc=0; [ 4323.918383] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4323.918383] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4323.918383] echo $(hostname -s): $val; [ 4323.918383] rc=$val; [ 4323.918383] fi; [ 4323.918383] exit $rc [ 4324.609603] Lustre: DEBUG MARKER: dmesg [ 4325.117794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 160e: changelog negative testing ====================================================== 17:13:47 \(1491412427\) [ 4325.311051] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing ====================================================== 17:13:47 (1491412427) [ 4325.484889] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4325.633377] Lustre: lustre-MDD0000: changelog on [ 4325.783707] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl77 [ 4325.931687] Lustre: 8018:0:(mdd_device.c:1439:mdd_changelog_user_purge()) lustre-MDD0000: No changelog for user 77; rc=0 [ 4326.126592] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4326.816718] Lustre: DEBUG MARKER: rc=0; [ 4326.816718] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4326.816718] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4326.816718] echo $(hostname -s): $val; [ 4326.816718] rc=$val; [ 4326.816718] fi; [ 4326.816718] exit $rc [ 4327.530316] Lustre: DEBUG MARKER: dmesg [ 4327.948402] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl4 [ 4328.097714] Lustre: lustre-MDD0000: changelog off [ 4328.329876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161a: link ea sanity ================================================================== 17:13:50 \(1491412430\) [ 4328.531712] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ================================================================== 17:13:50 (1491412430) [ 4331.274737] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4331.984380] Lustre: DEBUG MARKER: rc=0; [ 4331.984380] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4331.984380] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4331.984380] echo $(hostname -s): $val; [ 4331.984380] rc=$val; [ 4331.984380] fi; [ 4331.984380] exit $rc [ 4332.724359] Lustre: DEBUG MARKER: dmesg [ 4333.237253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161b: link ea sanity under remote directory =========================================== 17:13:55 \(1491412435\) [ 4333.414420] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory =========================================== 17:13:55 (1491412435) [ 4333.628929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_161b skipping remote directory test [ 4333.814612] Lustre: DEBUG MARKER: SKIP: sanity test_161b skipping remote directory test [ 4334.032930] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4334.702427] Lustre: DEBUG MARKER: rc=0; [ 4334.702427] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4334.702427] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4334.702427] echo $(hostname -s): $val; [ 4334.702427] rc=$val; [ 4334.702427] fi; [ 4334.702427] exit $rc [ 4335.380039] Lustre: DEBUG MARKER: dmesg [ 4335.889035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 17:13:57 \(1491412437\) [ 4336.092350] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags =================================== 17:13:57 (1491412437) [ 4336.288960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4336.586945] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 4336.730790] Lustre: lustre-MDD0000: changelog on [ 4336.995773] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl5 [ 4337.140874] Lustre: lustre-MDD0000: changelog off [ 4337.330844] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4337.995518] Lustre: DEBUG MARKER: rc=0; [ 4337.995518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4337.995518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4337.995518] echo $(hostname -s): $val; [ 4337.995518] rc=$val; [ 4337.995518] fi; [ 4337.995518] exit $rc [ 4338.676948] Lustre: DEBUG MARKER: dmesg [ 4339.193226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162a: path lookup sanity ============================================================== 17:14:01 \(1491412441\) [ 4339.386351] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ============================================================== 17:14:01 (1491412441) [ 4339.720081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4340.407917] Lustre: DEBUG MARKER: rc=0; [ 4340.407917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4340.407917] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4340.407917] echo $(hostname -s): $val; [ 4340.407917] rc=$val; [ 4340.407917] fi; [ 4340.407917] exit $rc [ 4341.102113] Lustre: DEBUG MARKER: dmesg [ 4341.591206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162b: striped directory path lookup sanity ============================================ 17:14:03 \(1491412443\) [ 4341.781617] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ============================================ 17:14:03 (1491412443) [ 4342.001394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_162b needs \>= 2 MDTs [ 4342.189478] Lustre: DEBUG MARKER: SKIP: sanity test_162b needs >= 2 MDTs [ 4342.399931] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4343.070135] Lustre: DEBUG MARKER: rc=0; [ 4343.070135] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4343.070135] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4343.070135] echo $(hostname -s): $val; [ 4343.070135] rc=$val; [ 4343.070135] fi; [ 4343.070135] exit $rc [ 4343.769458] Lustre: DEBUG MARKER: dmesg [ 4344.272565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 17:14:06 \(1491412446\) [ 4344.457658] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================== 17:14:06 (1491412446) [ 4344.629755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4347.271237] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4347.946220] Lustre: DEBUG MARKER: rc=0; [ 4347.946220] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4347.946220] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4347.946220] echo $(hostname -s): $val; [ 4347.946220] rc=$val; [ 4347.946220] fi; [ 4347.946220] exit $rc [ 4348.643873] Lustre: DEBUG MARKER: dmesg [ 4349.171149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 169: parallel read and truncate should not deadlock =================================== 17:14:11 \(1491412451\) [ 4349.338634] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock =================================== 17:14:11 (1491412451) [ 4349.572570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating a 10 Mb file [ 4349.754268] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4350.313769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark starting reads [ 4350.601659] Lustre: DEBUG MARKER: starting reads [ 4350.864471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating the file [ 4351.132538] Lustre: DEBUG MARKER: truncating the file [ 4351.374442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark killing dd [ 4351.583648] Lustre: DEBUG MARKER: killing dd [ 4351.805910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing the temporary file [ 4352.015019] Lustre: DEBUG MARKER: removing the temporary file [ 4352.247030] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4352.928543] Lustre: DEBUG MARKER: rc=0; [ 4352.928543] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4352.928543] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4352.928543] echo $(hostname -s): $val; [ 4352.928543] rc=$val; [ 4352.928543] fi; [ 4352.928543] exit $rc [ 4353.629039] Lustre: DEBUG MARKER: dmesg [ 4354.130413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 170: test lctl df to handle corrupted log ============================================= 17:14:16 \(1491412456\) [ 4354.327549] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log ============================================= 17:14:16 (1491412456) [ 4356.617364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4357.294664] Lustre: DEBUG MARKER: rc=0; [ 4357.294664] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4357.294664] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4357.294664] echo $(hostname -s): $val; [ 4357.294664] rc=$val; [ 4357.294664] fi; [ 4357.294664] exit $rc [ 4357.982053] Lustre: DEBUG MARKER: dmesg [ 4358.478830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit\(\) ============================== 17:14:20 \(1491412460\) [ 4358.666354] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ============================== 17:14:20 (1491412460) [ 4361.935541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4362.628504] Lustre: DEBUG MARKER: rc=0; [ 4362.628504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4362.628504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4362.628504] echo $(hostname -s): $val; [ 4362.628504] rc=$val; [ 4362.628504] fi; [ 4362.628504] exit $rc [ 4363.337871] Lustre: DEBUG MARKER: dmesg [ 4363.867283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180a skipping ALWAYS excluded test 180a \(base 180\) [ 4364.088422] Lustre: DEBUG MARKER: SKIP: sanity test_180a skipping ALWAYS excluded test 180a (base 180) [ 4364.330450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180b skipping ALWAYS excluded test 180b \(base 180\) [ 4364.536273] Lustre: DEBUG MARKER: SKIP: sanity test_180b skipping ALWAYS excluded test 180b (base 180) [ 4364.771324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_180c skipping ALWAYS excluded test 180c \(base 180\) [ 4364.982510] Lustre: DEBUG MARKER: SKIP: sanity test_180c skipping ALWAYS excluded test 180c (base 180) [ 4365.230246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 181: Test open-unlinked dir =========================================================== 17:14:27 \(1491412467\) [ 4365.420930] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir =========================================================== 17:14:27 (1491412467) [ 4374.986490] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4375.679726] Lustre: DEBUG MARKER: rc=0; [ 4375.679726] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4375.679726] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4375.679726] echo $(hostname -s): $val; [ 4375.679726] rc=$val; [ 4375.679726] fi; [ 4375.679726] exit $rc [ 4376.376860] Lustre: DEBUG MARKER: dmesg [ 4376.869559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 182: Test parallel modify metadata operations ========================================= 17:14:38 \(1491412478\) [ 4377.063142] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================= 17:14:38 (1491412478) [ 4385.874775] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4386.571349] Lustre: DEBUG MARKER: rc=0; [ 4386.571349] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4386.571349] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4386.571349] echo $(hostname -s): $val; [ 4386.571349] rc=$val; [ 4386.571349] fi; [ 4386.571349] exit $rc [ 4387.273870] Lustre: DEBUG MARKER: dmesg [ 4387.773145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 183: No crash or request leak in case of strange dispositions ========================= 17:14:49 \(1491412489\) [ 4387.975043] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ========================= 17:14:49 (1491412489) [ 4388.163064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4388.479347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x148 [ 4388.633519] Lustre: *** cfs_fail_loc=148, val=0*** [ 4388.785722] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4389.194575] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4389.964286] Lustre: DEBUG MARKER: rc=0; [ 4389.964286] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4389.964286] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4389.964286] echo $(hostname -s): $val; [ 4389.964286] rc=$val; [ 4389.964286] fi; [ 4389.964286] exit $rc [ 4390.668821] Lustre: DEBUG MARKER: dmesg [ 4391.176799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184a: Basic layout swap =============================================================== 17:14:53 \(1491412493\) [ 4391.369939] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =============================================================== 17:14:53 (1491412493) [ 4391.646528] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4392.330420] Lustre: DEBUG MARKER: rc=0; [ 4392.330420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4392.330420] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4392.330420] echo $(hostname -s): $val; [ 4392.330420] rc=$val; [ 4392.330420] fi; [ 4392.330420] exit $rc [ 4393.018629] Lustre: DEBUG MARKER: dmesg [ 4393.518072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184b: Forbidden layout swap \(will generate errors\) ==================================== 17:14:55 \(1491412495\) [ 4393.708337] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ==================================== 17:14:55 (1491412495) [ 4394.037918] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4394.733508] Lustre: DEBUG MARKER: rc=0; [ 4394.733508] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4394.733508] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4394.733508] echo $(hostname -s): $val; [ 4394.733508] rc=$val; [ 4394.733508] fi; [ 4394.733508] exit $rc [ 4395.511808] Lustre: DEBUG MARKER: dmesg [ 4396.209416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184c: Concurrent write and layout swap ================================================ 17:14:58 \(1491412498\) [ 4396.442973] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ================================================ 17:14:58 (1491412498) [ 4404.595286] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4405.246411] Lustre: DEBUG MARKER: rc=0; [ 4405.246411] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4405.246411] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4405.246411] echo $(hostname -s): $val; [ 4405.246411] rc=$val; [ 4405.246411] fi; [ 4405.246411] exit $rc [ 4405.936738] Lustre: DEBUG MARKER: dmesg [ 4406.431514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184d: allow stripeless layouts swap =================================================== 17:15:08 \(1491412508\) [ 4406.629668] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap =================================================== 17:15:08 (1491412508) [ 4406.887222] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4407.541432] Lustre: DEBUG MARKER: rc=0; [ 4407.541432] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4407.541432] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4407.541432] echo $(hostname -s): $val; [ 4407.541432] rc=$val; [ 4407.541432] fi; [ 4407.541432] exit $rc [ 4408.253323] Lustre: DEBUG MARKER: dmesg [ 4408.855094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 17:15:10 \(1491412510\) [ 4409.054143] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps =================================== 17:15:10 (1491412510) [ 4409.217176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4409.616155] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4410.307776] Lustre: DEBUG MARKER: rc=0; [ 4410.307776] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4410.307776] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4410.307776] echo $(hostname -s): $val; [ 4410.307776] rc=$val; [ 4410.307776] fi; [ 4410.307776] exit $rc [ 4410.995534] Lustre: DEBUG MARKER: dmesg [ 4411.495777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 185: Volatile file support ============================================================ 17:15:13 \(1491412513\) [ 4411.696505] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ============================================================ 17:15:13 (1491412513) [ 4411.867254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4413.239159] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4413.920152] Lustre: DEBUG MARKER: rc=0; [ 4413.920152] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4413.920152] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4413.920152] echo $(hostname -s): $val; [ 4413.920152] rc=$val; [ 4413.920152] fi; [ 4413.920152] exit $rc [ 4414.615389] Lustre: DEBUG MARKER: dmesg [ 4415.116261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 187a: Test data version change ======================================================== 17:15:17 \(1491412517\) [ 4415.308710] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ======================================================== 17:15:17 (1491412517) [ 4415.504301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4417.516398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4418.191860] Lustre: DEBUG MARKER: rc=0; [ 4418.191860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4418.191860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4418.191860] echo $(hostname -s): $val; [ 4418.191860] rc=$val; [ 4418.191860] fi; [ 4418.191860] exit $rc [ 4418.913114] Lustre: DEBUG MARKER: dmesg [ 4419.447529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 187b: Test data version change on volatile file ======================================= 17:15:21 \(1491412521\) [ 4419.641398] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ======================================= 17:15:21 (1491412521) [ 4419.834293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4420.619467] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4421.297915] Lustre: DEBUG MARKER: rc=0; [ 4421.297915] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4421.297915] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4421.297915] echo $(hostname -s): $val; [ 4421.297915] rc=$val; [ 4421.297915] fi; [ 4421.297915] exit $rc [ 4421.994505] Lustre: DEBUG MARKER: dmesg [ 4422.552460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 200: OST pools ======================================================================== 17:15:24 \(1491412524\) [ 4422.743537] Lustre: DEBUG MARKER: == sanity test 200: OST pools ======================================================================== 17:15:24 (1491412524) [ 4422.910340] Lustre: DEBUG MARKER: lctl pool_new lustre.cea1 [ 4433.214648] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 2>/dev/null || echo foo [ 4433.515376] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 2>/dev/null || echo foo [ 4433.857312] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.cea1 lustre-OST[0-1/2] [ 4444.157781] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 4444.157781] sort -u | tr '\n' ' ' [ 4444.452928] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 4444.452928] sort -u | tr '\n' ' ' [ 4444.971837] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.cea1 lustre-OST0000_UUID [ 4453.273402] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 4453.273402] grep lustre-OST0000_UUID [ 4453.585706] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 | [ 4453.585706] grep lustre-OST0000_UUID [ 4454.952163] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 [ 4455.244556] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.cea1 [ 4455.585081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.cea1 [ 4467.994344] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4468.740064] Lustre: DEBUG MARKER: rc=0; [ 4468.740064] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4468.740064] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4468.740064] echo $(hostname -s): $val; [ 4468.740064] rc=$val; [ 4468.740064] fi; [ 4468.740064] exit $rc [ 4469.460462] Lustre: DEBUG MARKER: dmesg [ 4470.083975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204a: Print default stripe attributes ================================================= 17:16:12 \(1491412572\) [ 4470.292045] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ================================================= 17:16:12 (1491412572) [ 4470.537698] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4471.211785] Lustre: DEBUG MARKER: rc=0; [ 4471.211785] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4471.211785] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4471.211785] echo $(hostname -s): $val; [ 4471.211785] rc=$val; [ 4471.211785] fi; [ 4471.211785] exit $rc [ 4471.917257] Lustre: DEBUG MARKER: dmesg [ 4472.409220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204b: Print default stripe size and offset =========================================== 17:16:14 \(1491412574\) [ 4472.610338] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset =========================================== 17:16:14 (1491412574) [ 4472.841667] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4473.535994] Lustre: DEBUG MARKER: rc=0; [ 4473.535994] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4473.535994] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4473.535994] echo $(hostname -s): $val; [ 4473.535994] rc=$val; [ 4473.535994] fi; [ 4473.535994] exit $rc [ 4474.244188] Lustre: DEBUG MARKER: dmesg [ 4474.786859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204c: Print default stripe count and offset =========================================== 17:16:16 \(1491412576\) [ 4474.982061] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset =========================================== 17:16:16 (1491412576) [ 4475.204267] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4475.891553] Lustre: DEBUG MARKER: rc=0; [ 4475.891553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4475.891553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4475.891553] echo $(hostname -s): $val; [ 4475.891553] rc=$val; [ 4475.891553] fi; [ 4475.891553] exit $rc [ 4476.639106] Lustre: DEBUG MARKER: dmesg [ 4477.138079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204d: Print default stripe count and size ============================================= 17:16:19 \(1491412579\) [ 4477.332879] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ============================================= 17:16:19 (1491412579) [ 4477.574598] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4478.264589] Lustre: DEBUG MARKER: rc=0; [ 4478.264589] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4478.264589] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4478.264589] echo $(hostname -s): $val; [ 4478.264589] rc=$val; [ 4478.264589] fi; [ 4478.264589] exit $rc [ 4478.969257] Lustre: DEBUG MARKER: dmesg [ 4479.468838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204e: Print raw stripe attributes ===================================================== 17:16:21 \(1491412581\) [ 4479.659792] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ===================================================== 17:16:21 (1491412581) [ 4479.905801] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4480.579992] Lustre: DEBUG MARKER: rc=0; [ 4480.579992] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4480.579992] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4480.579992] echo $(hostname -s): $val; [ 4480.579992] rc=$val; [ 4480.579992] fi; [ 4480.579992] exit $rc [ 4481.279180] Lustre: DEBUG MARKER: dmesg [ 4481.768631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204f: Print raw stripe size and offset =============================================== 17:16:23 \(1491412583\) [ 4481.976753] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset =============================================== 17:16:23 (1491412583) [ 4482.227740] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4482.902779] Lustre: DEBUG MARKER: rc=0; [ 4482.902779] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4482.902779] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4482.902779] echo $(hostname -s): $val; [ 4482.902779] rc=$val; [ 4482.902779] fi; [ 4482.902779] exit $rc [ 4483.586880] Lustre: DEBUG MARKER: dmesg [ 4484.091096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204g: Print raw stripe count and offset =============================================== 17:16:26 \(1491412586\) [ 4484.289260] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset =============================================== 17:16:26 (1491412586) [ 4484.529638] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4485.203124] Lustre: DEBUG MARKER: rc=0; [ 4485.203124] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4485.203124] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4485.203124] echo $(hostname -s): $val; [ 4485.203124] rc=$val; [ 4485.203124] fi; [ 4485.203124] exit $rc [ 4485.910786] Lustre: DEBUG MARKER: dmesg [ 4486.492906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 204h: Print raw stripe count and size ================================================= 17:16:28 \(1491412588\) [ 4486.690637] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ================================================= 17:16:28 (1491412588) [ 4486.908420] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4487.565787] Lustre: DEBUG MARKER: rc=0; [ 4487.565787] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4487.565787] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4487.565787] echo $(hostname -s): $val; [ 4487.565787] rc=$val; [ 4487.565787] fi; [ 4487.565787] exit $rc [ 4488.257923] Lustre: DEBUG MARKER: dmesg [ 4488.768522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 205: Verify job stats ================================================================= 17:16:30 \(1491412590\) [ 4488.956825] Lustre: DEBUG MARKER: == sanity test 205: Verify job stats ================================================================= 17:16:30 (1491412590) [ 4489.123543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4489.477956] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=nodelocal [ 4495.882995] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_register -n [ 4496.026955] Lustre: lustre-MDD0000: changelog on [ 4496.174874] Lustre: DEBUG MARKER: lctl get_param -n mdt.*.job_cleanup_interval [ 4496.461976] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.job_cleanup_interval=5 [ 4496.805868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mkdir \/mnt\/lustre\/d205.sanity [ 4497.002343] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity [ 4497.226112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mkdir.5927 [ 4497.426826] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.5927 [ 4497.584309] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4497.941227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: rmdir \/mnt\/lustre\/d205.sanity [ 4498.148613] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205.sanity [ 4498.366570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.rmdir.8771 [ 4498.559371] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rmdir.8771 [ 4498.732773] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4499.077523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mknod \/mnt\/lustre\/f205.sanity c 1 3 [ 4499.267791] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205.sanity c 1 3 [ 4499.493144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mknod.32482 [ 4499.700539] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mknod.32482 [ 4499.885904] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4500.243712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: rm -f \/mnt\/lustre\/f205.sanity [ 4500.452997] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205.sanity [ 4500.675996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.rm.14619 [ 4500.872864] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.rm.14619 [ 4501.050387] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4501.423168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: \/usr\/bin\/lfs setstripe -i 0 -c 1 \/mnt\/lustre\/f205.sanity [ 4501.614344] Lustre: DEBUG MARKER: Test: /usr/bin/lfs setstripe -i 0 -c 1 /mnt/lustre/f205.sanity [ 4501.856391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.lfs.17989 [ 4502.044833] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.lfs.17989 [ 4502.225923] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4502.610018] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: touch \/mnt\/lustre\/f205.sanity [ 4502.804642] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205.sanity [ 4503.009644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.touch.31262 [ 4503.205659] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.touch.31262 [ 4503.381721] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4504.089756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: dd if=\/dev\/zero of=\/mnt\/lustre\/f205.sanity bs=1M count=1 oflag=sync [ 4504.282209] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205.sanity bs=1M count=1 oflag=sync [ 4504.507725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.dd.20178 [ 4504.713838] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.20178 [ 4505.631480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: dd if=\/mnt\/lustre\/f205.sanity of=\/dev\/null bs=1M count=1 iflag=direct [ 4505.813749] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205.sanity of=/dev/null bs=1M count=1 iflag=direct [ 4506.034878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.dd.23380 [ 4506.226216] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.dd.23380 [ 4506.797033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: \/usr\/lib64\/lustre\/tests\/truncate \/mnt\/lustre\/f205.sanity 0 [ 4506.994225] Lustre: DEBUG MARKER: Test: /usr/lib64/lustre/tests/truncate /mnt/lustre/f205.sanity 0 [ 4507.227300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.truncate.4308 [ 4507.430257] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.truncate.4308 [ 4507.609133] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4508.294922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mv -f \/mnt\/lustre\/f205.sanity \/mnt\/lustre\/d205.sanity.rename [ 4508.479599] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205.sanity /mnt/lustre/d205.sanity.rename [ 4508.712634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mv.964 [ 4508.909791] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mv.964 [ 4509.088711] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4509.441743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Test: mkdir \/mnt\/lustre\/d205.sanity.expire [ 4509.629359] Lustre: DEBUG MARKER: Test: mkdir /mnt/lustre/d205.sanity.expire [ 4509.843333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using JobID environment variable nodelocal=id.205.mkdir.19445 [ 4510.039471] Lustre: DEBUG MARKER: Using JobID environment variable nodelocal=id.205.mkdir.19445 [ 4510.212392] Lustre: DEBUG MARKER: lctl get_param *.lustre-MDT0000.job_stats [ 4510.516382] Lustre: DEBUG MARKER: lctl get_param *.*.job_stats [ 4510.814610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4511.130499] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=disable [ 4520.594362] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.job_cleanup_interval=600 [ 4520.883096] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var=procname_uid [ 4528.327690] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl6 [ 4528.475375] Lustre: lustre-MDD0000: changelog off [ 4528.680542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4529.405809] Lustre: DEBUG MARKER: rc=0; [ 4529.405809] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4529.405809] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4529.405809] echo $(hostname -s): $val; [ 4529.405809] rc=$val; [ 4529.405809] fi; [ 4529.405809] exit $rc [ 4530.143184] Lustre: DEBUG MARKER: dmesg [ 4530.701870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 206: fail lov_init_raid0\(\) doesn\'t lbug =============================================== 17:17:12 \(1491412632\) [ 4530.889363] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug =============================================== 17:17:12 (1491412632) [ 4531.140778] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4531.814301] Lustre: DEBUG MARKER: rc=0; [ 4531.814301] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4531.814301] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4531.814301] echo $(hostname -s): $val; [ 4531.814301] rc=$val; [ 4531.814301] fi; [ 4531.814301] exit $rc [ 4532.509279] Lustre: DEBUG MARKER: dmesg [ 4532.997793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207a: can refresh layout at glimpse =================================================== 17:17:15 \(1491412635\) [ 4533.189381] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse =================================================== 17:17:15 (1491412635) [ 4533.413236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4534.079601] Lustre: DEBUG MARKER: rc=0; [ 4534.079601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4534.079601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4534.079601] echo $(hostname -s): $val; [ 4534.079601] rc=$val; [ 4534.079601] fi; [ 4534.079601] exit $rc [ 4534.764451] Lustre: DEBUG MARKER: dmesg [ 4535.303841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 207b: can refresh layout at open ====================================================== 17:17:17 \(1491412637\) [ 4535.494367] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ====================================================== 17:17:17 (1491412637) [ 4535.735473] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4536.404901] Lustre: DEBUG MARKER: rc=0; [ 4536.404901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4536.404901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4536.404901] echo $(hostname -s): $val; [ 4536.404901] rc=$val; [ 4536.404901] fi; [ 4536.404901] exit $rc [ 4537.131739] Lustre: DEBUG MARKER: dmesg [ 4537.663173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 208: Exclusive open =================================================================== 17:17:19 \(1491412639\) [ 4537.851358] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open =================================================================== 17:17:19 (1491412639) [ 4538.049508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4541.377050] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4541.673401] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4541.818551] Lustre: Failing over lustre-MDT0000 [ 4543.885748] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.890179] LustreError: Skipped 3 previous similar messages [ 4547.902040] Lustre: 23026:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412643/real 1491412643] req@ffff8800673e4300 x1563855296757952/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491412649 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4547.923091] Lustre: server umount lustre-MDT0000 complete [ 4548.070078] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4548.377749] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4548.377749] grep -q ^lustre-mdt1/ /proc/mounts || [ 4548.377749] zpool export lustre-mdt1 [ 4558.921704] Lustre: DEBUG MARKER: hostname [ 4559.296491] Lustre: DEBUG MARKER: modprobe zfs; [ 4559.296491] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4559.296491] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4559.921091] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4560.208147] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4560.399248] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 4560.401502] Lustre: Skipped 27 previous similar messages [ 4560.548114] LustreError: 24486:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4560.620646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.802616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4560.860141] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4561.128429] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4561.632381] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4561.941351] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4562.252987] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4562.561952] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4565.931527] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4566.393126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4566.397060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4566.610473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4566.625648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4568.961390] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4569.249781] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4569.395645] Lustre: Failing over lustre-MDT0000 [ 4571.366103] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4571.370993] LustreError: Skipped 2 previous similar messages [ 4575.540060] Lustre: 25307:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491412671/real 1491412671] req@ffff880063621e00 x1563855296758736/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491412677 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4575.560047] Lustre: server umount lustre-MDT0000 complete [ 4575.705902] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4576.011377] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4576.011377] grep -q ^lustre-mdt1/ /proc/mounts || [ 4576.011377] zpool export lustre-mdt1 [ 4586.441645] Lustre: DEBUG MARKER: hostname [ 4586.812490] Lustre: DEBUG MARKER: modprobe zfs; [ 4586.812490] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 4586.812490] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 4587.372140] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 4587.663481] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 4587.869072] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.912053] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.238@tcp (not set up) [ 4587.930674] LustreError: 26766:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 4587.978988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.148889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4588.446280] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4588.944140] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4589.234164] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4589.529642] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4589.848629] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 4597.855961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4597.933720] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4598.709979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4598.726766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4598.920742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4598.970827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4600.332585] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4601.037382] Lustre: DEBUG MARKER: rc=0; [ 4601.037382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4601.037382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4601.037382] echo $(hostname -s): $val; [ 4601.037382] rc=$val; [ 4601.037382] fi; [ 4601.037382] exit $rc [ 4601.733423] Lustre: DEBUG MARKER: dmesg [ 4602.251955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 209: read-only open\/close requests should be freed promptly =========================== 17:18:24 \(1491412704\) [ 4602.449221] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly =========================== 17:18:24 (1491412704) [ 4610.298943] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4612.844894] Lustre: DEBUG MARKER: rc=0; [ 4612.844894] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4612.844894] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4612.844894] echo $(hostname -s): $val; [ 4612.844894] rc=$val; [ 4612.844894] fi; [ 4612.844894] exit $rc [ 4613.566580] Lustre: DEBUG MARKER: dmesg [ 4614.207410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 212: Sendfile test ==================================================================== 17:18:36 \(1491412716\) [ 4614.399206] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ==================================================================== 17:18:36 (1491412716) [ 4614.976210] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4615.654457] Lustre: DEBUG MARKER: rc=0; [ 4615.654457] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4615.654457] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4615.654457] echo $(hostname -s): $val; [ 4615.654457] rc=$val; [ 4615.654457] fi; [ 4615.654457] exit $rc [ 4616.400022] Lustre: DEBUG MARKER: dmesg [ 4616.910150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 213: OSC lock completion and cancel race don\'t crash - bug 18829 ====================== 17:18:38 \(1491412718\) [ 4617.093373] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ====================== 17:18:38 (1491412718) [ 4617.385457] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4618.067526] Lustre: DEBUG MARKER: rc=0; [ 4618.067526] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4618.067526] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4618.067526] echo $(hostname -s): $val; [ 4618.067526] rc=$val; [ 4618.067526] fi; [ 4618.067526] exit $rc [ 4618.773429] Lustre: DEBUG MARKER: dmesg [ 4619.691469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 17:18:41 \(1491412721\) [ 4619.878731] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================== 17:18:41 (1491412721) [ 4621.726442] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4622.430259] Lustre: DEBUG MARKER: rc=0; [ 4622.430259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4622.430259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4622.430259] echo $(hostname -s): $val; [ 4622.430259] rc=$val; [ 4622.430259] fi; [ 4622.430259] exit $rc [ 4623.198067] Lustre: DEBUG MARKER: dmesg [ 4623.708275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 17:18:45 \(1491412725\) [ 4623.903680] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ==================== 17:18:45 (1491412725) [ 4624.265419] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4624.971727] Lustre: DEBUG MARKER: rc=0; [ 4624.971727] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4624.971727] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4624.971727] echo $(hostname -s): $val; [ 4624.971727] rc=$val; [ 4624.971727] fi; [ 4624.971727] exit $rc [ 4625.717782] Lustre: DEBUG MARKER: dmesg [ 4626.218874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 17:18:48 \(1491412728\) [ 4626.414601] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write works and updates file size and kms correctly ======== 17:18:48 (1491412728) [ 4633.621476] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4634.291931] Lustre: DEBUG MARKER: rc=0; [ 4634.291931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4634.291931] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4634.291931] echo $(hostname -s): $val; [ 4634.291931] rc=$val; [ 4634.291931] fi; [ 4634.291931] exit $rc [ 4634.995924] Lustre: DEBUG MARKER: dmesg [ 4635.483146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 217: check lctl ping for hostnames with hiphen \(\'-\'\) ================================== 17:18:57 \(1491412737\) [ 4635.673480] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ================================== 17:18:57 (1491412737) [ 4636.161523] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [ 4636.880674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4637.552970] Lustre: DEBUG MARKER: rc=0; [ 4637.552970] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4637.552970] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4637.552970] echo $(hostname -s): $val; [ 4637.552970] rc=$val; [ 4637.552970] fi; [ 4637.552970] exit $rc [ 4638.284614] Lustre: DEBUG MARKER: dmesg [ 4638.775458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 218: parallel read and truncate should not deadlock =================================== 17:19:00 \(1491412740\) [ 4638.968363] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock =================================== 17:19:00 (1491412740) [ 4639.197411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating a 10 Mb file [ 4639.397712] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4640.598413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark starting reads [ 4640.797374] Lustre: DEBUG MARKER: starting reads [ 4641.066537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating the file [ 4641.325835] Lustre: DEBUG MARKER: truncating the file [ 4641.546524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark killing dd [ 4641.727298] Lustre: DEBUG MARKER: killing dd [ 4641.945155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing the temporary file [ 4642.155020] Lustre: DEBUG MARKER: removing the temporary file [ 4642.363553] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4643.044295] Lustre: DEBUG MARKER: rc=0; [ 4643.044295] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4643.044295] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4643.044295] echo $(hostname -s): $val; [ 4643.044295] rc=$val; [ 4643.044295] fi; [ 4643.044295] exit $rc [ 4643.731617] Lustre: DEBUG MARKER: dmesg [ 4644.245193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 219: LU-394: Write partial won\'t cause uncontiguous pages vec at LND ================== 17:19:06 \(1491412746\) [ 4644.449664] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ================== 17:19:06 (1491412746) [ 4644.799359] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4645.472620] Lustre: DEBUG MARKER: rc=0; [ 4645.472620] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4645.472620] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4645.472620] echo $(hostname -s): $val; [ 4645.472620] rc=$val; [ 4645.472620] fi; [ 4645.472620] exit $rc [ 4646.197330] Lustre: DEBUG MARKER: dmesg [ 4646.723042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 220: preallocated MDS objects still used if ENOSPC from OST =========================== 17:19:08 \(1491412748\) [ 4646.922362] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST =========================== 17:19:08 (1491412748) [ 4647.190496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4647.495924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4647.796805] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 4648.092431] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 4648.389430] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 4649.331461] Lustre: DEBUG MARKER: lctl pool_new lustre.test_220 [ 4657.633514] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_220 2>/dev/null || echo foo [ 4657.930550] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_220 2>/dev/null || echo foo [ 4658.270827] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_220 lustre-OST0000 [ 4662.643713] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 4662.935772] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 4663.900805] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_220 lustre-OST0000 [ 4668.192371] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_220 [ 4676.594249] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4677.270401] Lustre: DEBUG MARKER: rc=0; [ 4677.270401] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4677.270401] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4677.270401] echo $(hostname -s): $val; [ 4677.270401] rc=$val; [ 4677.270401] fi; [ 4677.270401] exit $rc [ 4677.964830] Lustre: DEBUG MARKER: dmesg [ 4678.398576] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 4678.773936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 17:19:40 \(1491412780\) [ 4678.969545] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM =============================== 17:19:40 (1491412780) [ 4684.058090] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4684.721889] Lustre: DEBUG MARKER: rc=0; [ 4684.721889] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4684.721889] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4684.721889] echo $(hostname -s): $val; [ 4684.721889] rc=$val; [ 4684.721889] fi; [ 4684.721889] exit $rc [ 4685.410903] Lustre: DEBUG MARKER: dmesg [ 4685.932631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 17:19:47 \(1491412787\) [ 4686.118243] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ================================= 17:19:47 (1491412787) [ 4686.416682] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4687.129587] Lustre: DEBUG MARKER: rc=0; [ 4687.129587] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4687.129587] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4687.129587] echo $(hostname -s): $val; [ 4687.129587] rc=$val; [ 4687.129587] fi; [ 4687.129587] exit $rc [ 4687.836419] Lustre: DEBUG MARKER: dmesg [ 4688.351521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 17:19:50 \(1491412790\) [ 4688.551747] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ============================== 17:19:50 (1491412790) [ 4688.815584] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4689.507832] Lustre: DEBUG MARKER: rc=0; [ 4689.507832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4689.507832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4689.507832] echo $(hostname -s): $val; [ 4689.507832] rc=$val; [ 4689.507832] fi; [ 4689.507832] exit $rc [ 4690.207984] Lustre: DEBUG MARKER: dmesg [ 4690.704851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 17:19:52 \(1491412792\) [ 4690.897645] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ======================================== 17:19:52 (1491412792) [ 4691.167729] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4691.845613] Lustre: DEBUG MARKER: rc=0; [ 4691.845613] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4691.845613] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4691.845613] echo $(hostname -s): $val; [ 4691.845613] rc=$val; [ 4691.845613] fi; [ 4691.845613] exit $rc [ 4692.549351] Lustre: DEBUG MARKER: dmesg [ 4693.055307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224a: Don\'t panic on bulk IO failure ================================================== 17:19:55 \(1491412795\) [ 4693.249542] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ================================================== 17:19:55 (1491412795) [ 4693.790837] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4694.476736] Lustre: DEBUG MARKER: rc=0; [ 4694.476736] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4694.476736] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4694.476736] echo $(hostname -s): $val; [ 4694.476736] rc=$val; [ 4694.476736] fi; [ 4694.476736] exit $rc [ 4695.162681] Lustre: DEBUG MARKER: dmesg [ 4695.697078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224b: Don\'t panic on bulk IO failure ================================================== 17:19:57 \(1491412797\) [ 4695.877458] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ================================================== 17:19:57 (1491412797) [ 4696.110054] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4696.783098] Lustre: DEBUG MARKER: rc=0; [ 4696.783098] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4696.783098] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4696.783098] echo $(hostname -s): $val; [ 4696.783098] rc=$val; [ 4696.783098] fi; [ 4696.783098] exit $rc [ 4697.477773] Lustre: DEBUG MARKER: dmesg [ 4697.977871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 224c: Don\'t hang if one of md lost during large bulk RPC ============================== 17:19:59 \(1491412799\) [ 4698.171414] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ============================== 17:19:59 (1491412799) [ 4700.128509] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='0' [ 4706.548648] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.timeout='5' [ 4714.861162] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='600' [ 4725.320959] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.timeout='20' [ 4733.793679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4734.472481] Lustre: DEBUG MARKER: rc=0; [ 4734.472481] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4734.472481] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4734.472481] echo $(hostname -s): $val; [ 4734.472481] rc=$val; [ 4734.472481] fi; [ 4734.472481] exit $rc [ 4735.176628] Lustre: DEBUG MARKER: dmesg [ 4735.696477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 17:20:37 \(1491412837\) [ 4735.894863] Lustre: DEBUG MARKER: == sanity test 225a: Metadata survey sanity with zero-stripe ========================================= 17:20:37 (1491412837) [ 4736.094390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4736.399583] Lustre: DEBUG MARKER: lctl dl [ 4737.387613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4738.475243] LustreError: 2235:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [ 4738.477453] LustreError: 2235:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [ 4738.930265] LustreError: 2301:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [ 4742.301512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4743.001446] Lustre: DEBUG MARKER: rc=0; [ 4743.001446] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4743.001446] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4743.001446] echo $(hostname -s): $val; [ 4743.001446] rc=$val; [ 4743.001446] fi; [ 4743.001446] exit $rc [ 4743.694425] Lustre: DEBUG MARKER: dmesg [ 4744.205991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 17:20:46 \(1491412846\) [ 4744.403226] Lustre: DEBUG MARKER: == sanity test 225b: Metadata survey sanity with stripe_count = 1 ==================================== 17:20:46 (1491412846) [ 4744.596085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4744.901229] Lustre: DEBUG MARKER: lctl dl [ 4745.623827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4746.697867] LustreError: 3610:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [ 4746.700076] LustreError: 3610:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [ 4747.055714] LustreError: 3676:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [ 4747.060019] LustreError: 3676:0:(lod_object.c:1370:lod_verify_md_striping()) Skipped 1 previous similar message [ 4750.332430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4751.021312] Lustre: DEBUG MARKER: rc=0; [ 4751.021312] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4751.021312] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4751.021312] echo $(hostname -s): $val; [ 4751.021312] rc=$val; [ 4751.021312] fi; [ 4751.021312] exit $rc [ 4751.730605] Lustre: DEBUG MARKER: dmesg [ 4752.275928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 226a: call path2fid and fid2path on files of all type ================================= 17:20:54 \(1491412854\) [ 4752.469294] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ================================= 17:20:54 (1491412854) [ 4752.806264] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4753.495445] Lustre: DEBUG MARKER: rc=0; [ 4753.495445] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4753.495445] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4753.495445] echo $(hostname -s): $val; [ 4753.495445] rc=$val; [ 4753.495445] fi; [ 4753.495445] exit $rc [ 4754.218440] Lustre: DEBUG MARKER: dmesg [ 4754.737431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 17:20:56 \(1491412856\) [ 4754.934891] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ================ 17:20:56 (1491412856) [ 4755.156960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_226b needs \>= 2 MDTs [ 4755.351352] Lustre: DEBUG MARKER: SKIP: sanity test_226b needs >= 2 MDTs [ 4755.570386] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4756.248483] Lustre: DEBUG MARKER: rc=0; [ 4756.248483] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4756.248483] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4756.248483] echo $(hostname -s): $val; [ 4756.248483] rc=$val; [ 4756.248483] fi; [ 4756.248483] exit $rc [ 4756.958154] Lustre: DEBUG MARKER: dmesg [ 4757.470231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 227: running truncated executable does not cause OOM ================================== 17:20:59 \(1491412859\) [ 4757.673388] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ================================== 17:20:59 (1491412859) [ 4757.934190] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4758.613315] Lustre: DEBUG MARKER: rc=0; [ 4758.613315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4758.613315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4758.613315] echo $(hostname -s): $val; [ 4758.613315] rc=$val; [ 4758.613315] fi; [ 4758.613315] exit $rc [ 4759.312259] Lustre: DEBUG MARKER: dmesg [ 4759.823304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228a: try to reuse idle OI blocks ===================================================== 17:21:01 \(1491412861\) [ 4760.013326] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ===================================================== 17:21:01 (1491412861) [ 4760.251341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_228a non-ldiskfs backend [ 4760.443514] Lustre: DEBUG MARKER: SKIP: sanity test_228a non-ldiskfs backend [ 4760.666618] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4761.356486] Lustre: DEBUG MARKER: rc=0; [ 4761.356486] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4761.356486] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4761.356486] echo $(hostname -s): $val; [ 4761.356486] rc=$val; [ 4761.356486] fi; [ 4761.356486] exit $rc [ 4762.071276] Lustre: DEBUG MARKER: dmesg [ 4762.678516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228b: idle OI blocks can be reused after MDT restart ================================== 17:21:04 \(1491412864\) [ 4762.867243] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ================================== 17:21:04 (1491412864) [ 4763.137689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_228b non-ldiskfs backend [ 4763.328138] Lustre: DEBUG MARKER: SKIP: sanity test_228b non-ldiskfs backend [ 4763.533718] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4764.216351] Lustre: DEBUG MARKER: rc=0; [ 4764.216351] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4764.216351] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4764.216351] echo $(hostname -s): $val; [ 4764.216351] rc=$val; [ 4764.216351] fi; [ 4764.216351] exit $rc [ 4764.911253] Lustre: DEBUG MARKER: dmesg [ 4765.430651] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 17:21:07 \(1491412867\) [ 4765.638203] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ================= 17:21:07 (1491412867) [ 4765.873062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_228c non-ldiskfs backend [ 4766.067681] Lustre: DEBUG MARKER: SKIP: sanity test_228c non-ldiskfs backend [ 4766.281123] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4766.965118] Lustre: DEBUG MARKER: rc=0; [ 4766.965118] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4766.965118] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4766.965118] echo $(hostname -s): $val; [ 4766.965118] rc=$val; [ 4766.965118] fi; [ 4766.965118] exit $rc [ 4767.667691] Lustre: DEBUG MARKER: dmesg [ 4768.195810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 229: getstripe\/stat\/rm\/attr changes work on released files ============================ 17:21:10 \(1491412870\) [ 4768.386313] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 17:21:10 (1491412870) [ 4768.567163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4769.012127] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4769.704138] Lustre: DEBUG MARKER: rc=0; [ 4769.704138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4769.704138] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4769.704138] echo $(hostname -s): $val; [ 4769.704138] rc=$val; [ 4769.704138] fi; [ 4769.704138] exit $rc [ 4770.402076] Lustre: DEBUG MARKER: dmesg [ 4770.929201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230a: Create remote directory and files under the remote directory ==================== 17:21:12 \(1491412872\) [ 4771.110216] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ==================== 17:21:12 (1491412872) [ 4771.331349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230a needs \>= 2 MDTs [ 4771.526209] Lustre: DEBUG MARKER: SKIP: sanity test_230a needs >= 2 MDTs [ 4771.753900] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4772.407470] Lustre: DEBUG MARKER: rc=0; [ 4772.407470] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4772.407470] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4772.407470] echo $(hostname -s): $val; [ 4772.407470] rc=$val; [ 4772.407470] fi; [ 4772.407470] exit $rc [ 4773.107634] Lustre: DEBUG MARKER: dmesg [ 4773.674033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230b: migrate directory =============================================================== 17:21:15 \(1491412875\) [ 4773.869600] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =============================================================== 17:21:15 (1491412875) [ 4774.098822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230b needs \>= 2 MDTs [ 4774.294633] Lustre: DEBUG MARKER: SKIP: sanity test_230b needs >= 2 MDTs [ 4774.538612] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4775.215565] Lustre: DEBUG MARKER: rc=0; [ 4775.215565] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4775.215565] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4775.215565] echo $(hostname -s): $val; [ 4775.215565] rc=$val; [ 4775.215565] fi; [ 4775.215565] exit $rc [ 4775.911581] Lustre: DEBUG MARKER: dmesg [ 4776.431242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230c: check directory accessiblity if migration is failed ============================= 17:21:18 \(1491412878\) [ 4776.627930] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration is failed ============================= 17:21:18 (1491412878) [ 4776.877376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230c needs \>= 2 MDTs [ 4777.076983] Lustre: DEBUG MARKER: SKIP: sanity test_230c needs >= 2 MDTs [ 4777.289798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4778.020238] Lustre: DEBUG MARKER: rc=0; [ 4778.020238] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4778.020238] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4778.020238] echo $(hostname -s): $val; [ 4778.020238] rc=$val; [ 4778.020238] fi; [ 4778.020238] exit $rc [ 4778.736612] Lustre: DEBUG MARKER: dmesg [ 4779.285157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230d: check migrate big directory ===================================================== 17:21:21 \(1491412881\) [ 4779.497166] Lustre: DEBUG MARKER: == sanity test 230d: check migrate big directory ===================================================== 17:21:21 (1491412881) [ 4779.719470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230d needs \>= 2 MDTs [ 4779.916508] Lustre: DEBUG MARKER: SKIP: sanity test_230d needs >= 2 MDTs [ 4780.141714] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4780.828176] Lustre: DEBUG MARKER: rc=0; [ 4780.828176] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4780.828176] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4780.828176] echo $(hostname -s): $val; [ 4780.828176] rc=$val; [ 4780.828176] fi; [ 4780.828176] exit $rc [ 4781.544797] Lustre: DEBUG MARKER: dmesg [ 4782.060379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230e: migrate mulitple local link files =============================================== 17:21:24 \(1491412884\) [ 4782.251823] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files =============================================== 17:21:24 (1491412884) [ 4782.477421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230e needs \>= 2 MDTs [ 4782.660875] Lustre: DEBUG MARKER: SKIP: sanity test_230e needs >= 2 MDTs [ 4782.884787] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4783.556735] Lustre: DEBUG MARKER: rc=0; [ 4783.556735] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4783.556735] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4783.556735] echo $(hostname -s): $val; [ 4783.556735] rc=$val; [ 4783.556735] fi; [ 4783.556735] exit $rc [ 4784.261675] Lustre: DEBUG MARKER: dmesg [ 4784.766955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230f: migrate mulitple remote link files ============================================== 17:21:26 \(1491412886\) [ 4784.960361] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files ============================================== 17:21:26 (1491412886) [ 4785.186256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230f needs \>= 2 MDTs [ 4785.383112] Lustre: DEBUG MARKER: SKIP: sanity test_230f needs >= 2 MDTs [ 4785.597520] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4786.288353] Lustre: DEBUG MARKER: rc=0; [ 4786.288353] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4786.288353] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4786.288353] echo $(hostname -s): $val; [ 4786.288353] rc=$val; [ 4786.288353] fi; [ 4786.288353] exit $rc [ 4786.984634] Lustre: DEBUG MARKER: dmesg [ 4787.497107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230g: migrate dir to non-exist MDT ==================================================== 17:21:29 \(1491412889\) [ 4787.686984] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ==================================================== 17:21:29 (1491412889) [ 4787.914647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230g needs \>= 2 MDTs [ 4788.108155] Lustre: DEBUG MARKER: SKIP: sanity test_230g needs >= 2 MDTs [ 4788.332701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4789.020391] Lustre: DEBUG MARKER: rc=0; [ 4789.020391] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4789.020391] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4789.020391] echo $(hostname -s): $val; [ 4789.020391] rc=$val; [ 4789.020391] fi; [ 4789.020391] exit $rc [ 4789.727761] Lustre: DEBUG MARKER: dmesg [ 4790.241835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230h: migrate .. and root ============================================================= 17:21:32 \(1491412892\) [ 4790.431541] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ============================================================= 17:21:32 (1491412892) [ 4790.655059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230h needs \>= 2 MDTs [ 4790.837144] Lustre: DEBUG MARKER: SKIP: sanity test_230h needs >= 2 MDTs [ 4791.049455] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4791.719194] Lustre: DEBUG MARKER: rc=0; [ 4791.719194] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4791.719194] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4791.719194] echo $(hostname -s): $val; [ 4791.719194] rc=$val; [ 4791.719194] fi; [ 4791.719194] exit $rc [ 4792.416116] Lustre: DEBUG MARKER: dmesg [ 4792.924322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 17:21:34 \(1491412894\) [ 4793.122117] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ======================================= 17:21:34 (1491412894) [ 4793.346981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_230i needs \>= 2 MDTs [ 4793.535524] Lustre: DEBUG MARKER: SKIP: sanity test_230i needs >= 2 MDTs [ 4793.753152] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4794.420870] Lustre: DEBUG MARKER: rc=0; [ 4794.420870] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4794.420870] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4794.420870] echo $(hostname -s): $val; [ 4794.420870] rc=$val; [ 4794.420870] fi; [ 4794.420870] exit $rc [ 4795.112940] Lustre: DEBUG MARKER: dmesg [ 4795.607666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 231a: checking that reading\/writing of BRW RPC size results in one RPC ================ 17:21:37 \(1491412897\) [ 4795.803624] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ================ 17:21:37 (1491412897) [ 4797.337860] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4798.011018] Lustre: DEBUG MARKER: rc=0; [ 4798.011018] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4798.011018] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4798.011018] echo $(hostname -s): $val; [ 4798.011018] rc=$val; [ 4798.011018] fi; [ 4798.011018] exit $rc [ 4798.726793] Lustre: DEBUG MARKER: dmesg [ 4799.237085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 17:21:41 \(1491412901\) [ 4799.435980] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ============================ 17:21:41 (1491412901) [ 4802.171081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4802.866531] Lustre: DEBUG MARKER: rc=0; [ 4802.866531] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4802.866531] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4802.866531] echo $(hostname -s): $val; [ 4802.866531] rc=$val; [ 4802.866531] fi; [ 4802.866531] exit $rc [ 4803.603727] Lustre: DEBUG MARKER: dmesg [ 4804.102386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 232: failed lock should not block umount ============================================== 17:21:46 \(1491412906\) [ 4804.298930] Lustre: DEBUG MARKER: == sanity test 232: failed lock should not block umount ============================================== 17:21:46 (1491412906) [ 4804.999700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4805.683750] Lustre: DEBUG MARKER: rc=0; [ 4805.683750] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4805.683750] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4805.683750] echo $(hostname -s): $val; [ 4805.683750] rc=$val; [ 4805.683750] fi; [ 4805.683750] exit $rc [ 4806.377384] Lustre: DEBUG MARKER: dmesg [ 4806.902080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 17:21:48 \(1491412908\) [ 4807.089320] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ======================================= 17:21:48 (1491412908) [ 4807.266498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4807.617061] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4808.304974] Lustre: DEBUG MARKER: rc=0; [ 4808.304974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4808.304974] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4808.304974] echo $(hostname -s): $val; [ 4808.304974] rc=$val; [ 4808.304974] fi; [ 4808.304974] exit $rc [ 4809.028770] Lustre: DEBUG MARKER: dmesg [ 4809.532169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 17:21:51 \(1491412911\) [ 4809.729837] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 17:21:51 (1491412911) [ 4809.913122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4810.266679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4810.952374] Lustre: DEBUG MARKER: rc=0; [ 4810.952374] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4810.952374] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4810.952374] echo $(hostname -s): $val; [ 4810.952374] rc=$val; [ 4810.952374] fi; [ 4810.952374] exit $rc [ 4811.660211] Lustre: DEBUG MARKER: dmesg [ 4812.200331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 17:21:54 \(1491412914\) [ 4812.396797] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM =========================================== 17:21:54 (1491412914) [ 4812.923245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.46 too old [ 4813.124714] Lustre: DEBUG MARKER: SKIP: sanity test_234 LU-3703: attr version getfattr 2.4.46 too old [ 4813.375898] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4814.086843] Lustre: DEBUG MARKER: rc=0; [ 4814.086843] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4814.086843] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4814.086843] echo $(hostname -s): $val; [ 4814.086843] rc=$val; [ 4814.086843] fi; [ 4814.086843] exit $rc [ 4814.810818] Lustre: DEBUG MARKER: dmesg [ 4815.331391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 17:21:57 \(1491412917\) [ 4815.532083] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================= 17:21:57 (1491412917) [ 4815.696313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4818.116809] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4818.802045] Lustre: DEBUG MARKER: rc=0; [ 4818.802045] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4818.802045] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4818.802045] echo $(hostname -s): $val; [ 4818.802045] rc=$val; [ 4818.802045] fi; [ 4818.802045] exit $rc [ 4819.508492] Lustre: DEBUG MARKER: dmesg [ 4820.046913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 236: Layout swap on open unlinked file ================================================ 17:22:02 \(1491412922\) [ 4820.237439] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ================================================ 17:22:02 (1491412922) [ 4820.511610] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4821.195824] Lustre: DEBUG MARKER: rc=0; [ 4821.195824] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4821.195824] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4821.195824] echo $(hostname -s): $val; [ 4821.195824] rc=$val; [ 4821.195824] fi; [ 4821.195824] exit $rc [ 4821.894430] Lustre: DEBUG MARKER: dmesg [ 4822.426916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 237: Verify name_to_handle_at\/open_by_handle_at syscalls ============================== 17:22:04 \(1491412924\) [ 4822.643595] Lustre: DEBUG MARKER: == sanity test 237: Verify name_to_handle_at/open_by_handle_at syscalls ============================== 17:22:04 (1491412924) [ 4822.926045] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4823.604365] Lustre: DEBUG MARKER: rc=0; [ 4823.604365] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4823.604365] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4823.604365] echo $(hostname -s): $val; [ 4823.604365] rc=$val; [ 4823.604365] fi; [ 4823.604365] exit $rc [ 4824.321909] Lustre: DEBUG MARKER: dmesg [ 4824.837234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 238: Verify linkea consistency ======================================================== 17:22:06 \(1491412926\) [ 4825.030259] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ======================================================== 17:22:06 (1491412926) [ 4825.203696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4825.635188] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4826.316050] Lustre: DEBUG MARKER: rc=0; [ 4826.316050] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4826.316050] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4826.316050] echo $(hostname -s): $val; [ 4826.316050] rc=$val; [ 4826.316050] fi; [ 4826.316050] exit $rc [ 4827.002238] Lustre: DEBUG MARKER: dmesg [ 4827.531692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239: osp_sync test ==================================================================== 17:22:09 \(1491412929\) [ 4827.728220] Lustre: DEBUG MARKER: == sanity test 239: osp_sync test ==================================================================== 17:22:09 (1491412929) [ 4827.901302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4839.957979] Lustre: DEBUG MARKER: lctl set_param -n osp*.*.sync_changes 1 [ 4844.448798] Lustre: DEBUG MARKER: lctl get_param -n osc.*MDT*.sync_changes osc.*MDT*.sync_in_flight [ 4844.828436] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4845.509987] Lustre: DEBUG MARKER: rc=0; [ 4845.509987] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4845.509987] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4845.509987] echo $(hostname -s): $val; [ 4845.509987] rc=$val; [ 4845.509987] fi; [ 4845.509987] exit $rc [ 4846.235718] Lustre: DEBUG MARKER: dmesg [ 4846.765511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239a: process invalid osp sync record correctly ======================================= 17:22:28 \(1491412948\) [ 4846.955366] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ======================================= 17:22:28 (1491412948) [ 4847.145892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x2100 [ 4847.467361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4847.760593] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 4848.089439] Lustre: *** cfs_fail_loc=2100, val=0*** [ 4848.240827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4849.559505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4850.889810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4852.224943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4853.559902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4859.946673] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4860.648707] Lustre: DEBUG MARKER: rc=0; [ 4860.648707] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4860.648707] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4860.648707] echo $(hostname -s): $val; [ 4860.648707] rc=$val; [ 4860.648707] fi; [ 4860.648707] exit $rc [ 4861.343135] Lustre: DEBUG MARKER: dmesg [ 4861.871271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 17:22:43 \(1491412963\) [ 4862.067663] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ============================= 17:22:43 (1491412963) [ 4862.268655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x2101 [ 4862.632320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4868.936149] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 4869.352110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4875.710731] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4876.391200] Lustre: DEBUG MARKER: rc=0; [ 4876.391200] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4876.391200] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4876.391200] echo $(hostname -s): $val; [ 4876.391200] rc=$val; [ 4876.391200] fi; [ 4876.391200] exit $rc [ 4877.085719] Lustre: DEBUG MARKER: dmesg [ 4877.608547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 240: race between ldlm enqueue and the connection RPC \(no ASSERT\) ===================== 17:22:59 \(1491412979\) [ 4877.799806] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ===================== 17:22:59 (1491412979) [ 4878.022689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_240 needs \>= 2 MDTs [ 4878.215678] Lustre: DEBUG MARKER: SKIP: sanity test_240 needs >= 2 MDTs [ 4878.433985] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4879.115459] Lustre: DEBUG MARKER: rc=0; [ 4879.115459] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4879.115459] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4879.115459] echo $(hostname -s): $val; [ 4879.115459] rc=$val; [ 4879.115459] fi; [ 4879.115459] exit $rc [ 4879.810636] Lustre: DEBUG MARKER: dmesg [ 4880.306902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241a: bio vs dio ====================================================================== 17:23:02 \(1491412982\) [ 4880.508326] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ====================================================================== 17:23:02 (1491412982) [ 4889.573058] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4890.280708] Lustre: DEBUG MARKER: rc=0; [ 4890.280708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4890.280708] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4890.280708] echo $(hostname -s): $val; [ 4890.280708] rc=$val; [ 4890.280708] fi; [ 4890.280708] exit $rc [ 4890.995953] Lustre: DEBUG MARKER: dmesg [ 4891.516647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 241b: dio vs dio ====================================================================== 17:23:13 \(1491412993\) [ 4891.710674] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ====================================================================== 17:23:13 (1491412993) [ 4897.873380] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4898.548756] Lustre: DEBUG MARKER: rc=0; [ 4898.548756] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4898.548756] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4898.548756] echo $(hostname -s): $val; [ 4898.548756] rc=$val; [ 4898.548756] fi; [ 4898.548756] exit $rc [ 4899.253606] Lustre: DEBUG MARKER: dmesg [ 4899.756989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 17:23:21 \(1491413001\) [ 4899.949812] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ======================= 17:23:21 (1491413001) [ 4900.144596] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x105 [ 4900.295179] Lustre: *** cfs_fail_loc=105, val=0*** [ 4900.445985] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 4900.787384] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4901.467377] Lustre: DEBUG MARKER: rc=0; [ 4901.467377] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4901.467377] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4901.467377] echo $(hostname -s): $val; [ 4901.467377] rc=$val; [ 4901.467377] fi; [ 4901.467377] exit $rc [ 4902.175497] Lustre: DEBUG MARKER: dmesg [ 4902.687947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 243: various group lock tests ========================================================= 17:23:24 \(1491413004\) [ 4902.887134] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ========================================================= 17:23:24 (1491413004) [ 4917.527045] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4918.232204] Lustre: DEBUG MARKER: rc=0; [ 4918.232204] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4918.232204] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4918.232204] echo $(hostname -s): $val; [ 4918.232204] rc=$val; [ 4918.232204] fi; [ 4918.232204] exit $rc [ 4918.962809] Lustre: DEBUG MARKER: dmesg [ 4919.506590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 244: sendfile with group lock tests =================================================== 17:23:41 \(1491413021\) [ 4919.702263] Lustre: DEBUG MARKER: == sanity test 244: sendfile with group lock tests =================================================== 17:23:41 (1491413021) [ 4932.633080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4933.317269] Lustre: DEBUG MARKER: rc=0; [ 4933.317269] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4933.317269] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4933.317269] echo $(hostname -s): $val; [ 4933.317269] rc=$val; [ 4933.317269] fi; [ 4933.317269] exit $rc [ 4934.033759] Lustre: DEBUG MARKER: dmesg [ 4934.571420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 245: check mdc connection flag\/data: multiple modify RPCs ============================= 17:23:56 \(1491413036\) [ 4934.759733] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ============================= 17:23:56 (1491413036) [ 4934.984951] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4935.658649] Lustre: DEBUG MARKER: rc=0; [ 4935.658649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4935.658649] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4935.658649] echo $(hostname -s): $val; [ 4935.658649] rc=$val; [ 4935.658649] fi; [ 4935.658649] exit $rc [ 4936.351028] Lustre: DEBUG MARKER: dmesg [ 4936.858577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 246: Read file of size 4095 should return right length ================================ 17:23:58 \(1491413038\) [ 4937.052025] Lustre: DEBUG MARKER: == sanity test 246: Read file of size 4095 should return right length ================================ 17:23:58 (1491413038) [ 4938.320393] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4939.035936] Lustre: DEBUG MARKER: rc=0; [ 4939.035936] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4939.035936] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4939.035936] echo $(hostname -s): $val; [ 4939.035936] rc=$val; [ 4939.035936] fi; [ 4939.035936] exit $rc [ 4939.767378] Lustre: DEBUG MARKER: dmesg [ 4940.269355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247a: mount subdir as fileset ========================================================= 17:24:02 \(1491413042\) [ 4940.467750] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ========================================================= 17:24:02 (1491413042) [ 4941.149301] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4941.833551] Lustre: DEBUG MARKER: rc=0; [ 4941.833551] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4941.833551] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4941.833551] echo $(hostname -s): $val; [ 4941.833551] rc=$val; [ 4941.833551] fi; [ 4941.833551] exit $rc [ 4942.547440] Lustre: DEBUG MARKER: dmesg [ 4943.090447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247b: mount subdir that dose not exist ================================================ 17:24:05 \(1491413045\) [ 4943.285559] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ================================================ 17:24:05 (1491413045) [ 4943.563214] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4944.240960] Lustre: DEBUG MARKER: rc=0; [ 4944.240960] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4944.240960] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4944.240960] echo $(hostname -s): $val; [ 4944.240960] rc=$val; [ 4944.240960] fi; [ 4944.240960] exit $rc [ 4944.944485] Lustre: DEBUG MARKER: dmesg [ 4945.474110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247c: running fid2path outside root =================================================== 17:24:07 \(1491413047\) [ 4945.672800] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside root =================================================== 17:24:07 (1491413047) [ 4946.176262] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4946.857429] Lustre: DEBUG MARKER: rc=0; [ 4946.857429] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4946.857429] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4946.857429] echo $(hostname -s): $val; [ 4946.857429] rc=$val; [ 4946.857429] fi; [ 4946.857429] exit $rc [ 4947.571277] Lustre: DEBUG MARKER: dmesg [ 4948.173108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247d: running fid2path inside root ==================================================== 17:24:10 \(1491413050\) [ 4948.393069] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside root ==================================================== 17:24:10 (1491413050) [ 4948.905643] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4949.577754] Lustre: DEBUG MARKER: rc=0; [ 4949.577754] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4949.577754] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4949.577754] echo $(hostname -s): $val; [ 4949.577754] rc=$val; [ 4949.577754] fi; [ 4949.577754] exit $rc [ 4950.290423] Lustre: DEBUG MARKER: dmesg [ 4950.814145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 247e: mount .. as fileset ============================================================= 17:24:12 \(1491413052\) [ 4951.013109] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ============================================================= 17:24:12 (1491413052) [ 4951.362590] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4952.042055] Lustre: DEBUG MARKER: rc=0; [ 4952.042055] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4952.042055] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4952.042055] echo $(hostname -s): $val; [ 4952.042055] rc=$val; [ 4952.042055] fi; [ 4952.042055] exit $rc [ 4952.739288] Lustre: DEBUG MARKER: dmesg [ 4953.308139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 248: fast read verification =========================================================== 17:24:15 \(1491413055\) [ 4953.499386] Lustre: DEBUG MARKER: == sanity test 248: fast read verification =========================================================== 17:24:15 (1491413055) [ 4963.182367] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4963.865350] Lustre: DEBUG MARKER: rc=0; [ 4963.865350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4963.865350] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4963.865350] echo $(hostname -s): $val; [ 4963.865350] rc=$val; [ 4963.865350] fi; [ 4963.865350] exit $rc [ 4964.565140] Lustre: DEBUG MARKER: dmesg [ 4965.080452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 249: Write above 2T file size ========================================================= 17:24:27 \(1491413067\) [ 4965.276725] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ========================================================= 17:24:27 (1491413067) [ 4965.458438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4965.808222] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4966.503315] Lustre: DEBUG MARKER: rc=0; [ 4966.503315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4966.503315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4966.503315] echo $(hostname -s): $val; [ 4966.503315] rc=$val; [ 4966.503315] fi; [ 4966.503315] exit $rc [ 4967.197136] Lustre: DEBUG MARKER: dmesg [ 4967.728158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 250: Write above 16T limit ============================================================ 17:24:29 \(1491413069\) [ 4967.921406] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ============================================================ 17:24:29 (1491413069) [ 4968.153272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_250 no 16TB file size limit on ZFS [ 4968.346132] Lustre: DEBUG MARKER: SKIP: sanity test_250 no 16TB file size limit on ZFS [ 4968.570203] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4969.247403] Lustre: DEBUG MARKER: rc=0; [ 4969.247403] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4969.247403] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4969.247403] echo $(hostname -s): $val; [ 4969.247403] rc=$val; [ 4969.247403] fi; [ 4969.247403] exit $rc [ 4969.934994] Lustre: DEBUG MARKER: dmesg [ 4970.456695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 251: Handling short read and write correctly ========================================== 17:24:32 \(1491413072\) [ 4970.650022] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================== 17:24:32 (1491413072) [ 4970.923992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4971.602108] Lustre: DEBUG MARKER: rc=0; [ 4971.602108] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4971.602108] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4971.602108] echo $(hostname -s): $val; [ 4971.602108] rc=$val; [ 4971.602108] fi; [ 4971.602108] exit $rc [ 4972.304867] Lustre: DEBUG MARKER: dmesg [ 4972.806598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 252: check lr_reader tool ============================================================= 17:24:34 \(1491413074\) [ 4973.001358] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ============================================================= 17:24:34 (1491413074) [ 4973.292685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_252 can only run lr_reader on ldiskfs target [ 4973.501485] Lustre: DEBUG MARKER: SKIP: sanity test_252 can only run lr_reader on ldiskfs target [ 4973.710308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4974.388599] Lustre: DEBUG MARKER: rc=0; [ 4974.388599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4974.388599] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4974.388599] echo $(hostname -s): $val; [ 4974.388599] rc=$val; [ 4974.388599] fi; [ 4974.388599] exit $rc [ 4975.092588] Lustre: DEBUG MARKER: dmesg [ 4975.611925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 253: Check object allocation limit ==================================================== 17:24:37 \(1491413077\) [ 4975.815091] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ==================================================== 17:24:37 (1491413077) [ 4976.015104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4976.317805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 4976.615451] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 4976.905926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_high [ 4977.220550] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 4977.512343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 4979.845857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 4986.151782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_high [ 4986.442078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.reserved_mb_low [ 4986.728851] Lustre: DEBUG MARKER: lctl pool_new lustre.test_253 [ 4997.026138] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_253 2>/dev/null || echo foo [ 4997.310798] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_253 2>/dev/null || echo foo [ 4997.641944] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_253 lustre-OST0000 [ 5006.188979] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=7716 [ 5006.475653] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=7711 [ 5044.783437] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5055.089901] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5065.393953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status [ 5065.560133] LustreError: 27312:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x20000234c:0x13b8:0x0]: have 0 want 1 [ 5066.006382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5066.308124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 5066.752854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5068.070263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5069.384841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5070.697684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5072.014427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5078.342967] Lustre: DEBUG MARKER: lctl get_param -n lov.*.qos_maxage [ 5088.733928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status [ 5089.026229] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_high=15 [ 5089.317774] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=7 [ 5089.606321] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_253 lustre-OST0000 [ 5099.909692] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_253 [ 5110.266806] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5110.990109] Lustre: DEBUG MARKER: rc=0; [ 5110.990109] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5110.990109] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5110.990109] echo $(hostname -s): $val; [ 5110.990109] rc=$val; [ 5110.990109] fi; [ 5110.990109] exit $rc [ 5111.713711] Lustre: DEBUG MARKER: dmesg [ 5112.174564] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 5112.567742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 254: Check changelog size ============================================================= 17:26:54 \(1491413214\) [ 5112.753553] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ============================================================= 17:26:54 (1491413214) [ 5112.952875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5113.241387] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5113.385063] Lustre: lustre-MDD0000: changelog on [ 5113.539601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5113.887395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_size [ 5114.181170] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_deregister cl1 [ 5114.325240] Lustre: lustre-MDD0000: changelog off [ 5114.518776] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5115.235401] Lustre: DEBUG MARKER: rc=0; [ 5115.235401] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5115.235401] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5115.235401] echo $(hostname -s): $val; [ 5115.235401] rc=$val; [ 5115.235401] fi; [ 5115.235401] exit $rc [ 5115.962342] Lustre: DEBUG MARKER: dmesg [ 5116.479941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 255a: check \'lfs ladvise -a willread\' ================================================= 17:26:58 \(1491413218\) [ 5116.683953] Lustre: DEBUG MARKER: == sanity test 255a: check 'lfs ladvise -a willread' ================================================= 17:26:58 (1491413218) [ 5611.740493] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5612.546739] Lustre: DEBUG MARKER: rc=0; [ 5612.546739] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5612.546739] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5612.546739] echo $(hostname -s): $val; [ 5612.546739] rc=$val; [ 5612.546739] fi; [ 5612.546739] exit $rc [ 5613.250885] Lustre: DEBUG MARKER: dmesg [ 5613.969655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 255b: check \'lfs ladvise -a dontneed\' ================================================= 17:35:15 \(1491413715\) [ 5614.164734] Lustre: DEBUG MARKER: == sanity test 255b: check 'lfs ladvise -a dontneed' ================================================= 17:35:15 (1491413715) [ 5614.744237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_255b zfs-osd does not support dontneed advice [ 5614.937737] Lustre: DEBUG MARKER: SKIP: sanity test_255b zfs-osd does not support dontneed advice [ 5615.159214] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5615.849646] Lustre: DEBUG MARKER: rc=0; [ 5615.849646] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5615.849646] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5615.849646] echo $(hostname -s): $val; [ 5615.849646] rc=$val; [ 5615.849646] fi; [ 5615.849646] exit $rc [ 5616.548417] Lustre: DEBUG MARKER: dmesg [ 5617.098173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 256: Check llog delete for empty and not full state =================================== 17:35:19 \(1491413719\) [ 5617.283999] Lustre: DEBUG MARKER: == sanity test 256: Check llog delete for empty and not full state =================================== 17:35:19 (1491413719) [ 5617.530294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_256 non-ldiskfs backend [ 5617.717648] Lustre: DEBUG MARKER: SKIP: sanity test_256 non-ldiskfs backend [ 5617.938777] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5618.631860] Lustre: DEBUG MARKER: rc=0; [ 5618.631860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5618.631860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5618.631860] echo $(hostname -s): $val; [ 5618.631860] rc=$val; [ 5618.631860] fi; [ 5618.631860] exit $rc [ 5619.324722] Lustre: DEBUG MARKER: dmesg [ 5619.848941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 257: xattr locks are not lost ========================================================= 17:35:21 \(1491413721\) [ 5620.038051] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ========================================================= 17:35:21 (1491413721) [ 5620.228318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5620.558077] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x80000161 [ 5620.726205] Lustre: *** cfs_fail_loc=161, val=0*** [ 5620.880972] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5621.172548] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5621.318170] Lustre: Failing over lustre-MDT0000 [ 5621.910794] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.192177] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.199419] LustreError: Skipped 2 previous similar messages [ 5627.449049] Lustre: 7710:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491413723/real 1491413723] req@ffff88005ee90c00 x1563855296881072/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491413729 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5627.470454] Lustre: server umount lustre-MDT0000 complete [ 5627.621329] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5627.937322] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 5627.937322] grep -q ^lustre-mdt1/ /proc/mounts || [ 5627.937322] zpool export lustre-mdt1 [ 5628.428403] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 5628.721903] Lustre: DEBUG MARKER: modprobe zfs; [ 5628.721903] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 5628.721903] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 5629.296792] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 5629.583594] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 5629.766743] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 5629.769206] Lustre: Skipped 28 previous similar messages [ 5630.003993] LustreError: 9169:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 5630.068538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.237222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5630.539629] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5631.048286] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5631.346289] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5631.636871] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5631.934195] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 5632.224388] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5632.561364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5633.263712] Lustre: DEBUG MARKER: rc=0; [ 5633.263712] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5633.263712] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5633.263712] echo $(hostname -s): $val; [ 5633.263712] rc=$val; [ 5633.263712] fi; [ 5633.263712] exit $rc [ 5633.958588] Lustre: DEBUG MARKER: dmesg [ 5634.484218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 260: Check mdc_close fail ============================================================= 17:35:36 \(1491413736\) [ 5634.678442] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ============================================================= 17:35:36 (1491413736) [ 5636.177990] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5644.010682] Lustre: lustre-MDT0000: Recovery over after 0:08, of 2 clients 2 recovered and 0 were evicted. [ 5644.291945] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5644.972706] Lustre: DEBUG MARKER: rc=0; [ 5644.972706] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5644.972706] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5644.972706] echo $(hostname -s): $val; [ 5644.972706] rc=$val; [ 5644.972706] fi; [ 5644.972706] exit $rc [ 5645.678471] Lustre: DEBUG MARKER: dmesg [ 5646.190706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300a: basic striped dir sanity test =================================================== 17:35:48 \(1491413748\) [ 5646.388736] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test =================================================== 17:35:48 (1491413748) [ 5646.569701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5646.917458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300a needs \>= 2 MDTs [ 5647.110085] Lustre: DEBUG MARKER: SKIP: sanity test_300a needs >= 2 MDTs [ 5647.340134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5648.022520] Lustre: DEBUG MARKER: rc=0; [ 5648.022520] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5648.022520] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5648.022520] echo $(hostname -s): $val; [ 5648.022520] rc=$val; [ 5648.022520] fi; [ 5648.022520] exit $rc [ 5648.727171] Lustre: DEBUG MARKER: dmesg [ 5649.235351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300b: check ctime\/mtime for striped dir =============================================== 17:35:51 \(1491413751\) [ 5649.428467] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir =============================================== 17:35:51 (1491413751) [ 5649.603190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5649.951713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300b needs \>= 2 MDTs [ 5650.149576] Lustre: DEBUG MARKER: SKIP: sanity test_300b needs >= 2 MDTs [ 5650.371229] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5651.065274] Lustre: DEBUG MARKER: rc=0; [ 5651.065274] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5651.065274] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5651.065274] echo $(hostname -s): $val; [ 5651.065274] rc=$val; [ 5651.065274] fi; [ 5651.065274] exit $rc [ 5651.770737] Lustre: DEBUG MARKER: dmesg [ 5652.384998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300c: chown && check ls under striped directory ======================================= 17:35:54 \(1491413754\) [ 5652.581464] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 5652.756995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5653.112264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300c needs \>= 2 MDTs [ 5653.303251] Lustre: DEBUG MARKER: SKIP: sanity test_300c needs >= 2 MDTs [ 5653.526280] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5654.203792] Lustre: DEBUG MARKER: rc=0; [ 5654.203792] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5654.203792] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5654.203792] echo $(hostname -s): $val; [ 5654.203792] rc=$val; [ 5654.203792] fi; [ 5654.203792] exit $rc [ 5654.890734] Lustre: DEBUG MARKER: dmesg [ 5655.415793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300d: check default stripe under striped directory ==================================== 17:35:57 \(1491413757\) [ 5655.606937] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ==================================== 17:35:57 (1491413757) [ 5655.827457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5656.175285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300d needs \>= 2 MDTs [ 5656.367130] Lustre: DEBUG MARKER: SKIP: sanity test_300d needs >= 2 MDTs [ 5656.586693] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5657.303441] Lustre: DEBUG MARKER: rc=0; [ 5657.303441] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5657.303441] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5657.303441] echo $(hostname -s): $val; [ 5657.303441] rc=$val; [ 5657.303441] fi; [ 5657.303441] exit $rc [ 5658.003846] Lustre: DEBUG MARKER: dmesg [ 5658.519087] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300e: check rename under striped directory ============================================ 17:36:00 \(1491413760\) [ 5658.707582] Lustre: DEBUG MARKER: == sanity test 300e: check rename under striped directory ============================================ 17:36:00 (1491413760) [ 5658.885661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5659.241216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300e needs \>= 2 MDTs [ 5659.432550] Lustre: DEBUG MARKER: SKIP: sanity test_300e needs >= 2 MDTs [ 5659.647678] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5660.350382] Lustre: DEBUG MARKER: rc=0; [ 5660.350382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5660.350382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5660.350382] echo $(hostname -s): $val; [ 5660.350382] rc=$val; [ 5660.350382] fi; [ 5660.350382] exit $rc [ 5661.044597] Lustre: DEBUG MARKER: dmesg [ 5661.592874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300f: check rename cross striped directory ============================================ 17:36:03 \(1491413763\) [ 5661.791026] Lustre: DEBUG MARKER: == sanity test 300f: check rename cross striped directory ============================================ 17:36:03 (1491413763) [ 5661.972044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5662.328872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300f needs \>= 2 MDTs [ 5662.551973] Lustre: DEBUG MARKER: SKIP: sanity test_300f needs >= 2 MDTs [ 5662.768798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5663.466262] Lustre: DEBUG MARKER: rc=0; [ 5663.466262] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5663.466262] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5663.466262] echo $(hostname -s): $val; [ 5663.466262] rc=$val; [ 5663.466262] fi; [ 5663.466262] exit $rc [ 5664.170456] Lustre: DEBUG MARKER: dmesg [ 5664.694698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300g: check default striped directory for normal directory ============================ 17:36:06 \(1491413766\) [ 5664.898936] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ============================ 17:36:06 (1491413766) [ 5665.075976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5665.434113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300g needs \>= 2 MDTs [ 5665.637570] Lustre: DEBUG MARKER: SKIP: sanity test_300g needs >= 2 MDTs [ 5665.862162] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5666.550543] Lustre: DEBUG MARKER: rc=0; [ 5666.550543] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5666.550543] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5666.550543] echo $(hostname -s): $val; [ 5666.550543] rc=$val; [ 5666.550543] fi; [ 5666.550543] exit $rc [ 5667.250901] Lustre: DEBUG MARKER: dmesg [ 5667.762601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300h: check default striped directory for striped directory =========================== 17:36:09 \(1491413769\) [ 5667.966995] Lustre: DEBUG MARKER: == sanity test 300h: check default striped directory for striped directory =========================== 17:36:09 (1491413769) [ 5668.135323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5668.493364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300h needs \>= 2 MDTs [ 5668.688971] Lustre: DEBUG MARKER: SKIP: sanity test_300h needs >= 2 MDTs [ 5668.904921] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5669.602481] Lustre: DEBUG MARKER: rc=0; [ 5669.602481] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5669.602481] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5669.602481] echo $(hostname -s): $val; [ 5669.602481] rc=$val; [ 5669.602481] fi; [ 5669.602481] exit $rc [ 5670.312562] Lustre: DEBUG MARKER: dmesg [ 5670.847704] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300i: client handle unknown hash type striped directory =============================== 17:36:12 \(1491413772\) [ 5671.060366] Lustre: DEBUG MARKER: == sanity test 300i: client handle unknown hash type striped directory =============================== 17:36:12 (1491413772) [ 5671.220283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5671.573598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300i needs \>= 2 MDTs [ 5671.762488] Lustre: DEBUG MARKER: SKIP: sanity test_300i needs >= 2 MDTs [ 5671.988279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5672.702314] Lustre: DEBUG MARKER: rc=0; [ 5672.702314] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5672.702314] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5672.702314] echo $(hostname -s): $val; [ 5672.702314] rc=$val; [ 5672.702314] fi; [ 5672.702314] exit $rc [ 5673.408911] Lustre: DEBUG MARKER: dmesg [ 5674.013862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300j: test large update record ======================================================== 17:36:15 \(1491413775\) [ 5674.197336] Lustre: DEBUG MARKER: == sanity test 300j: test large update record ======================================================== 17:36:15 (1491413775) [ 5674.381159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5674.748125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300j needs \>= 2 MDTs [ 5674.939988] Lustre: DEBUG MARKER: SKIP: sanity test_300j needs >= 2 MDTs [ 5675.153971] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5675.863460] Lustre: DEBUG MARKER: rc=0; [ 5675.863460] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5675.863460] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5675.863460] echo $(hostname -s): $val; [ 5675.863460] rc=$val; [ 5675.863460] fi; [ 5675.863460] exit $rc [ 5676.591484] Lustre: DEBUG MARKER: dmesg [ 5677.175290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300k: test large striped directory ==================================================== 17:36:19 \(1491413779\) [ 5677.371340] Lustre: DEBUG MARKER: == sanity test 300k: test large striped directory ==================================================== 17:36:19 (1491413779) [ 5677.556739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5677.926725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300k needs \>= 2 MDTs [ 5678.122914] Lustre: DEBUG MARKER: SKIP: sanity test_300k needs >= 2 MDTs [ 5678.331041] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5679.039917] Lustre: DEBUG MARKER: rc=0; [ 5679.039917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5679.039917] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5679.039917] echo $(hostname -s): $val; [ 5679.039917] rc=$val; [ 5679.039917] fi; [ 5679.039917] exit $rc [ 5679.766155] Lustre: DEBUG MARKER: dmesg [ 5680.289731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 17:36:22 \(1491413782\) [ 5680.487121] Lustre: DEBUG MARKER: == sanity test 300l: non-root user to create dir under striped dir with stale layout ================= 17:36:22 (1491413782) [ 5680.654784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5681.040408] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300l needs \>= 2 MDTs [ 5681.224621] Lustre: DEBUG MARKER: SKIP: sanity test_300l needs >= 2 MDTs [ 5681.437213] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5682.104118] Lustre: DEBUG MARKER: rc=0; [ 5682.104118] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5682.104118] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5682.104118] echo $(hostname -s): $val; [ 5682.104118] rc=$val; [ 5682.104118] fi; [ 5682.104118] exit $rc [ 5682.806547] Lustre: DEBUG MARKER: dmesg [ 5683.364296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300m: setstriped directory on single MDT FS =========================================== 17:36:25 \(1491413785\) [ 5683.566757] Lustre: DEBUG MARKER: == sanity test 300m: setstriped directory on single MDT FS =========================================== 17:36:25 (1491413785) [ 5683.740386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5684.130936] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5684.810832] Lustre: DEBUG MARKER: rc=0; [ 5684.810832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5684.810832] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5684.810832] echo $(hostname -s): $val; [ 5684.810832] rc=$val; [ 5684.810832] fi; [ 5684.810832] exit $rc [ 5685.533412] Lustre: DEBUG MARKER: dmesg [ 5686.076795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 17:36:28 \(1491413788\) [ 5686.277640] Lustre: DEBUG MARKER: == sanity test 300n: non-root user to create dir under striped dir with default EA =================== 17:36:28 (1491413788) [ 5686.450234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5686.812189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300n needs \>= 2 MDTs [ 5687.015229] Lustre: DEBUG MARKER: SKIP: sanity test_300n needs >= 2 MDTs [ 5687.230766] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5687.906807] Lustre: DEBUG MARKER: rc=0; [ 5687.906807] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5687.906807] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5687.906807] echo $(hostname -s): $val; [ 5687.906807] rc=$val; [ 5687.906807] fi; [ 5687.906807] exit $rc [ 5688.606379] Lustre: DEBUG MARKER: dmesg [ 5689.140840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300o skipping SLOW test 300o [ 5689.363195] Lustre: DEBUG MARKER: SKIP: sanity test_300o skipping SLOW test 300o [ 5689.627454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300p: create striped directory without space ========================================== 17:36:31 \(1491413791\) [ 5689.822696] Lustre: DEBUG MARKER: == sanity test 300p: create striped directory without space ========================================== 17:36:31 (1491413791) [ 5690.037647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300p needs \>= 2 MDTs [ 5690.234156] Lustre: DEBUG MARKER: SKIP: sanity test_300p needs >= 2 MDTs [ 5690.467309] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5691.186134] Lustre: DEBUG MARKER: rc=0; [ 5691.186134] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5691.186134] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5691.186134] echo $(hostname -s): $val; [ 5691.186134] rc=$val; [ 5691.186134] fi; [ 5691.186134] exit $rc [ 5691.879100] Lustre: DEBUG MARKER: dmesg [ 5692.397740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 300q: create remote directory under orphan directory ================================== 17:36:34 \(1491413794\) [ 5692.588441] Lustre: DEBUG MARKER: == sanity test 300q: create remote directory under orphan directory ================================== 17:36:34 (1491413794) [ 5692.813328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_300q needs \>= 2 MDTs [ 5693.001628] Lustre: DEBUG MARKER: SKIP: sanity test_300q needs >= 2 MDTs [ 5693.232431] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5693.908535] Lustre: DEBUG MARKER: rc=0; [ 5693.908535] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5693.908535] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5693.908535] echo $(hostname -s): $val; [ 5693.908535] rc=$val; [ 5693.908535] fi; [ 5693.908535] exit $rc [ 5694.615775] Lustre: DEBUG MARKER: dmesg [ 5695.146537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310a: open unlink remote file ========================================================= 17:36:37 \(1491413797\) [ 5695.335670] Lustre: DEBUG MARKER: == sanity test 310a: open unlink remote file ========================================================= 17:36:37 (1491413797) [ 5695.549365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310a needs \>= 4 MDTs [ 5695.745272] Lustre: DEBUG MARKER: SKIP: sanity test_310a needs >= 4 MDTs [ 5695.961792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5696.656823] Lustre: DEBUG MARKER: rc=0; [ 5696.656823] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5696.656823] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5696.656823] echo $(hostname -s): $val; [ 5696.656823] rc=$val; [ 5696.656823] fi; [ 5696.656823] exit $rc [ 5697.353354] Lustre: DEBUG MARKER: dmesg [ 5697.909077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310b: unlink remote file with multiple links while open =============================== 17:36:39 \(1491413799\) [ 5698.101172] Lustre: DEBUG MARKER: == sanity test 310b: unlink remote file with multiple links while open =============================== 17:36:39 (1491413799) [ 5698.336328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310b needs \>= 4 MDTs [ 5698.537078] Lustre: DEBUG MARKER: SKIP: sanity test_310b needs >= 4 MDTs [ 5698.761774] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5699.440752] Lustre: DEBUG MARKER: rc=0; [ 5699.440752] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5699.440752] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5699.440752] echo $(hostname -s): $val; [ 5699.440752] rc=$val; [ 5699.440752] fi; [ 5699.440752] exit $rc [ 5700.138645] Lustre: DEBUG MARKER: dmesg [ 5700.690826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 310c: open-unlink remote file with multiple links ===================================== 17:36:42 \(1491413802\) [ 5700.907219] Lustre: DEBUG MARKER: == sanity test 310c: open-unlink remote file with multiple links ===================================== 17:36:42 (1491413802) [ 5701.120365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_310c needs \>= 4 MDTs [ 5701.324212] Lustre: DEBUG MARKER: SKIP: sanity test_310c needs >= 4 MDTs [ 5701.561711] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5702.241929] Lustre: DEBUG MARKER: rc=0; [ 5702.241929] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5702.241929] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5702.241929] echo $(hostname -s): $val; [ 5702.241929] rc=$val; [ 5702.241929] fi; [ 5702.241929] exit $rc [ 5702.982723] Lustre: DEBUG MARKER: dmesg [ 5703.530574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 17:36:45 \(1491413805\) [ 5703.733089] Lustre: DEBUG MARKER: == sanity test 311: disable OSP precreate, and unlink should destroy objs ============================ 17:36:45 (1491413805) [ 5703.909675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5705.339410] Lustre: DEBUG MARKER: lctl get_param -n osp.*OST0000*MDT0000.create_count [ 5705.628475] Lustre: DEBUG MARKER: lctl get_param -n osp.*OST0000*MDT0000.max_create_count [ 5705.921970] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.max_create_count=0 [ 5707.422596] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.max_create_count=20000 [ 5707.709847] Lustre: DEBUG MARKER: lctl set_param -n osp.*OST0000*MDT000?.create_count=32 [ 5708.051545] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5708.746307] Lustre: DEBUG MARKER: rc=0; [ 5708.746307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5708.746307] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5708.746307] echo $(hostname -s): $val; [ 5708.746307] rc=$val; [ 5708.746307] fi; [ 5708.746307] exit $rc [ 5709.541499] Lustre: DEBUG MARKER: dmesg [ 5710.092796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 17:36:52 \(1491413812\) [ 5710.290829] Lustre: DEBUG MARKER: == sanity test 312: make sure ZFS adjusts its block size by write pattern ============================ 17:36:52 (1491413812) [ 5739.523860] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5740.214663] Lustre: DEBUG MARKER: rc=0; [ 5740.214663] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5740.214663] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5740.214663] echo $(hostname -s): $val; [ 5740.214663] rc=$val; [ 5740.214663] fi; [ 5740.214663] exit $rc [ 5740.918223] Lustre: DEBUG MARKER: dmesg [ 5741.435150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 313: io should fail after last_rcvd update fail ======================================= 17:37:23 \(1491413843\) [ 5741.635616] Lustre: DEBUG MARKER: == sanity test 313: io should fail after last_rcvd update fail ======================================= 17:37:23 (1491413843) [ 5742.544640] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5743.213558] Lustre: DEBUG MARKER: rc=0; [ 5743.213558] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5743.213558] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5743.213558] echo $(hostname -s): $val; [ 5743.213558] rc=$val; [ 5743.213558] fi; [ 5743.213558] exit $rc [ 5743.906611] Lustre: DEBUG MARKER: dmesg [ 5744.422361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 399a: fake write should not be slower than normal write =============================== 17:37:26 \(1491413846\) [ 5744.619250] Lustre: DEBUG MARKER: == sanity test 399a: fake write should not be slower than normal write =============================== 17:37:26 (1491413846) [ 6176.251541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6176.991890] Lustre: DEBUG MARKER: rc=0; [ 6176.991890] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6176.991890] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6176.991890] echo $(hostname -s): $val; [ 6176.991890] rc=$val; [ 6176.991890] fi; [ 6176.991890] exit $rc [ 6177.706536] Lustre: DEBUG MARKER: dmesg [ 6178.236298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 399b: fake read should not be slower than normal read ================================= 17:44:40 \(1491414280\) [ 6178.430931] Lustre: DEBUG MARKER: == sanity test 399b: fake read should not be slower than normal read ================================= 17:44:40 (1491414280) [ 6178.655389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_399b only for ldiskfs [ 6178.852674] Lustre: DEBUG MARKER: SKIP: sanity test_399b only for ldiskfs [ 6179.081101] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6179.757049] Lustre: DEBUG MARKER: rc=0; [ 6179.757049] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6179.757049] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6179.757049] echo $(hostname -s): $val; [ 6179.757049] rc=$val; [ 6179.757049] fi; [ 6179.757049] exit $rc [ 6180.462042] Lustre: DEBUG MARKER: dmesg [ 6181.010226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 400a: Lustre client api program can compile and link ================================== 17:44:43 \(1491414283\) [ 6181.210077] Lustre: DEBUG MARKER: == sanity test 400a: Lustre client api program can compile and link ================================== 17:44:43 (1491414283) [ 6183.642398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6184.327413] Lustre: DEBUG MARKER: rc=0; [ 6184.327413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6184.327413] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6184.327413] echo $(hostname -s): $val; [ 6184.327413] rc=$val; [ 6184.327413] fi; [ 6184.327413] exit $rc [ 6185.029100] Lustre: DEBUG MARKER: dmesg [ 6185.566299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 400b: packaged headers can be compiled ================================================ 17:44:47 \(1491414287\) [ 6185.767089] Lustre: DEBUG MARKER: == sanity test 400b: packaged headers can be compiled ================================================ 17:44:47 (1491414287) [ 6186.105418] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6186.771417] Lustre: DEBUG MARKER: rc=0; [ 6186.771417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6186.771417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6186.771417] echo $(hostname -s): $val; [ 6186.771417] rc=$val; [ 6186.771417] fi; [ 6186.771417] exit $rc [ 6187.480499] Lustre: DEBUG MARKER: dmesg [ 6188.007190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401a: Verify if \'lctl list_param -R\' can list parameters recursively ================== 17:44:50 \(1491414290\) [ 6188.213489] Lustre: DEBUG MARKER: == sanity test 401a: Verify if 'lctl list_param -R' can list parameters recursively ================== 17:44:50 (1491414290) [ 6188.527343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6189.220035] Lustre: DEBUG MARKER: rc=0; [ 6189.220035] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6189.220035] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6189.220035] echo $(hostname -s): $val; [ 6189.220035] rc=$val; [ 6189.220035] fi; [ 6189.220035] exit $rc [ 6189.919613] Lustre: DEBUG MARKER: dmesg [ 6190.449028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401b: Verify \'lctl {get,set}_param\' continue after error ============================== 17:44:52 \(1491414292\) [ 6190.641741] Lustre: DEBUG MARKER: == sanity test 401b: Verify 'lctl get_param' set_param' continue after error ============================== 17:44:52 (1491414292) [ 6190.873746] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6191.569079] Lustre: DEBUG MARKER: rc=0; [ 6191.569079] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6191.569079] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6191.569079] echo $(hostname -s): $val; [ 6191.569079] rc=$val; [ 6191.569079] fi; [ 6191.569079] exit $rc [ 6192.258952] Lustre: DEBUG MARKER: dmesg [ 6192.804476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401c: Verify \'lctl set_param\' without value fails in either format. =================== 17:44:54 \(1491414294\) [ 6193.007193] Lustre: DEBUG MARKER: == sanity test 401c: Verify 'lctl set_param' without value fails in either format. =================== 17:44:54 (1491414294) [ 6193.228454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6193.922286] Lustre: DEBUG MARKER: rc=0; [ 6193.922286] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6193.922286] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6193.922286] echo $(hostname -s): $val; [ 6193.922286] rc=$val; [ 6193.922286] fi; [ 6193.922286] exit $rc [ 6194.625906] Lustre: DEBUG MARKER: dmesg [ 6195.142491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 401d: Verify \'lctl set_param\' accepts values containing \'=\' =========================== 17:44:57 \(1491414297\) [ 6195.343875] Lustre: DEBUG MARKER: == sanity test 401d: Verify 'lctl set_param' accepts values containing '=' =========================== 17:44:57 (1491414297) [ 6195.584615] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6196.269982] Lustre: DEBUG MARKER: rc=0; [ 6196.269982] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6196.269982] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6196.269982] echo $(hostname -s): $val; [ 6196.269982] rc=$val; [ 6196.269982] fi; [ 6196.269982] exit $rc [ 6196.970870] Lustre: DEBUG MARKER: dmesg [ 6197.511886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 17:44:59 \(1491414299\) [ 6197.764195] Lustre: DEBUG MARKER: == sanity test 402: Return ENOENT to lod_generate_and_set_lovea ====================================== 17:44:59 (1491414299) [ 6197.937609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 6198.259867] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000015c [ 6198.411181] Lustre: *** cfs_fail_loc=15c, val=0*** [ 6198.413056] LustreError: 10061:0:(lod_lov.c:715:lod_generate_and_set_lovea()) lustre-MDT0000-mdtlov: Can not locate [0x100010000:0x8824:0x0]: rc = -2 [ 6198.620174] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6199.336453] Lustre: DEBUG MARKER: rc=0; [ 6199.336453] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6199.336453] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6199.336453] echo $(hostname -s): $val; [ 6199.336453] rc=$val; [ 6199.336453] fi; [ 6199.336453] exit $rc [ 6200.041776] Lustre: DEBUG MARKER: dmesg [ 6200.571029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 17:45:02 \(1491414302\) [ 6200.760260] Lustre: DEBUG MARKER: == sanity test 403: i_nlink should not drop to zero due to aliasing ================================== 17:45:02 (1491414302) [ 6201.088591] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6201.777851] Lustre: DEBUG MARKER: rc=0; [ 6201.777851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6201.777851] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6201.777851] echo $(hostname -s): $val; [ 6201.777851] rc=$val; [ 6201.777851] fi; [ 6201.777851] exit $rc [ 6202.481279] Lustre: DEBUG MARKER: dmesg [ 6203.127637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 17:45:05 \(1491414305\) [ 6203.332724] Lustre: DEBUG MARKER: == sanity test 404: validate manual {de}activated works properly for OSPs ============================ 17:45:05 (1491414305) [ 6203.550114] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6203.837241] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0000-osc-MDT0000 deactivate [ 6203.981114] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6204.127867] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6204.417130] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0000-osc-MDT0000 activate [ 6204.561733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6204.568692] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.719891] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6205.008668] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0001-osc-MDT0000 deactivate [ 6205.153404] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6205.304768] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6205.592104] Lustre: DEBUG MARKER: /usr/sbin/lctl --device %lustre-OST0001-osc-MDT0000 activate [ 6205.735658] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6205.742906] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6205.899053] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 6206.239917] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6206.926881] Lustre: DEBUG MARKER: rc=0; [ 6206.926881] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6206.926881] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6206.926881] echo $(hostname -s): $val; [ 6206.926881] rc=$val; [ 6206.926881] fi; [ 6206.926881] exit $rc [ 6207.633108] Lustre: DEBUG MARKER: dmesg [ 6208.157532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 405: Various layout swap lock tests =================================================== 17:45:10 \(1491414310\) [ 6208.346610] Lustre: DEBUG MARKER: == sanity test 405: Various layout swap lock tests =================================================== 17:45:10 (1491414310) [ 6208.526841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 6222.552770] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6223.251425] Lustre: DEBUG MARKER: rc=0; [ 6223.251425] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6223.251425] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6223.251425] echo $(hostname -s): $val; [ 6223.251425] rc=$val; [ 6223.251425] fi; [ 6223.251425] exit $rc [ 6223.958142] Lustre: DEBUG MARKER: dmesg [ 6224.501379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 406: DNE support fs default striping ================================================== 17:45:26 \(1491414326\) [ 6224.698634] Lustre: DEBUG MARKER: == sanity test 406: DNE support fs default striping ================================================== 17:45:26 (1491414326) [ 6224.927725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_406 needs \>= 2 MDTs [ 6225.126128] Lustre: DEBUG MARKER: SKIP: sanity test_406 needs >= 2 MDTs [ 6225.349203] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6226.035300] Lustre: DEBUG MARKER: rc=0; [ 6226.035300] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6226.035300] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6226.035300] echo $(hostname -s): $val; [ 6226.035300] rc=$val; [ 6226.035300] fi; [ 6226.035300] exit $rc [ 6226.737036] Lustre: DEBUG MARKER: dmesg [ 6227.292403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 6227.505879] Lustre: DEBUG MARKER: SKIP: sanity test_407 skipping ALWAYS excluded test 407 [ 6227.747915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 408: drop_caches should not hang due to page leaks ==================================== 17:45:29 \(1491414329\) [ 6227.944449] Lustre: DEBUG MARKER: == sanity test 408: drop_caches should not hang due to page leaks ==================================== 17:45:29 (1491414329) [ 6228.759999] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6229.445616] Lustre: DEBUG MARKER: rc=0; [ 6229.445616] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6229.445616] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6229.445616] echo $(hostname -s): $val; [ 6229.445616] rc=$val; [ 6229.445616] fi; [ 6229.445616] exit $rc [ 6230.150715] Lustre: DEBUG MARKER: dmesg [ 6230.682406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 409: Large amount of cross-MDTs hard links on the same file =========================== 17:45:32 \(1491414332\) [ 6230.875666] Lustre: DEBUG MARKER: == sanity test 409: Large amount of cross-MDTs hard links on the same file =========================== 17:45:32 (1491414332) [ 6231.104694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity test_409 We need at least 2 MDTs for this test [ 6231.299717] Lustre: DEBUG MARKER: SKIP: sanity test_409 We need at least 2 MDTs for this test [ 6231.523704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6232.209408] Lustre: DEBUG MARKER: rc=0; [ 6232.209408] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6232.209408] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6232.209408] echo $(hostname -s): $val; [ 6232.209408] rc=$val; [ 6232.209408] fi; [ 6232.209408] exit $rc [ 6232.904223] Lustre: DEBUG MARKER: dmesg [ 6233.432948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 900: umount should not race with any mgc requeue thread =============================== 17:45:35 \(1491414335\) [ 6233.626842] Lustre: DEBUG MARKER: == sanity test 900: umount should not race with any mgc requeue thread =============================== 17:45:35 (1491414335) [ 6234.492393] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6234.793487] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6235.408848] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 6235.791043] LustreError: 5924:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880067542400 x1563855296908560/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6240.405933] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.976060] Lustre: 24640:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491414338/real 1491414338] req@ffff880067542400 x1563855296908640/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491414344 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6242.996984] Lustre: server umount lustre-MDT0000 complete [ 6243.142659] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6243.445957] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6243.445957] grep -q ^lustre-mdt1/ /proc/mounts || [ 6243.445957] zpool export lustre-mdt1 [ 6263.793494] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6263.793494] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6263.793494] if [ $running -ne $mpts ]; then [ 6263.793494] echo $(hostname) env are INSANE!; [ 6263.793494] exit 1; [ 6263.793494] fi [ 6264.118555] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6264.118555] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6264.118555] if [ $running -ne $mpts ]; then [ 6264.118555] echo $(hostname) env are INSANE!; [ 6264.118555] exit 1; [ 6264.118555] fi [ 6265.101598] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 6265.402361] Lustre: DEBUG MARKER: modprobe zfs; [ 6265.402361] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6265.402361] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6265.942272] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6266.226004] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6266.408935] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 6266.411167] Lustre: Skipped 10 previous similar messages [ 6266.521359] LustreError: 26249:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 6266.560875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.762982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6267.066062] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6267.569430] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6267.860024] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6268.148547] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6268.447305] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6268.741941] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6271.528077] Lustre: 5923:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491414368/real 1491414368] req@ffff880051632100 x1563855296908816/t0(0) o8->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491414373 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6280.803236] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 6281.156254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 6281.347498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6281.521881] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 6286.904681] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6287.605985] Lustre: DEBUG MARKER: rc=0; [ 6287.605985] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6287.605985] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6287.605985] echo $(hostname -s): $val; [ 6287.605985] rc=$val; [ 6287.605985] fi; [ 6287.605985] exit $rc [ 6288.323716] Lustre: DEBUG MARKER: dmesg [ 6288.882266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test complete, duration 5589 sec =========================================================== 17:46:30 \(1491414390\) [ 6289.101744] Lustre: DEBUG MARKER: == sanity test complete, duration 5589 sec =========================================================== 17:46:30 (1491414390) [ 6293.783938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanityn ============----- Wed Apr 5 17:46:35 UTC 2017 [ 6293.997042] Lustre: DEBUG MARKER: -----============= acceptance-small: sanityn ============----- Wed Apr 5 17:46:35 UTC 2017 [ 6294.457760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test complete, duration -o sec ============================================================ 17:46:36 \(1491414396\) [ 6294.648721] Lustre: DEBUG MARKER: == sanityn test complete, duration -o sec ============================================================ 17:46:36 (1491414396) [ 6297.273936] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6297.273936] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6297.273936] if [ $running -ne $mpts ]; then [ 6297.273936] echo $(hostname) env are INSANE!; [ 6297.273936] exit 1; [ 6297.273936] fi [ 6297.609895] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 6297.609895] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 6297.609895] if [ $running -ne $mpts ]; then [ 6297.609895] echo $(hostname) env are INSANE!; [ 6297.609895] exit 1; [ 6297.609895] fi [ 6298.929979] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6299.224534] Lustre: DEBUG MARKER: cat /proc/mounts [ 6301.003679] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 6301.329156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 6301.531186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6301.707229] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 6302.061085] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 6302.688638] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6303.341948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 14b 18c 19 28 29 35 36 51a [ 6303.545397] Lustre: DEBUG MARKER: excepting tests: 14b 18c 19 28 29 35 36 51a [ 6303.748584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 33a [ 6303.956990] Lustre: DEBUG MARKER: skipping tests SLOW=no: 33a [ 6304.182714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 1a: check create on 2 mtpt\'s ========================================================= 17:46:46 \(1491414406\) [ 6304.377150] Lustre: DEBUG MARKER: == sanityn test 1a: check create on 2 mtpt's ========================================================= 17:46:46 (1491414406) [ 6304.589141] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6305.283086] Lustre: DEBUG MARKER: rc=0; [ 6305.283086] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6305.283086] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6305.283086] echo $(hostname -s): $val; [ 6305.283086] rc=$val; [ 6305.283086] fi; [ 6305.283086] exit $rc [ 6306.002718] Lustre: DEBUG MARKER: dmesg [ 6306.525226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 1b: check attribute updates on 2 mtpt\'s ============================================== 17:46:48 \(1491414408\) [ 6306.708276] Lustre: DEBUG MARKER: == sanityn test 1b: check attribute updates on 2 mtpt's ============================================== 17:46:48 (1491414408) [ 6306.929517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6307.596424] Lustre: DEBUG MARKER: rc=0; [ 6307.596424] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6307.596424] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6307.596424] echo $(hostname -s): $val; [ 6307.596424] rc=$val; [ 6307.596424] fi; [ 6307.596424] exit $rc [ 6308.285545] Lustre: DEBUG MARKER: dmesg [ 6308.781593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 1c: check after remount attribute updates on 2 mtpt\'s ================================ 17:46:50 \(1491414410\) [ 6308.965293] Lustre: DEBUG MARKER: == sanityn test 1c: check after remount attribute updates on 2 mtpt's ================================ 17:46:50 (1491414410) [ 6309.171389] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6309.821857] Lustre: DEBUG MARKER: rc=0; [ 6309.821857] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6309.821857] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6309.821857] echo $(hostname -s): $val; [ 6309.821857] rc=$val; [ 6309.821857] fi; [ 6309.821857] exit $rc [ 6310.493687] Lustre: DEBUG MARKER: dmesg [ 6310.979217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 17:46:53 \(1491414413\) [ 6311.168868] Lustre: DEBUG MARKER: == sanityn test 1d: unlink on one mountpoint removes file on other =================================== 17:46:53 (1491414413) [ 6311.382827] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6312.027944] Lustre: DEBUG MARKER: rc=0; [ 6312.027944] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6312.027944] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6312.027944] echo $(hostname -s): $val; [ 6312.027944] rc=$val; [ 6312.027944] fi; [ 6312.027944] exit $rc [ 6312.709088] Lustre: DEBUG MARKER: dmesg [ 6313.230772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2a: check cached attribute updates on 2 mtpt\'s ======================================= 17:46:55 \(1491414415\) [ 6313.411303] Lustre: DEBUG MARKER: == sanityn test 2a: check cached attribute updates on 2 mtpt's ======================================= 17:46:55 (1491414415) [ 6313.624361] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6314.282914] Lustre: DEBUG MARKER: rc=0; [ 6314.282914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6314.282914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6314.282914] echo $(hostname -s): $val; [ 6314.282914] rc=$val; [ 6314.282914] fi; [ 6314.282914] exit $rc [ 6314.957594] Lustre: DEBUG MARKER: dmesg [ 6315.462916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2b: check cached attribute updates on 2 mtpt\'s ======================================= 17:46:57 \(1491414417\) [ 6315.652221] Lustre: DEBUG MARKER: == sanityn test 2b: check cached attribute updates on 2 mtpt's ======================================= 17:46:57 (1491414417) [ 6315.860933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6316.519601] Lustre: DEBUG MARKER: rc=0; [ 6316.519601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6316.519601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6316.519601] echo $(hostname -s): $val; [ 6316.519601] rc=$val; [ 6316.519601] fi; [ 6316.519601] exit $rc [ 6317.203299] Lustre: DEBUG MARKER: dmesg [ 6317.726283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2c: check cached attribute updates on 2 mtpt\'s root ================================== 17:46:59 \(1491414419\) [ 6317.908155] Lustre: DEBUG MARKER: == sanityn test 2c: check cached attribute updates on 2 mtpt's root ================================== 17:46:59 (1491414419) [ 6318.122761] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6318.800516] Lustre: DEBUG MARKER: rc=0; [ 6318.800516] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6318.800516] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6318.800516] echo $(hostname -s): $val; [ 6318.800516] rc=$val; [ 6318.800516] fi; [ 6318.800516] exit $rc [ 6319.481741] Lustre: DEBUG MARKER: dmesg [ 6319.976823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2d: check cached attribute updates on 2 mtpt\'s root ================================== 17:47:01 \(1491414421\) [ 6320.153887] Lustre: DEBUG MARKER: == sanityn test 2d: check cached attribute updates on 2 mtpt's root ================================== 17:47:01 (1491414421) [ 6320.404585] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6321.060846] Lustre: DEBUG MARKER: rc=0; [ 6321.060846] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6321.060846] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6321.060846] echo $(hostname -s): $val; [ 6321.060846] rc=$val; [ 6321.060846] fi; [ 6321.060846] exit $rc [ 6321.742543] Lustre: DEBUG MARKER: dmesg [ 6322.247894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2e: check chmod on root is propagated to others ====================================== 17:47:04 \(1491414424\) [ 6322.434715] Lustre: DEBUG MARKER: == sanityn test 2e: check chmod on root is propagated to others ====================================== 17:47:04 (1491414424) [ 6322.713112] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6323.366216] Lustre: DEBUG MARKER: rc=0; [ 6323.366216] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6323.366216] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6323.366216] echo $(hostname -s): $val; [ 6323.366216] rc=$val; [ 6323.366216] fi; [ 6323.366216] exit $rc [ 6324.039405] Lustre: DEBUG MARKER: dmesg [ 6324.540457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2f: check attr\/owner updates on DNE with 2 mtpt\'s ==================================== 17:47:06 \(1491414426\) [ 6324.722982] Lustre: DEBUG MARKER: == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ==================================== 17:47:06 (1491414426) [ 6324.922866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_2f needs \>= 2 MDTs [ 6325.111614] Lustre: DEBUG MARKER: SKIP: sanityn test_2f needs >= 2 MDTs [ 6325.318388] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6325.965459] Lustre: DEBUG MARKER: rc=0; [ 6325.965459] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6325.965459] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6325.965459] echo $(hostname -s): $val; [ 6325.965459] rc=$val; [ 6325.965459] fi; [ 6325.965459] exit $rc [ 6326.628739] Lustre: DEBUG MARKER: dmesg [ 6327.211018] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 2g: check blocks update on sync write ================================================ 17:47:09 \(1491414429\) [ 6327.390436] Lustre: DEBUG MARKER: == sanityn test 2g: check blocks update on sync write ================================================ 17:47:09 (1491414429) [ 6328.062397] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6328.706680] Lustre: DEBUG MARKER: rc=0; [ 6328.706680] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6328.706680] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6328.706680] echo $(hostname -s): $val; [ 6328.706680] rc=$val; [ 6328.706680] fi; [ 6328.706680] exit $rc [ 6329.381920] Lustre: DEBUG MARKER: dmesg [ 6329.896496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 17:47:11 \(1491414431\) [ 6330.081127] Lustre: DEBUG MARKER: == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 17:47:11 (1491414431) [ 6330.283184] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6330.948265] Lustre: DEBUG MARKER: rc=0; [ 6330.948265] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6330.948265] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6330.948265] echo $(hostname -s): $val; [ 6330.948265] rc=$val; [ 6330.948265] fi; [ 6330.948265] exit $rc [ 6331.614449] Lustre: DEBUG MARKER: dmesg [ 6332.118258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 4: fstat validation on multiple mount points ========================================= 17:47:14 \(1491414434\) [ 6332.304164] Lustre: DEBUG MARKER: == sanityn test 4: fstat validation on multiple mount points ========================================= 17:47:14 (1491414434) [ 6333.601982] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6334.236119] Lustre: DEBUG MARKER: rc=0; [ 6334.236119] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6334.236119] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6334.236119] echo $(hostname -s): $val; [ 6334.236119] rc=$val; [ 6334.236119] fi; [ 6334.236119] exit $rc [ 6334.916799] Lustre: DEBUG MARKER: dmesg [ 6335.414410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 5: create a file on one mount, truncate it on the other ============================== 17:47:17 \(1491414437\) [ 6335.602512] Lustre: DEBUG MARKER: == sanityn test 5: create a file on one mount, truncate it on the other ============================== 17:47:17 (1491414437) [ 6335.813319] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6336.482417] Lustre: DEBUG MARKER: rc=0; [ 6336.482417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6336.482417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6336.482417] echo $(hostname -s): $val; [ 6336.482417] rc=$val; [ 6336.482417] fi; [ 6336.482417] exit $rc [ 6337.152573] Lustre: DEBUG MARKER: dmesg [ 6337.659028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 6: remove of open file on other node ================================================= 17:47:19 \(1491414439\) [ 6337.844864] Lustre: DEBUG MARKER: == sanityn test 6: remove of open file on other node ================================================= 17:47:19 (1491414439) [ 6338.077596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6338.729454] Lustre: DEBUG MARKER: rc=0; [ 6338.729454] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6338.729454] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6338.729454] echo $(hostname -s): $val; [ 6338.729454] rc=$val; [ 6338.729454] fi; [ 6338.729454] exit $rc [ 6339.405677] Lustre: DEBUG MARKER: dmesg [ 6339.921924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 7: remove of open directory on other node ============================================ 17:47:21 \(1491414441\) [ 6340.108963] Lustre: DEBUG MARKER: == sanityn test 7: remove of open directory on other node ============================================ 17:47:21 (1491414441) [ 6340.330114] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6340.976860] Lustre: DEBUG MARKER: rc=0; [ 6340.976860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6340.976860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6340.976860] echo $(hostname -s): $val; [ 6340.976860] rc=$val; [ 6340.976860] fi; [ 6340.976860] exit $rc [ 6341.655537] Lustre: DEBUG MARKER: dmesg [ 6342.160763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 8: remove of open special file on other node ========================================= 17:47:24 \(1491414444\) [ 6342.339925] Lustre: DEBUG MARKER: == sanityn test 8: remove of open special file on other node ========================================= 17:47:24 (1491414444) [ 6342.547262] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6343.192439] Lustre: DEBUG MARKER: rc=0; [ 6343.192439] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6343.192439] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6343.192439] echo $(hostname -s): $val; [ 6343.192439] rc=$val; [ 6343.192439] fi; [ 6343.192439] exit $rc [ 6343.863112] Lustre: DEBUG MARKER: dmesg [ 6344.360196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 17:47:26 \(1491414446\) [ 6344.554495] Lustre: DEBUG MARKER: == sanityn test 9: append of file with sub-page size on multiple mounts ============================== 17:47:26 (1491414446) [ 6344.818302] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6345.477800] Lustre: DEBUG MARKER: rc=0; [ 6345.477800] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6345.477800] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6345.477800] echo $(hostname -s): $val; [ 6345.477800] rc=$val; [ 6345.477800] fi; [ 6345.477800] exit $rc [ 6346.159658] Lustre: DEBUG MARKER: dmesg [ 6346.656575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 17:47:28 \(1491414448\) [ 6346.840168] Lustre: DEBUG MARKER: == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 17:47:28 (1491414448) [ 6347.199559] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6347.865465] Lustre: DEBUG MARKER: rc=0; [ 6347.865465] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6347.865465] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6347.865465] echo $(hostname -s): $val; [ 6347.865465] rc=$val; [ 6347.865465] fi; [ 6347.865465] exit $rc [ 6348.540242] Lustre: DEBUG MARKER: dmesg [ 6349.025610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 17:47:31 \(1491414451\) [ 6349.222745] Lustre: DEBUG MARKER: == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 17:47:31 (1491414451) [ 6349.443091] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6350.096545] Lustre: DEBUG MARKER: rc=0; [ 6350.096545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6350.096545] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6350.096545] echo $(hostname -s): $val; [ 6350.096545] rc=$val; [ 6350.096545] fi; [ 6350.096545] exit $rc [ 6350.784193] Lustre: DEBUG MARKER: dmesg [ 6351.306478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 11: execution of file opened for write should return error =========================== 17:47:33 \(1491414453\) [ 6351.494837] Lustre: DEBUG MARKER: == sanityn test 11: execution of file opened for write should return error =========================== 17:47:33 (1491414453) [ 6351.728379] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6352.385636] Lustre: DEBUG MARKER: rc=0; [ 6352.385636] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6352.385636] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6352.385636] echo $(hostname -s): $val; [ 6352.385636] rc=$val; [ 6352.385636] fi; [ 6352.385636] exit $rc [ 6353.060773] Lustre: DEBUG MARKER: dmesg [ 6353.564502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 12: test lock ordering \(link, stat, unlink\) ========================================== 17:47:35 \(1491414455\) [ 6353.755447] Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ========================================== 17:47:35 (1491414455) [ 6479.975542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6480.630695] Lustre: DEBUG MARKER: rc=0; [ 6480.630695] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6480.630695] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6480.630695] echo $(hostname -s): $val; [ 6480.630695] rc=$val; [ 6480.630695] fi; [ 6480.630695] exit $rc [ 6481.330653] Lustre: DEBUG MARKER: dmesg [ 6481.843881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 13: test directory page revocation =================================================== 17:49:43 \(1491414583\) [ 6482.028223] Lustre: DEBUG MARKER: == sanityn test 13: test directory page revocation =================================================== 17:49:43 (1491414583) [ 6482.261438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6482.960522] Lustre: DEBUG MARKER: rc=0; [ 6482.960522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6482.960522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6482.960522] echo $(hostname -s): $val; [ 6482.960522] rc=$val; [ 6482.960522] fi; [ 6482.960522] exit $rc [ 6483.658830] Lustre: DEBUG MARKER: dmesg [ 6484.185800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14aa: execution of file open for write returns -ETXTBSY ============================== 17:49:46 \(1491414586\) [ 6484.380021] Lustre: DEBUG MARKER: == sanityn test 14aa: execution of file open for write returns -ETXTBSY ============================== 17:49:46 (1491414586) [ 6484.666915] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6485.325085] Lustre: DEBUG MARKER: rc=0; [ 6485.325085] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6485.325085] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6485.325085] echo $(hostname -s): $val; [ 6485.325085] rc=$val; [ 6485.325085] fi; [ 6485.325085] exit $rc [ 6486.020438] Lustre: DEBUG MARKER: dmesg [ 6486.525937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14ab: open\(RDWR\) of executing file returns -ETXTBSY ================================== 17:49:48 \(1491414588\) [ 6486.732396] Lustre: DEBUG MARKER: == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ================================== 17:49:48 (1491414588) [ 6486.968357] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6487.687322] Lustre: DEBUG MARKER: rc=0; [ 6487.687322] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6487.687322] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6487.687322] echo $(hostname -s): $val; [ 6487.687322] rc=$val; [ 6487.687322] fi; [ 6487.687322] exit $rc [ 6488.389754] Lustre: DEBUG MARKER: dmesg [ 6488.981360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_14b skipping ALWAYS excluded test 14b [ 6489.167545] Lustre: DEBUG MARKER: SKIP: sanityn test_14b skipping ALWAYS excluded test 14b [ 6489.398450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14c: open\(O_TRUNC\) of executing file return -ETXTBSY ================================= 17:49:51 \(1491414591\) [ 6489.579379] Lustre: DEBUG MARKER: == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ================================= 17:49:51 (1491414591) [ 6489.808446] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6490.477959] Lustre: DEBUG MARKER: rc=0; [ 6490.477959] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6490.477959] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6490.477959] echo $(hostname -s): $val; [ 6490.477959] rc=$val; [ 6490.477959] fi; [ 6490.477959] exit $rc [ 6491.153698] Lustre: DEBUG MARKER: dmesg [ 6491.656891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 14d: chmod of executing file is still possible ======================================= 17:49:53 \(1491414593\) [ 6491.841215] Lustre: DEBUG MARKER: == sanityn test 14d: chmod of executing file is still possible ======================================= 17:49:53 (1491414593) [ 6492.061208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark chmod [ 6492.251167] Lustre: DEBUG MARKER: chmod [ 6492.459798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6493.109507] Lustre: DEBUG MARKER: rc=0; [ 6493.109507] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6493.109507] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6493.109507] echo $(hostname -s): $val; [ 6493.109507] rc=$val; [ 6493.109507] fi; [ 6493.109507] exit $rc [ 6493.782398] Lustre: DEBUG MARKER: dmesg [ 6494.322928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 15: test out-of-space with multiple writers ========================================== 17:49:56 \(1491414596\) [ 6494.497076] Lustre: DEBUG MARKER: == sanityn test 15: test out-of-space with multiple writers ========================================== 17:49:56 (1491414596) [ 6495.900796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_15 oos2.sh: 15988864kB free gt MAXFREE 800000kB, increase 800000 \(or reduce test fs size\) to proceed [ 6496.107546] Lustre: DEBUG MARKER: SKIP: sanityn test_15 oos2.sh: 15988864kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed [ 6496.319400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6496.609546] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 6496.910077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6498.213491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6504.597537] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6505.275241] Lustre: DEBUG MARKER: rc=0; [ 6505.275241] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6505.275241] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6505.275241] echo $(hostname -s): $val; [ 6505.275241] rc=$val; [ 6505.275241] fi; [ 6505.275241] exit $rc [ 6505.954160] Lustre: DEBUG MARKER: dmesg [ 6506.517915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 16: 500 iterations of dual-mount fsx ================================================= 17:50:08 \(1491414608\) [ 6506.692962] Lustre: DEBUG MARKER: == sanityn test 16: 500 iterations of dual-mount fsx ================================================= 17:50:08 (1491414608) [ 6513.248182] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6513.925418] Lustre: DEBUG MARKER: rc=0; [ 6513.925418] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6513.925418] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6513.925418] echo $(hostname -s): $val; [ 6513.925418] rc=$val; [ 6513.925418] fi; [ 6513.925418] exit $rc [ 6514.611369] Lustre: DEBUG MARKER: dmesg [ 6515.226460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 17: resource creation\/LVB creation race ============================================== 17:50:17 \(1491414617\) [ 6515.416891] Lustre: DEBUG MARKER: == sanityn test 17: resource creation/LVB creation race ============================================== 17:50:17 (1491414617) [ 6518.134456] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6518.822983] Lustre: DEBUG MARKER: rc=0; [ 6518.822983] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6518.822983] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6518.822983] echo $(hostname -s): $val; [ 6518.822983] rc=$val; [ 6518.822983] fi; [ 6518.822983] exit $rc [ 6519.533983] Lustre: DEBUG MARKER: dmesg [ 6520.037181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 18: mmap sanity check ================================================================ 17:50:22 \(1491414622\) [ 6520.233940] Lustre: DEBUG MARKER: == sanityn test 18: mmap sanity check ================================================================ 17:50:22 (1491414622) [ 6524.015876] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6524.672607] Lustre: DEBUG MARKER: rc=0; [ 6524.672607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6524.672607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6524.672607] echo $(hostname -s): $val; [ 6524.672607] rc=$val; [ 6524.672607] fi; [ 6524.672607] exit $rc [ 6525.356685] Lustre: DEBUG MARKER: dmesg [ 6525.840344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_19 skipping ALWAYS excluded test 19 [ 6526.036188] Lustre: DEBUG MARKER: SKIP: sanityn test_19 skipping ALWAYS excluded test 19 [ 6526.274534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 20: test extra readahead page left in cache ========================================== 17:50:28 \(1491414628\) [ 6526.488856] Lustre: DEBUG MARKER: == sanityn test 20: test extra readahead page left in cache ========================================== 17:50:28 (1491414628) [ 6526.850702] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6527.552506] Lustre: DEBUG MARKER: rc=0; [ 6527.552506] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6527.552506] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6527.552506] echo $(hostname -s): $val; [ 6527.552506] rc=$val; [ 6527.552506] fi; [ 6527.552506] exit $rc [ 6528.239847] Lustre: DEBUG MARKER: dmesg [ 6528.760531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 21: Try to remove mountpoint on another dir ========================================= 17:50:30 \(1491414630\) [ 6528.945979] Lustre: DEBUG MARKER: == sanityn test 21: Try to remove mountpoint on another dir ========================================= 17:50:30 (1491414630) [ 6529.182225] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6529.853977] Lustre: DEBUG MARKER: rc=0; [ 6529.853977] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6529.853977] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6529.853977] echo $(hostname -s): $val; [ 6529.853977] rc=$val; [ 6529.853977] fi; [ 6529.853977] exit $rc [ 6530.584095] Lustre: DEBUG MARKER: dmesg [ 6531.253321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 23: others should see updated atime while another read=============================== 17:50:33 \(1491414633\) [ 6531.428746] Lustre: DEBUG MARKER: == sanityn test 23: others should see updated atime while another read=============================== 17:50:33 (1491414633) [ 6531.604267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.*MDT0000*.atime_diff [ 6592.955122] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6593.621020] Lustre: DEBUG MARKER: rc=0; [ 6593.621020] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6593.621020] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6593.621020] echo $(hostname -s): $val; [ 6593.621020] rc=$val; [ 6593.621020] fi; [ 6593.621020] exit $rc [ 6594.311751] Lustre: DEBUG MARKER: dmesg [ 6594.834286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 17:51:36 \(1491414696\) [ 6595.024770] Lustre: DEBUG MARKER: == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 17:51:36 (1491414696) [ 6595.279772] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6595.943386] Lustre: DEBUG MARKER: rc=0; [ 6595.943386] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6595.943386] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6595.943386] echo $(hostname -s): $val; [ 6595.943386] rc=$val; [ 6595.943386] fi; [ 6595.943386] exit $rc [ 6596.624024] Lustre: DEBUG MARKER: dmesg [ 6597.142179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 24b: lfs df should show both filesystems ============================================= 17:51:39 \(1491414699\) [ 6597.317160] Lustre: DEBUG MARKER: == sanityn test 24b: lfs df should show both filesystems ============================================= 17:51:39 (1491414699) [ 6597.537103] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6598.188514] Lustre: DEBUG MARKER: rc=0; [ 6598.188514] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6598.188514] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6598.188514] echo $(hostname -s): $val; [ 6598.188514] rc=$val; [ 6598.188514] fi; [ 6598.188514] exit $rc [ 6598.920849] Lustre: DEBUG MARKER: dmesg [ 6599.445595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 25a: change ACL on one mountpoint be seen on another ================================= 17:51:41 \(1491414701\) [ 6599.639622] Lustre: DEBUG MARKER: == sanityn test 25a: change ACL on one mountpoint be seen on another ================================= 17:51:41 (1491414701) [ 6600.211405] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6600.869417] Lustre: DEBUG MARKER: rc=0; [ 6600.869417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6600.869417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6600.869417] echo $(hostname -s): $val; [ 6600.869417] rc=$val; [ 6600.869417] fi; [ 6600.869417] exit $rc [ 6601.552408] Lustre: DEBUG MARKER: dmesg [ 6602.063071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ================ 17:51:44 \(1491414704\) [ 6602.267859] Lustre: DEBUG MARKER: == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ================ 17:51:44 (1491414704) [ 6602.460264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_25b needs \>= 2 MDTs [ 6602.664572] Lustre: DEBUG MARKER: SKIP: sanityn test_25b needs >= 2 MDTs [ 6602.884964] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6603.542457] Lustre: DEBUG MARKER: rc=0; [ 6603.542457] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6603.542457] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6603.542457] echo $(hostname -s): $val; [ 6603.542457] rc=$val; [ 6603.542457] fi; [ 6603.542457] exit $rc [ 6604.244629] Lustre: DEBUG MARKER: dmesg [ 6604.747537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 26a: allow mtime to get older ======================================================== 17:51:46 \(1491414706\) [ 6604.935290] Lustre: DEBUG MARKER: == sanityn test 26a: allow mtime to get older ======================================================== 17:51:46 (1491414706) [ 6606.148479] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6606.801768] Lustre: DEBUG MARKER: rc=0; [ 6606.801768] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6606.801768] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6606.801768] echo $(hostname -s): $val; [ 6606.801768] rc=$val; [ 6606.801768] fi; [ 6606.801768] exit $rc [ 6607.484390] Lustre: DEBUG MARKER: dmesg [ 6607.994403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 26b: sync mtime between ost and mds ================================================== 17:51:50 \(1491414710\) [ 6608.189680] Lustre: DEBUG MARKER: == sanityn test 26b: sync mtime between ost and mds ================================================== 17:51:50 (1491414710) [ 6610.410705] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6611.089310] Lustre: DEBUG MARKER: rc=0; [ 6611.089310] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6611.089310] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6611.089310] echo $(hostname -s): $val; [ 6611.089310] rc=$val; [ 6611.089310] fi; [ 6611.089310] exit $rc [ 6611.764790] Lustre: DEBUG MARKER: dmesg [ 6612.282128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 27: align non-overlapping extent locks from request ================================== 17:51:54 \(1491414714\) [ 6612.473540] Lustre: DEBUG MARKER: == sanityn test 27: align non-overlapping extent locks from request ================================== 17:51:54 (1491414714) [ 6612.998254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 1 started [ 6613.197739] Lustre: DEBUG MARKER: dd 1 started [ 6613.422158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 2 started [ 6613.800083] Lustre: DEBUG MARKER: dd 2 started [ 6615.009171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark dd 3 finished [ 6615.193538] Lustre: DEBUG MARKER: dd 3 finished [ 6615.402247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6616.065946] Lustre: DEBUG MARKER: rc=0; [ 6616.065946] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6616.065946] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6616.065946] echo $(hostname -s): $val; [ 6616.065946] rc=$val; [ 6616.065946] fi; [ 6616.065946] exit $rc [ 6616.759378] Lustre: DEBUG MARKER: dmesg [ 6617.258710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_28 skipping ALWAYS excluded test 28 [ 6617.446448] Lustre: DEBUG MARKER: SKIP: sanityn test_28 skipping ALWAYS excluded test 28 [ 6617.674114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_29 skipping ALWAYS excluded test 29 [ 6617.878183] Lustre: DEBUG MARKER: SKIP: sanityn test_29 skipping ALWAYS excluded test 29 [ 6618.104604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 30: recreate file race =============================================================== 17:52:00 \(1491414720\) [ 6618.303726] Lustre: DEBUG MARKER: == sanityn test 30: recreate file race =============================================================== 17:52:00 (1491414720) [ 6620.546961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6621.199142] Lustre: DEBUG MARKER: rc=0; [ 6621.199142] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6621.199142] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6621.199142] echo $(hostname -s): $val; [ 6621.199142] rc=$val; [ 6621.199142] fi; [ 6621.199142] exit $rc [ 6621.879067] Lustre: DEBUG MARKER: dmesg [ 6622.369732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 31a: voluntary cancel \/ blocking ast race============================================= 17:52:04 \(1491414724\) [ 6622.562998] Lustre: DEBUG MARKER: == sanityn test 31a: voluntary cancel / blocking ast race============================================= 17:52:04 (1491414724) [ 6622.795850] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6623.456326] Lustre: DEBUG MARKER: rc=0; [ 6623.456326] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6623.456326] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6623.456326] echo $(hostname -s): $val; [ 6623.456326] rc=$val; [ 6623.456326] fi; [ 6623.456326] exit $rc [ 6624.158674] Lustre: DEBUG MARKER: dmesg [ 6624.665316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 31b: voluntary OST cancel \/ blocking ast race========================================= 17:52:06 \(1491414726\) [ 6624.856351] Lustre: DEBUG MARKER: == sanityn test 31b: voluntary OST cancel / blocking ast race========================================= 17:52:06 (1491414726) [ 6625.073609] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 6625.377482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 6627.709474] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6634.742187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6635.410705] Lustre: DEBUG MARKER: rc=0; [ 6635.410705] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6635.410705] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6635.410705] echo $(hostname -s): $val; [ 6635.410705] rc=$val; [ 6635.410705] fi; [ 6635.410705] exit $rc [ 6636.096203] Lustre: DEBUG MARKER: dmesg [ 6636.628406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 32a: lockless truncate =============================================================== 17:52:18 \(1491414738\) [ 6636.814220] Lustre: DEBUG MARKER: == sanityn test 32a: lockless truncate =============================================================== 17:52:18 (1491414738) [ 6637.141755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark checking cached lockless truncate [ 6637.332510] Lustre: DEBUG MARKER: checking cached lockless truncate [ 6637.565278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark checking not cached lockless truncate [ 6637.752826] Lustre: DEBUG MARKER: checking not cached lockless truncate [ 6638.198440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark disabled lockless truncate [ 6638.394947] Lustre: DEBUG MARKER: disabled lockless truncate [ 6638.663478] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6639.318094] Lustre: DEBUG MARKER: rc=0; [ 6639.318094] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6639.318094] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6639.318094] echo $(hostname -s): $val; [ 6639.318094] rc=$val; [ 6639.318094] fi; [ 6639.318094] exit $rc [ 6639.992697] Lustre: DEBUG MARKER: dmesg [ 6640.504774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 32b: lockless i\/o ==================================================================== 17:52:22 \(1491414742\) [ 6640.694448] Lustre: DEBUG MARKER: == sanityn test 32b: lockless i/o ==================================================================== 17:52:22 (1491414742) [ 6647.737195] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6648.408323] Lustre: DEBUG MARKER: rc=0; [ 6648.408323] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6648.408323] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6648.408323] echo $(hostname -s): $val; [ 6648.408323] rc=$val; [ 6648.408323] fi; [ 6648.408323] exit $rc [ 6649.091738] Lustre: DEBUG MARKER: dmesg [ 6649.673289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_33a skipping SLOW test 33a [ 6649.841405] Lustre: DEBUG MARKER: SKIP: sanityn test_33a skipping SLOW test 33a [ 6650.073321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 33b: COS: cross create\/delete, 2 clients, benchmark under remote dir ================= 17:52:32 \(1491414752\) [ 6650.260278] Lustre: DEBUG MARKER: == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ================= 17:52:32 (1491414752) [ 6650.490235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_33b needs \>= 2 MDTs [ 6650.675951] Lustre: DEBUG MARKER: SKIP: sanityn test_33b needs >= 2 MDTs [ 6650.880252] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6651.551574] Lustre: DEBUG MARKER: rc=0; [ 6651.551574] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6651.551574] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6651.551574] echo $(hostname -s): $val; [ 6651.551574] rc=$val; [ 6651.551574] fi; [ 6651.551574] exit $rc [ 6652.267271] Lustre: DEBUG MARKER: dmesg [ 6653.032650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel =========================== 17:52:35 \(1491414755\) [ 6653.217537] Lustre: DEBUG MARKER: == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel =========================== 17:52:35 (1491414755) [ 6653.423979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_33c needs \>= 2 MDTs [ 6653.607291] Lustre: DEBUG MARKER: SKIP: sanityn test_33c needs >= 2 MDTs [ 6653.823089] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6654.504476] Lustre: DEBUG MARKER: rc=0; [ 6654.504476] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6654.504476] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6654.504476] echo $(hostname -s): $val; [ 6654.504476] rc=$val; [ 6654.504476] fi; [ 6654.504476] exit $rc [ 6655.186235] Lustre: DEBUG MARKER: dmesg [ 6655.708422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 33d: DNE distributed operation should trigger COS ==================================== 17:52:37 \(1491414757\) [ 6655.901460] Lustre: DEBUG MARKER: == sanityn test 33d: DNE distributed operation should trigger COS ==================================== 17:52:37 (1491414757) [ 6656.116735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_33d needs \>= 2 MDTs [ 6656.311832] Lustre: DEBUG MARKER: SKIP: sanityn test_33d needs >= 2 MDTs [ 6656.532366] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6657.224656] Lustre: DEBUG MARKER: rc=0; [ 6657.224656] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6657.224656] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6657.224656] echo $(hostname -s): $val; [ 6657.224656] rc=$val; [ 6657.224656] fi; [ 6657.224656] exit $rc [ 6657.931951] Lustre: DEBUG MARKER: dmesg [ 6658.479993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 33e: DNE local operation shouldn\'t trigger COS ======================================= 17:52:40 \(1491414760\) [ 6658.660670] Lustre: DEBUG MARKER: == sanityn test 33e: DNE local operation shouldn't trigger COS ======================================= 17:52:40 (1491414760) [ 6658.867145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_33e needs \>= 2 MDTs [ 6659.057198] Lustre: DEBUG MARKER: SKIP: sanityn test_33e needs >= 2 MDTs [ 6659.278465] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6659.933391] Lustre: DEBUG MARKER: rc=0; [ 6659.933391] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6659.933391] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6659.933391] echo $(hostname -s): $val; [ 6659.933391] rc=$val; [ 6659.933391] fi; [ 6659.933391] exit $rc [ 6660.639275] Lustre: DEBUG MARKER: dmesg [ 6661.144720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 34: no lock timeout under IO ========================================================= 17:52:43 \(1491414763\) [ 6661.344671] Lustre: DEBUG MARKER: == sanityn test 34: no lock timeout under IO ========================================================= 17:52:43 (1491414763) [ 6694.818877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6695.007990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6695.430315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6695.599225] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6696.378420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6696.578141] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6696.980350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6697.160110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6697.367832] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6698.034648] Lustre: DEBUG MARKER: rc=0; [ 6698.034648] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6698.034648] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6698.034648] echo $(hostname -s): $val; [ 6698.034648] rc=$val; [ 6698.034648] fi; [ 6698.034648] exit $rc [ 6698.723298] Lustre: DEBUG MARKER: dmesg [ 6699.752049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6699.944129] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6700.359322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6700.549541] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880037896800.ost_server_uuid in FULL state after 0 sec [ 6700.768447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_35 skipping ALWAYS excluded test 35 [ 6700.962727] Lustre: DEBUG MARKER: SKIP: sanityn test_35 skipping ALWAYS excluded test 35 [ 6701.180705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_36 skipping ALWAYS excluded test 36 [ 6701.372061] Lustre: DEBUG MARKER: SKIP: sanityn test_36 skipping ALWAYS excluded test 36 [ 6701.598625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 37: check i_size is not updated for directory on close \(bug 18695\) =================== 17:53:23 \(1491414803\) [ 6701.796490] Lustre: DEBUG MARKER: == sanityn test 37: check i_size is not updated for directory on close (bug 18695) =================== 17:53:23 (1491414803) [ 6708.194709] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6708.859772] Lustre: DEBUG MARKER: rc=0; [ 6708.859772] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6708.859772] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6708.859772] echo $(hostname -s): $val; [ 6708.859772] rc=$val; [ 6708.859772] fi; [ 6708.859772] exit $rc [ 6709.559621] Lustre: DEBUG MARKER: dmesg [ 6710.111021] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39a: test from 11063 ================================================================= 17:53:32 \(1491414812\) [ 6710.302190] Lustre: DEBUG MARKER: == sanityn test 39a: test from 11063 ================================================================= 17:53:32 (1491414812) [ 6711.553805] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6712.236022] Lustre: DEBUG MARKER: rc=0; [ 6712.236022] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6712.236022] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6712.236022] echo $(hostname -s): $val; [ 6712.236022] rc=$val; [ 6712.236022] fi; [ 6712.236022] exit $rc [ 6712.925456] Lustre: DEBUG MARKER: dmesg [ 6713.467880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39b: 11063 problem 1 ================================================================= 17:53:35 \(1491414815\) [ 6713.657157] Lustre: DEBUG MARKER: == sanityn test 39b: 11063 problem 1 ================================================================= 17:53:35 (1491414815) [ 6715.599111] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6716.258875] Lustre: DEBUG MARKER: rc=0; [ 6716.258875] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6716.258875] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6716.258875] echo $(hostname -s): $val; [ 6716.258875] rc=$val; [ 6716.258875] fi; [ 6716.258875] exit $rc [ 6716.944620] Lustre: DEBUG MARKER: dmesg [ 6717.456339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39c: check truncate mtime update ===================================================== 17:53:39 \(1491414819\) [ 6717.653388] Lustre: DEBUG MARKER: == sanityn test 39c: check truncate mtime update ===================================================== 17:53:39 (1491414819) [ 6719.617705] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6720.285738] Lustre: DEBUG MARKER: rc=0; [ 6720.285738] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6720.285738] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6720.285738] echo $(hostname -s): $val; [ 6720.285738] rc=$val; [ 6720.285738] fi; [ 6720.285738] exit $rc [ 6720.983138] Lustre: DEBUG MARKER: dmesg [ 6721.506153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 39d: sync write should update mtime ================================================== 17:53:43 \(1491414823\) [ 6721.701882] Lustre: DEBUG MARKER: == sanityn test 39d: sync write should update mtime ================================================== 17:53:43 (1491414823) [ 6722.042764] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6722.704706] Lustre: DEBUG MARKER: rc=0; [ 6722.704706] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6722.704706] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6722.704706] echo $(hostname -s): $val; [ 6722.704706] rc=$val; [ 6722.704706] fi; [ 6722.704706] exit $rc [ 6723.414687] Lustre: DEBUG MARKER: dmesg [ 6723.912584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 40a: pdirops: create vs others ======================================================= 17:53:45 \(1491414825\) [ 6724.110573] Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ======================================================= 17:53:45 (1491414825) [ 6724.285114] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6724.440562] LustreError: 27356:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6739.442043] LustreError: 27356:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6739.635352] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6740.314810] Lustre: DEBUG MARKER: rc=0; [ 6740.314810] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6740.314810] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6740.314810] echo $(hostname -s): $val; [ 6740.314810] rc=$val; [ 6740.314810] fi; [ 6740.314810] exit $rc [ 6741.012690] Lustre: DEBUG MARKER: dmesg [ 6741.531582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 40b: pdirops: open\|create and others ================================================= 17:54:03 \(1491414843\) [ 6741.723851] Lustre: DEBUG MARKER: == sanityn test 40b: pdirops: open|create and others ================================================= 17:54:03 (1491414843) [ 6741.904548] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6742.053446] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6757.055025] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6757.250387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6757.915084] Lustre: DEBUG MARKER: rc=0; [ 6757.915084] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6757.915084] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6757.915084] echo $(hostname -s): $val; [ 6757.915084] rc=$val; [ 6757.915084] fi; [ 6757.915084] exit $rc [ 6758.605485] Lustre: DEBUG MARKER: dmesg [ 6759.137382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 40c: pdirops: link and others ======================================================== 17:54:21 \(1491414861\) [ 6759.331556] Lustre: DEBUG MARKER: == sanityn test 40c: pdirops: link and others ======================================================== 17:54:21 (1491414861) [ 6759.513595] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6759.664854] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6774.667036] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6774.855928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6775.534802] Lustre: DEBUG MARKER: rc=0; [ 6775.534802] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6775.534802] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6775.534802] echo $(hostname -s): $val; [ 6775.534802] rc=$val; [ 6775.534802] fi; [ 6775.534802] exit $rc [ 6776.221158] Lustre: DEBUG MARKER: dmesg [ 6776.740385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 40d: pdirops: unlink and others ====================================================== 17:54:38 \(1491414878\) [ 6776.926863] Lustre: DEBUG MARKER: == sanityn test 40d: pdirops: unlink and others ====================================================== 17:54:38 (1491414878) [ 6777.132466] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6777.282976] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6792.285053] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6792.472623] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6793.136667] Lustre: DEBUG MARKER: rc=0; [ 6793.136667] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6793.136667] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6793.136667] echo $(hostname -s): $val; [ 6793.136667] rc=$val; [ 6793.136667] fi; [ 6793.136667] exit $rc [ 6793.816438] Lustre: DEBUG MARKER: dmesg [ 6794.330728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 40e: pdirops: rename and others ====================================================== 17:54:56 \(1491414896\) [ 6794.508191] Lustre: DEBUG MARKER: == sanityn test 40e: pdirops: rename and others ====================================================== 17:54:56 (1491414896) [ 6794.693117] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6794.844151] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6809.846044] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6810.047662] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6810.707904] Lustre: DEBUG MARKER: rc=0; [ 6810.707904] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6810.707904] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6810.707904] echo $(hostname -s): $val; [ 6810.707904] rc=$val; [ 6810.707904] fi; [ 6810.707904] exit $rc [ 6811.402753] Lustre: DEBUG MARKER: dmesg [ 6811.903345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41a: pdirops: create vs mkdir ======================================================== 17:55:13 \(1491414913\) [ 6812.085433] Lustre: DEBUG MARKER: == sanityn test 41a: pdirops: create vs mkdir ======================================================== 17:55:13 (1491414913) [ 6812.281085] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6812.431488] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6827.433047] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6828.634199] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6829.310457] Lustre: DEBUG MARKER: rc=0; [ 6829.310457] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6829.310457] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6829.310457] echo $(hostname -s): $val; [ 6829.310457] rc=$val; [ 6829.310457] fi; [ 6829.310457] exit $rc [ 6830.008469] Lustre: DEBUG MARKER: dmesg [ 6830.521643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41b: pdirops: create vs create ======================================================= 17:55:32 \(1491414932\) [ 6830.705821] Lustre: DEBUG MARKER: == sanityn test 41b: pdirops: create vs create ======================================================= 17:55:32 (1491414932) [ 6830.877139] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6831.027793] LustreError: 27354:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6846.030034] LustreError: 27354:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6847.230469] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6847.902861] Lustre: DEBUG MARKER: rc=0; [ 6847.902861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6847.902861] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6847.902861] echo $(hostname -s): $val; [ 6847.902861] rc=$val; [ 6847.902861] fi; [ 6847.902861] exit $rc [ 6848.585469] Lustre: DEBUG MARKER: dmesg [ 6849.088755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41c: pdirops: create vs link ========================================================= 17:55:51 \(1491414951\) [ 6849.271799] Lustre: DEBUG MARKER: == sanityn test 41c: pdirops: create vs link ========================================================= 17:55:51 (1491414951) [ 6849.441558] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6865.791285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6866.462663] Lustre: DEBUG MARKER: rc=0; [ 6866.462663] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6866.462663] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6866.462663] echo $(hostname -s): $val; [ 6866.462663] rc=$val; [ 6866.462663] fi; [ 6866.462663] exit $rc [ 6867.151377] Lustre: DEBUG MARKER: dmesg [ 6867.705963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41d: pdirops: create vs unlink ======================================================= 17:56:09 \(1491414969\) [ 6867.885096] Lustre: DEBUG MARKER: == sanityn test 41d: pdirops: create vs unlink ======================================================= 17:56:09 (1491414969) [ 6868.055867] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6868.207925] LustreError: 27357:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6868.210246] LustreError: 27357:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 6883.212033] LustreError: 27357:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6883.214154] LustreError: 27357:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 6884.416700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6885.092680] Lustre: DEBUG MARKER: rc=0; [ 6885.092680] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6885.092680] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6885.092680] echo $(hostname -s): $val; [ 6885.092680] rc=$val; [ 6885.092680] fi; [ 6885.092680] exit $rc [ 6885.779566] Lustre: DEBUG MARKER: dmesg [ 6886.352727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41e: pdirops: create and rename \(tgt\) ================================================ 17:56:28 \(1491414988\) [ 6886.543574] Lustre: DEBUG MARKER: == sanityn test 41e: pdirops: create and rename (tgt) ================================================ 17:56:28 (1491414988) [ 6886.715882] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6903.061679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6903.745450] Lustre: DEBUG MARKER: rc=0; [ 6903.745450] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6903.745450] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6903.745450] echo $(hostname -s): $val; [ 6903.745450] rc=$val; [ 6903.745450] fi; [ 6903.745450] exit $rc [ 6904.443259] Lustre: DEBUG MARKER: dmesg [ 6904.944266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41f: pdirops: create and rename \(src\) ================================================ 17:56:46 \(1491415006\) [ 6905.145859] Lustre: DEBUG MARKER: == sanityn test 41f: pdirops: create and rename (src) ================================================ 17:56:46 (1491415006) [ 6905.309972] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6921.665817] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6922.334651] Lustre: DEBUG MARKER: rc=0; [ 6922.334651] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6922.334651] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6922.334651] echo $(hostname -s): $val; [ 6922.334651] rc=$val; [ 6922.334651] fi; [ 6922.334651] exit $rc [ 6923.028369] Lustre: DEBUG MARKER: dmesg [ 6923.565253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41g: pdirops: create vs getattr ====================================================== 17:57:05 \(1491415025\) [ 6923.757262] Lustre: DEBUG MARKER: == sanityn test 41g: pdirops: create vs getattr ====================================================== 17:57:05 (1491415025) [ 6923.931405] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6940.292758] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6940.947962] Lustre: DEBUG MARKER: rc=0; [ 6940.947962] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6940.947962] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6940.947962] echo $(hostname -s): $val; [ 6940.947962] rc=$val; [ 6940.947962] fi; [ 6940.947962] exit $rc [ 6941.643844] Lustre: DEBUG MARKER: dmesg [ 6942.160658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 41h: pdirops: create vs readdir ====================================================== 17:57:24 \(1491415044\) [ 6942.347774] Lustre: DEBUG MARKER: == sanityn test 41h: pdirops: create vs readdir ====================================================== 17:57:24 (1491415044) [ 6942.516584] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6942.669555] LustreError: 26229:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 6942.671860] LustreError: 26229:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 6957.673029] LustreError: 26229:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 6957.675250] LustreError: 26229:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 6958.901032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6959.561042] Lustre: DEBUG MARKER: rc=0; [ 6959.561042] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6959.561042] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6959.561042] echo $(hostname -s): $val; [ 6959.561042] rc=$val; [ 6959.561042] fi; [ 6959.561042] exit $rc [ 6960.264444] Lustre: DEBUG MARKER: dmesg [ 6960.792789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 17:57:42 \(1491415062\) [ 6960.971728] Lustre: DEBUG MARKER: == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 17:57:42 (1491415062) [ 6961.146790] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6977.497026] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6978.167226] Lustre: DEBUG MARKER: rc=0; [ 6978.167226] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6978.167226] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6978.167226] echo $(hostname -s): $val; [ 6978.167226] rc=$val; [ 6978.167226] fi; [ 6978.167226] exit $rc [ 6978.874213] Lustre: DEBUG MARKER: dmesg [ 6979.438939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42b: pdirops: mkdir vs create ======================================================== 17:58:01 \(1491415081\) [ 6979.637079] Lustre: DEBUG MARKER: == sanityn test 42b: pdirops: mkdir vs create ======================================================== 17:58:01 (1491415081) [ 6979.802760] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 6996.158029] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6996.816691] Lustre: DEBUG MARKER: rc=0; [ 6996.816691] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6996.816691] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6996.816691] echo $(hostname -s): $val; [ 6996.816691] rc=$val; [ 6996.816691] fi; [ 6996.816691] exit $rc [ 6997.495466] Lustre: DEBUG MARKER: dmesg [ 6998.010690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42c: pdirops: mkdir vs link ========================================================== 17:58:20 \(1491415100\) [ 6998.194795] Lustre: DEBUG MARKER: == sanityn test 42c: pdirops: mkdir vs link ========================================================== 17:58:20 (1491415100) [ 6998.387771] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7014.745686] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7015.447161] Lustre: DEBUG MARKER: rc=0; [ 7015.447161] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7015.447161] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7015.447161] echo $(hostname -s): $val; [ 7015.447161] rc=$val; [ 7015.447161] fi; [ 7015.447161] exit $rc [ 7016.144432] Lustre: DEBUG MARKER: dmesg [ 7016.682139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 17:58:38 \(1491415118\) [ 7016.890209] Lustre: DEBUG MARKER: == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 17:58:38 (1491415118) [ 7017.050348] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7033.398936] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7034.059820] Lustre: DEBUG MARKER: rc=0; [ 7034.059820] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7034.059820] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7034.059820] echo $(hostname -s): $val; [ 7034.059820] rc=$val; [ 7034.059820] fi; [ 7034.059820] exit $rc [ 7034.734848] Lustre: DEBUG MARKER: dmesg [ 7035.249510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42e: pdirops: mkdir and rename \(tgt\) ================================================= 17:58:57 \(1491415137\) [ 7035.457342] Lustre: DEBUG MARKER: == sanityn test 42e: pdirops: mkdir and rename (tgt) ================================================= 17:58:57 (1491415137) [ 7035.631557] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7051.996302] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7052.660258] Lustre: DEBUG MARKER: rc=0; [ 7052.660258] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7052.660258] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7052.660258] echo $(hostname -s): $val; [ 7052.660258] rc=$val; [ 7052.660258] fi; [ 7052.660258] exit $rc [ 7053.363097] Lustre: DEBUG MARKER: dmesg [ 7053.867440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42f: pdirops: mkdir and rename \(src\) ================================================= 17:59:15 \(1491415155\) [ 7054.052663] Lustre: DEBUG MARKER: == sanityn test 42f: pdirops: mkdir and rename (src) ================================================= 17:59:15 (1491415155) [ 7054.222495] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7070.573408] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7071.257896] Lustre: DEBUG MARKER: rc=0; [ 7071.257896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7071.257896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7071.257896] echo $(hostname -s): $val; [ 7071.257896] rc=$val; [ 7071.257896] fi; [ 7071.257896] exit $rc [ 7071.947965] Lustre: DEBUG MARKER: dmesg [ 7072.480321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 17:59:34 \(1491415174\) [ 7072.659900] Lustre: DEBUG MARKER: == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 17:59:34 (1491415174) [ 7072.834130] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7072.989167] LustreError: 27357:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms [ 7072.991439] LustreError: 27357:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 7087.993033] LustreError: 27357:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 awake [ 7087.995140] LustreError: 27357:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 7089.197697] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7089.875527] Lustre: DEBUG MARKER: rc=0; [ 7089.875527] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7089.875527] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7089.875527] echo $(hostname -s): $val; [ 7089.875527] rc=$val; [ 7089.875527] fi; [ 7089.875527] exit $rc [ 7090.562934] Lustre: DEBUG MARKER: dmesg [ 7091.085101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 17:59:53 \(1491415193\) [ 7091.270152] Lustre: DEBUG MARKER: == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 17:59:53 (1491415193) [ 7091.450643] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7107.817225] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7108.503974] Lustre: DEBUG MARKER: rc=0; [ 7108.503974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7108.503974] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7108.503974] echo $(hostname -s): $val; [ 7108.503974] rc=$val; [ 7108.503974] fi; [ 7108.503974] exit $rc [ 7109.203680] Lustre: DEBUG MARKER: dmesg [ 7109.733654] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 18:00:11 \(1491415211\) [ 7109.920184] Lustre: DEBUG MARKER: == sanityn test 43a: pdirops: unlink vs mkdir ======================================================== 18:00:11 (1491415211) [ 7110.116213] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7126.486946] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7127.179489] Lustre: DEBUG MARKER: rc=0; [ 7127.179489] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7127.179489] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7127.179489] echo $(hostname -s): $val; [ 7127.179489] rc=$val; [ 7127.179489] fi; [ 7127.179489] exit $rc [ 7127.868844] Lustre: DEBUG MARKER: dmesg [ 7128.426901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43b: pdirops: unlink vs create ======================================================= 18:00:30 \(1491415230\) [ 7128.630753] Lustre: DEBUG MARKER: == sanityn test 43b: pdirops: unlink vs create ======================================================= 18:00:30 (1491415230) [ 7128.827574] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7145.201869] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7145.874751] Lustre: DEBUG MARKER: rc=0; [ 7145.874751] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7145.874751] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7145.874751] echo $(hostname -s): $val; [ 7145.874751] rc=$val; [ 7145.874751] fi; [ 7145.874751] exit $rc [ 7146.570411] Lustre: DEBUG MARKER: dmesg [ 7147.099501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43c: pdirops: unlink vs link ========================================================= 18:00:49 \(1491415249\) [ 7147.310859] Lustre: DEBUG MARKER: == sanityn test 43c: pdirops: unlink vs link ========================================================= 18:00:49 (1491415249) [ 7147.469757] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7163.830906] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7164.519111] Lustre: DEBUG MARKER: rc=0; [ 7164.519111] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7164.519111] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7164.519111] echo $(hostname -s): $val; [ 7164.519111] rc=$val; [ 7164.519111] fi; [ 7164.519111] exit $rc [ 7165.212150] Lustre: DEBUG MARKER: dmesg [ 7165.752443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 18:01:07 \(1491415267\) [ 7165.953197] Lustre: DEBUG MARKER: == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 18:01:07 (1491415267) [ 7166.124879] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7182.474637] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7183.144173] Lustre: DEBUG MARKER: rc=0; [ 7183.144173] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7183.144173] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7183.144173] echo $(hostname -s): $val; [ 7183.144173] rc=$val; [ 7183.144173] fi; [ 7183.144173] exit $rc [ 7183.825518] Lustre: DEBUG MARKER: dmesg [ 7184.324139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43e: pdirops: unlink and rename \(tgt\) ================================================ 18:01:26 \(1491415286\) [ 7184.509840] Lustre: DEBUG MARKER: == sanityn test 43e: pdirops: unlink and rename (tgt) ================================================ 18:01:26 (1491415286) [ 7184.685394] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7201.031831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7201.691922] Lustre: DEBUG MARKER: rc=0; [ 7201.691922] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7201.691922] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7201.691922] echo $(hostname -s): $val; [ 7201.691922] rc=$val; [ 7201.691922] fi; [ 7201.691922] exit $rc [ 7202.377837] Lustre: DEBUG MARKER: dmesg [ 7203.000852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43f: pdirops: unlink and rename \(src\) ================================================ 18:01:45 \(1491415305\) [ 7203.185743] Lustre: DEBUG MARKER: == sanityn test 43f: pdirops: unlink and rename (src) ================================================ 18:01:45 (1491415305) [ 7203.352726] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7219.704230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7220.374171] Lustre: DEBUG MARKER: rc=0; [ 7220.374171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7220.374171] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7220.374171] echo $(hostname -s): $val; [ 7220.374171] rc=$val; [ 7220.374171] fi; [ 7220.374171] exit $rc [ 7221.061777] Lustre: DEBUG MARKER: dmesg [ 7221.648355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 18:02:03 \(1491415323\) [ 7221.831512] Lustre: DEBUG MARKER: == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 18:02:03 (1491415323) [ 7222.002305] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7238.345242] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7239.020868] Lustre: DEBUG MARKER: rc=0; [ 7239.020868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7239.020868] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7239.020868] echo $(hostname -s): $val; [ 7239.020868] rc=$val; [ 7239.020868] fi; [ 7239.020868] exit $rc [ 7239.701396] Lustre: DEBUG MARKER: dmesg [ 7240.240417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 18:02:22 \(1491415342\) [ 7240.432712] Lustre: DEBUG MARKER: == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 18:02:22 (1491415342) [ 7240.599385] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7256.962542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7257.638125] Lustre: DEBUG MARKER: rc=0; [ 7257.638125] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7257.638125] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7257.638125] echo $(hostname -s): $val; [ 7257.638125] rc=$val; [ 7257.638125] fi; [ 7257.638125] exit $rc [ 7258.330886] Lustre: DEBUG MARKER: dmesg [ 7258.847128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 43i: pdirops: unlink vs remote mkdir ================================================= 18:02:40 \(1491415360\) [ 7259.029080] Lustre: DEBUG MARKER: == sanityn test 43i: pdirops: unlink vs remote mkdir ================================================= 18:02:40 (1491415360) [ 7259.250456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_43i needs \>= 2 MDTs [ 7259.445630] Lustre: DEBUG MARKER: SKIP: sanityn test_43i needs >= 2 MDTs [ 7259.661141] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7260.343194] Lustre: DEBUG MARKER: rc=0; [ 7260.343194] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7260.343194] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7260.343194] echo $(hostname -s): $val; [ 7260.343194] rc=$val; [ 7260.343194] fi; [ 7260.343194] exit $rc [ 7261.049412] Lustre: DEBUG MARKER: dmesg [ 7261.603172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 18:02:43 \(1491415363\) [ 7261.789361] Lustre: DEBUG MARKER: == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 18:02:43 (1491415363) [ 7261.965027] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7273.316118] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7273.963880] Lustre: DEBUG MARKER: rc=0; [ 7273.963880] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7273.963880] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7273.963880] echo $(hostname -s): $val; [ 7273.963880] rc=$val; [ 7273.963880] fi; [ 7273.963880] exit $rc [ 7274.644906] Lustre: DEBUG MARKER: dmesg [ 7275.155403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44b: pdirops: rename tgt vs create =================================================== 18:02:57 \(1491415377\) [ 7275.337778] Lustre: DEBUG MARKER: == sanityn test 44b: pdirops: rename tgt vs create =================================================== 18:02:57 (1491415377) [ 7275.510523] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7286.889699] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7287.554169] Lustre: DEBUG MARKER: rc=0; [ 7287.554169] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7287.554169] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7287.554169] echo $(hostname -s): $val; [ 7287.554169] rc=$val; [ 7287.554169] fi; [ 7287.554169] exit $rc [ 7288.270305] Lustre: DEBUG MARKER: dmesg [ 7288.843582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 18:03:10 \(1491415390\) [ 7289.046324] Lustre: DEBUG MARKER: == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 18:03:10 (1491415390) [ 7289.227628] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7300.577346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7301.236065] Lustre: DEBUG MARKER: rc=0; [ 7301.236065] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7301.236065] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7301.236065] echo $(hostname -s): $val; [ 7301.236065] rc=$val; [ 7301.236065] fi; [ 7301.236065] exit $rc [ 7301.921357] Lustre: DEBUG MARKER: dmesg [ 7302.442273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 18:03:24 \(1491415404\) [ 7302.630369] Lustre: DEBUG MARKER: == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 18:03:24 (1491415404) [ 7302.800154] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7314.163576] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7314.842374] Lustre: DEBUG MARKER: rc=0; [ 7314.842374] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7314.842374] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7314.842374] echo $(hostname -s): $val; [ 7314.842374] rc=$val; [ 7314.842374] fi; [ 7314.842374] exit $rc [ 7315.544378] Lustre: DEBUG MARKER: dmesg [ 7316.059946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44e: pdirops: rename tgt and rename \(tgt\) ============================================ 18:03:38 \(1491415418\) [ 7316.258347] Lustre: DEBUG MARKER: == sanityn test 44e: pdirops: rename tgt and rename (tgt) ============================================ 18:03:38 (1491415418) [ 7316.437600] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7327.799770] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7328.465658] Lustre: DEBUG MARKER: rc=0; [ 7328.465658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7328.465658] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7328.465658] echo $(hostname -s): $val; [ 7328.465658] rc=$val; [ 7328.465658] fi; [ 7328.465658] exit $rc [ 7329.154021] Lustre: DEBUG MARKER: dmesg [ 7329.688356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44f: pdirops: rename tgt and rename \(src\) ============================================ 18:03:51 \(1491415431\) [ 7329.877160] Lustre: DEBUG MARKER: == sanityn test 44f: pdirops: rename tgt and rename (src) ============================================ 18:03:51 (1491415431) [ 7330.053632] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7330.209071] LustreError: 27358:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 146 sleeping for 10000ms [ 7330.211349] LustreError: 27358:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 14 previous similar messages [ 7341.421863] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7342.100910] Lustre: DEBUG MARKER: rc=0; [ 7342.100910] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7342.100910] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7342.100910] echo $(hostname -s): $val; [ 7342.100910] rc=$val; [ 7342.100910] fi; [ 7342.100910] exit $rc [ 7342.786247] Lustre: DEBUG MARKER: dmesg [ 7343.311964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 18:04:05 \(1491415445\) [ 7343.494731] Lustre: DEBUG MARKER: == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 18:04:05 (1491415445) [ 7343.677629] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7353.826038] LustreError: 26230:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 146 awake [ 7353.828173] LustreError: 26230:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 15 previous similar messages [ 7355.030066] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7355.708654] Lustre: DEBUG MARKER: rc=0; [ 7355.708654] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7355.708654] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7355.708654] echo $(hostname -s): $val; [ 7355.708654] rc=$val; [ 7355.708654] fi; [ 7355.708654] exit $rc [ 7356.381699] Lustre: DEBUG MARKER: dmesg [ 7356.913752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 18:04:18 \(1491415458\) [ 7357.101921] Lustre: DEBUG MARKER: == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 18:04:18 (1491415458) [ 7357.273018] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000146 [ 7368.633155] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7369.286344] Lustre: DEBUG MARKER: rc=0; [ 7369.286344] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7369.286344] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7369.286344] echo $(hostname -s): $val; [ 7369.286344] rc=$val; [ 7369.286344] fi; [ 7369.286344] exit $rc [ 7369.965369] Lustre: DEBUG MARKER: dmesg [ 7370.493080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 44i: pdirops: rename tgt vs remote mkdir ============================================= 18:04:32 \(1491415472\) [ 7370.675474] Lustre: DEBUG MARKER: == sanityn test 44i: pdirops: rename tgt vs remote mkdir ============================================= 18:04:32 (1491415472) [ 7370.888516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_44i needs \>= 2 MDTs [ 7371.072471] Lustre: DEBUG MARKER: SKIP: sanityn test_44i needs >= 2 MDTs [ 7371.288219] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7371.958863] Lustre: DEBUG MARKER: rc=0; [ 7371.958863] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7371.958863] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7371.958863] echo $(hostname -s): $val; [ 7371.958863] rc=$val; [ 7371.958863] fi; [ 7371.958863] exit $rc [ 7372.634894] Lustre: DEBUG MARKER: dmesg [ 7373.156952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 18:04:35 \(1491415475\) [ 7373.344545] Lustre: DEBUG MARKER: == sanityn test 45a: pdirops: rename src vs mkdir ==================================================== 18:04:35 (1491415475) [ 7373.513076] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7389.874035] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7390.524670] Lustre: DEBUG MARKER: rc=0; [ 7390.524670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7390.524670] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7390.524670] echo $(hostname -s): $val; [ 7390.524670] rc=$val; [ 7390.524670] fi; [ 7390.524670] exit $rc [ 7391.213858] Lustre: DEBUG MARKER: dmesg [ 7391.731671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45b: pdirops: rename src vs create =================================================== 18:04:53 \(1491415493\) [ 7391.918299] Lustre: DEBUG MARKER: == sanityn test 45b: pdirops: rename src vs create =================================================== 18:04:53 (1491415493) [ 7392.087733] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7408.439051] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7409.101197] Lustre: DEBUG MARKER: rc=0; [ 7409.101197] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7409.101197] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7409.101197] echo $(hostname -s): $val; [ 7409.101197] rc=$val; [ 7409.101197] fi; [ 7409.101197] exit $rc [ 7409.790939] Lustre: DEBUG MARKER: dmesg [ 7410.343854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45c: pdirops: rename src vs link ===================================================== 18:05:12 \(1491415512\) [ 7410.539926] Lustre: DEBUG MARKER: == sanityn test 45c: pdirops: rename src vs link ===================================================== 18:05:12 (1491415512) [ 7410.726961] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7427.080838] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7427.747821] Lustre: DEBUG MARKER: rc=0; [ 7427.747821] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7427.747821] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7427.747821] echo $(hostname -s): $val; [ 7427.747821] rc=$val; [ 7427.747821] fi; [ 7427.747821] exit $rc [ 7428.449592] Lustre: DEBUG MARKER: dmesg [ 7429.078141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45d: pdirops: rename src vs unlink =================================================== 18:05:31 \(1491415531\) [ 7429.264313] Lustre: DEBUG MARKER: == sanityn test 45d: pdirops: rename src vs unlink =================================================== 18:05:31 (1491415531) [ 7429.431948] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7445.791302] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7446.461697] Lustre: DEBUG MARKER: rc=0; [ 7446.461697] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7446.461697] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7446.461697] echo $(hostname -s): $val; [ 7446.461697] rc=$val; [ 7446.461697] fi; [ 7446.461697] exit $rc [ 7447.137201] Lustre: DEBUG MARKER: dmesg [ 7447.676079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45e: pdirops: rename src and rename \(tgt\) ============================================ 18:05:49 \(1491415549\) [ 7447.865594] Lustre: DEBUG MARKER: == sanityn test 45e: pdirops: rename src and rename (tgt) ============================================ 18:05:49 (1491415549) [ 7448.042622] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7464.407639] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7465.092002] Lustre: DEBUG MARKER: rc=0; [ 7465.092002] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7465.092002] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7465.092002] echo $(hostname -s): $val; [ 7465.092002] rc=$val; [ 7465.092002] fi; [ 7465.092002] exit $rc [ 7465.788484] Lustre: DEBUG MARKER: dmesg [ 7466.339822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45f: pdirops: rename src and rename \(src\) ============================================ 18:06:08 \(1491415568\) [ 7466.534531] Lustre: DEBUG MARKER: == sanityn test 45f: pdirops: rename src and rename (src) ============================================ 18:06:08 (1491415568) [ 7466.706862] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7483.062780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7483.733728] Lustre: DEBUG MARKER: rc=0; [ 7483.733728] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7483.733728] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7483.733728] echo $(hostname -s): $val; [ 7483.733728] rc=$val; [ 7483.733728] fi; [ 7483.733728] exit $rc [ 7484.436658] Lustre: DEBUG MARKER: dmesg [ 7484.961344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45g: pdirops: rename src vs getattr ================================================== 18:06:26 \(1491415586\) [ 7485.159336] Lustre: DEBUG MARKER: == sanityn test 45g: pdirops: rename src vs getattr ================================================== 18:06:26 (1491415586) [ 7485.312433] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7501.683134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7502.336297] Lustre: DEBUG MARKER: rc=0; [ 7502.336297] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7502.336297] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7502.336297] echo $(hostname -s): $val; [ 7502.336297] rc=$val; [ 7502.336297] fi; [ 7502.336297] exit $rc [ 7503.018450] Lustre: DEBUG MARKER: dmesg [ 7503.527490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 18:06:45 \(1491415605\) [ 7503.720757] Lustre: DEBUG MARKER: == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 18:06:45 (1491415605) [ 7503.891097] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7520.252087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7520.927791] Lustre: DEBUG MARKER: rc=0; [ 7520.927791] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7520.927791] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7520.927791] echo $(hostname -s): $val; [ 7520.927791] rc=$val; [ 7520.927791] fi; [ 7520.927791] exit $rc [ 7521.616279] Lustre: DEBUG MARKER: dmesg [ 7522.144930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 45i: pdirops: rename src vs remote mkdir ============================================= 18:07:04 \(1491415624\) [ 7522.333967] Lustre: DEBUG MARKER: == sanityn test 45i: pdirops: rename src vs remote mkdir ============================================= 18:07:04 (1491415624) [ 7522.536505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_45i needs \>= 2 MDTs [ 7522.720865] Lustre: DEBUG MARKER: SKIP: sanityn test_45i needs >= 2 MDTs [ 7522.936107] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7523.594522] Lustre: DEBUG MARKER: rc=0; [ 7523.594522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7523.594522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7523.594522] echo $(hostname -s): $val; [ 7523.594522] rc=$val; [ 7523.594522] fi; [ 7523.594522] exit $rc [ 7524.280991] Lustre: DEBUG MARKER: dmesg [ 7524.827944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46a: pdirops: link vs mkdir ========================================================== 18:07:06 \(1491415626\) [ 7525.026460] Lustre: DEBUG MARKER: == sanityn test 46a: pdirops: link vs mkdir ========================================================== 18:07:06 (1491415626) [ 7525.187200] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7541.548804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7542.214766] Lustre: DEBUG MARKER: rc=0; [ 7542.214766] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7542.214766] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7542.214766] echo $(hostname -s): $val; [ 7542.214766] rc=$val; [ 7542.214766] fi; [ 7542.214766] exit $rc [ 7542.900121] Lustre: DEBUG MARKER: dmesg [ 7543.428053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46b: pdirops: link vs create ========================================================= 18:07:25 \(1491415645\) [ 7543.615321] Lustre: DEBUG MARKER: == sanityn test 46b: pdirops: link vs create ========================================================= 18:07:25 (1491415645) [ 7543.789826] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7560.145925] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7560.843048] Lustre: DEBUG MARKER: rc=0; [ 7560.843048] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7560.843048] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7560.843048] echo $(hostname -s): $val; [ 7560.843048] rc=$val; [ 7560.843048] fi; [ 7560.843048] exit $rc [ 7561.551363] Lustre: DEBUG MARKER: dmesg [ 7562.129686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46c: pdirops: link vs link =========================================================== 18:07:44 \(1491415664\) [ 7562.326420] Lustre: DEBUG MARKER: == sanityn test 46c: pdirops: link vs link =========================================================== 18:07:44 (1491415664) [ 7562.494995] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7578.854429] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7579.522470] Lustre: DEBUG MARKER: rc=0; [ 7579.522470] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7579.522470] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7579.522470] echo $(hostname -s): $val; [ 7579.522470] rc=$val; [ 7579.522470] fi; [ 7579.522470] exit $rc [ 7580.249341] Lustre: DEBUG MARKER: dmesg [ 7580.772321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46d: pdirops: link vs unlink ========================================================= 18:08:02 \(1491415682\) [ 7580.968975] Lustre: DEBUG MARKER: == sanityn test 46d: pdirops: link vs unlink ========================================================= 18:08:02 (1491415682) [ 7581.166044] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7597.534475] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7598.205152] Lustre: DEBUG MARKER: rc=0; [ 7598.205152] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7598.205152] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7598.205152] echo $(hostname -s): $val; [ 7598.205152] rc=$val; [ 7598.205152] fi; [ 7598.205152] exit $rc [ 7598.893079] Lustre: DEBUG MARKER: dmesg [ 7599.435627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46e: pdirops: link and rename \(tgt\) ================================================== 18:08:21 \(1491415701\) [ 7599.621236] Lustre: DEBUG MARKER: == sanityn test 46e: pdirops: link and rename (tgt) ================================================== 18:08:21 (1491415701) [ 7599.794503] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7616.155137] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7616.826509] Lustre: DEBUG MARKER: rc=0; [ 7616.826509] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7616.826509] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7616.826509] echo $(hostname -s): $val; [ 7616.826509] rc=$val; [ 7616.826509] fi; [ 7616.826509] exit $rc [ 7617.527520] Lustre: DEBUG MARKER: dmesg [ 7618.062989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46f: pdirops: link and rename \(src\) ================================================== 18:08:40 \(1491415720\) [ 7618.253670] Lustre: DEBUG MARKER: == sanityn test 46f: pdirops: link and rename (src) ================================================== 18:08:40 (1491415720) [ 7618.443181] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7634.801308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7635.475202] Lustre: DEBUG MARKER: rc=0; [ 7635.475202] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7635.475202] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7635.475202] echo $(hostname -s): $val; [ 7635.475202] rc=$val; [ 7635.475202] fi; [ 7635.475202] exit $rc [ 7636.153661] Lustre: DEBUG MARKER: dmesg [ 7636.671343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46g: pdirops: link vs getattr ======================================================== 18:08:58 \(1491415738\) [ 7636.853865] Lustre: DEBUG MARKER: == sanityn test 46g: pdirops: link vs getattr ======================================================== 18:08:58 (1491415738) [ 7637.029980] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7653.387712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7654.052354] Lustre: DEBUG MARKER: rc=0; [ 7654.052354] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7654.052354] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7654.052354] echo $(hostname -s): $val; [ 7654.052354] rc=$val; [ 7654.052354] fi; [ 7654.052354] exit $rc [ 7654.745908] Lustre: DEBUG MARKER: dmesg [ 7655.269037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46h: pdirops: link vs readdir ======================================================== 18:09:17 \(1491415757\) [ 7655.456984] Lustre: DEBUG MARKER: == sanityn test 46h: pdirops: link vs readdir ======================================================== 18:09:17 (1491415757) [ 7655.630058] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000145 [ 7671.998378] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7672.691065] Lustre: DEBUG MARKER: rc=0; [ 7672.691065] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7672.691065] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7672.691065] echo $(hostname -s): $val; [ 7672.691065] rc=$val; [ 7672.691065] fi; [ 7672.691065] exit $rc [ 7673.404543] Lustre: DEBUG MARKER: dmesg [ 7673.934577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 46i: pdirops: link vs remote mkdir =================================================== 18:09:35 \(1491415775\) [ 7674.121051] Lustre: DEBUG MARKER: == sanityn test 46i: pdirops: link vs remote mkdir =================================================== 18:09:35 (1491415775) [ 7674.347813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_46i needs \>= 2 MDTs [ 7674.531487] Lustre: DEBUG MARKER: SKIP: sanityn test_46i needs >= 2 MDTs [ 7674.739279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7675.402569] Lustre: DEBUG MARKER: rc=0; [ 7675.402569] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7675.402569] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7675.402569] echo $(hostname -s): $val; [ 7675.402569] rc=$val; [ 7675.402569] fi; [ 7675.402569] exit $rc [ 7676.080940] Lustre: DEBUG MARKER: dmesg [ 7676.599536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47a: pdirops: remote mkdir vs mkdir ================================================== 18:09:38 \(1491415778\) [ 7676.780605] Lustre: DEBUG MARKER: == sanityn test 47a: pdirops: remote mkdir vs mkdir ================================================== 18:09:38 (1491415778) [ 7676.991229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47a needs \>= 2 MDTs [ 7677.183714] Lustre: DEBUG MARKER: SKIP: sanityn test_47a needs >= 2 MDTs [ 7677.404370] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7678.066919] Lustre: DEBUG MARKER: rc=0; [ 7678.066919] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7678.066919] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7678.066919] echo $(hostname -s): $val; [ 7678.066919] rc=$val; [ 7678.066919] fi; [ 7678.066919] exit $rc [ 7678.747609] Lustre: DEBUG MARKER: dmesg [ 7679.372242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47b: pdirops: remote mkdir vs create ================================================= 18:09:41 \(1491415781\) [ 7679.559359] Lustre: DEBUG MARKER: == sanityn test 47b: pdirops: remote mkdir vs create ================================================= 18:09:41 (1491415781) [ 7679.778321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47b needs \>= 2 MDTs [ 7679.975386] Lustre: DEBUG MARKER: SKIP: sanityn test_47b needs >= 2 MDTs [ 7680.193592] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7680.854668] Lustre: DEBUG MARKER: rc=0; [ 7680.854668] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7680.854668] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7680.854668] echo $(hostname -s): $val; [ 7680.854668] rc=$val; [ 7680.854668] fi; [ 7680.854668] exit $rc [ 7681.535416] Lustre: DEBUG MARKER: dmesg [ 7682.119920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47c: pdirops: remote mkdir vs link =================================================== 18:09:44 \(1491415784\) [ 7682.293802] Lustre: DEBUG MARKER: == sanityn test 47c: pdirops: remote mkdir vs link =================================================== 18:09:44 (1491415784) [ 7682.508148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47c needs \>= 2 MDTs [ 7682.693298] Lustre: DEBUG MARKER: SKIP: sanityn test_47c needs >= 2 MDTs [ 7682.915689] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7683.588895] Lustre: DEBUG MARKER: rc=0; [ 7683.588895] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7683.588895] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7683.588895] echo $(hostname -s): $val; [ 7683.588895] rc=$val; [ 7683.588895] fi; [ 7683.588895] exit $rc [ 7684.277820] Lustre: DEBUG MARKER: dmesg [ 7684.812751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47d: pdirops: remote mkdir vs unlink ================================================= 18:09:46 \(1491415786\) [ 7684.995153] Lustre: DEBUG MARKER: == sanityn test 47d: pdirops: remote mkdir vs unlink ================================================= 18:09:46 (1491415786) [ 7685.204835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47d needs \>= 2 MDTs [ 7685.388466] Lustre: DEBUG MARKER: SKIP: sanityn test_47d needs >= 2 MDTs [ 7685.600667] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7686.259645] Lustre: DEBUG MARKER: rc=0; [ 7686.259645] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7686.259645] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7686.259645] echo $(hostname -s): $val; [ 7686.259645] rc=$val; [ 7686.259645] fi; [ 7686.259645] exit $rc [ 7686.960826] Lustre: DEBUG MARKER: dmesg [ 7687.481687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47e: pdirops: remote mkdir and rename \(tgt\) ========================================== 18:09:49 \(1491415789\) [ 7687.668130] Lustre: DEBUG MARKER: == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================== 18:09:49 (1491415789) [ 7687.888020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47e needs \>= 2 MDTs [ 7688.081971] Lustre: DEBUG MARKER: SKIP: sanityn test_47e needs >= 2 MDTs [ 7688.301257] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7688.970156] Lustre: DEBUG MARKER: rc=0; [ 7688.970156] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7688.970156] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7688.970156] echo $(hostname -s): $val; [ 7688.970156] rc=$val; [ 7688.970156] fi; [ 7688.970156] exit $rc [ 7689.663395] Lustre: DEBUG MARKER: dmesg [ 7690.187274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47f: pdirops: remote mkdir and rename \(src\) ========================================== 18:09:52 \(1491415792\) [ 7690.375243] Lustre: DEBUG MARKER: == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================== 18:09:52 (1491415792) [ 7690.578679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47f needs \>= 2 MDTs [ 7690.761442] Lustre: DEBUG MARKER: SKIP: sanityn test_47f needs >= 2 MDTs [ 7690.980385] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7691.637626] Lustre: DEBUG MARKER: rc=0; [ 7691.637626] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7691.637626] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7691.637626] echo $(hostname -s): $val; [ 7691.637626] rc=$val; [ 7691.637626] fi; [ 7691.637626] exit $rc [ 7692.320625] Lustre: DEBUG MARKER: dmesg [ 7692.847294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 47g: pdirops: remote mkdir vs getattr ================================================ 18:09:54 \(1491415794\) [ 7693.051683] Lustre: DEBUG MARKER: == sanityn test 47g: pdirops: remote mkdir vs getattr ================================================ 18:09:54 (1491415794) [ 7693.272121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_47g needs \>= 2 MDTs [ 7693.479314] Lustre: DEBUG MARKER: SKIP: sanityn test_47g needs >= 2 MDTs [ 7693.696171] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7694.391637] Lustre: DEBUG MARKER: rc=0; [ 7694.391637] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7694.391637] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7694.391637] echo $(hostname -s): $val; [ 7694.391637] rc=$val; [ 7694.391637] fi; [ 7694.391637] exit $rc [ 7695.094748] Lustre: DEBUG MARKER: dmesg [ 7695.668956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 18:09:57 \(1491415797\) [ 7695.856395] Lustre: DEBUG MARKER: == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 18:09:57 (1491415797) [ 7701.114319] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7701.839378] Lustre: DEBUG MARKER: rc=0; [ 7701.839378] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7701.839378] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7701.839378] echo $(hostname -s): $val; [ 7701.839378] rc=$val; [ 7701.839378] fi; [ 7701.839378] exit $rc [ 7702.531877] Lustre: DEBUG MARKER: dmesg [ 7703.116461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_51a skipping ALWAYS excluded test 51a [ 7703.304747] Lustre: DEBUG MARKER: SKIP: sanityn test_51a skipping ALWAYS excluded test 51a [ 7703.531492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ================ 18:10:05 \(1491415805\) [ 7703.718949] Lustre: DEBUG MARKER: == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ================ 18:10:05 (1491415805) [ 7703.917408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 7710.324591] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7711.062293] Lustre: DEBUG MARKER: rc=0; [ 7711.062293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7711.062293] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7711.062293] echo $(hostname -s): $val; [ 7711.062293] rc=$val; [ 7711.062293] fi; [ 7711.062293] exit $rc [ 7711.775072] Lustre: DEBUG MARKER: dmesg [ 7712.318586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned =============== 18:10:14 \(1491415814\) [ 7712.520044] Lustre: DEBUG MARKER: == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned =============== 18:10:14 (1491415814) [ 7712.726450] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x172 [ 7717.098845] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7717.819695] Lustre: DEBUG MARKER: rc=0; [ 7717.819695] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7717.819695] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7717.819695] echo $(hostname -s): $val; [ 7717.819695] rc=$val; [ 7717.819695] fi; [ 7717.819695] exit $rc [ 7718.560999] Lustre: DEBUG MARKER: dmesg [ 7719.132245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 51d: layout lock: losing layout lock should clean up memory map region =============== 18:10:21 \(1491415821\) [ 7719.330186] Lustre: DEBUG MARKER: == sanityn test 51d: layout lock: losing layout lock should clean up memory map region =============== 18:10:21 (1491415821) [ 7720.592023] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7721.314213] Lustre: DEBUG MARKER: rc=0; [ 7721.314213] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7721.314213] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7721.314213] echo $(hostname -s): $val; [ 7721.314213] rc=$val; [ 7721.314213] fi; [ 7721.314213] exit $rc [ 7722.046357] Lustre: DEBUG MARKER: dmesg [ 7722.598559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 54: rename locking =================================================================== 18:10:24 \(1491415824\) [ 7722.790402] Lustre: DEBUG MARKER: == sanityn test 54: rename locking =================================================================== 18:10:24 (1491415824) [ 7722.987586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000153 [ 7728.316104] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000154 [ 7733.660619] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000153 [ 7739.011628] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000154 [ 7744.439288] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7745.122111] Lustre: DEBUG MARKER: rc=0; [ 7745.122111] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7745.122111] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7745.122111] echo $(hostname -s): $val; [ 7745.122111] rc=$val; [ 7745.122111] fi; [ 7745.122111] exit $rc [ 7745.846845] Lustre: DEBUG MARKER: dmesg [ 7746.406972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 55a: rename vs unlink target dir ===================================================== 18:10:48 \(1491415848\) [ 7746.585956] Lustre: DEBUG MARKER: == sanityn test 55a: rename vs unlink target dir ===================================================== 18:10:48 (1491415848) [ 7746.771680] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000156 [ 7751.937154] LustreError: 27359:0:(mdt_reint.c:1329:mdt_is_subdir()) lustre-MDT0000: failed subdir check in [0x200002b10:0xaa:0x0] for [0x200002b10:0xa9:0x0]: rc = -2 [ 7752.138714] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7752.846454] Lustre: DEBUG MARKER: rc=0; [ 7752.846454] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7752.846454] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7752.846454] echo $(hostname -s): $val; [ 7752.846454] rc=$val; [ 7752.846454] fi; [ 7752.846454] exit $rc [ 7753.602599] Lustre: DEBUG MARKER: dmesg [ 7754.154083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 55b: rename vs unlink source dir ===================================================== 18:10:56 \(1491415856\) [ 7754.357914] Lustre: DEBUG MARKER: == sanityn test 55b: rename vs unlink source dir ===================================================== 18:10:56 (1491415856) [ 7754.517755] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000156 [ 7759.869961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7760.603806] Lustre: DEBUG MARKER: rc=0; [ 7760.603806] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7760.603806] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7760.603806] echo $(hostname -s): $val; [ 7760.603806] rc=$val; [ 7760.603806] fi; [ 7760.603806] exit $rc [ 7761.326364] Lustre: DEBUG MARKER: dmesg [ 7761.908531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 55c: rename vs unlink orphan target dir ============================================== 18:11:03 \(1491415863\) [ 7762.106023] Lustre: DEBUG MARKER: == sanityn test 55c: rename vs unlink orphan target dir ============================================== 18:11:03 (1491415863) [ 7762.281771] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x156 [ 7772.439115] LustreError: 27359:0:(mdt_reint.c:1329:mdt_is_subdir()) lustre-MDT0000: failed subdir check in [0x200002b10:0xb0:0x0] for [0x200002b10:0xaf:0x0]: rc = -2 [ 7772.649805] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7773.321188] Lustre: DEBUG MARKER: rc=0; [ 7773.321188] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7773.321188] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7773.321188] echo $(hostname -s): $val; [ 7773.321188] rc=$val; [ 7773.321188] fi; [ 7773.321188] exit $rc [ 7774.044374] Lustre: DEBUG MARKER: dmesg [ 7774.631962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 55d: rename file vs link ============================================================= 18:11:16 \(1491415876\) [ 7774.845623] Lustre: DEBUG MARKER: == sanityn test 55d: rename file vs link ============================================================= 18:11:16 (1491415876) [ 7775.038534] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x155 [ 7782.408622] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7783.156815] Lustre: DEBUG MARKER: rc=0; [ 7783.156815] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7783.156815] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7783.156815] echo $(hostname -s): $val; [ 7783.156815] rc=$val; [ 7783.156815] fi; [ 7783.156815] exit $rc [ 7783.867790] Lustre: DEBUG MARKER: dmesg [ 7784.420706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 60: Verify data_version behaviour ==================================================== 18:11:26 \(1491415886\) [ 7784.599931] Lustre: DEBUG MARKER: == sanityn test 60: Verify data_version behaviour ==================================================== 18:11:26 (1491415886) [ 7784.789332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 7785.201059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7785.897072] Lustre: DEBUG MARKER: rc=0; [ 7785.897072] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7785.897072] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7785.897072] echo $(hostname -s): $val; [ 7785.897072] rc=$val; [ 7785.897072] fi; [ 7785.897072] exit $rc [ 7786.615550] Lustre: DEBUG MARKER: dmesg [ 7787.171766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 70a: cd directory && rm directory ==================================================== 18:11:29 \(1491415889\) [ 7787.370614] Lustre: DEBUG MARKER: == sanityn test 70a: cd directory [ 7787.600784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7788.329649] Lustre: DEBUG MARKER: rc=0; [ 7788.329649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7788.329649] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7788.329649] echo $(hostname -s): $val; [ 7788.329649] rc=$val; [ 7788.329649] fi; [ 7788.329649] exit $rc [ 7789.048348] Lustre: DEBUG MARKER: dmesg [ 7789.596347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 70b: remove files after calling rm_entry ============================================= 18:11:31 \(1491415891\) [ 7789.790878] Lustre: DEBUG MARKER: == sanityn test 70b: remove files after calling rm_entry ============================================= 18:11:31 (1491415891) [ 7790.103137] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7790.827429] Lustre: DEBUG MARKER: rc=0; [ 7790.827429] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7790.827429] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7790.827429] echo $(hostname -s): $val; [ 7790.827429] rc=$val; [ 7790.827429] fi; [ 7790.827429] exit $rc [ 7791.551315] Lustre: DEBUG MARKER: dmesg [ 7792.093493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 71: correct file map just after write operation is finished ========================== 18:11:34 \(1491415894\) [ 7792.282053] Lustre: DEBUG MARKER: == sanityn test 71: correct file map just after write operation is finished ========================== 18:11:34 (1491415894) [ 7792.455978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 7792.881157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_71 ORI-366\/LU-1941: FIEMAP unimplemented on ZFS [ 7793.060359] Lustre: DEBUG MARKER: SKIP: sanityn test_71 ORI-366/LU-1941: FIEMAP unimplemented on ZFS [ 7793.307530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7794.068072] Lustre: DEBUG MARKER: rc=0; [ 7794.068072] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7794.068072] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7794.068072] echo $(hostname -s): $val; [ 7794.068072] rc=$val; [ 7794.068072] fi; [ 7794.068072] exit $rc [ 7794.773823] Lustre: DEBUG MARKER: dmesg [ 7795.365549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 72: getxattr\/setxattr cache should be consistent between nodes ======================= 18:11:37 \(1491415897\) [ 7795.555721] Lustre: DEBUG MARKER: == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ======================= 18:11:37 (1491415897) [ 7795.845935] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7796.573914] Lustre: DEBUG MARKER: rc=0; [ 7796.573914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7796.573914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7796.573914] echo $(hostname -s): $val; [ 7796.573914] rc=$val; [ 7796.573914] fi; [ 7796.573914] exit $rc [ 7797.317887] Lustre: DEBUG MARKER: dmesg [ 7797.868816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 73: getxattr should not cause xattr lock cancellation ================================ 18:11:39 \(1491415899\) [ 7798.072213] Lustre: DEBUG MARKER: == sanityn test 73: getxattr should not cause xattr lock cancellation ================================ 18:11:39 (1491415899) [ 7798.347234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7799.055705] Lustre: DEBUG MARKER: rc=0; [ 7799.055705] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7799.055705] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7799.055705] echo $(hostname -s): $val; [ 7799.055705] rc=$val; [ 7799.055705] fi; [ 7799.055705] exit $rc [ 7799.774598] Lustre: DEBUG MARKER: dmesg [ 7800.369364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 74: flock deadlock: different mounts ================================================= 18:11:42 \(1491415902\) [ 7800.594138] Lustre: DEBUG MARKER: == sanityn test 74: flock deadlock: different mounts ================================================= 18:11:42 (1491415902) [ 7800.801666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 7805.178833] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7805.834803] Lustre: DEBUG MARKER: rc=0; [ 7805.834803] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7805.834803] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7805.834803] echo $(hostname -s): $val; [ 7805.834803] rc=$val; [ 7805.834803] fi; [ 7805.834803] exit $rc [ 7806.526883] Lustre: DEBUG MARKER: dmesg [ 7807.052046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 75: osc: upcall after unuse lock====================================================== 18:11:49 \(1491415909\) [ 7807.239377] Lustre: DEBUG MARKER: == sanityn test 75: osc: upcall after unuse lock====================================================== 18:11:49 (1491415909) [ 7812.558462] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7813.231668] Lustre: DEBUG MARKER: rc=0; [ 7813.231668] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7813.231668] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7813.231668] echo $(hostname -s): $val; [ 7813.231668] rc=$val; [ 7813.231668] fi; [ 7813.231668] exit $rc [ 7813.918331] Lustre: DEBUG MARKER: dmesg [ 7814.476511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 76: Verify open file for 2048 files ================================================== 18:11:56 \(1491415916\) [ 7814.664653] Lustre: DEBUG MARKER: == sanityn test 76: Verify open file for 2048 files ================================================== 18:11:56 (1491415916) [ 7814.838642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 7950.987045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.*.exports.'10\.9\.5\.237@tcp'.open_files [ 7967.113637] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7967.821849] Lustre: DEBUG MARKER: rc=0; [ 7967.821849] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7967.821849] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7967.821849] echo $(hostname -s): $val; [ 7967.821849] rc=$val; [ 7967.821849] fi; [ 7967.821849] exit $rc [ 7968.563494] Lustre: DEBUG MARKER: dmesg [ 7969.111403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77a: check FIFO NRS policy =========================================================== 18:14:31 \(1491416071\) [ 7969.297639] Lustre: DEBUG MARKER: == sanityn test 77a: check FIFO NRS policy =========================================================== 18:14:31 (1491416071) [ 7978.402998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7979.080524] Lustre: DEBUG MARKER: rc=0; [ 7979.080524] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7979.080524] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7979.080524] echo $(hostname -s): $val; [ 7979.080524] rc=$val; [ 7979.080524] fi; [ 7979.080524] exit $rc [ 7979.792572] Lustre: DEBUG MARKER: dmesg [ 7980.323253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77b: check CRR-N NRS policy ========================================================== 18:14:42 \(1491416082\) [ 7980.509070] Lustre: DEBUG MARKER: == sanityn test 77b: check CRR-N NRS policy ========================================================== 18:14:42 (1491416082) [ 7998.205651] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7998.889349] Lustre: DEBUG MARKER: rc=0; [ 7998.889349] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7998.889349] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7998.889349] echo $(hostname -s): $val; [ 7998.889349] rc=$val; [ 7998.889349] fi; [ 7998.889349] exit $rc [ 7999.582180] Lustre: DEBUG MARKER: dmesg [ 8000.139612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77c: check ORR NRS policy ============================================================ 18:15:02 \(1491416102\) [ 8000.324453] Lustre: DEBUG MARKER: == sanityn test 77c: check ORR NRS policy ============================================================ 18:15:02 (1491416102) [ 8027.414152] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8028.096297] Lustre: DEBUG MARKER: rc=0; [ 8028.096297] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8028.096297] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8028.096297] echo $(hostname -s): $val; [ 8028.096297] rc=$val; [ 8028.096297] fi; [ 8028.096297] exit $rc [ 8028.788700] Lustre: DEBUG MARKER: dmesg [ 8029.333375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77d: check TRR nrs policy ============================================================ 18:15:31 \(1491416131\) [ 8029.523586] Lustre: DEBUG MARKER: == sanityn test 77d: check TRR nrs policy ============================================================ 18:15:31 (1491416131) [ 8056.226836] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8056.931035] Lustre: DEBUG MARKER: rc=0; [ 8056.931035] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8056.931035] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8056.931035] echo $(hostname -s): $val; [ 8056.931035] rc=$val; [ 8056.931035] fi; [ 8056.931035] exit $rc [ 8057.644459] Lustre: DEBUG MARKER: dmesg [ 8058.317926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77e: check TBF NID nrs policy ======================================================== 18:16:00 \(1491416160\) [ 8058.508648] Lustre: DEBUG MARKER: == sanityn test 77e: check TBF NID nrs policy ======================================================== 18:16:00 (1491416160) [ 8097.169630] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8097.873601] Lustre: DEBUG MARKER: rc=0; [ 8097.873601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8097.873601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8097.873601] echo $(hostname -s): $val; [ 8097.873601] rc=$val; [ 8097.873601] fi; [ 8097.873601] exit $rc [ 8098.565963] Lustre: DEBUG MARKER: dmesg [ 8099.103252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77f: check TBF JobID nrs policy ====================================================== 18:16:41 \(1491416201\) [ 8099.290744] Lustre: DEBUG MARKER: == sanityn test 77f: check TBF JobID nrs policy ====================================================== 18:16:41 (1491416201) [ 8138.573719] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8139.259820] Lustre: DEBUG MARKER: rc=0; [ 8139.259820] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8139.259820] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8139.259820] echo $(hostname -s): $val; [ 8139.259820] rc=$val; [ 8139.259820] fi; [ 8139.259820] exit $rc [ 8139.958475] Lustre: DEBUG MARKER: dmesg [ 8140.513484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77g: Change TBF type directly ======================================================== 18:17:22 \(1491416242\) [ 8140.704260] Lustre: DEBUG MARKER: == sanityn test 77g: Change TBF type directly ======================================================== 18:17:22 (1491416242) [ 8142.992557] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8143.667368] Lustre: DEBUG MARKER: rc=0; [ 8143.667368] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8143.667368] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8143.667368] echo $(hostname -s): $val; [ 8143.667368] rc=$val; [ 8143.667368] fi; [ 8143.667368] exit $rc [ 8144.380982] Lustre: DEBUG MARKER: dmesg [ 8144.931105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77h: Wrong policy name should report error, not LBUG ================================= 18:17:26 \(1491416246\) [ 8145.118834] Lustre: DEBUG MARKER: == sanityn test 77h: Wrong policy name should report error, not LBUG ================================= 18:17:26 (1491416246) [ 8147.947619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8148.624554] Lustre: DEBUG MARKER: rc=0; [ 8148.624554] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8148.624554] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8148.624554] echo $(hostname -s): $val; [ 8148.624554] rc=$val; [ 8148.624554] fi; [ 8148.624554] exit $rc [ 8149.314879] Lustre: DEBUG MARKER: dmesg [ 8149.910100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77i: Change rank of TBF rule ========================================================= 18:17:31 \(1491416251\) [ 8150.105570] Lustre: DEBUG MARKER: == sanityn test 77i: Change rank of TBF rule ========================================================= 18:17:31 (1491416251) [ 8156.587878] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8157.262763] Lustre: DEBUG MARKER: rc=0; [ 8157.262763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8157.262763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8157.262763] echo $(hostname -s): $val; [ 8157.262763] rc=$val; [ 8157.262763] fi; [ 8157.262763] exit $rc [ 8157.965767] Lustre: DEBUG MARKER: dmesg [ 8158.540892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 77j: check TBF-OPCode NRS policy ===================================================== 18:17:40 \(1491416260\) [ 8158.731779] Lustre: DEBUG MARKER: == sanityn test 77j: check TBF-OPCode NRS policy ===================================================== 18:17:40 (1491416260) [ 8206.623559] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8207.311985] Lustre: DEBUG MARKER: rc=0; [ 8207.311985] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8207.311985] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8207.311985] echo $(hostname -s): $val; [ 8207.311985] rc=$val; [ 8207.311985] fi; [ 8207.311985] exit $rc [ 8208.015647] Lustre: DEBUG MARKER: dmesg [ 8208.676523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 78: Enable policy and specify tunings right away ===================================== 18:18:30 \(1491416310\) [ 8208.865282] Lustre: DEBUG MARKER: == sanityn test 78: Enable policy and specify tunings right away ===================================== 18:18:30 (1491416310) [ 8210.360585] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8211.039451] Lustre: DEBUG MARKER: rc=0; [ 8211.039451] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8211.039451] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8211.039451] echo $(hostname -s): $val; [ 8211.039451] rc=$val; [ 8211.039451] fi; [ 8211.039451] exit $rc [ 8211.762155] Lustre: DEBUG MARKER: dmesg [ 8212.454791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 79: xattr: intent error ============================================================== 18:18:34 \(1491416314\) [ 8212.645927] Lustre: DEBUG MARKER: == sanityn test 79: xattr: intent error ============================================================== 18:18:34 (1491416314) [ 8213.163551] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x80000160 [ 8213.330739] LustreError: 26229:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 160 sleeping for 10000ms [ 8213.333090] LustreError: 26229:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 30 previous similar messages [ 8215.484896] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x80000131 [ 8223.335050] LustreError: 26229:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 160 awake [ 8223.337352] LustreError: 26229:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 29 previous similar messages [ 8223.339677] Lustre: *** cfs_fail_loc=131, val=0*** [ 8223.539656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8224.217477] Lustre: DEBUG MARKER: rc=0; [ 8224.217477] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8224.217477] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8224.217477] echo $(hostname -s): $val; [ 8224.217477] rc=$val; [ 8224.217477] fi; [ 8224.217477] exit $rc [ 8224.926984] Lustre: DEBUG MARKER: dmesg [ 8225.494668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 80a: migrate directory when some children is being opened ============================ 18:18:47 \(1491416327\) [ 8225.684999] Lustre: DEBUG MARKER: == sanityn test 80a: migrate directory when some children is being opened ============================ 18:18:47 (1491416327) [ 8225.902740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_80a needs \>= 2 MDTs [ 8226.105870] Lustre: DEBUG MARKER: SKIP: sanityn test_80a needs >= 2 MDTs [ 8226.326081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8227.001590] Lustre: DEBUG MARKER: rc=0; [ 8227.001590] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8227.001590] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8227.001590] echo $(hostname -s): $val; [ 8227.001590] rc=$val; [ 8227.001590] fi; [ 8227.001590] exit $rc [ 8227.690553] Lustre: DEBUG MARKER: dmesg [ 8228.304530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 80b: Accessing directory during migration ============================================ 18:18:50 \(1491416330\) [ 8228.495070] Lustre: DEBUG MARKER: == sanityn test 80b: Accessing directory during migration ============================================ 18:18:50 (1491416330) [ 8228.710473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_80b needs \>= 2 MDTs [ 8228.900892] Lustre: DEBUG MARKER: SKIP: sanityn test_80b needs >= 2 MDTs [ 8229.122091] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8229.796724] Lustre: DEBUG MARKER: rc=0; [ 8229.796724] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8229.796724] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8229.796724] echo $(hostname -s): $val; [ 8229.796724] rc=$val; [ 8229.796724] fi; [ 8229.796724] exit $rc [ 8230.503265] Lustre: DEBUG MARKER: dmesg [ 8231.071648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 81: rename and stat under striped directory ========================================== 18:18:53 \(1491416333\) [ 8231.275937] Lustre: DEBUG MARKER: == sanityn test 81: rename and stat under striped directory ========================================== 18:18:53 (1491416333) [ 8231.487277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_81 needs \>= 2 MDTs [ 8231.665326] Lustre: DEBUG MARKER: SKIP: sanityn test_81 needs >= 2 MDTs [ 8231.880502] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8232.567607] Lustre: DEBUG MARKER: rc=0; [ 8232.567607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8232.567607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8232.567607] echo $(hostname -s): $val; [ 8232.567607] rc=$val; [ 8232.567607] fi; [ 8232.567607] exit $rc [ 8233.267920] Lustre: DEBUG MARKER: dmesg [ 8233.857236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================== 18:18:55 \(1491416335\) [ 8234.044960] Lustre: DEBUG MARKER: == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================== 18:18:55 (1491416335) [ 8234.235299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 8234.631754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8235.335346] Lustre: DEBUG MARKER: rc=0; [ 8235.335346] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8235.335346] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8235.335346] echo $(hostname -s): $val; [ 8235.335346] rc=$val; [ 8235.335346] fi; [ 8235.335346] exit $rc [ 8236.055678] Lustre: DEBUG MARKER: dmesg [ 8236.593787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 83: access striped directory while it is being created\/unlinked ====================== 18:18:58 \(1491416338\) [ 8236.787481] Lustre: DEBUG MARKER: == sanityn test 83: access striped directory while it is being created/unlinked ====================== 18:18:58 (1491416338) [ 8237.016677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_83 needs \>= 2 MDTs [ 8237.208762] Lustre: DEBUG MARKER: SKIP: sanityn test_83 needs >= 2 MDTs [ 8237.432396] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8238.118263] Lustre: DEBUG MARKER: rc=0; [ 8238.118263] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8238.118263] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8238.118263] echo $(hostname -s): $val; [ 8238.118263] rc=$val; [ 8238.118263] fi; [ 8238.118263] exit $rc [ 8238.830861] Lustre: DEBUG MARKER: dmesg [ 8239.402925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 90: open\/create and unlink striped directory ========================================= 18:19:01 \(1491416341\) [ 8239.596049] Lustre: DEBUG MARKER: == sanityn test 90: open/create and unlink striped directory ========================================= 18:19:01 (1491416341) [ 8239.817725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_90 needs \>= 2 MDTs [ 8240.008285] Lustre: DEBUG MARKER: SKIP: sanityn test_90 needs >= 2 MDTs [ 8240.215801] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8240.932120] Lustre: DEBUG MARKER: rc=0; [ 8240.932120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8240.932120] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8240.932120] echo $(hostname -s): $val; [ 8240.932120] rc=$val; [ 8240.932120] fi; [ 8240.932120] exit $rc [ 8241.640256] Lustre: DEBUG MARKER: dmesg [ 8242.170985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 91: chmod and unlink striped directory =============================================== 18:19:04 \(1491416344\) [ 8242.358460] Lustre: DEBUG MARKER: == sanityn test 91: chmod and unlink striped directory =============================================== 18:19:04 (1491416344) [ 8242.566314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_91 needs \>= 2 MDTs [ 8242.749471] Lustre: DEBUG MARKER: SKIP: sanityn test_91 needs >= 2 MDTs [ 8242.964988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8243.640775] Lustre: DEBUG MARKER: rc=0; [ 8243.640775] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8243.640775] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8243.640775] echo $(hostname -s): $val; [ 8243.640775] rc=$val; [ 8243.640775] fi; [ 8243.640775] exit $rc [ 8244.333419] Lustre: DEBUG MARKER: dmesg [ 8244.889696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 92: create remote directory under orphan directory =================================== 18:19:06 \(1491416346\) [ 8245.084385] Lustre: DEBUG MARKER: == sanityn test 92: create remote directory under orphan directory =================================== 18:19:06 (1491416346) [ 8245.302052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_92 needs \>= 2 MDTs [ 8245.488417] Lustre: DEBUG MARKER: SKIP: sanityn test_92 needs >= 2 MDTs [ 8245.692755] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8246.369705] Lustre: DEBUG MARKER: rc=0; [ 8246.369705] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8246.369705] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8246.369705] echo $(hostname -s): $val; [ 8246.369705] rc=$val; [ 8246.369705] fi; [ 8246.369705] exit $rc [ 8247.062885] Lustre: DEBUG MARKER: dmesg [ 8247.615967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 93: alloc_rr should not allocate on same ost ========================================= 18:19:09 \(1491416349\) [ 8247.807722] Lustre: DEBUG MARKER: == sanityn test 93: alloc_rr should not allocate on same ost ========================================= 18:19:09 (1491416349) [ 8247.987722] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT*/qos_threshold_rr [ 8248.282539] Lustre: DEBUG MARKER: lctl set_param -n lod.lustre-MDT*/qos_threshold_rr 100 [ 8248.576270] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x00000163 [ 8253.877407] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 8254.180875] Lustre: DEBUG MARKER: lctl set_param -n 'lod.lustre-MDT*/qos_threshold_rr' 17 [ 8254.543516] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8255.219233] Lustre: DEBUG MARKER: rc=0; [ 8255.219233] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8255.219233] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8255.219233] echo $(hostname -s): $val; [ 8255.219233] rc=$val; [ 8255.219233] fi; [ 8255.219233] exit $rc [ 8255.916795] Lustre: DEBUG MARKER: dmesg [ 8256.460604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cleanup: ====================================================== [ 8256.658682] Lustre: DEBUG MARKER: cleanup: ====================================================== [ 8257.046942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test complete, duration 1963 sec ========================================================== 18:19:19 \(1491416359\) [ 8257.237454] Lustre: DEBUG MARKER: == sanityn test complete, duration 1963 sec ========================================================== 18:19:19 (1491416359) [ 8267.167814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-quota ============----- Wed Apr 5 18:19:29 UTC 2017 [ 8267.358923] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Wed Apr 5 18:19:29 UTC 2017 [ 8267.714551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test complete, duration -o sec ======================================================= 18:19:29 \(1491416369\) [ 8267.892112] Lustre: DEBUG MARKER: == sanity-quota test complete, duration -o sec ======================================================= 18:19:29 (1491416369) [ 8269.403407] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8269.403407] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8269.403407] if [ $running -ne $mpts ]; then [ 8269.403407] echo $(hostname) env are INSANE!; [ 8269.403407] exit 1; [ 8269.403407] fi [ 8269.750649] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8269.750649] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8269.750649] if [ $running -ne $mpts ]; then [ 8269.750649] echo $(hostname) env are INSANE!; [ 8269.750649] exit 1; [ 8269.750649] fi [ 8270.656747] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 8270.953402] Lustre: DEBUG MARKER: cat /proc/mounts [ 8272.619690] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 8272.950565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 8273.134017] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8273.294202] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 8273.646473] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 8274.221982] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8274.561647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 3 4a 6 11 33 34 35 [ 8274.746732] Lustre: DEBUG MARKER: excepting tests: 3 4a 6 11 33 34 35 [ 8274.950036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 12a 9 [ 8275.144880] Lustre: DEBUG MARKER: skipping tests SLOW=no: 12a 9 [ 8275.929028] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8280.415333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 0: Test basic quota performance ================================================= 18:19:42 \(1491416382\) [ 8280.614483] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ================================================= 18:19:42 (1491416382) [ 8280.795892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8287.140232] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8287.488994] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8290.450656] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8298.810705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8299.115689] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8299.518906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8300.847432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8302.177256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8303.495183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8304.804640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8311.140877] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8312.952627] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8313.611554] Lustre: DEBUG MARKER: rc=0; [ 8313.611554] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8313.611554] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8313.611554] echo $(hostname -s): $val; [ 8313.611554] rc=$val; [ 8313.611554] fi; [ 8313.611554] exit $rc [ 8314.286655] Lustre: DEBUG MARKER: dmesg [ 8314.809931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 1: Block hard limit \(normal use and out of quota\) =============================== 18:20:16 \(1491416416\) [ 8314.999259] Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) =============================== 18:20:16 (1491416416) [ 8315.187180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8321.523026] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8321.867796] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8324.152904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(block hardlimit:10 MB\) [ 8324.336566] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 8324.788896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write... [ 8324.977571] Lustre: DEBUG MARKER: Write... [ 8325.471887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8325.669928] Lustre: DEBUG MARKER: Write out of block quota ... [ 8326.307308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8326.606460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8327.029696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8328.346361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8329.657948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8330.965865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8332.292671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8333.611573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8334.926676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8336.240703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8342.573590] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8344.406156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8344.590292] Lustre: DEBUG MARKER: -------------------------------------- [ 8344.796313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(block hardlimit:10 MB\) [ 8344.983302] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 8345.272110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write ... [ 8345.452717] Lustre: DEBUG MARKER: Write ... [ 8345.725541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8345.936261] Lustre: DEBUG MARKER: Write out of block quota ... [ 8346.558568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8346.865629] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8347.301754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8348.615441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8349.928378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8351.237860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8352.546911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8353.857884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8355.184551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8361.524642] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8363.340911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8363.993616] Lustre: DEBUG MARKER: rc=0; [ 8363.993616] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8363.993616] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8363.993616] echo $(hostname -s): $val; [ 8363.993616] rc=$val; [ 8363.993616] fi; [ 8363.993616] exit $rc [ 8364.680057] Lustre: DEBUG MARKER: dmesg [ 8365.243676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 2: File hard limit \(normal use and out of quota\) ================================ 18:21:07 \(1491416467\) [ 8365.430317] Lustre: DEBUG MARKER: == sanity-quota test 2: File hard limit (normal use and out of quota) ================================ 18:21:07 (1491416467) [ 8365.625685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8371.964108] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8372.316272] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8372.616773] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8372.908333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8373.197371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8374.492971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8375.791078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8377.086114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8378.384095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8379.678944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8380.018388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(inode hardlimit:1024 files\) [ 8380.209215] Lustre: DEBUG MARKER: User quota (inode hardlimit:1024 files) [ 8380.569143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8380.756295] Lustre: DEBUG MARKER: Create 1024 files ... [ 8381.645137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8381.847366] Lustre: DEBUG MARKER: Create out of file quota ... [ 8383.193526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8390.720015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8390.910189] Lustre: DEBUG MARKER: -------------------------------------- [ 8391.122473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(inode hardlimit:1024 files\) [ 8391.306020] Lustre: DEBUG MARKER: Group quota (inode hardlimit:1024 files) [ 8391.855176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8392.053650] Lustre: DEBUG MARKER: Create 1024 files ... [ 8392.911963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8393.121503] Lustre: DEBUG MARKER: Create out of file quota ... [ 8394.416410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8400.755560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8407.074628] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8408.954500] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8409.610655] Lustre: DEBUG MARKER: rc=0; [ 8409.610655] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8409.610655] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8409.610655] echo $(hostname -s): $val; [ 8409.610655] rc=$val; [ 8409.610655] fi; [ 8409.610655] exit $rc [ 8410.296436] Lustre: DEBUG MARKER: dmesg [ 8410.835585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_3 skipping ALWAYS excluded test 3 [ 8411.019438] Lustre: DEBUG MARKER: SKIP: sanity-quota test_3 skipping ALWAYS excluded test 3 [ 8411.225846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_4a skipping ALWAYS excluded test 4a [ 8411.412387] Lustre: DEBUG MARKER: SKIP: sanity-quota test_4a skipping ALWAYS excluded test 4a [ 8411.628242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 4b: Grace time strings handling ================================================= 18:21:53 \(1491416513\) [ 8411.821087] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ================================================= 18:21:53 (1491416513) [ 8412.001457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8419.220750] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8419.881498] Lustre: DEBUG MARKER: rc=0; [ 8419.881498] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8419.881498] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8419.881498] echo $(hostname -s): $val; [ 8419.881498] rc=$val; [ 8419.881498] fi; [ 8419.881498] exit $rc [ 8420.563620] Lustre: DEBUG MARKER: dmesg [ 8421.101935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 5: Chown & chgrp successfully even out of block\/file quota ====================== 18:22:03 \(1491416523\) [ 8421.300620] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 8421.780812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8428.123194] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8428.480954] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8428.784363] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8429.091442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8429.390902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 8429.705198] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8432.903207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8433.202496] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8433.610448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8434.935753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8436.257923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8437.569539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8438.881992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8440.207677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8441.527257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8447.865131] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8450.919723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8451.571138] Lustre: DEBUG MARKER: rc=0; [ 8451.571138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8451.571138] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8451.571138] echo $(hostname -s): $val; [ 8451.571138] rc=$val; [ 8451.571138] fi; [ 8451.571138] exit $rc [ 8452.258472] Lustre: DEBUG MARKER: dmesg [ 8452.820789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_6 skipping ALWAYS excluded test 6 [ 8453.017895] Lustre: DEBUG MARKER: SKIP: sanity-quota test_6 skipping ALWAYS excluded test 6 [ 8453.246494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7a: Quota reintegration \(global index\) ========================================== 18:22:35 \(1491416555\) [ 8453.432292] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================== 18:22:35 (1491416555) [ 8453.617707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8459.959377] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8460.329977] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8463.825169] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8470.461771] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8470.687762] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8470.690724] LustreError: Skipped 1 previous similar message [ 8470.692990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8476.166513] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 8476.168693] Lustre: Skipped 10 previous similar messages [ 8478.548497] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8542.284245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8542.587405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8543.018495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8544.337826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8545.646960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8546.962487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8548.273518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8549.584169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8555.922334] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8562.613419] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 8562.615505] Lustre: Skipped 1 previous similar message [ 8563.551666] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8563.553689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8565.087103] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8574.402830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8574.701056] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8575.119102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8576.430971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8577.765509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8579.092519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8580.418341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8586.763663] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8588.490356] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8589.155079] Lustre: DEBUG MARKER: rc=0; [ 8589.155079] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8589.155079] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8589.155079] echo $(hostname -s): $val; [ 8589.155079] rc=$val; [ 8589.155079] fi; [ 8589.155079] exit $rc [ 8589.838802] Lustre: DEBUG MARKER: dmesg [ 8590.508062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7b: Quota reintegration \(slave index\) =========================================== 18:24:52 \(1491416692\) [ 8590.695096] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) =========================================== 18:24:52 (1491416692) [ 8590.885155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8597.220849] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8597.587250] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8600.997622] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8608.898604] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0xa02 [ 8609.215977] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8616.839267] Lustre: *** cfs_fail_loc=a02, val=0*** [ 8617.263818] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8620.319830] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8620.321865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8622.875549] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8629.956700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8630.258925] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8630.679891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8631.991741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8638.325318] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8640.179374] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8640.872814] Lustre: DEBUG MARKER: rc=0; [ 8640.872814] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8640.872814] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8640.872814] echo $(hostname -s): $val; [ 8640.872814] rc=$val; [ 8640.872814] fi; [ 8640.872814] exit $rc [ 8641.578167] Lustre: DEBUG MARKER: dmesg [ 8642.151506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7c: Quota reintegration \(restart mds during reintegration\) ====================== 18:25:44 \(1491416744\) [ 8642.350559] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ====================== 18:25:44 (1491416744) [ 8642.531841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8648.873160] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8649.247243] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8661.361243] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8672.041382] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8672.346486] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8672.506900] Lustre: Failing over lustre-MDT0000 [ 8675.167133] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8678.546107] Lustre: 31776:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491416774/real 1491416774] req@ffff880049b2d200 x1563855297791536/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491416780 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8678.552583] Lustre: 31776:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8678.569640] Lustre: server umount lustre-MDT0000 complete [ 8678.714756] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8679.019004] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 8679.019004] grep -q ^lustre-mdt1/ /proc/mounts || [ 8679.019004] zpool export lustre-mdt1 [ 8679.873454] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 8680.159210] Lustre: DEBUG MARKER: modprobe zfs; [ 8680.159210] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 8680.159210] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 8680.740651] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 8681.037604] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 8681.364072] LustreError: 844:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 8681.409129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.580302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8681.883714] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 8682.170113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8682.248587] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8682.399745] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8682.696406] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8683.011234] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8683.326398] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 8683.638119] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8683.971376] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8856.038791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8856.336499] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8856.711545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8858.028257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8859.337244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8860.651300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8861.975401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8868.318677] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8870.127580] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8870.814648] Lustre: DEBUG MARKER: rc=0; [ 8870.814648] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8870.814648] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8870.814648] echo $(hostname -s): $val; [ 8870.814648] rc=$val; [ 8870.814648] fi; [ 8870.814648] exit $rc [ 8871.519658] Lustre: DEBUG MARKER: dmesg [ 8872.080465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7d: Quota reintegration \(Transfer index in multiple bulks\) ====================== 18:29:34 \(1491416974\) [ 8872.268786] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ====================== 18:29:34 (1491416974) [ 8872.452984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8878.799508] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8879.170134] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8889.793503] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x608 [ 8890.119147] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 8899.934609] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9195.520336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9195.828348] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9196.249177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9197.559574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9198.875033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9200.186891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9201.504878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9202.820222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9209.160905] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9212.149028] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9212.833024] Lustre: DEBUG MARKER: rc=0; [ 9212.833024] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9212.833024] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9212.833024] echo $(hostname -s): $val; [ 9212.833024] rc=$val; [ 9212.833024] fi; [ 9212.833024] exit $rc [ 9213.509155] Lustre: DEBUG MARKER: dmesg [ 9214.075428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7e: Quota reintegration \(inode limits\) ========================================== 18:35:16 \(1491417316\) [ 9214.270423] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================== 18:35:16 (1491417316) [ 9214.496074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_7e Required more MDTs [ 9214.682176] Lustre: DEBUG MARKER: SKIP: sanity-quota test_7e Required more MDTs [ 9214.886048] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9215.550828] Lustre: DEBUG MARKER: rc=0; [ 9215.550828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9215.550828] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9215.550828] echo $(hostname -s): $val; [ 9215.550828] rc=$val; [ 9215.550828] fi; [ 9215.550828] exit $rc [ 9216.227386] Lustre: DEBUG MARKER: dmesg [ 9216.796053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 8: Run dbench with quota enabled ================================================ 18:35:18 \(1491417318\) [ 9216.985536] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ================================================ 18:35:18 (1491417318) [ 9217.170358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9223.506147] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9223.870432] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9224.171888] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9224.466975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 9224.773713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-zfs.lustre-MDT0000.quota_slave.enabled [ 9225.082835] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9381.970561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9382.296323] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9383.279840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9384.646868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9385.996831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9387.336302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9393.699176] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9394.518782] LustreError: 5925:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x8 qsd:lustre-MDT0000 qtype:usr id:60000 enforced:1 granted:262141 pending:0 waiting:0 req:1 usage:18446744073709551613 qunit:262144 qtune:65536 edquot:0 [ 9397.065131] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9397.734159] Lustre: DEBUG MARKER: rc=0; [ 9397.734159] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9397.734159] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9397.734159] echo $(hostname -s): $val; [ 9397.734159] rc=$val; [ 9397.734159] fi; [ 9397.734159] exit $rc [ 9398.431393] Lustre: DEBUG MARKER: dmesg [ 9399.046306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_9 skipping SLOW test 9 [ 9399.245866] Lustre: DEBUG MARKER: SKIP: sanity-quota test_9 skipping SLOW test 9 [ 9399.472941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 10: Test quota for root user ==================================================== 18:38:21 \(1491417501\) [ 9399.669961] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ==================================================== 18:38:21 (1491417501) [ 9399.852444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9406.201403] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9406.570977] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9409.538654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9409.844098] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9410.256775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9411.577079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9412.886129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9414.204381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9415.517802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9416.827987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9418.153985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9424.492550] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9426.313231] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9427.007868] Lustre: DEBUG MARKER: rc=0; [ 9427.007868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9427.007868] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9427.007868] echo $(hostname -s): $val; [ 9427.007868] rc=$val; [ 9427.007868] fi; [ 9427.007868] exit $rc [ 9427.704024] Lustre: DEBUG MARKER: dmesg [ 9428.241673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_11 skipping ALWAYS excluded test 11 [ 9428.434536] Lustre: DEBUG MARKER: SKIP: sanity-quota test_11 skipping ALWAYS excluded test 11 [ 9428.662415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_12a skipping SLOW test 12a [ 9428.872775] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12a skipping SLOW test 12a [ 9429.103927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 12b: Inode quota rebalancing ==================================================== 18:38:51 \(1491417531\) [ 9429.302580] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ==================================================== 18:38:51 (1491417531) [ 9429.507957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-quota test_12b skipping rebalancing test [ 9429.685816] Lustre: DEBUG MARKER: SKIP: sanity-quota test_12b skipping rebalancing test [ 9429.901940] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9430.608276] Lustre: DEBUG MARKER: rc=0; [ 9430.608276] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9430.608276] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9430.608276] echo $(hostname -s): $val; [ 9430.608276] rc=$val; [ 9430.608276] fi; [ 9430.608276] exit $rc [ 9431.310907] Lustre: DEBUG MARKER: dmesg [ 9431.866507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 18:38:53 \(1491417533\) [ 9432.060318] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 18:38:53 (1491417533) [ 9432.242065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9438.574909] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9438.937125] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9462.420587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9462.731063] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9463.107738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9464.431387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9465.752660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9467.073958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9468.391612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9474.726857] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9475.446523] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9476.117763] Lustre: DEBUG MARKER: rc=0; [ 9476.117763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9476.117763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9476.117763] echo $(hostname -s): $val; [ 9476.117763] rc=$val; [ 9476.117763] fi; [ 9476.117763] exit $rc [ 9476.824386] Lustre: DEBUG MARKER: dmesg [ 9477.376756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 15: Set over 4T block quota ===================================================== 18:39:39 \(1491417579\) [ 9477.563929] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ===================================================== 18:39:39 (1491417579) [ 9477.748581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9484.071212] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9487.159572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9487.833769] Lustre: DEBUG MARKER: rc=0; [ 9487.833769] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9487.833769] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9487.833769] echo $(hostname -s): $val; [ 9487.833769] rc=$val; [ 9487.833769] fi; [ 9487.833769] exit $rc [ 9488.520959] Lustre: DEBUG MARKER: dmesg [ 9489.076060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 17: DQACQ return recoverable error ============================================== 18:39:51 \(1491417591\) [ 9489.261316] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error ============================================== 18:39:51 (1491417591) [ 9489.439728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9495.787458] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9496.160482] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9508.981707] Lustre: DEBUG MARKER: lctl set_param fail_val=37 fail_loc=0xa04 [ 9509.183642] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9510.217058] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9510.218685] Lustre: Skipped 1 previous similar message [ 9511.320752] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9512.690865] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9513.500163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9513.795904] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9514.230454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9515.542754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9516.853761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9518.174303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9519.503707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9520.822496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9522.136436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9523.448078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9529.780810] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9531.584671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9537.944283] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9538.296773] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9541.746846] Lustre: DEBUG MARKER: lctl set_param fail_val=11 fail_loc=0xa04 [ 9541.951699] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9541.953270] Lustre: Skipped 1 previous similar message [ 9544.089104] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9545.457094] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9546.227057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9546.526685] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9546.966787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9548.278293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9549.590489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9550.901419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9552.235823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9553.545317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9559.883286] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9561.693151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9568.030092] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9568.415417] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9571.874079] Lustre: DEBUG MARKER: lctl set_param fail_val=110 fail_loc=0xa04 [ 9572.072864] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9572.074275] Lustre: Skipped 1 previous similar message [ 9574.213072] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9575.615294] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9576.406521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9576.703959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9577.128242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9578.445489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9579.756736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9581.065257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9582.395533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9583.726937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9590.085413] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9591.944766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9598.289775] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9598.649560] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9602.053436] Lustre: DEBUG MARKER: lctl set_param fail_val=107 fail_loc=0xa04 [ 9602.254908] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9602.256120] Lustre: Skipped 1 previous similar message [ 9604.391785] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9605.742715] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9606.563427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9606.864728] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9607.270764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9608.593708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9609.903318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9611.215151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9612.537724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9613.851083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9620.182367] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9621.956993] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9622.641880] Lustre: DEBUG MARKER: rc=0; [ 9622.641880] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9622.641880] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9622.641880] echo $(hostname -s): $val; [ 9622.641880] rc=$val; [ 9622.641880] fi; [ 9622.641880] exit $rc [ 9623.352058] Lustre: DEBUG MARKER: dmesg [ 9623.904327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 18: MDS failover while writing, no watchdog triggered \(b14840\) ================== 18:42:05 \(1491417725\) [ 9624.098531] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ================== 18:42:05 (1491417725) [ 9624.721947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9631.061701] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9631.411445] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9641.822582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [ 9641.993375] Lustre: DEBUG MARKER: User quota (limit: 200m) [ 9642.297516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(buffered\) ... [ 9642.481872] Lustre: DEBUG MARKER: Write 100M (buffered) ... [ 9642.689470] Lustre: DEBUG MARKER: sync; sync; sync [ 9644.161388] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 9644.686445] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 9644.920803] LustreError: 14526:0:(osd_handler.c:651:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 9645.067867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 9645.214570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9645.411513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [ 9645.609083] Lustre: DEBUG MARKER: Fail mds for 0 seconds [ 9645.791386] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9646.082665] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 9646.228003] Lustre: Failing over lustre-MDT0000 [ 9647.682190] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9647.685792] LustreError: Skipped 3 previous similar messages [ 9648.510843] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.217757] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.262036] Lustre: 14817:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491417748/real 1491417748] req@ffff88006365c300 x1563855298891328/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491417754 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9652.281294] Lustre: server umount lustre-MDT0000 complete [ 9652.427101] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9652.728372] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 9652.728372] grep -q ^lustre-mdt1/ /proc/mounts || [ 9652.728372] zpool export lustre-mdt1 [ 9663.157593] Lustre: DEBUG MARKER: hostname [ 9663.495363] Lustre: DEBUG MARKER: modprobe zfs; [ 9663.495363] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 9663.495363] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 9664.102188] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 9664.390627] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 9664.578877] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 9664.581326] Lustre: Skipped 12 previous similar messages [ 9664.659798] LustreError: 16299:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 9664.842352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.009622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9665.309377] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 9665.812983] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9666.105968] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9666.401242] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9666.701904] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 9669.184052] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9674.314173] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 9674.535608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 9674.535609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 9674.824549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 9674.824561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 9676.181131] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 9676.473856] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 9676.818355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=28722, time=0, timeout=600\) [ 9677.015485] Lustre: DEBUG MARKER: (dd_pid=28722, time=0, timeout=600) [ 9677.310382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9677.611950] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9678.012142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9679.343620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9680.655222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9681.979043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9683.298600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9684.614430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9690.965217] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9692.789706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9699.129701] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9699.505968] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9701.904210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [ 9702.075292] Lustre: DEBUG MARKER: User quota (limit: 200m) [ 9702.402502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(directio\) ... [ 9702.595554] Lustre: DEBUG MARKER: Write 100M (directio) ... [ 9702.772937] Lustre: DEBUG MARKER: sync; sync; sync [ 9703.753258] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 9704.068551] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 9704.317537] LustreError: 18487:0:(osd_handler.c:651:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 9704.472958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 9704.630137] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9704.846617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [ 9705.118335] Lustre: DEBUG MARKER: Fail mds for 0 seconds [ 9705.361351] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9705.660439] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 9705.809212] Lustre: Failing over lustre-MDT0000 [ 9706.624181] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.847087] Lustre: 18779:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491417807/real 1491417807] req@ffff880046ef7300 x1563855298892992/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491417813 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9711.867036] Lustre: server umount lustre-MDT0000 complete [ 9712.012163] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9712.317929] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 9712.317929] grep -q ^lustre-mdt1/ /proc/mounts || [ 9712.317929] zpool export lustre-mdt1 [ 9722.750986] Lustre: DEBUG MARKER: hostname [ 9723.088247] Lustre: DEBUG MARKER: modprobe zfs; [ 9723.088247] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 9723.088247] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 9723.666952] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 9723.958620] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 9724.148143] Lustre: MGS: Connection restored to 606155ce-7931-bd92-9d83-ea7b7aec62fd (at 0@lo) [ 9724.150193] Lustre: Skipped 8 previous similar messages [ 9724.296502] LustreError: 20261:0:(obd_config.c:1393:class_process_proc_param()) lov.: error writing proc entry 'stripeoffset': rc = -34 [ 9724.361523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.524993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9724.818249] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 9725.308986] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9725.604372] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9725.898961] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9726.201034] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 9733.939166] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9735.036535] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9736.053952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9736.069300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9736.274858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9736.281447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9737.602281] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 9737.892623] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 9749.349682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=31290, time=11, timeout=600\) [ 9749.535375] Lustre: DEBUG MARKER: (dd_pid=31290, time=11, timeout=600) [ 9749.793947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9750.101259] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9750.413580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9751.725746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9753.040700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9754.355442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9755.665739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9762.003611] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9764.167560] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9764.845044] Lustre: DEBUG MARKER: rc=0; [ 9764.845044] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9764.845044] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9764.845044] echo $(hostname -s): $val; [ 9764.845044] rc=$val; [ 9764.845044] fi; [ 9764.845044] exit $rc [ 9765.544998] Lustre: DEBUG MARKER: dmesg [ 9766.059203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 19: Updating admin limits doesn\'t zero operational limits\(b14790\) =============== 18:44:28 \(1491417868\) [ 9766.247695] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) =============== 18:44:28 (1491417868) [ 9766.435602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9772.773936] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9773.135186] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9785.054318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Files for user \(quota_usr\): [ 9785.242399] Lustre: DEBUG MARKER: Files for user (quota_usr): [ 9785.470102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Block quota isn\'t 0 \(u:quota_usr:2\). [ 9785.666786] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 9785.975436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Files for user \(quota_usr\): [ 9786.152710] Lustre: DEBUG MARKER: Files for user (quota_usr): [ 9786.372798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Block quota isn\'t 0 \(u:quota_usr:2\). [ 9786.569662] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2). [ 9787.088823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9787.386941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9787.766931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9789.077803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9790.384594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9791.698556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9793.007858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9794.323126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9795.637039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9801.981721] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9803.813116] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9804.472382] Lustre: DEBUG MARKER: rc=0; [ 9804.472382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9804.472382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9804.472382] echo $(hostname -s): $val; [ 9804.472382] rc=$val; [ 9804.472382] fi; [ 9804.472382] exit $rc [ 9805.171467] Lustre: DEBUG MARKER: dmesg [ 9805.672762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 20: Test if setquota specifiers work properly \(b15754\) ========================== 18:45:07 \(1491417907\) [ 9805.866544] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================== 18:45:07 (1491417907) [ 9808.596075] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9809.272564] Lustre: DEBUG MARKER: rc=0; [ 9809.272564] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9809.272564] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9809.272564] echo $(hostname -s): $val; [ 9809.272564] rc=$val; [ 9809.272564] fi; [ 9809.272564] exit $rc [ 9809.963956] Lustre: DEBUG MARKER: dmesg [ 9810.463766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 21: Setquota while writing & deleting \(b16053\) ================================== 18:45:12 \(1491417912\) [ 9810.650206] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [ 9810.834704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9817.174460] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9817.537593] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9819.897162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for user: quota_usr [ 9820.089678] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [ 9820.443511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for group: quota_usr [ 9820.638199] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [ 9821.079122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 1 times [ 9821.262743] Lustre: DEBUG MARKER: Set quota for 1 times [ 9822.880109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 2 times [ 9823.078786] Lustre: DEBUG MARKER: Set quota for 2 times [ 9824.675598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 3 times [ 9824.863676] Lustre: DEBUG MARKER: Set quota for 3 times [ 9826.480660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 4 times [ 9826.669963] Lustre: DEBUG MARKER: Set quota for 4 times [ 9828.238030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 5 times [ 9828.427415] Lustre: DEBUG MARKER: Set quota for 5 times [ 9830.075734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 6 times [ 9830.280448] Lustre: DEBUG MARKER: Set quota for 6 times [ 9831.925181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 7 times [ 9832.113796] Lustre: DEBUG MARKER: Set quota for 7 times [ 9833.718580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 8 times [ 9833.903704] Lustre: DEBUG MARKER: Set quota for 8 times [ 9835.535635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 9 times [ 9835.720883] Lustre: DEBUG MARKER: Set quota for 9 times [ 9837.345199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 10 times [ 9837.522949] Lustre: DEBUG MARKER: Set quota for 10 times [ 9839.110658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 11 times [ 9839.306701] Lustre: DEBUG MARKER: Set quota for 11 times [ 9840.914334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 12 times [ 9841.110193] Lustre: DEBUG MARKER: Set quota for 12 times [ 9842.701859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 13 times [ 9842.899904] Lustre: DEBUG MARKER: Set quota for 13 times [ 9844.629145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 14 times [ 9844.825319] Lustre: DEBUG MARKER: Set quota for 14 times [ 9846.410760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 15 times [ 9846.603584] Lustre: DEBUG MARKER: Set quota for 15 times [ 9848.246257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 16 times [ 9848.421473] Lustre: DEBUG MARKER: Set quota for 16 times [ 9850.043579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 17 times [ 9850.223849] Lustre: DEBUG MARKER: Set quota for 17 times [ 9912.680461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Files for user \(quota_usr\): [ 9912.870416] Lustre: DEBUG MARKER: Files for user (quota_usr): [13539.215959] SysRq : Show State [13539.217521] task PC stack pid father [13539.219215] systemd S 0000000000000000 0 1 0 0x00000000 [13539.221119] ffff88007c7bbdb8 0000000000000086 ffff88007c7b0000 ffff88007c7bbfd8 [13539.222899] ffff88007c7bbfd8 ffff88007c7bbfd8 ffff88007c7b0000 0000000000000000 [13539.224669] 0000000000000000 ffff88003681f060 ffff88007c7b0000 0000000000000000 [13539.226446] Call Trace: [13539.227738] [] schedule+0x29/0x70 [13539.229260] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13539.230923] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13539.232556] [] schedule_hrtimeout_range+0x13/0x20 [13539.234078] [] ep_poll+0x23e/0x360 [13539.235557] [] ? __fput+0x183/0x260 [13539.236983] [] ? wake_up_state+0x20/0x20 [13539.238530] [] SyS_epoll_wait+0xed/0x120 [13539.239993] [] system_call_fastpath+0x16/0x1b [13539.241551] kthreadd S ffff88007c7b0fb0 0 2 0 0x00000000 [13539.243163] ffff88007c027e70 0000000000000046 ffff88007c7b0fb0 ffff88007c027fd8 [13539.244875] ffff88007c027fd8 ffff88007c027fd8 ffff88007c7b0fb0 ffffffff819f9d00 [13539.246610] ffff880062d23980 ffff88007c7b0fb0 ffff88007c7b0fb0 ffff88007c7b0fb0 [13539.248292] Call Trace: [13539.249504] [] schedule+0x29/0x70 [13539.251019] [] kthreadd+0x2e6/0x2f0 [13539.252475] [] ? kthread_create_on_cpu+0x60/0x60 [13539.253987] [] ret_from_fork+0x58/0x90 [13539.255531] [] ? kthread_create_on_cpu+0x60/0x60 [13539.257030] ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 [13539.258675] ffff88007c037e80 0000000000000046 ffff88007c7b1f60 ffff88007c037fd8 [13539.260371] ffff88007c037fd8 ffff88007c037fd8 ffff88007c7b1f60 ffff88007c7b1f60 [13539.262071] ffff88007d002380 ffffffff819ed0c0 0000000000000000 0000000000000000 [13539.263715] Call Trace: [13539.264962] [] schedule+0x29/0x70 [13539.266417] [] smpboot_thread_fn+0xdd/0x180 [13539.267890] [] ? lg_double_unlock+0x90/0x90 [13539.269428] [] kthread+0xcf/0xe0 [13539.270864] [] ? kthread_create_on_node+0x140/0x140 [13539.272382] [] ret_from_fork+0x58/0x90 [13539.273869] [] ? kthread_create_on_node+0x140/0x140 [13539.275443] kworker/0:0H S ffff88007fc16840 0 5 2 0x00000000 [13539.277047] ffff88007c03fe50 0000000000000046 ffff88007c7b3ec0 ffff88007c03ffd8 [13539.278770] ffff88007c03ffd8 ffff88007c03ffd8 ffff88007c7b3ec0 ffff88007fc16860 [13539.280504] ffff88007d03f9b0 ffff88007c7b3ec0 ffff88007d03f980 ffff88007fc16840 [13539.282200] Call Trace: [13539.283409] [] schedule+0x29/0x70 [13539.284893] [] worker_thread+0x1d9/0x410 [13539.286392] [] ? rescuer_thread+0x460/0x460 [13539.287903] [] kthread+0xcf/0xe0 [13539.289283] [] ? kthread_create_on_node+0x140/0x140 [13539.290854] [] ret_from_fork+0x58/0x90 [13539.292271] [] ? kthread_create_on_node+0x140/0x140 [13539.293878] kworker/u4:0 S ffff88007c060800 0 6 2 0x00000000 [13539.295465] ffff88007c067e50 0000000000000046 ffff88007c7b4e70 ffff88007c067fd8 [13539.297154] ffff88007c067fd8 ffff88007c067fd8 ffff88007c7b4e70 ffff88007c060820 [13539.298785] ffff88007d03fab0 ffff88007c7b4e70 ffff88007d03fa80 ffff88007c060800 [13539.300458] Call Trace: [13539.301646] [] schedule+0x29/0x70 [13539.303111] [] worker_thread+0x1d9/0x410 [13539.304579] [] ? rescuer_thread+0x460/0x460 [13539.306083] [] kthread+0xcf/0xe0 [13539.307438] [] ? kthread_create_on_node+0x140/0x140 [13539.309012] [] ret_from_fork+0x58/0x90 [13539.310466] [] ? kthread_create_on_node+0x140/0x140 [13539.312033] migration/0 S 0000000000000000 0 7 2 0x00000000 [13539.313627] ffff88007c06be80 0000000000000046 ffff88007c7b5e20 ffff88007c06bfd8 [13539.315339] ffff88007c06bfd8 ffff88007c06bfd8 ffff88007c7b5e20 ffff88007c7b5e20 [13539.316967] ffff88007d002390 ffffffff81a04f00 0000000000000000 0000000000000000 [13539.318651] Call Trace: [13539.319873] [] schedule+0x29/0x70 [13539.321337] [] smpboot_thread_fn+0xdd/0x180 [13539.322847] [] ? lg_double_unlock+0x90/0x90 [13539.324304] [] kthread+0xcf/0xe0 [13539.325744] [] ? kthread_create_on_node+0x140/0x140 [13539.327308] [] ret_from_fork+0x58/0x90 [13539.328723] [] ? kthread_create_on_node+0x140/0x140 [13539.330297] rcu_bh S 0000000000000000 0 8 2 0x00000000 [13539.331884] ffff88007c06fe10 0000000000000046 ffff88007c7b6dd0 ffff88007c06ffd8 [13539.333566] ffff88007c06ffd8 ffff88007c06ffd8 ffff88007c7b6dd0 ffffffff81a05d40 [13539.335268] ffff88007c7b6dd0 ffffffff81a1a3a0 0000000000000000 0000000000000000 [13539.336890] Call Trace: [13539.338132] [] schedule+0x29/0x70 [13539.339530] [] rcu_gp_kthread+0xb2/0x7e0 [13539.341039] [] ? __schedule+0x3b0/0x990 [13539.342565] [] ? wake_up_atomic_t+0x30/0x30 [13539.344071] [] ? rcu_gp_fqs+0xb0/0xb0 [13539.345538] [] kthread+0xcf/0xe0 [13539.346911] [] ? kthread_create_on_node+0x140/0x140 [13539.348476] [] ret_from_fork+0x58/0x90 [13539.349864] [] ? kthread_create_on_node+0x140/0x140 [13539.351429] rcu_sched S 000000000000f888 0 9 2 0x00000000 [13539.353007] ffff88007c07bd60 0000000000000046 ffff88007c070000 ffff88007c07bfd8 [13539.354682] ffff88007c07bfd8 ffff88007c07bfd8 ffff88007c070000 ffffffff81e380c0 [13539.356354] ffff88007c07bd90 0000000100ca043e ffffffff81e380c0 000000000000f888 [13539.357998] Call Trace: [13539.359243] [] schedule+0x29/0x70 [13539.360699] [] schedule_timeout+0x175/0x2d0 [13539.362169] [] ? internal_add_timer+0x70/0x70 [13539.363721] [] ? prepare_to_wait+0x56/0x90 [13539.365231] [] rcu_gp_kthread+0x349/0x7e0 [13539.366716] [] ? wake_up_atomic_t+0x30/0x30 [13539.368213] [] ? rcu_gp_fqs+0xb0/0xb0 [13539.369650] [] kthread+0xcf/0xe0 [13539.371025] [] ? kthread_create_on_node+0x140/0x140 [13539.372574] [] ret_from_fork+0x58/0x90 [13539.374024] [] ? kthread_create_on_node+0x140/0x140 [13539.375561] watchdog/0 S 0000000000000000 0 10 2 0x00000000 [13539.377145] ffff88007c07fe80 0000000000000046 ffff88007c070fb0 ffff88007c07ffd8 [13539.378819] ffff88007c07ffd8 ffff88007c07ffd8 ffff88007c070fb0 ffff88007c070fb0 [13539.380550] ffff88007d002890 ffffffff81a057e0 0000000000000000 0000000000000000 [13539.382250] Call Trace: [13539.383440] [] schedule+0x29/0x70 [13539.384866] [] smpboot_thread_fn+0xdd/0x180 [13539.386381] [] ? lg_double_unlock+0x90/0x90 [13539.387905] [] kthread+0xcf/0xe0 [13539.389314] [] ? kthread_create_on_node+0x140/0x140 [13539.390841] [] ret_from_fork+0x58/0x90 [13539.392249] [] ? kthread_create_on_node+0x140/0x140 [13539.393804] watchdog/1 S 0000000000000000 0 11 2 0x00000000 [13539.395389] ffff88007c0c7e80 0000000000000046 ffff88007c072f10 ffff88007c0c7fd8 [13539.397070] ffff88007c0c7fd8 ffff88007c0c7fd8 ffff88007c072f10 ffff88007c072f10 [13539.398683] ffff88007d0028a0 ffffffff81a057e0 0000000000000000 0000000000000000 [13539.400377] Call Trace: [13539.401568] [] schedule+0x29/0x70 [13539.403031] [] smpboot_thread_fn+0xdd/0x180 [13539.404467] [] ? lg_double_unlock+0x90/0x90 [13539.405972] [] kthread+0xcf/0xe0 [13539.407377] [] ? kthread_create_on_node+0x140/0x140 [13539.408909] [] ret_from_fork+0x58/0x90 [13539.410293] [] ? kthread_create_on_node+0x140/0x140 [13539.411836] migration/1 S 0000000000000000 0 12 2 0x00000000 [13539.413461] ffff88007c0cbe80 0000000000000046 ffff88007c073ec0 ffff88007c0cbfd8 [13539.415139] ffff88007c0cbfd8 ffff88007c0cbfd8 ffff88007c073ec0 ffff88007c073ec0 [13539.416753] ffff88007d0028b0 ffffffff81a04f00 0000000000000000 0000000000000000 [13539.418444] Call Trace: [13539.419649] [] schedule+0x29/0x70 [13539.421120] [] smpboot_thread_fn+0xdd/0x180 [13539.422625] [] ? lg_double_unlock+0x90/0x90 [13539.424121] [] kthread+0xcf/0xe0 [13539.425527] [] ? kthread_create_on_node+0x140/0x140 [13539.427024] [] ret_from_fork+0x58/0x90 [13539.428495] [] ? kthread_create_on_node+0x140/0x140 [13539.429976] ksoftirqd/1 R running task 0 13 2 0x00000000 [13539.431592] ffff88007c0cfe80 0000000000000046 ffff88007c074e70 ffff88007c0cffd8 [13539.433282] ffff88007c0cffd8 ffff88007c0cffd8 ffff88007c074e70 ffff88007c074e70 [13539.434904] ffff88007d0028c0 ffffffff819ed0c0 0000000000000000 0000000000000000 [13539.436610] Call Trace: [13539.437817] [] schedule+0x29/0x70 [13539.439293] [] smpboot_thread_fn+0xdd/0x180 [13539.440769] [] ? lg_double_unlock+0x90/0x90 [13539.442333] [] kthread+0xcf/0xe0 [13539.443727] [] ? kthread_create_on_node+0x140/0x140 [13539.445317] [] ret_from_fork+0x58/0x90 [13539.446746] [] ? kthread_create_on_node+0x140/0x140 [13539.448318] kworker/1:0H S ffff88007fd16840 0 15 2 0x00000000 [13539.449917] ffff88007c0dbe50 0000000000000046 ffff88007c076dd0 ffff88007c0dbfd8 [13539.451640] ffff88007c0dbfd8 ffff88007c0dbfd8 ffff88007c076dd0 ffff88007fd16860 [13539.453346] ffff88007c780030 ffff88007c076dd0 ffff88007c780000 ffff88007fd16840 [13539.454990] Call Trace: [13539.456219] [] schedule+0x29/0x70 [13539.457617] [] worker_thread+0x1d9/0x410 [13539.459141] [] ? rescuer_thread+0x460/0x460 [13539.460602] [] kthread+0xcf/0xe0 [13539.462070] [] ? kthread_create_on_node+0x140/0x140 [13539.463633] [] ret_from_fork+0x58/0x90 [13539.465078] [] ? kthread_create_on_node+0x140/0x140 [13539.466649] kdevtmpfs S 0000000000000000 0 17 2 0x00000000 [13539.468308] ffff88007c147e70 0000000000000046 ffff88007c128fb0 ffff88007c147fd8 [13539.469972] ffff88007c147fd8 ffff88007c147fd8 ffff88007c128fb0 0000000000000000 [13539.471679] 0000000000000000 ffff88007c128fb0 0000000000000000 0000000000000000 [13539.473369] Call Trace: [13539.474582] [] schedule+0x29/0x70 [13539.476063] [] devtmpfsd+0x168/0x180 [13539.477548] [] ? handle_create.isra.2+0x220/0x220 [13539.479127] [] kthread+0xcf/0xe0 [13539.480524] [] ? kthread_create_on_node+0x140/0x140 [13539.482121] [] ret_from_fork+0x58/0x90 [13539.483629] [] ? kthread_create_on_node+0x140/0x140 [13539.485194] netns S ffff88007c780300 0 18 2 0x00000000 [13539.486794] ffff88007c187e08 0000000000000046 ffff88007c129f60 ffff88007c187fd8 [13539.488519] ffff88007c187fd8 ffff88007c187fd8 ffff88007c129f60 ffff88007c129f60 [13539.490220] ffff88007c780300 ffff88007c0ff080 ffff88007c780330 ffff88007c780300 [13539.491858] Call Trace: [13539.493090] [] schedule+0x29/0x70 [13539.494575] [] rescuer_thread+0x359/0x460 [13539.496079] [] ? __schedule+0x3b0/0x990 [13539.497514] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.499111] [] kthread+0xcf/0xe0 [13539.500557] [] ? kthread_create_on_node+0x140/0x140 [13539.502090] [] ret_from_fork+0x58/0x90 [13539.503585] [] ? kthread_create_on_node+0x140/0x140 [13539.505092] khungtaskd S 000000000000f888 0 19 2 0x00000000 [13539.506733] ffff88007c1dfd90 0000000000000046 ffff88007c12af10 ffff88007c1dffd8 [13539.508430] ffff88007c1dffd8 ffff88007c1dffd8 ffff88007c12af10 ffff88007c0dc000 [13539.510106] ffff88007c1dfdc0 0000000100ca553b ffff88007c0dc000 000000000000f888 [13539.511723] Call Trace: [13539.512966] [] schedule+0x29/0x70 [13539.514361] [] schedule_timeout+0x175/0x2d0 [13539.515907] [] ? internal_add_timer+0x70/0x70 [13539.517379] [] schedule_timeout_interruptible+0x1e/0x20 [13539.519015] [] watchdog+0x4c/0x2c0 [13539.520469] [] ? reset_hung_task_detector+0x20/0x20 [13539.521991] [] kthread+0xcf/0xe0 [13539.523421] [] ? kthread_create_on_node+0x140/0x140 [13539.524941] [] ret_from_fork+0x58/0x90 [13539.526448] [] ? kthread_create_on_node+0x140/0x140 [13539.528002] writeback S ffff88007c1c0280 0 20 2 0x00000000 [13539.529602] ffff88007c1e3e08 0000000000000046 ffff88007c12bec0 ffff88007c1e3fd8 [13539.531336] ffff88007c1e3fd8 ffff88007c1e3fd8 ffff88007c12bec0 ffff88007c12bec0 [13539.532999] ffff88007c1c0280 ffff88007c1da880 ffff88007c1c02b0 ffff88007c1c0280 [13539.534707] Call Trace: [13539.535941] [] schedule+0x29/0x70 [13539.537414] [] rescuer_thread+0x359/0x460 [13539.538869] [] ? __schedule+0x3b0/0x990 [13539.540385] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.541891] [] kthread+0xcf/0xe0 [13539.543348] [] ? kthread_create_on_node+0x140/0x140 [13539.544859] [] ret_from_fork+0x58/0x90 [13539.546348] [] ? kthread_create_on_node+0x140/0x140 [13539.547898] kintegrityd S ffff88007c1c0380 0 21 2 0x00000000 [13539.549512] ffff88007c1ebe08 0000000000000046 ffff88007c12ce70 ffff88007c1ebfd8 [13539.551227] ffff88007c1ebfd8 ffff88007c1ebfd8 ffff88007c12ce70 ffff88007c12ce70 [13539.552878] ffff88007c1c0380 ffff88007c1daa80 ffff88007c1c03b0 ffff88007c1c0380 [13539.554594] Call Trace: [13539.555803] [] schedule+0x29/0x70 [13539.557282] [] rescuer_thread+0x359/0x460 [13539.558737] [] ? __schedule+0x3b0/0x990 [13539.560228] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.561804] [] kthread+0xcf/0xe0 [13539.563269] [] ? kthread_create_on_node+0x140/0x140 [13539.564774] [] ret_from_fork+0x58/0x90 [13539.566263] [] ? kthread_create_on_node+0x140/0x140 [13539.567816] bioset S ffff88007c1c0500 0 22 2 0x00000000 [13539.569431] ffff88007c1efe08 0000000000000046 ffff88007c12de20 ffff88007c1effd8 [13539.571129] ffff88007c1effd8 ffff88007c1effd8 ffff88007c12de20 ffff88007c12de20 [13539.572784] ffff88007c1c0500 ffff88007c1dac80 ffff88007c1c0530 ffff88007c1c0500 [13539.574481] Call Trace: [13539.575690] [] schedule+0x29/0x70 [13539.577165] [] rescuer_thread+0x359/0x460 [13539.578604] [] ? __schedule+0x3b0/0x990 [13539.580094] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.581618] [] kthread+0xcf/0xe0 [13539.583055] [] ? kthread_create_on_node+0x140/0x140 [13539.584597] [] ret_from_fork+0x58/0x90 [13539.586011] [] ? kthread_create_on_node+0x140/0x140 [13539.587571] kblockd S ffff88007c1c0600 0 23 2 0x00000000 [13539.589165] ffff88007c1fbe08 0000000000000046 ffff88007c12edd0 ffff88007c1fbfd8 [13539.590865] ffff88007c1fbfd8 ffff88007c1fbfd8 ffff88007c12edd0 ffff88007c12edd0 [13539.592556] ffff88007c1c0600 ffff88007c1db080 ffff88007c1c0630 ffff88007c1c0600 [13539.594227] Call Trace: [13539.595423] [] schedule+0x29/0x70 [13539.596885] [] rescuer_thread+0x359/0x460 [13539.598345] [] ? __schedule+0x3b0/0x990 [13539.599833] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.601348] [] kthread+0xcf/0xe0 [13539.602774] [] ? kthread_create_on_node+0x140/0x140 [13539.604335] [] ret_from_fork+0x58/0x90 [13539.605779] [] ? kthread_create_on_node+0x140/0x140 [13539.607344] md S ffff88007c243a80 0 24 2 0x00000000 [13539.608941] ffff88007c293e08 0000000000000046 ffff88007c128000 ffff88007c293fd8 [13539.610654] ffff88007c293fd8 ffff88007c293fd8 ffff88007c128000 ffff88007c128000 [13539.612354] ffff88007c243a80 ffff88007c28e880 ffff88007c243ab0 ffff88007c243a80 [13539.613992] Call Trace: [13539.615238] [] schedule+0x29/0x70 [13539.616628] [] rescuer_thread+0x359/0x460 [13539.618144] [] ? __schedule+0x3b0/0x990 [13539.619581] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.621137] [] kthread+0xcf/0xe0 [13539.622501] [] ? kthread_create_on_node+0x140/0x140 [13539.624070] [] ret_from_fork+0x58/0x90 [13539.625484] [] ? kthread_create_on_node+0x140/0x140 [13539.627043] kswapd0 S 0000000000000000 0 31 2 0x00000000 [13539.628644] ffff880079193e08 0000000000000046 ffff88007c36de20 ffff880079193fd8 [13539.630379] ffff880079193fd8 ffff880079193fd8 ffff88007c36de20 ffff88007ffd6000 [13539.632062] ffff88007fffc4e8 0000000000000000 ffff880079193e80 0000000000000000 [13539.633684] Call Trace: [13539.634954] [] schedule+0x29/0x70 [13539.636418] [] kswapd+0x394/0x450 [13539.637855] [] ? wake_up_atomic_t+0x30/0x30 [13539.639332] [] ? balance_pgdat+0x5e0/0x5e0 [13539.640845] [] kthread+0xcf/0xe0 [13539.642252] [] ? kthread_create_on_node+0x140/0x140 [13539.643797] [] ret_from_fork+0x58/0x90 [13539.645268] [] ? kthread_create_on_node+0x140/0x140 [13539.646755] ksmd S 0000000000000000 0 32 2 0x00000000 [13539.648375] ffff88007919be50 0000000000000046 ffff88007c36edd0 ffff88007919bfd8 [13539.650045] ffff88007919bfd8 ffff88007919bfd8 ffff88007c36edd0 ffff88007c36edd0 [13539.651678] ffff88007c36edd0 ffff88007919be88 ffff88007c36edd0 0000000000000000 [13539.653371] Call Trace: [13539.654560] [] schedule+0x29/0x70 [13539.656037] [] ksm_scan_thread+0x16a/0x240 [13539.657540] [] ? wake_up_atomic_t+0x30/0x30 [13539.658986] [] ? ksm_do_scan+0x11e0/0x11e0 [13539.660476] [] kthread+0xcf/0xe0 [13539.661879] [] ? kthread_create_on_node+0x140/0x140 [13539.663367] [] ret_from_fork+0x58/0x90 [13539.664841] [] ? kthread_create_on_node+0x140/0x140 [13539.666369] khugepaged S 000000000000f888 0 33 2 0x00000000 [13539.667944] ffff8800791dfd80 0000000000000046 ffff88007c36ce70 ffff8800791dffd8 [13539.669638] ffff8800791dffd8 ffff8800791dffd8 ffff88007c36ce70 ffff88007c0dc000 [13539.671314] ffff8800791dfdb0 0000000100ca0ee8 ffff88007c0dc000 000000000000f888 [13539.672950] Call Trace: [13539.674167] [] schedule+0x29/0x70 [13539.675558] [] schedule_timeout+0x175/0x2d0 [13539.677105] [] ? internal_add_timer+0x70/0x70 [13539.678558] [] ? prepare_to_wait+0x56/0x90 [13539.680070] [] khugepaged+0x1e3/0x480 [13539.681491] [] ? wake_up_atomic_t+0x30/0x30 [13539.682960] [] ? khugepaged_scan_mm_slot+0xc70/0xc70 [13539.684467] [] kthread+0xcf/0xe0 [13539.685870] [] ? kthread_create_on_node+0x140/0x140 [13539.687405] [] ret_from_fork+0x58/0x90 [13539.688798] [] ? kthread_create_on_node+0x140/0x140 [13539.690358] fsnotify_mark S ffff8800791e3e00 0 34 2 0x00000000 [13539.691943] ffff8800791e3e28 0000000000000046 ffff88007c36bec0 ffff8800791e3fd8 [13539.693663] ffff8800791e3fd8 ffff8800791e3fd8 ffff88007c36bec0 ffff8800791e3e58 [13539.695352] ffff8800791e3e58 ffff88007c36bec0 ffff8800793d8258 ffff8800791e3e00 [13539.696976] Call Trace: [13539.698202] [] schedule+0x29/0x70 [13539.699567] [] fsnotify_mark_destroy+0x141/0x180 [13539.701103] [] ? wake_up_atomic_t+0x30/0x30 [13539.702530] [] ? fsnotify_put_mark+0x40/0x40 [13539.704040] [] kthread+0xcf/0xe0 [13539.705429] [] ? kthread_create_on_node+0x140/0x140 [13539.706897] [] ret_from_fork+0x58/0x90 [13539.708358] [] ? kthread_create_on_node+0x140/0x140 [13539.709924] crypto S ffff88007c3da880 0 35 2 0x00000000 [13539.711562] ffff88007920be08 0000000000000046 ffff88007c36af10 ffff88007920bfd8 [13539.713254] ffff88007920bfd8 ffff88007920bfd8 ffff88007c36af10 ffff88007c36af10 [13539.714894] ffff88007c3da880 ffff8800791da680 ffff88007c3da8b0 ffff88007c3da880 [13539.716599] Call Trace: [13539.717786] [] schedule+0x29/0x70 [13539.719232] [] rescuer_thread+0x359/0x460 [13539.720709] [] ? __schedule+0x3b0/0x990 [13539.722165] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.723721] [] kthread+0xcf/0xe0 [13539.725126] [] ? kthread_create_on_node+0x140/0x140 [13539.726658] [] ret_from_fork+0x58/0x90 [13539.728122] [] ? kthread_create_on_node+0x140/0x140 [13539.729659] kthrotld S ffff88007c3dad80 0 43 2 0x00000000 [13539.731305] ffff88007923be08 0000000000000046 ffff88007921de20 ffff88007923bfd8 [13539.732945] ffff88007923bfd8 ffff88007923bfd8 ffff88007921de20 ffff88007921de20 [13539.734646] ffff88007c3dad80 ffff8800791daa80 ffff88007c3dadb0 ffff88007c3dad80 [13539.736344] Call Trace: [13539.737541] [] schedule+0x29/0x70 [13539.738992] [] rescuer_thread+0x359/0x460 [13539.740521] [] ? __schedule+0x3b0/0x990 [13539.741990] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.743549] [] kthread+0xcf/0xe0 [13539.744913] [] ? kthread_create_on_node+0x140/0x140 [13539.746462] [] ret_from_fork+0x58/0x90 [13539.747929] [] ? kthread_create_on_node+0x140/0x140 [13539.749424] kmpath_rdacd S ffff88007c3da180 0 45 2 0x00000000 [13539.751076] ffff880079273e08 0000000000000046 ffff88007921ce70 ffff880079273fd8 [13539.752779] ffff880079273fd8 ffff880079273fd8 ffff88007921ce70 ffff88007921ce70 [13539.754477] ffff88007c3da180 ffff8800791db280 ffff88007c3da1b0 ffff88007c3da180 [13539.756166] Call Trace: [13539.757371] [] schedule+0x29/0x70 [13539.758829] [] rescuer_thread+0x359/0x460 [13539.760302] [] ? __schedule+0x3b0/0x990 [13539.761799] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.763365] [] kthread+0xcf/0xe0 [13539.764777] [] ? kthread_create_on_node+0x140/0x140 [13539.766298] [] ret_from_fork+0x58/0x90 [13539.767787] [] ? kthread_create_on_node+0x140/0x140 [13539.769335] kpsmoused S ffff88007c152200 0 46 2 0x00000000 [13539.770933] ffff88007928fe08 0000000000000046 ffff88007921bec0 ffff88007928ffd8 [13539.772645] ffff88007928ffd8 ffff88007928ffd8 ffff88007921bec0 ffff88007921bec0 [13539.774348] ffff88007c152200 ffff880079288080 ffff88007c152230 ffff88007c152200 [13539.775994] Call Trace: [13539.777230] [] schedule+0x29/0x70 [13539.778681] [] rescuer_thread+0x359/0x460 [13539.780195] [] ? __schedule+0x3b0/0x990 [13539.781674] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.783241] [] kthread+0xcf/0xe0 [13539.784724] [] ? kthread_create_on_node+0x140/0x140 [13539.786252] [] ret_from_fork+0x58/0x90 [13539.787745] [] ? kthread_create_on_node+0x140/0x140 [13539.789295] ipv6_addrconf S ffff88007c152b80 0 48 2 0x00000000 [13539.790898] ffff8800792abe08 0000000000000046 ffff880079219f60 ffff8800792abfd8 [13539.792616] ffff8800792abfd8 ffff8800792abfd8 ffff880079219f60 ffff880079219f60 [13539.794329] ffff88007c152b80 ffff880079289080 ffff88007c152bb0 ffff88007c152b80 [13539.795984] Call Trace: [13539.797224] [] schedule+0x29/0x70 [13539.798684] [] rescuer_thread+0x359/0x460 [13539.800199] [] ? __schedule+0x3b0/0x990 [13539.801683] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.803258] [] kthread+0xcf/0xe0 [13539.804675] [] ? kthread_create_on_node+0x140/0x140 [13539.806252] [] ret_from_fork+0x58/0x90 [13539.807677] [] ? kthread_create_on_node+0x140/0x140 [13539.809255] deferwq S ffff880079258f00 0 67 2 0x00000000 [13539.810834] ffff8800792bfe08 0000000000000046 ffff880079308fb0 ffff8800792bffd8 [13539.812556] ffff8800792bffd8 ffff8800792bffd8 ffff880079308fb0 ffff880079308fb0 [13539.814243] ffff880079258f00 ffff8800791dbe80 ffff880079258f30 ffff880079258f00 [13539.815893] Call Trace: [13539.817134] [] schedule+0x29/0x70 [13539.818546] [] rescuer_thread+0x359/0x460 [13539.820065] [] ? __schedule+0x3b0/0x990 [13539.821557] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.823115] [] kthread+0xcf/0xe0 [13539.824496] [] ? kthread_create_on_node+0x140/0x140 [13539.826060] [] ret_from_fork+0x58/0x90 [13539.827499] [] ? kthread_create_on_node+0x140/0x140 [13539.829061] kauditd S 0000000000000000 0 102 2 0x00000000 [13539.830653] ffff88003684fe48 0000000000000046 ffff88007930de20 ffff88003684ffd8 [13539.832354] ffff88003684ffd8 ffff88003684ffd8 ffff88007930de20 0000000000000000 [13539.834012] ffff88007930de20 ffff88007930de20 ffff88007930de20 0000000000000000 [13539.835689] Call Trace: [13539.836936] [] schedule+0x29/0x70 [13539.838338] [] kauditd_thread+0x184/0x1e0 [13539.839843] [] ? wake_up_state+0x20/0x20 [13539.841301] [] ? audit_printk_skb+0x70/0x70 [13539.842800] [] kthread+0xcf/0xe0 [13539.844216] [] ? kthread_create_on_node+0x140/0x140 [13539.845702] [] ret_from_fork+0x58/0x90 [13539.847179] [] ? kthread_create_on_node+0x140/0x140 [13539.848719] ata_sff S ffff88007936c980 0 238 2 0x00000000 [13539.850339] ffff880036b8fe08 0000000000000046 ffff8800369e9f60 ffff880036b8ffd8 [13539.851970] ffff880036b8ffd8 ffff880036b8ffd8 ffff8800369e9f60 ffff8800369e9f60 [13539.853667] ffff88007936c980 ffff8800369f5280 ffff88007936c9b0 ffff88007936c980 [13539.855362] Call Trace: [13539.856557] [] schedule+0x29/0x70 [13539.858029] [] rescuer_thread+0x359/0x460 [13539.859533] [] ? __schedule+0x3b0/0x990 [13539.861017] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.862509] [] kthread+0xcf/0xe0 [13539.863945] [] ? kthread_create_on_node+0x140/0x140 [13539.865484] [] ret_from_fork+0x58/0x90 [13539.866960] [] ? kthread_create_on_node+0x140/0x140 [13539.868491] scsi_eh_0 S 0000000000000000 0 240 2 0x00000000 [13539.870075] ffff880036b8be18 0000000000000046 ffff8800792faf10 ffff880036b8bfd8 [13539.871746] ffff880036b8bfd8 ffff880036b8bfd8 ffff8800792faf10 ffff8800793c6000 [13539.873454] ffff8800792faf10 ffff8800792fb588 0000000000000000 0000000000000000 [13539.875138] Call Trace: [13539.876337] [] schedule+0x29/0x70 [13539.877798] [] scsi_error_handler+0xa3/0x8b0 [13539.879286] [] ? scsi_eh_get_sense+0x250/0x250 [13539.880758] [] kthread+0xcf/0xe0 [13539.882134] [] ? kthread_create_on_node+0x140/0x140 [13539.883629] [] ret_from_fork+0x58/0x90 [13539.885052] [] ? kthread_create_on_node+0x140/0x140 [13539.886549] scsi_tmf_0 S ffff880036a77780 0 241 2 0x00000000 [13539.888140] ffff88003643be08 0000000000000046 ffff8800792fce70 ffff88003643bfd8 [13539.889806] ffff88003643bfd8 ffff88003643bfd8 ffff8800792fce70 ffff8800792fce70 [13539.891505] ffff880036a77780 ffff880036acd680 ffff880036a777b0 ffff880036a77780 [13539.893183] Call Trace: [13539.894457] [] schedule+0x29/0x70 [13539.895852] [] rescuer_thread+0x359/0x460 [13539.897306] [] ? __schedule+0x3b0/0x990 [13539.898734] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.900248] [] kthread+0xcf/0xe0 [13539.901620] [] ? kthread_create_on_node+0x140/0x140 [13539.903128] [] ret_from_fork+0x58/0x90 [13539.904545] [] ? kthread_create_on_node+0x140/0x140 [13539.906070] scsi_eh_1 S 0000000000000000 0 242 2 0x00000000 [13539.907644] ffff88003647fe18 0000000000000046 ffff8800792f8000 ffff88003647ffd8 [13539.909292] ffff88003647ffd8 ffff88003647ffd8 ffff8800792f8000 ffff8800793c6800 [13539.910927] ffff8800792f8000 ffff8800792f8678 0000000000000000 0000000000000000 [13539.912579] Call Trace: [13539.913798] [] schedule+0x29/0x70 [13539.915221] [] scsi_error_handler+0xa3/0x8b0 [13539.916715] [] ? scsi_eh_get_sense+0x250/0x250 [13539.918214] [] kthread+0xcf/0xe0 [13539.919617] [] ? kthread_create_on_node+0x140/0x140 [13539.921139] [] ret_from_fork+0x58/0x90 [13539.922561] [] ? kthread_create_on_node+0x140/0x140 [13539.924111] scsi_tmf_1 S ffff880036a77980 0 243 2 0x00000000 [13539.925721] ffff8800364a3e08 0000000000000046 ffff8800792f8fb0 ffff8800364a3fd8 [13539.927442] ffff8800364a3fd8 ffff8800364a3fd8 ffff8800792f8fb0 ffff8800792f8fb0 [13539.929130] ffff880036a77980 ffff880036acdc80 ffff880036a779b0 ffff880036a77980 [13539.930772] Call Trace: [13539.932041] [] schedule+0x29/0x70 [13539.933545] [] rescuer_thread+0x359/0x460 [13539.935022] [] ? __schedule+0x3b0/0x990 [13539.936542] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.938072] [] kthread+0xcf/0xe0 [13539.939541] [] ? kthread_create_on_node+0x140/0x140 [13539.941073] [] ret_from_fork+0x58/0x90 [13539.942563] [] ? kthread_create_on_node+0x140/0x140 [13539.944146] ttm_swap S ffff8800364e9e00 0 252 2 0x00000000 [13539.945742] ffff880036553e08 0000000000000046 ffff880036954e70 ffff880036553fd8 [13539.947466] ffff880036553fd8 ffff880036553fd8 ffff880036954e70 ffff880036954e70 [13539.949172] ffff8800364e9e00 ffff880036b71a80 ffff8800364e9e30 ffff8800364e9e00 [13539.950827] Call Trace: [13539.952078] [] schedule+0x29/0x70 [13539.953552] [] rescuer_thread+0x359/0x460 [13539.955015] [] ? __schedule+0x3b0/0x990 [13539.956533] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.958063] [] kthread+0xcf/0xe0 [13539.959511] [] ? kthread_create_on_node+0x140/0x140 [13539.961032] [] ret_from_fork+0x58/0x90 [13539.962511] [] ? kthread_create_on_node+0x140/0x140 [13539.964091] jbd2/vda1-8 S ffff8800361d2000 0 273 2 0x00000000 [13539.965672] ffff8800360c7e30 0000000000000046 ffff880036951f60 ffff8800360c7fd8 [13539.967396] ffff8800360c7fd8 ffff8800360c7fd8 ffff880036951f60 ffff8800361d2028 [13539.969080] ffff8800361d23e8 ffff8800361d20a8 ffff8800360c7e80 ffff8800361d2000 [13539.970748] Call Trace: [13539.971993] [] schedule+0x29/0x70 [13539.973509] [] kjournald2+0x203/0x260 [jbd2] [13539.975010] [] ? wake_up_atomic_t+0x30/0x30 [13539.976545] [] ? commit_timeout+0x10/0x10 [jbd2] [13539.978097] [] kthread+0xcf/0xe0 [13539.979544] [] ? kthread_create_on_node+0x140/0x140 [13539.981108] [] ret_from_fork+0x58/0x90 [13539.982597] [] ? kthread_create_on_node+0x140/0x140 [13539.984113] ext4-rsv-conver S ffff880079325b00 0 274 2 0x00000000 [13539.985744] ffff880036157e08 0000000000000046 ffff880036952f10 ffff880036157fd8 [13539.987631] ffff880036157fd8 ffff880036157fd8 ffff880036952f10 ffff880036952f10 [13539.989295] ffff880079325b00 ffff880036acd080 ffff880079325b30 ffff880079325b00 [13539.990933] Call Trace: [13539.992154] [] schedule+0x29/0x70 [13539.993554] [] rescuer_thread+0x359/0x460 [13539.995018] [] ? __schedule+0x3b0/0x990 [13539.996460] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13539.997969] [] kthread+0xcf/0xe0 [13539.999338] [] ? kthread_create_on_node+0x140/0x140 [13540.000846] [] ret_from_fork+0x58/0x90 [13540.002273] [] ? kthread_create_on_node+0x140/0x140 [13540.003767] systemd-journal R running task 0 356 1 0x00000000 [13540.005364] ffff8800792e7db8 0000000000000086 ffff880036924e70 ffff8800792e7fd8 [13540.007009] ffff8800792e7fd8 ffff8800792e7fd8 ffff880036924e70 0000000000000000 [13540.008658] 0000000000000000 ffff88007a3c0de0 ffff880036924e70 0000000000000000 [13540.010318] Call Trace: [13540.011536] [] schedule+0x29/0x70 [13540.012945] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.014513] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13540.016124] [] schedule_hrtimeout_range+0x13/0x20 [13540.017654] [] ep_poll+0x23e/0x360 [13540.019064] [] ? do_truncate+0x88/0xc0 [13540.020486] [] ? __sb_start_write+0x58/0x110 [13540.021968] [] ? wake_up_state+0x20/0x20 [13540.023392] [] SyS_epoll_wait+0xed/0x120 [13540.024813] [] system_call_fastpath+0x16/0x1b [13540.026296] rpciod S ffff880036a72500 0 367 2 0x00000000 [13540.027881] ffff88007a3e3e08 0000000000000046 ffff880036955e20 ffff88007a3e3fd8 [13540.029545] ffff88007a3e3fd8 ffff88007a3e3fd8 ffff880036955e20 ffff880036955e20 [13540.031200] ffff880036a72500 ffff88003685ea80 ffff880036a72530 ffff880036a72500 [13540.032830] Call Trace: [13540.034040] [] schedule+0x29/0x70 [13540.035437] [] rescuer_thread+0x359/0x460 [13540.036880] [] ? __schedule+0x3b0/0x990 [13540.038299] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.039818] [] kthread+0xcf/0xe0 [13540.041207] [] ? kthread_create_on_node+0x140/0x140 [13540.042701] [] ret_from_fork+0x58/0x90 [13540.044120] [] ? kthread_create_on_node+0x140/0x140 [13540.045625] lvmetad S 0000000000000004 0 377 1 0x00000000 [13540.047219] ffff880079dab8e8 0000000000000086 ffff8800369eedd0 ffff880079dabfd8 [13540.048862] ffff880079dabfd8 ffff880079dabfd8 ffff8800369eedd0 0000000000000000 [13540.050506] ffff880079dabad0 00000000fffffffc 0000000000000004 0000000000000004 [13540.052163] Call Trace: [13540.053367] [] schedule+0x29/0x70 [13540.054769] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.056322] [] ? add_wait_queue+0x3c/0x50 [13540.057754] [] ? __pollwait+0x7f/0xf0 [13540.059178] [] schedule_hrtimeout_range+0x13/0x20 [13540.060672] [] poll_schedule_timeout+0x55/0xb0 [13540.062150] [] do_select+0x6d1/0x7c0 [13540.063545] [] ? poll_select_copy_remaining+0x150/0x150 [13540.065084] [] ? radix_tree_insert+0x41/0xf0 [13540.066554] [] ? __inc_zone_page_state+0x32/0x60 [13540.068040] [] ? shmem_add_to_page_cache.isra.19+0xf0/0x170 [13540.069604] [] ? zone_statistics+0x89/0xa0 [13540.071060] [] ? zone_statistics+0x89/0xa0 [13540.072494] [] ? list_del+0xd/0x30 [13540.073870] [] ? __rmqueue+0x8a/0x460 [13540.075259] [] ? get_partial_node.isra.49+0x1df/0x230 [13540.076734] [] ? zone_statistics+0x89/0xa0 [13540.078155] [] ? ___slab_alloc+0x209/0x4f0 [13540.079560] [] ? ___slab_alloc+0x209/0x4f0 [13540.080952] [] ? audit_alloc+0xe3/0x180 [13540.082315] [] ? kvm_clock_read+0x1f/0x30 [13540.083688] [] ? sched_clock+0x9/0x10 [13540.085036] [] core_sys_select+0x1db/0x300 [13540.086412] [] ? x2apic_send_IPI_mask+0x13/0x20 [13540.087816] [] ? native_smp_send_reschedule+0x4c/0x70 [13540.089263] [] ? resched_task+0xb8/0xd0 [13540.090610] [] ? wake_up_new_task+0x10c/0x170 [13540.092012] [] ? do_fork+0xe8/0x2c0 [13540.093338] [] SyS_select+0xba/0x110 [13540.094671] [] ? SyS_clone+0x16/0x20 [13540.095992] [] system_call_fastpath+0x16/0x1b [13540.097356] systemd-udevd S ffff88007b67fea8 0 383 1 0x00000000 [13540.098839] ffff88007b67fdb8 0000000000000082 ffff8800369b6dd0 ffff88007b67ffd8 [13540.100411] ffff88007b67ffd8 ffff88007b67ffd8 ffff8800369b6dd0 ffff88007b67fea8 [13540.101955] 00000000002dc6bf 0000000000000000 ffff8800369b6dd0 ffff88007b67fea8 [13540.103504] Call Trace: [13540.104614] [] schedule+0x29/0x70 [13540.105924] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.107386] [] ? hrtimer_get_res+0x50/0x50 [13540.108754] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.110319] [] schedule_hrtimeout_range+0x13/0x20 [13540.111781] [] ep_poll+0x23e/0x360 [13540.113107] [] ? do_unlinkat+0xf1/0x2b0 [13540.114447] [] ? wake_up_state+0x20/0x20 [13540.115782] [] SyS_epoll_wait+0xed/0x120 [13540.117134] [] system_call_fastpath+0x16/0x1b [13540.118503] vballoon S 0000000000000000 0 401 2 0x00000000 [13540.119995] ffff880079987e20 0000000000000046 ffff8800369b0fb0 ffff880079987fd8 [13540.121538] ffff880079987fd8 ffff880079987fd8 ffff8800369b0fb0 ffff880079293800 [13540.123093] ffff880079987e64 ffff880079293820 ffff8800369b0fb0 0000000000000000 [13540.124617] Call Trace: [13540.125716] [] schedule+0x29/0x70 [13540.127024] [] balloon+0x1ea/0x2ac [virtio_balloon] [13540.128443] [] ? wake_up_atomic_t+0x30/0x30 [13540.129819] [] ? update_balloon_stats+0x130/0x130 [virtio_balloon] [13540.131348] [] kthread+0xcf/0xe0 [13540.132646] [] ? kthread_create_on_node+0x140/0x140 [13540.134092] [] ret_from_fork+0x58/0x90 [13540.135432] [] ? kthread_create_on_node+0x140/0x140 [13540.136856] kworker/u5:0 S ffff88007b131c00 0 477 2 0x00000000 [13540.138399] ffff880079977e50 0000000000000046 ffff880079d65e20 ffff880079977fd8 [13540.140022] ffff880079977fd8 ffff880079977fd8 ffff880079d65e20 ffff88007b131c20 [13540.141610] ffff880036095030 ffff880079d65e20 ffff880036095000 ffff88007b131c00 [13540.143216] Call Trace: [13540.144368] [] schedule+0x29/0x70 [13540.145718] [] worker_thread+0x1d9/0x410 [13540.147142] [] ? rescuer_thread+0x460/0x460 [13540.148580] [] kthread+0xcf/0xe0 [13540.149945] [] ? kthread_create_on_node+0x140/0x140 [13540.151435] [] ret_from_fork+0x58/0x90 [13540.152839] [] ? kthread_create_on_node+0x140/0x140 [13540.154361] ib-comp-wq S ffff880036095280 0 478 2 0x00000000 [13540.155942] ffff880079393e08 0000000000000046 ffff880079d62f10 ffff880079393fd8 [13540.157600] ffff880079393fd8 ffff880079393fd8 ffff880079d62f10 ffff880079d62f10 [13540.159247] ffff880036095280 ffff8800364bb280 ffff8800360952b0 ffff880036095280 [13540.160878] Call Trace: [13540.162097] [] schedule+0x29/0x70 [13540.163512] [] rescuer_thread+0x359/0x460 [13540.164978] [] ? __schedule+0x3b0/0x990 [13540.166415] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.167930] [] kthread+0xcf/0xe0 [13540.169329] [] ? kthread_create_on_node+0x140/0x140 [13540.170844] [] ret_from_fork+0x58/0x90 [13540.172293] [] ? kthread_create_on_node+0x140/0x140 [13540.173801] ib_addr S ffff880036095d00 0 480 2 0x00000000 [13540.175417] ffff8800368f7e08 0000000000000046 ffff880079d64e70 ffff8800368f7fd8 [13540.177078] ffff8800368f7fd8 ffff8800368f7fd8 ffff880079d64e70 ffff880079d64e70 [13540.178727] ffff880036095d00 ffff8800364bbe80 ffff880036095d30 ffff880036095d00 [13540.180369] Call Trace: [13540.181575] [] schedule+0x29/0x70 [13540.182984] [] rescuer_thread+0x359/0x460 [13540.184453] [] ? __schedule+0x3b0/0x990 [13540.185899] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.187435] [] kthread+0xcf/0xe0 [13540.188836] [] ? kthread_create_on_node+0x140/0x140 [13540.190355] [] ret_from_fork+0x58/0x90 [13540.191794] [] ? kthread_create_on_node+0x140/0x140 [13540.193338] ib_mcast S ffff880036095e80 0 481 2 0x00000000 [13540.194938] ffff88007aec3e08 0000000000000046 ffff880079d63ec0 ffff88007aec3fd8 [13540.196613] ffff88007aec3fd8 ffff88007aec3fd8 ffff880079d63ec0 ffff880079d63ec0 [13540.198279] ffff880036095e80 ffff88003608fe80 ffff880036095eb0 ffff880036095e80 [13540.199937] Call Trace: [13540.201150] [] schedule+0x29/0x70 [13540.202549] [] rescuer_thread+0x359/0x460 [13540.204010] [] ? __schedule+0x3b0/0x990 [13540.205449] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.206965] [] kthread+0xcf/0xe0 [13540.208351] [] ? kthread_create_on_node+0x140/0x140 [13540.209851] [] ret_from_fork+0x58/0x90 [13540.211268] [] ? kthread_create_on_node+0x140/0x140 [13540.212780] ib_nl_sa_wq S ffff880036095d80 0 482 2 0x00000000 [13540.214376] ffff880079e27e08 0000000000000046 ffff880079d60000 ffff880079e27fd8 [13540.216033] ffff880079e27fd8 ffff880079e27fd8 ffff880079d60000 ffff880079d60000 [13540.217672] ffff880036095d80 ffff88003608fc80 ffff880036095db0 ffff880036095d80 [13540.219331] Call Trace: [13540.220544] [] schedule+0x29/0x70 [13540.221954] [] rescuer_thread+0x359/0x460 [13540.223412] [] ? __schedule+0x3b0/0x990 [13540.224875] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.226393] [] kthread+0xcf/0xe0 [13540.227767] [] ? kthread_create_on_node+0x140/0x140 [13540.229288] [] ret_from_fork+0x58/0x90 [13540.230709] [] ? kthread_create_on_node+0x140/0x140 [13540.232221] iw_cm_wq S ffff880079387780 0 496 2 0x00000000 [13540.233801] ffff88003613fe08 0000000000000046 ffff8800369e8fb0 ffff88003613ffd8 [13540.235468] ffff88003613ffd8 ffff88003613ffd8 ffff8800369e8fb0 ffff8800369e8fb0 [13540.237126] ffff880079387780 ffff880036845680 ffff8800793877b0 ffff880079387780 [13540.238773] Call Trace: [13540.239990] [] schedule+0x29/0x70 [13540.241400] [] rescuer_thread+0x359/0x460 [13540.242865] [] ? __schedule+0x3b0/0x990 [13540.244312] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.245816] [] kthread+0xcf/0xe0 [13540.247200] [] ? kthread_create_on_node+0x140/0x140 [13540.248710] [] ret_from_fork+0x58/0x90 [13540.250138] [] ? kthread_create_on_node+0x140/0x140 [13540.251643] auditd S ffff88007a8a7ea8 0 499 1 0x00000000 [13540.253232] ffff88007a8a7db8 0000000000000086 ffff8800369ece70 ffff88007a8a7fd8 [13540.254883] ffff88007a8a7fd8 ffff88007a8a7fd8 ffff8800369ece70 ffff88007a8a7ea8 [13540.256534] 00000000038f9b17 0000000000000000 ffff8800369ece70 ffff88007a8a7ea8 [13540.258192] Call Trace: [13540.259401] [] schedule+0x29/0x70 [13540.260804] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.262374] [] ? hrtimer_get_res+0x50/0x50 [13540.263835] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.265417] [] schedule_hrtimeout_range+0x13/0x20 [13540.266918] [] ep_poll+0x23e/0x360 [13540.268322] [] ? wake_up_state+0x20/0x20 [13540.269775] [] SyS_epoll_wait+0xed/0x120 [13540.271209] [] system_call_fastpath+0x16/0x1b [13540.272665] auditd S ffffc90000383400 0 505 1 0x00000000 [13540.274261] ffff88007afffcd0 0000000000000086 ffff880036920000 ffff88007affffd8 [13540.275920] ffff88007affffd8 ffff88007affffd8 ffff880036920000 ffff880036920000 [13540.277571] ffff880036920000 0000000000000000 ffff88007afffde0 ffffc90000383400 [13540.279226] Call Trace: [13540.280443] [] schedule+0x29/0x70 [13540.281868] [] futex_wait_queue_me+0xc4/0x120 [13540.283374] [] futex_wait+0x179/0x280 [13540.284799] [] ? set_next_entity+0x3c/0xe0 [13540.286251] [] ? __switch_to+0x15c/0x4c0 [13540.287680] [] do_futex+0xfe/0x5b0 [13540.289072] [] ? wake_up_atomic_t+0x30/0x30 [13540.290516] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [13540.292069] [] SyS_futex+0x80/0x180 [13540.293467] [] ? do_fsync+0x76/0xa0 [13540.294860] [] system_call_fastpath+0x16/0x1b [13540.296318] ib_cm S ffff880036095700 0 515 2 0x00000080 [13540.297892] ffff880079dc3e08 0000000000000046 ffff880036950000 ffff880079dc3fd8 [13540.299547] ffff880079dc3fd8 ffff880079dc3fd8 ffff880036950000 ffff880036950000 [13540.301192] ffff880036095700 ffff88003608f080 ffff880036095730 ffff880036095700 [13540.302947] Call Trace: [13540.304143] [] schedule+0x29/0x70 [13540.305520] [] rescuer_thread+0x359/0x460 [13540.306955] [] ? __schedule+0x3b0/0x990 [13540.308378] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.309865] [] kthread+0xcf/0xe0 [13540.311236] [] ? kthread_create_on_node+0x140/0x140 [13540.312732] [] ret_from_fork+0x58/0x90 [13540.314143] [] ? kthread_create_on_node+0x140/0x140 [13540.315641] rdma_cm S ffff880079387d80 0 523 2 0x00000080 [13540.317247] ffff88007998fe08 0000000000000046 ffff880036953ec0 ffff88007998ffd8 [13540.318882] ffff88007998ffd8 ffff88007998ffd8 ffff880036953ec0 ffff880036953ec0 [13540.320535] ffff880079387d80 ffff880036b04280 ffff880079387db0 ffff880079387d80 [13540.322181] Call Trace: [13540.323398] [] schedule+0x29/0x70 [13540.324774] [] rescuer_thread+0x359/0x460 [13540.326216] [] ? __schedule+0x3b0/0x990 [13540.327652] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.329152] [] kthread+0xcf/0xe0 [13540.330528] [] ? kthread_create_on_node+0x140/0x140 [13540.332025] [] ret_from_fork+0x58/0x90 [13540.333421] [] ? kthread_create_on_node+0x140/0x140 [13540.334918] dbus-daemon S 0000000000000000 0 538 1 0x00000080 [13540.336491] ffff8800799b7db8 0000000000000082 ffff880079950000 ffff8800799b7fd8 [13540.338134] ffff8800799b7fd8 ffff8800799b7fd8 ffff880079950000 0000000000000000 [13540.339750] 0000000000000000 ffff880036792720 ffff880079950000 0000000000000000 [13540.341394] Call Trace: [13540.342591] [] schedule+0x29/0x70 [13540.343974] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.345520] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13540.347092] [] schedule_hrtimeout_range+0x13/0x20 [13540.348629] [] ep_poll+0x23e/0x360 [13540.350010] [] ? __fput+0x183/0x260 [13540.351398] [] ? wake_up_state+0x20/0x20 [13540.352809] [] SyS_epoll_wait+0xed/0x120 [13540.354226] [] system_call_fastpath+0x16/0x1b [13540.355679] gssproxy S ffff88003629fea8 0 541 1 0x00000080 [13540.357263] ffff88003629fdb8 0000000000000082 ffff880079fc6dd0 ffff88003629ffd8 [13540.358908] ffff88003629ffd8 ffff88003629ffd8 ffff880079fc6dd0 ffff88003629fea8 [13540.360550] 0000000001c9c37f 0000000000000000 ffff880079fc6dd0 ffff88003629fea8 [13540.362195] Call Trace: [13540.363394] [] schedule+0x29/0x70 [13540.364782] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.366342] [] ? hrtimer_get_res+0x50/0x50 [13540.367782] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.369351] [] schedule_hrtimeout_range+0x13/0x20 [13540.370839] [] ep_poll+0x23e/0x360 [13540.372250] [] ? unix_poll+0x29/0xc0 [13540.373658] [] ? wake_up_state+0x20/0x20 [13540.375084] [] SyS_epoll_wait+0xed/0x120 [13540.376498] [] system_call_fastpath+0x16/0x1b [13540.377961] gssproxy S ffffc9000037c800 0 544 1 0x00000080 [13540.379540] ffff880079143cd0 0000000000000082 ffff880079953ec0 ffff880079143fd8 [13540.381202] ffff880079143fd8 ffff880079143fd8 ffff880079953ec0 ffff880079953ec0 [13540.383297] ffff880079953ec0 0000000000000000 ffff880079143de0 ffffc9000037c800 [13540.385838] Call Trace: [13540.387464] [] schedule+0x29/0x70 [13540.389171] [] futex_wait_queue_me+0xc4/0x120 [13540.390642] [] futex_wait+0x179/0x280 [13540.392049] [] ? dequeue_entity+0x11c/0x5d0 [13540.393506] [] ? kvm_clock_read+0x1f/0x30 [13540.394917] [] ? dequeue_task_fair+0x41e/0x660 [13540.396378] [] do_futex+0xfe/0x5b0 [13540.397738] [] ? schedule+0x29/0x70 [13540.399125] [] SyS_futex+0x80/0x180 [13540.400485] [] system_call_fastpath+0x16/0x1b [13540.401924] gssproxy S ffffc9000037f840 0 545 1 0x00000080 [13540.403492] ffff88007b12bcd0 0000000000000082 ffff880079954e70 ffff88007b12bfd8 [13540.405122] ffff88007b12bfd8 ffff88007b12bfd8 ffff880079954e70 ffff880079954e70 [13540.406722] ffff880079954e70 0000000000000000 ffff88007b12bde0 ffffc9000037f840 [13540.408342] Call Trace: [13540.409501] [] schedule+0x29/0x70 [13540.410867] [] futex_wait_queue_me+0xc4/0x120 [13540.412318] [] futex_wait+0x179/0x280 [13540.413701] [] do_futex+0xfe/0x5b0 [13540.415051] [] SyS_futex+0x80/0x180 [13540.416412] [] system_call_fastpath+0x16/0x1b [13540.417818] gssproxy S ffffc90000382100 0 546 1 0x00000080 [13540.419368] ffff88007b12fcd0 0000000000000082 ffff880079955e20 ffff88007b12ffd8 [13540.420968] ffff88007b12ffd8 ffff88007b12ffd8 ffff880079955e20 ffff880079955e20 [13540.422598] ffff880079955e20 0000000000000000 ffff88007b12fde0 ffffc90000382100 [13540.424257] Call Trace: [13540.425437] [] schedule+0x29/0x70 [13540.426799] [] futex_wait_queue_me+0xc4/0x120 [13540.428252] [] futex_wait+0x179/0x280 [13540.429617] [] do_futex+0xfe/0x5b0 [13540.430976] [] SyS_futex+0x80/0x180 [13540.432332] [] system_call_fastpath+0x16/0x1b [13540.433743] gssproxy S ffffc9000037e880 0 547 1 0x00000080 [13540.435283] ffff88007b9a3cd0 0000000000000082 ffff880079956dd0 ffff88007b9a3fd8 [13540.436877] ffff88007b9a3fd8 ffff88007b9a3fd8 ffff880079956dd0 ffff880079956dd0 [13540.438479] ffff880079956dd0 0000000000000000 ffff88007b9a3de0 ffffc9000037e880 [13540.440067] Call Trace: [13540.441213] [] schedule+0x29/0x70 [13540.442569] [] futex_wait_queue_me+0xc4/0x120 [13540.443995] [] futex_wait+0x179/0x280 [13540.445388] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [13540.446873] [] ? unlock_page+0x2b/0x30 [13540.448265] [] do_futex+0xfe/0x5b0 [13540.449606] [] ? __do_page_fault+0x171/0x450 [13540.451024] [] SyS_futex+0x80/0x180 [13540.452364] [] ? do_page_fault+0x35/0x90 [13540.453748] [] system_call_fastpath+0x16/0x1b [13540.455145] gssproxy S ffffc90000381a00 0 548 1 0x00000080 [13540.456668] ffff88007b9a7cd0 0000000000000082 ffff880079952f10 ffff88007b9a7fd8 [13540.458256] ffff88007b9a7fd8 ffff88007b9a7fd8 ffff880079952f10 ffff880079952f10 [13540.459837] ffff880079952f10 0000000000000000 ffff88007b9a7de0 ffffc90000381a00 [13540.461428] Call Trace: [13540.462591] [] schedule+0x29/0x70 [13540.463957] [] futex_wait_queue_me+0xc4/0x120 [13540.465393] [] futex_wait+0x179/0x280 [13540.466780] [] ? __sb_end_write+0x31/0x60 [13540.468195] [] ? pipe_write+0x46a/0x5b0 [13540.469588] [] do_futex+0xfe/0x5b0 [13540.470946] [] SyS_futex+0x80/0x180 [13540.472283] [] system_call_fastpath+0x16/0x1b [13540.473689] NetworkManager S ffff88007917bbdc 0 551 1 0x00000080 [13540.475251] ffff88007917ba30 0000000000000082 ffff8800369e8000 ffff88007917bfd8 [13540.476839] ffff88007917bfd8 ffff88007917bfd8 ffff8800369e8000 ffff88007917bb88 [13540.478424] 0000000005f5e100 0000000000000000 0000000000000000 ffff88007917bbdc [13540.480025] Call Trace: [13540.481182] [] schedule+0x29/0x70 [13540.482539] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.484072] [] ? hrtimer_get_res+0x50/0x50 [13540.485498] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.487047] [] schedule_hrtimeout_range+0x13/0x20 [13540.488510] [] poll_schedule_timeout+0x55/0xb0 [13540.489974] [] do_sys_poll+0x4cd/0x580 [13540.491378] [] ? sock_recvmsg+0xbf/0x100 [13540.492778] [] ? update_curr+0x104/0x190 [13540.494167] [] ? __enqueue_entity+0x78/0x80 [13540.495578] [] ? poll_select_copy_remaining+0x150/0x150 [13540.497077] [] ? poll_select_copy_remaining+0x150/0x150 [13540.498560] [] ? poll_select_copy_remaining+0x150/0x150 [13540.500040] [] ? poll_select_copy_remaining+0x150/0x150 [13540.501492] [] ? poll_select_copy_remaining+0x150/0x150 [13540.502932] [] ? poll_select_copy_remaining+0x150/0x150 [13540.504335] [] ? poll_select_copy_remaining+0x150/0x150 [13540.505721] [] ? poll_select_copy_remaining+0x150/0x150 [13540.507122] [] ? kvm_clock_get_cycles+0x1f/0x30 [13540.508436] [] ? ktime_get_ts64+0x4c/0xf0 [13540.509703] [] SyS_poll+0x74/0x110 [13540.510928] [] system_call_fastpath+0x16/0x1b [13540.512238] gmain S ffff88007abbbbac 0 573 1 0x00000080 [13540.513656] ffff88007abbba30 0000000000000082 ffff880079fc0fb0 ffff88007abbbfd8 [13540.515150] ffff88007abbbfd8 ffff88007abbbfd8 ffff880079fc0fb0 0000000000000000 [13540.516639] ffff88007abbbc90 00000000fffffffc 0000000000000000 ffff88007abbbbac [13540.518137] Call Trace: [13540.519197] [] schedule+0x29/0x70 [13540.520459] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.521895] [] ? add_wait_queue+0x3c/0x50 [13540.523204] [] schedule_hrtimeout_range+0x13/0x20 [13540.524576] [] poll_schedule_timeout+0x55/0xb0 [13540.525915] [] do_sys_poll+0x4cd/0x580 [13540.527203] [] ? kvm_clock_read+0x1f/0x30 [13540.528487] [] ? dequeue_task_fair+0x41e/0x660 [13540.529813] [] ? set_next_entity+0x3c/0xe0 [13540.531109] [] ? __switch_to+0x15c/0x4c0 [13540.532386] [] ? __schedule+0x3b0/0x990 [13540.533658] [] ? poll_select_copy_remaining+0x150/0x150 [13540.535055] [] ? poll_select_copy_remaining+0x150/0x150 [13540.536436] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [13540.537809] [] ? get_futex_key+0x1c8/0x2c0 [13540.539110] [] ? futex_wake+0x80/0x160 [13540.540363] [] ? do_futex+0x122/0x5b0 [13540.541598] [] SyS_poll+0x74/0x110 [13540.542827] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.544146] [] system_call_fastpath+0x16/0x1b [13540.545419] gdbus S ffff880079843bb4 0 579 1 0x00000080 [13540.546814] ffff880079843a30 0000000000000082 ffff880079fbaf10 ffff880079843fd8 [13540.548303] ffff880079843fd8 ffff880079843fd8 ffff880079fbaf10 0000000000000000 [13540.549754] ffff880079843c90 00000000fffffffc 0000000000000000 ffff880079843bb4 [13540.551223] Call Trace: [13540.552249] [] schedule+0x29/0x70 [13540.553471] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.554857] [] ? add_wait_queue+0x3c/0x50 [13540.556150] [] ? __pollwait+0x7f/0xf0 [13540.557412] [] ? add_wait_queue+0x3c/0x50 [13540.558698] [] schedule_hrtimeout_range+0x13/0x20 [13540.560046] [] poll_schedule_timeout+0x55/0xb0 [13540.561384] [] do_sys_poll+0x4cd/0x580 [13540.562693] [] ? unix_stream_recvmsg+0x54/0x70 [13540.564066] [] ? unix_state_double_unlock+0x50/0x50 [13540.565474] [] ? sock_recvmsg+0xbf/0x100 [13540.566857] [] ? update_curr+0x104/0x190 [13540.568687] [] ? __enqueue_entity+0x78/0x80 [13540.571312] [] ? poll_select_copy_remaining+0x150/0x150 [13540.572855] [] ? poll_select_copy_remaining+0x150/0x150 [13540.574396] [] ? poll_select_copy_remaining+0x150/0x150 [13540.575901] [] ? try_to_wake_up+0x1c8/0x320 [13540.577324] [] ? default_wake_function+0x12/0x20 [13540.578779] [] ? pollwake+0x73/0x90 [13540.580128] [] ? wake_up_state+0x20/0x20 [13540.581515] [] ? eventfd_ctx_read+0x67/0x210 [13540.582917] [] ? vfs_read+0xf7/0x170 [13540.584261] [] SyS_poll+0x74/0x110 [13540.585530] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.586959] [] system_call_fastpath+0x16/0x1b [13540.588297] polkitd S ffff88007a86bbac 0 552 1 0x00000080 [13540.589810] ffff88007a86ba30 0000000000000086 ffff880036950fb0 ffff88007a86bfd8 [13540.591384] ffff88007a86bfd8 ffff88007a86bfd8 ffff880036950fb0 0000000000000000 [13540.592943] ffff88007a86bc90 00000000fffffffc 0000000000000000 ffff88007a86bbac [13540.594473] Call Trace: [13540.595572] [] schedule+0x29/0x70 [13540.596939] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.598452] [] ? add_wait_queue+0x3c/0x50 [13540.599808] [] schedule_hrtimeout_range+0x13/0x20 [13540.601291] [] poll_schedule_timeout+0x55/0xb0 [13540.602753] [] do_sys_poll+0x4cd/0x580 [13540.604163] [] ? kvm_clock_read+0x1f/0x30 [13540.605530] [] ? dequeue_task_fair+0x41e/0x660 [13540.607017] [] ? sched_clock_cpu+0x98/0xc0 [13540.608402] [] ? __switch_to+0x15c/0x4c0 [13540.609841] [] ? update_curr+0x104/0x190 [13540.611259] [] ? __enqueue_entity+0x78/0x80 [13540.612683] [] ? poll_select_copy_remaining+0x150/0x150 [13540.614185] [] ? poll_select_copy_remaining+0x150/0x150 [13540.615658] [] ? ttwu_do_wakeup+0x19/0xd0 [13540.616997] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [13540.618494] [] ? inotify_free_event+0xe/0x10 [13540.619881] [] ? eventfd_ctx_read+0x67/0x210 [13540.621220] [] ? vfs_read+0xf7/0x170 [13540.622564] [] SyS_poll+0x74/0x110 [13540.623825] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.625258] [] system_call_fastpath+0x16/0x1b [13540.626643] gmain S ffff88007b987bac 0 568 1 0x00000080 [13540.628154] ffff88007b987a30 0000000000000086 ffff88007b54de20 ffff88007b987fd8 [13540.629683] ffff88007b987fd8 ffff88007b987fd8 ffff88007b54de20 0000000000000000 [13540.631267] ffff88007b987c90 00000000fffffffc 0000000000000000 ffff88007b987bac [13540.632793] Call Trace: [13540.633932] [] schedule+0x29/0x70 [13540.635285] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.636797] [] ? add_wait_queue+0x3c/0x50 [13540.638166] [] schedule_hrtimeout_range+0x13/0x20 [13540.639649] [] poll_schedule_timeout+0x55/0xb0 [13540.641071] [] do_sys_poll+0x4cd/0x580 [13540.642505] [] ? clear_page+0x12/0x40 [13540.643871] [] ? get_page_from_freelist+0x51a/0x9f0 [13540.645395] [] ? poll_select_copy_remaining+0x150/0x150 [13540.646935] [] ? poll_select_copy_remaining+0x150/0x150 [13540.648422] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [13540.649964] [] ? lru_cache_add+0xe/0x10 [13540.651394] [] ? page_add_new_anon_rmap+0x91/0x150 [13540.652908] [] ? eventfd_ctx_read+0x67/0x210 [13540.654367] [] ? vfs_read+0xf7/0x170 [13540.655767] [] SyS_poll+0x74/0x110 [13540.657107] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.658592] [] system_call_fastpath+0x16/0x1b [13540.660040] gdbus S ffff880079fafbac 0 572 1 0x00000080 [13540.661546] ffff880079fafa30 0000000000000086 ffff8800792f9f60 ffff880079faffd8 [13540.663184] ffff880079faffd8 ffff880079faffd8 ffff8800792f9f60 0000000000000000 [13540.664738] ffff880079fafc90 00000000fffffffc 0000000000000000 ffff880079fafbac [13540.666379] Call Trace: [13540.667511] [] schedule+0x29/0x70 [13540.668917] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.670452] [] ? add_wait_queue+0x3c/0x50 [13540.671841] [] ? add_wait_queue+0x3c/0x50 [13540.673274] [] schedule_hrtimeout_range+0x13/0x20 [13540.674701] [] poll_schedule_timeout+0x55/0xb0 [13540.676183] [] do_sys_poll+0x4cd/0x580 [13540.677544] [] ? unix_stream_recvmsg+0x54/0x70 [13540.679030] [] ? unix_state_double_unlock+0x50/0x50 [13540.680501] [] ? sock_recvmsg+0xbf/0x100 [13540.681939] [] ? update_curr+0x104/0x190 [13540.683331] [] ? __enqueue_entity+0x78/0x80 [13540.684798] [] ? poll_select_copy_remaining+0x150/0x150 [13540.686339] [] ? poll_select_copy_remaining+0x150/0x150 [13540.687858] [] ? ttwu_do_wakeup+0x19/0xd0 [13540.689289] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [13540.690818] [] ? try_to_wake_up+0x1c8/0x320 [13540.692234] [] ? default_wake_function+0x12/0x20 [13540.693719] [] ? pollwake+0x73/0x90 [13540.695112] [] ? wake_up_state+0x20/0x20 [13540.696530] [] ? eventfd_ctx_read+0x67/0x210 [13540.697956] [] ? vfs_read+0xf7/0x170 [13540.699321] [] SyS_poll+0x74/0x110 [13540.700664] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.702066] [] system_call_fastpath+0x16/0x1b [13540.703483] JS GC Helper S ffffc90000381140 0 574 1 0x00000080 [13540.704976] ffff880079fb3cd0 0000000000000086 ffff880079950fb0 ffff880079fb3fd8 [13540.706574] ffff880079fb3fd8 ffff880079fb3fd8 ffff880079950fb0 ffff880079950fb0 [13540.708154] ffff880079950fb0 0000000000000000 ffff880079fb3de0 ffffc90000381140 [13540.709710] Call Trace: [13540.710862] [] schedule+0x29/0x70 [13540.712185] [] futex_wait_queue_me+0xc4/0x120 [13540.713682] [] futex_wait+0x179/0x280 [13540.715089] [] ? page_add_file_rmap+0x8c/0xc0 [13540.716504] [] ? unlock_page+0x2b/0x30 [13540.717999] [] do_futex+0xfe/0x5b0 [13540.719335] [] ? __do_page_fault+0x171/0x450 [13540.720771] [] ? strlcpy+0x42/0x60 [13540.722108] [] SyS_futex+0x80/0x180 [13540.723482] [] ? do_page_fault+0x35/0x90 [13540.724826] [] system_call_fastpath+0x16/0x1b [13540.726275] JS Sour~ Thread S ffffc9000037c580 0 576 1 0x00000080 [13540.727778] ffff880079803cd0 0000000000000086 ffff880079fb8000 ffff880079803fd8 [13540.729405] ffff880079803fd8 ffff880079803fd8 ffff880079fb8000 ffff880079fb8000 [13540.730936] ffff880079fb8000 0000000000000000 ffff880079803de0 ffffc9000037c580 [13540.732497] Call Trace: [13540.733613] [] schedule+0x29/0x70 [13540.734919] [] futex_wait_queue_me+0xc4/0x120 [13540.736284] [] futex_wait+0x179/0x280 [13540.737605] [] ? wake_up_state+0x10/0x20 [13540.738927] [] ? wake_futex+0x66/0x80 [13540.740236] [] ? get_futex_key+0x1c8/0x2c0 [13540.741584] [] do_futex+0xfe/0x5b0 [13540.742882] [] ? set_next_entity+0x3c/0xe0 [13540.744215] [] ? __switch_to+0x15c/0x4c0 [13540.745530] [] SyS_futex+0x80/0x180 [13540.746812] [] system_call_fastpath+0x16/0x1b [13540.748166] runaway-killer- S ffff88007981fba4 0 578 1 0x00000080 [13540.749650] ffff88007981fa30 0000000000000086 ffff880079fb9f60 ffff88007981ffd8 [13540.751196] ffff88007981ffd8 ffff88007981ffd8 ffff880079fb9f60 0000000000000000 [13540.752742] ffff88007981fc90 00000000fffffffc 0000000000000000 ffff88007981fba4 [13540.754317] Call Trace: [13540.755442] [] schedule+0x29/0x70 [13540.756764] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.758252] [] ? add_wait_queue+0x3c/0x50 [13540.759647] [] schedule_hrtimeout_range+0x13/0x20 [13540.761082] [] poll_schedule_timeout+0x55/0xb0 [13540.762497] [] do_sys_poll+0x4cd/0x580 [13540.763864] [] ? zone_statistics+0x89/0xa0 [13540.765274] [] ? zone_statistics+0x89/0xa0 [13540.766665] [] ? get_page_from_freelist+0x51a/0x9f0 [13540.768117] [] ? update_curr+0x104/0x190 [13540.769481] [] ? __enqueue_entity+0x78/0x80 [13540.770862] [] ? poll_select_copy_remaining+0x150/0x150 [13540.772313] [] ? check_preempt_curr+0x85/0xa0 [13540.773696] [] ? ttwu_do_wakeup+0x19/0xd0 [13540.775058] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [13540.776500] [] ? try_to_wake_up+0x1c8/0x320 [13540.777857] [] ? wake_up_state+0x10/0x20 [13540.779198] [] ? wake_futex+0x66/0x80 [13540.780513] [] ? futex_wake+0x80/0x160 [13540.781822] [] ? eventfd_ctx_read+0x67/0x210 [13540.783184] [] ? vfs_read+0xf7/0x170 [13540.784503] [] SyS_poll+0x74/0x110 [13540.785800] [] ? __audit_syscall_exit+0x1e6/0x280 [13540.787195] [] system_call_fastpath+0x16/0x1b [13540.788565] irqbalance S 00007fff568e7250 0 553 1 0x00000080 [13540.790062] ffff880079f97e68 0000000000000082 ffff880036956dd0 ffff880079f97fd8 [13540.791617] ffff880079f97fd8 ffff880079f97fd8 ffff880036956dd0 ffff880036956dd0 [13540.793177] ffff880079f97eb8 ffff880036956dd0 ffff880079f97f58 00007fff568e7250 [13540.794716] Call Trace: [13540.795829] [] schedule+0x29/0x70 [13540.797145] [] do_nanosleep+0x86/0xf0 [13540.798463] [] hrtimer_nanosleep+0xbb/0x180 [13540.799817] [] ? hrtimer_get_res+0x50/0x50 [13540.801169] [] ? do_nanosleep+0x72/0xf0 [13540.802491] [] SyS_nanosleep+0x7e/0x90 [13540.803827] [] system_call_fastpath+0x16/0x1b [13540.805218] sssd S ffff88007b163ea8 0 559 1 0x00000080 [13540.806725] ffff88007b163db8 0000000000000086 ffff880079fc3ec0 ffff88007b163fd8 [13540.808320] ffff88007b163fd8 ffff88007b163fd8 ffff880079fc3ec0 ffff88007b163ea8 [13540.809883] 0000000001c9c37f 0000000000000000 ffff880079fc3ec0 ffff88007b163ea8 [13540.811465] Call Trace: [13540.812608] [] schedule+0x29/0x70 [13540.813956] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.815452] [] ? hrtimer_get_res+0x50/0x50 [13540.816834] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.818339] [] schedule_hrtimeout_range+0x13/0x20 [13540.819777] [] ep_poll+0x23e/0x360 [13540.821099] [] ? wake_up_state+0x20/0x20 [13540.822452] [] SyS_epoll_wait+0xed/0x120 [13540.823801] [] system_call_fastpath+0x16/0x1b [13540.825184] sssd_be S ffff88007a8fbea8 0 564 559 0x00000080 [13540.826680] ffff88007a8fbdb8 0000000000000086 ffff88007b54bec0 ffff88007a8fbfd8 [13540.828252] ffff88007a8fbfd8 ffff88007a8fbfd8 ffff88007b54bec0 ffff88007a8fbea8 [13540.829802] 0000000000513127 0000000000000000 ffff88007b54bec0 ffff88007a8fbea8 [13540.831370] Call Trace: [13540.832517] [] schedule+0x29/0x70 [13540.833858] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.835359] [] ? hrtimer_get_res+0x50/0x50 [13540.836778] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.838308] [] schedule_hrtimeout_range+0x13/0x20 [13540.839768] [] ep_poll+0x23e/0x360 [13540.841140] [] ? wake_up_state+0x20/0x20 [13540.842546] [] SyS_epoll_wait+0xed/0x120 [13540.843943] [] system_call_fastpath+0x16/0x1b [13540.845371] chronyd S 0000000000000007 0 567 1 0x00000080 [13540.846929] ffff880079fa38e8 0000000000000082 ffff880079fc1f60 ffff880079fa3fd8 [13540.848562] ffff880079fa3fd8 ffff880079fa3fd8 ffff880079fc1f60 ffff880079fa3ac8 [13540.850203] 0000000005f5e100 0000000000000000 0000000000000007 0000000000000007 [13540.851826] Call Trace: [13540.853038] [] schedule+0x29/0x70 [13540.854438] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.855994] [] ? hrtimer_get_res+0x50/0x50 [13540.857447] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.859009] [] schedule_hrtimeout_range+0x13/0x20 [13540.860508] [] poll_schedule_timeout+0x55/0xb0 [13540.861995] [] do_select+0x6d1/0x7c0 [13540.863408] [] ? udp_send_skb+0x13c/0x380 [13540.864836] [] ? poll_select_copy_remaining+0x150/0x150 [13540.866393] [] ? poll_select_copy_remaining+0x150/0x150 [13540.867915] [] ? poll_select_copy_remaining+0x150/0x150 [13540.869428] [] ? poll_select_copy_remaining+0x150/0x150 [13540.870923] [] ? poll_select_copy_remaining+0x150/0x150 [13540.872406] [] ? poll_select_copy_remaining+0x150/0x150 [13540.873879] [] core_sys_select+0x1db/0x300 [13540.875247] [] ? ip4_datagram_connect+0x269/0x300 [13540.876659] [] ? inet_dgram_connect+0x2e/0x80 [13540.878063] [] ? kvm_clock_get_cycles+0x1f/0x30 [13540.879479] [] ? ktime_get_ts64+0x4c/0xf0 [13540.880843] [] SyS_select+0xba/0x110 [13540.882164] [] system_call_fastpath+0x16/0x1b [13540.883536] rpc.gssd S 0000000000000000 0 570 1 0x00000080 [13540.885038] ffff880079fa7db8 0000000000000086 ffff880079fc4e70 ffff880079fa7fd8 [13540.886583] ffff880079fa7fd8 ffff880079fa7fd8 ffff880079fc4e70 0000000000000000 [13540.888600] 0000000000000000 ffff88007a8f22a0 ffff880079fc4e70 0000000000000000 [13540.890611] Call Trace: [13540.891726] [] schedule+0x29/0x70 [13540.893116] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13540.894592] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13540.896104] [] schedule_hrtimeout_range+0x13/0x20 [13540.897557] [] ep_poll+0x23e/0x360 [13540.898917] [] ? wake_up_state+0x20/0x20 [13540.900338] [] SyS_epoll_wait+0xed/0x120 [13540.901685] [] system_call_fastpath+0x16/0x1b [13540.903121] ipoib_flush S ffff88007b5ab380 0 580 2 0x00000080 [13540.904630] ffff88007985be08 0000000000000046 ffff880079fbbec0 ffff88007985bfd8 [13540.906257] ffff88007985bfd8 ffff88007985bfd8 ffff880079fbbec0 ffff880079fbbec0 [13540.907809] ffff88007b5ab380 ffff880036875480 ffff88007b5ab3b0 ffff88007b5ab380 [13540.909425] Call Trace: [13540.910545] [] schedule+0x29/0x70 [13540.911928] [] rescuer_thread+0x359/0x460 [13540.913302] [] ? __schedule+0x3b0/0x990 [13540.914716] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13540.916148] [] kthread+0xcf/0xe0 [13540.917502] [] ? kthread_create_on_node+0x140/0x140 [13540.918939] [] ret_from_fork+0x58/0x90 [13540.920333] [] ? kthread_create_on_node+0x140/0x140 [13540.921742] sssd_nss S ffff880079effea8 0 599 559 0x00000080 [13540.923295] ffff880079effdb8 0000000000000082 ffff880036183ec0 ffff880079efffd8 [13540.924851] ffff880079efffd8 ffff880079efffd8 ffff880036183ec0 ffff880079effea8 [13540.926463] 00000000004d87a7 0000000000000000 ffff880036183ec0 ffff880079effea8 [13540.928035] Call Trace: [13540.929144] [] schedule+0x29/0x70 [13540.930494] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.931994] [] ? hrtimer_get_res+0x50/0x50 [13540.933396] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.934911] [] schedule_hrtimeout_range+0x13/0x20 [13540.936336] [] ep_poll+0x23e/0x360 [13540.937746] [] ? wake_up_state+0x20/0x20 [13540.939196] [] SyS_epoll_wait+0xed/0x120 [13540.940651] [] system_call_fastpath+0x16/0x1b [13540.942110] sssd_pam S ffff880036197ea8 0 600 559 0x00000080 [13540.943728] ffff880036197db8 0000000000000082 ffff880036182f10 ffff880036197fd8 [13540.945404] ffff880036197fd8 ffff880036197fd8 ffff880036182f10 ffff880036197ea8 [13540.947080] 000000000098967f 0000000000000000 ffff880036182f10 ffff880036197ea8 [13540.948722] Call Trace: [13540.949958] [] schedule+0x29/0x70 [13540.951406] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.953021] [] ? hrtimer_get_res+0x50/0x50 [13540.954528] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.956140] [] schedule_hrtimeout_range+0x13/0x20 [13540.957704] [] ep_poll+0x23e/0x360 [13540.959132] [] ? wake_up_state+0x20/0x20 [13540.960633] [] SyS_epoll_wait+0xed/0x120 [13540.962128] [] system_call_fastpath+0x16/0x1b [13540.963646] sssd_ssh S ffff880079077ea8 0 601 559 0x00000080 [13540.965276] ffff880079077db8 0000000000000086 ffff880036181f60 ffff880079077fd8 [13540.966937] ffff880079077fd8 ffff880079077fd8 ffff880036181f60 ffff880079077ea8 [13540.968629] 000000000098967f 0000000000000000 ffff880036181f60 ffff880079077ea8 [13540.970303] Call Trace: [13540.971514] [] schedule+0x29/0x70 [13540.972993] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.974603] [] ? hrtimer_get_res+0x50/0x50 [13540.976103] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.977725] [] schedule_hrtimeout_range+0x13/0x20 [13540.979249] [] ep_poll+0x23e/0x360 [13540.980713] [] ? wake_up_state+0x20/0x20 [13540.982167] [] SyS_epoll_wait+0xed/0x120 [13540.983669] [] system_call_fastpath+0x16/0x1b [13540.985156] sssd_pac S ffff88007a977ea8 0 602 559 0x00000080 [13540.986801] ffff88007a977db8 0000000000000086 ffff880036185e20 ffff88007a977fd8 [13540.988516] ffff88007a977fd8 ffff88007a977fd8 ffff880036185e20 ffff88007a977ea8 [13540.990232] 000000000098967f 0000000000000000 ffff880036185e20 ffff88007a977ea8 [13540.991875] Call Trace: [13540.993120] [] schedule+0x29/0x70 [13540.994521] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13540.996137] [] ? hrtimer_get_res+0x50/0x50 [13540.997647] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13540.999272] [] schedule_hrtimeout_range+0x13/0x20 [13541.000835] [] ep_poll+0x23e/0x360 [13541.002299] [] ? wake_up_state+0x20/0x20 [13541.003802] [] SyS_epoll_wait+0xed/0x120 [13541.005291] [] system_call_fastpath+0x16/0x1b [13541.006757] srp_remove S ffff88007ab9ea00 0 604 2 0x00000080 [13541.008408] ffff88007b3fbe08 0000000000000046 ffff880079fbde20 ffff88007b3fbfd8 [13541.010099] ffff88007b3fbfd8 ffff88007b3fbfd8 ffff880079fbde20 ffff880079fbde20 [13541.011743] ffff88007ab9ea00 ffff880036874a80 ffff88007ab9ea30 ffff88007ab9ea00 [13541.013453] Call Trace: [13541.014710] [] schedule+0x29/0x70 [13541.016214] [] rescuer_thread+0x359/0x460 [13541.017706] [] ? __schedule+0x3b0/0x990 [13541.019194] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.020766] [] kthread+0xcf/0xe0 [13541.022199] [] ? kthread_create_on_node+0x140/0x140 [13541.023762] [] ret_from_fork+0x58/0x90 [13541.025191] [] ? kthread_create_on_node+0x140/0x140 [13541.026758] target_completi S ffff8800362b1400 0 612 2 0x00000080 [13541.028410] ffff880079d1fe08 0000000000000046 ffff88007ae88fb0 ffff880079d1ffd8 [13541.030102] ffff880079d1ffd8 ffff880079d1ffd8 ffff88007ae88fb0 ffff88007ae88fb0 [13541.031756] ffff8800362b1400 ffff88003608e880 ffff8800362b1430 ffff8800362b1400 [13541.033453] Call Trace: [13541.034666] [] schedule+0x29/0x70 [13541.036155] [] rescuer_thread+0x359/0x460 [13541.037658] [] ? __schedule+0x3b0/0x990 [13541.039087] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.040680] [] kthread+0xcf/0xe0 [13541.042079] [] ? kthread_create_on_node+0x140/0x140 [13541.043659] [] ret_from_fork+0x58/0x90 [13541.045134] [] ? kthread_create_on_node+0x140/0x140 [13541.046642] tmr-rd_mcp S ffff8800362b1780 0 613 2 0x00000080 [13541.048343] ffff88007b3c3e08 0000000000000046 ffff88007ae8bec0 ffff88007b3c3fd8 [13541.049995] ffff88007b3c3fd8 ffff88007b3c3fd8 ffff88007ae8bec0 ffff88007ae8bec0 [13541.051670] ffff8800362b1780 ffff88003608f680 ffff8800362b17b0 ffff8800362b1780 [13541.053367] Call Trace: [13541.054582] [] schedule+0x29/0x70 [13541.056063] [] rescuer_thread+0x359/0x460 [13541.057528] [] ? __schedule+0x3b0/0x990 [13541.059034] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.060586] [] kthread+0xcf/0xe0 [13541.062033] [] ? kthread_create_on_node+0x140/0x140 [13541.063609] [] ret_from_fork+0x58/0x90 [13541.065042] [] ? kthread_create_on_node+0x140/0x140 [13541.066616] xcopy_wq S ffff8800362b1880 0 615 2 0x00000080 [13541.068223] ffff880079d27e08 0000000000000046 ffff88007ae8ce70 ffff880079d27fd8 [13541.069930] ffff880079d27fd8 ffff880079d27fd8 ffff88007ae8ce70 ffff88007ae8ce70 [13541.071642] ffff8800362b1880 ffff88003608e680 ffff8800362b18b0 ffff8800362b1880 [13541.073345] Call Trace: [13541.074558] [] schedule+0x29/0x70 [13541.076023] [] rescuer_thread+0x359/0x460 [13541.077478] [] ? __schedule+0x3b0/0x990 [13541.078994] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.080555] [] kthread+0xcf/0xe0 [13541.081938] [] ? kthread_create_on_node+0x140/0x140 [13541.083515] [] ret_from_fork+0x58/0x90 [13541.084998] [] ? kthread_create_on_node+0x140/0x140 [13541.086504] dhclient S 0000000000000007 0 630 551 0x00000080 [13541.088123] ffff88007a8378e8 0000000000000086 ffff88007ab04e70 ffff88007a837fd8 [13541.089815] ffff88007a837fd8 ffff88007a837fd8 ffff88007ab04e70 ffff88007a837ac8 [13541.091518] 0000000005f5e100 0000000000000000 0000000000000007 0000000000000007 [13541.093218] Call Trace: [13541.094438] [] schedule+0x29/0x70 [13541.095934] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13541.097503] [] ? hrtimer_get_res+0x50/0x50 [13541.099051] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13541.100690] [] schedule_hrtimeout_range+0x13/0x20 [13541.102214] [] poll_schedule_timeout+0x55/0xb0 [13541.103781] [] do_select+0x6d1/0x7c0 [13541.105229] [] ? poll_select_copy_remaining+0x150/0x150 [13541.106846] [] ? poll_select_copy_remaining+0x150/0x150 [13541.108436] [] ? kmem_cache_free+0x1bb/0x1f0 [13541.109963] [] ? kfree_skbmem+0x37/0x90 [13541.111448] [] ? __kfree_skb+0x1a/0x20 [13541.112868] [] ? skb_free_datagram_locked+0x93/0xd0 [13541.114448] [] ? udp_recvmsg+0x29e/0x490 [13541.115886] [] ? inet_recvmsg+0x7b/0xa0 [13541.117369] [] ? sock_recvmsg+0xbf/0x100 [13541.118827] [] core_sys_select+0x1db/0x300 [13541.120286] [] ? handle_mm_fault+0x72c/0xfe0 [13541.121761] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.123229] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.124725] [] ? ktime_get_ts64+0x4c/0xf0 [13541.126112] [] SyS_select+0xba/0x110 [13541.127511] [] system_call_fastpath+0x16/0x1b [13541.128922] systemd-logind S 0000000000000000 0 631 1 0x00000080 [13541.130512] ffff880079d03db8 0000000000000086 ffff88007b548fb0 ffff880079d03fd8 [13541.132145] ffff880079d03fd8 ffff880079d03fd8 ffff88007b548fb0 0000000000000000 [13541.133727] 0000000000000000 ffff8800793fd120 ffff88007b548fb0 0000000000000000 [13541.135376] Call Trace: [13541.136518] [] schedule+0x29/0x70 [13541.137933] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.139422] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13541.140948] [] schedule_hrtimeout_range+0x13/0x20 [13541.142428] [] ep_poll+0x23e/0x360 [13541.143765] [] ? wake_up_state+0x20/0x20 [13541.145209] [] SyS_epoll_wait+0xed/0x120 [13541.146631] [] system_call_fastpath+0x16/0x1b [13541.148073] iscsi_eh S ffff8800362b1c00 0 632 2 0x00000080 [13541.149603] ffff880000067e08 0000000000000046 ffff88007ab03ec0 ffff880000067fd8 [13541.151250] ffff880000067fd8 ffff880000067fd8 ffff88007ab03ec0 ffff88007ab03ec0 [13541.152827] ffff8800362b1c00 ffff880079f7fe80 ffff8800362b1c30 ffff8800362b1c00 [13541.154486] Call Trace: [13541.155630] [] schedule+0x29/0x70 [13541.157038] [] rescuer_thread+0x359/0x460 [13541.158479] [] ? __schedule+0x3b0/0x990 [13541.159887] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.161331] [] kthread+0xcf/0xe0 [13541.162696] [] ? finish_task_switch+0x53/0x180 [13541.164105] [] ? kthread_create_on_node+0x140/0x140 [13541.165590] [] ret_from_fork+0x58/0x90 [13541.166980] [] ? kthread_create_on_node+0x140/0x140 [13541.168424] frwr_recovery S ffff88007ab55f80 0 649 2 0x00000080 [13541.169971] ffff880079557e08 0000000000000046 ffff88007ab05e20 ffff880079557fd8 [13541.171599] ffff880079557fd8 ffff880079557fd8 ffff88007ab05e20 ffff88007ab05e20 [13541.173193] ffff88007ab55f80 ffff880079a22480 ffff88007ab55fb0 ffff88007ab55f80 [13541.174742] Call Trace: [13541.175911] [] schedule+0x29/0x70 [13541.177288] [] rescuer_thread+0x359/0x460 [13541.178713] [] ? __schedule+0x3b0/0x990 [13541.180091] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.181598] [] kthread+0xcf/0xe0 [13541.182926] [] ? kthread_create_on_node+0x140/0x140 [13541.184444] [] ret_from_fork+0x58/0x90 [13541.185885] [] ? kthread_create_on_node+0x140/0x140 [13541.187395] xprtrdma_receiv S ffff88007a6a9080 0 650 2 0x00000080 [13541.188968] ffff880079543e08 0000000000000046 ffff88007ab00fb0 ffff880079543fd8 [13541.190636] ffff880079543fd8 ffff880079543fd8 ffff88007ab00fb0 ffff88007ab00fb0 [13541.192323] ffff88007a6a9080 ffff880079a23480 ffff88007a6a90b0 ffff88007a6a9080 [13541.193943] Call Trace: [13541.195168] [] schedule+0x29/0x70 [13541.196568] [] rescuer_thread+0x359/0x460 [13541.198093] [] ? __schedule+0x3b0/0x990 [13541.199562] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.201074] [] kthread+0xcf/0xe0 [13541.202527] [] ? kthread_create_on_node+0x140/0x140 [13541.204095] [] ret_from_fork+0x58/0x90 [13541.205536] [] ? kthread_create_on_node+0x140/0x140 [13541.207183] kworker/0:1H S ffff88007fc16840 0 704 2 0x00000080 [13541.208903] ffff88003614fe50 0000000000000046 ffff88007ab02f10 ffff88003614ffd8 [13541.210615] ffff88003614ffd8 ffff88003614ffd8 ffff88007ab02f10 ffff88007fc16860 [13541.212282] ffff880079e91830 ffff88007ab02f10 ffff880079e91800 ffff88007fc16840 [13541.213920] Call Trace: [13541.215176] [] schedule+0x29/0x70 [13541.216615] [] worker_thread+0x1d9/0x410 [13541.218060] [] ? rescuer_thread+0x460/0x460 [13541.219586] [] kthread+0xcf/0xe0 [13541.221003] [] ? kthread_create_on_node+0x140/0x140 [13541.222556] [] ret_from_fork+0x58/0x90 [13541.223982] [] ? kthread_create_on_node+0x140/0x140 [13541.225527] rsyslogd S 0000000000000000 0 899 1 0x00000080 [13541.227115] ffff88003684b8e8 0000000000000082 ffff88007930ce70 ffff88003684bfd8 [13541.228785] ffff88003684bfd8 ffff88003684bfd8 ffff88007930ce70 ffff88003684bac8 [13541.230476] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [13541.232152] Call Trace: [13541.233357] [] schedule+0x29/0x70 [13541.234823] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13541.236382] [] ? hrtimer_get_res+0x50/0x50 [13541.237913] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13541.239537] [] schedule_hrtimeout_range+0x13/0x20 [13541.241049] [] poll_schedule_timeout+0x55/0xb0 [13541.242598] [] do_select+0x6d1/0x7c0 [13541.244064] [] ? ep_poll_callback+0xee/0x210 [13541.245597] [] ? __wake_up_sync_key+0x4f/0x60 [13541.247119] [] ? sock_def_readable+0x3a/0x70 [13541.248623] [] ? unix_dgram_sendmsg+0x3a8/0x720 [13541.250118] [] ? sock_sendmsg+0xb0/0xf0 [13541.251610] [] ? __switch_to+0x15c/0x4c0 [13541.253042] [] ? update_curr+0x104/0x190 [13541.254537] [] ? __enqueue_entity+0x78/0x80 [13541.256032] [] core_sys_select+0x1db/0x300 [13541.257518] [] ? __d_free+0x35/0x40 [13541.258903] [] ? d_free+0x60/0x70 [13541.260305] [] ? dentry_kill+0x14e/0x1b0 [13541.261681] [] ? mntput_no_expire+0x3e/0x120 [13541.263153] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.264631] [] ? ktime_get_ts64+0x4c/0xf0 [13541.266060] [] SyS_select+0xba/0x110 [13541.267406] [] system_call_fastpath+0x16/0x1b [13541.268873] in:imjournal S ffff88003616bba4 0 914 1 0x00000080 [13541.270410] ffff88003616ba30 0000000000000082 ffff880036922f10 ffff88003616bfd8 [13541.272050] ffff88003616bfd8 ffff88003616bfd8 ffff880036922f10 0000000000000000 [13541.273632] ffff88003616bc90 00000000fffffffc 0000000000000000 ffff88003616bba4 [13541.275298] Call Trace: [13541.276463] [] schedule+0x29/0x70 [13541.277890] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.279429] [] ? ext4_free_blocks+0x67e/0xb80 [ext4] [13541.280976] [] ? add_wait_queue+0x3c/0x50 [13541.282426] [] schedule_hrtimeout_range+0x13/0x20 [13541.283951] [] poll_schedule_timeout+0x55/0xb0 [13541.285427] [] do_sys_poll+0x4cd/0x580 [13541.286810] [] ? jbd2_journal_release_jbd_inode+0x100/0x130 [jbd2] [13541.288459] [] ? __wake_up+0x44/0x50 [13541.289894] [] ? update_curr+0x104/0x190 [13541.291344] [] ? __enqueue_entity+0x78/0x80 [13541.292835] [] ? enqueue_entity+0x26c/0xb60 [13541.294256] [] ? kvm_clock_read+0x1f/0x30 [13541.295720] [] ? check_preempt_wakeup+0x148/0x250 [13541.297178] [] ? poll_select_copy_remaining+0x150/0x150 [13541.298716] [] ? call_rcu_sched+0x1d/0x20 [13541.300162] [] ? try_to_wake_up+0x1c8/0x320 [13541.301564] [] ? wake_up_state+0x10/0x20 [13541.302985] [] ? wake_futex+0x66/0x80 [13541.304318] [] ? futex_wake_op+0x332/0x570 [13541.305753] [] ? do_futex+0x172/0x5b0 [13541.307080] [] ? wake_up_atomic_t+0x30/0x30 [13541.308506] [] SyS_poll+0x74/0x110 [13541.309795] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.311255] [] system_call_fastpath+0x16/0x1b [13541.312628] rs:main Q:Reg S ffffc9000037f600 0 915 1 0x00000080 [13541.314223] ffff880036a2fcd0 0000000000000082 ffff88007930bec0 ffff880036a2ffd8 [13541.315785] ffff880036a2ffd8 ffff880036a2ffd8 ffff88007930bec0 ffff88007930bec0 [13541.317388] ffff88007930bec0 0000000000000000 ffff880036a2fde0 ffffc9000037f600 [13541.318981] Call Trace: [13541.320103] [] schedule+0x29/0x70 [13541.321483] [] futex_wait_queue_me+0xc4/0x120 [13541.322944] [] futex_wait+0x179/0x280 [13541.324315] [] ? ext4_file_write+0xdb/0x470 [ext4] [13541.325825] [] do_futex+0xfe/0x5b0 [13541.327215] [] SyS_futex+0x80/0x180 [13541.328614] [] system_call_fastpath+0x16/0x1b [13541.330052] tuned S 0000000000000000 0 901 1 0x00000080 [13541.331643] ffff880036a9f8e8 0000000000000082 ffff88007930edd0 ffff880036a9ffd8 [13541.333329] ffff880036a9ffd8 ffff880036a9ffd8 ffff88007930edd0 ffff880036a9fac8 [13541.334931] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [13541.336625] Call Trace: [13541.337812] [] schedule+0x29/0x70 [13541.339259] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13541.340792] [] ? hrtimer_get_res+0x50/0x50 [13541.342283] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13541.343815] [] schedule_hrtimeout_range+0x13/0x20 [13541.345358] [] poll_schedule_timeout+0x55/0xb0 [13541.346853] [] do_select+0x6d1/0x7c0 [13541.348274] [] ? check_preempt_wakeup+0x11d/0x250 [13541.349795] [] ? move_task+0x4b/0x50 [13541.351213] [] ? update_curr+0x104/0x190 [13541.352649] [] ? account_entity_dequeue+0xae/0xd0 [13541.354168] [] ? dequeue_entity+0x11c/0x5d0 [13541.355576] [] ? kvm_clock_read+0x1f/0x30 [13541.357042] [] ? dequeue_task_fair+0x41e/0x660 [13541.358462] [] ? set_next_entity+0x3c/0xe0 [13541.359913] [] ? __switch_to+0x15c/0x4c0 [13541.361331] [] ? unlock_page+0x2b/0x30 [13541.362738] [] ? do_wp_page+0x17b/0x530 [13541.364103] [] ? handle_mm_fault+0x72c/0xfe0 [13541.365553] [] core_sys_select+0x1db/0x300 [13541.366938] [] ? do_page_fault+0x35/0x90 [13541.368365] [] ? handle_mm_fault+0x72c/0xfe0 [13541.369754] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.371231] [] ? ktime_get_ts64+0x4c/0xf0 [13541.372669] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.374137] [] ? ktime_get_ts64+0x4c/0xf0 [13541.375509] [] SyS_select+0xba/0x110 [13541.376931] [] system_call_fastpath+0x16/0x1b [13541.378338] gmain S ffff88007901fba4 0 968 1 0x00000080 [13541.379914] ffff88007901fa30 0000000000000082 ffff880079218000 ffff88007901ffd8 [13541.381562] ffff88007901ffd8 ffff88007901ffd8 ffff880079218000 0000000000000000 [13541.383178] ffff88007901fc90 00000000fffffffc 0000000000000000 ffff88007901fba4 [13541.384751] Call Trace: [13541.385927] [] schedule+0x29/0x70 [13541.387269] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.388815] [] ? add_wait_queue+0x3c/0x50 [13541.390257] [] schedule_hrtimeout_range+0x13/0x20 [13541.391694] [] poll_schedule_timeout+0x55/0xb0 [13541.393189] [] do_sys_poll+0x4cd/0x580 [13541.394616] [] ? zone_statistics+0x89/0xa0 [13541.396058] [] ? zone_statistics+0x89/0xa0 [13541.397485] [] ? get_page_from_freelist+0x51a/0x9f0 [13541.398990] [] ? poll_select_copy_remaining+0x150/0x150 [13541.400497] [] ? filemap_fault+0x215/0x410 [13541.401946] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [13541.403460] [] ? unlock_page+0x2b/0x30 [13541.404856] [] ? do_read_fault.isra.42+0xe4/0x130 [13541.406271] [] ? eventfd_ctx_read+0x67/0x210 [13541.407713] [] ? vfs_read+0xf7/0x170 [13541.409099] [] SyS_poll+0x74/0x110 [13541.410444] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.411849] [] system_call_fastpath+0x16/0x1b [13541.413314] tuned S ffff880079c63bac 0 969 1 0x00000080 [13541.414795] ffff880079c63a30 0000000000000082 ffff8800368a3ec0 ffff880079c63fd8 [13541.416413] ffff880079c63fd8 ffff880079c63fd8 ffff8800368a3ec0 0000000000000000 [13541.417996] ffff880079c63c90 00000000fffffffc 0000000000000000 ffff880079c63bac [13541.419565] Call Trace: [13541.420702] [] schedule+0x29/0x70 [13541.422117] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.423623] [] ? add_wait_queue+0x3c/0x50 [13541.425083] [] ? add_wait_queue+0x3c/0x50 [13541.426471] [] schedule_hrtimeout_range+0x13/0x20 [13541.427966] [] poll_schedule_timeout+0x55/0xb0 [13541.429398] [] do_sys_poll+0x4cd/0x580 [13541.430823] [] ? unix_stream_recvmsg+0x54/0x70 [13541.432311] [] ? unix_state_double_unlock+0x50/0x50 [13541.433832] [] ? zone_statistics+0x89/0xa0 [13541.435236] [] ? get_page_from_freelist+0x51a/0x9f0 [13541.436770] [] ? poll_select_copy_remaining+0x150/0x150 [13541.438272] [] ? poll_select_copy_remaining+0x150/0x150 [13541.439814] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [13541.441352] [] ? lru_cache_add+0xe/0x10 [13541.442777] [] ? page_add_new_anon_rmap+0x91/0x150 [13541.444285] [] ? eventfd_ctx_read+0x67/0x210 [13541.445708] [] ? wake_up_state+0x20/0x20 [13541.447164] [] ? eventfd_read+0x4c/0x90 [13541.448578] [] ? vfs_read+0x9e/0x170 [13541.449978] [] SyS_poll+0x74/0x110 [13541.451301] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.452776] [] system_call_fastpath+0x16/0x1b [13541.454208] tuned S 0000000000000000 0 971 1 0x00000080 [13541.455720] ffff880079d0b8e8 0000000000000082 ffff8800369eaf10 ffff880079d0bfd8 [13541.457344] ffff880079d0bfd8 ffff880079d0bfd8 ffff8800369eaf10 ffff880079d0bac8 [13541.458913] 00000000000f423d 0000000000000000 0000000000000000 0000000000000000 [13541.460530] Call Trace: [13541.461657] [] schedule+0x29/0x70 [13541.463067] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13541.464597] [] ? hrtimer_get_res+0x50/0x50 [13541.466015] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13541.467594] [] schedule_hrtimeout_range+0x13/0x20 [13541.469065] [] poll_schedule_timeout+0x55/0xb0 [13541.470557] [] do_select+0x6d1/0x7c0 [13541.471971] [] ? ep_poll_callback+0xee/0x210 [13541.473399] [] ? __wake_up_common+0x58/0x90 [13541.474873] [] ? __wake_up_sync_key+0x4f/0x60 [13541.476289] [] ? sock_def_readable+0x3a/0x70 [13541.477769] [] ? unix_stream_sendmsg+0x3fd/0x430 [13541.479217] [] ? sock_sendmsg+0xb0/0xf0 [13541.480660] [] core_sys_select+0x1db/0x300 [13541.482069] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.483545] [] ? ktime_get_ts64+0x4c/0xf0 [13541.484962] [] ? kvm_clock_get_cycles+0x1f/0x30 [13541.486421] [] ? ktime_get_ts64+0x4c/0xf0 [13541.487823] [] SyS_select+0xba/0x110 [13541.489136] [] system_call_fastpath+0x16/0x1b [13541.490559] tuned S 0000000000000000 0 972 1 0x00000080 [13541.492062] ffff88003681bdb8 0000000000000082 ffff8800369ede20 ffff88003681bfd8 [13541.493644] ffff88003681bfd8 ffff88003681bfd8 ffff8800369ede20 0000000000000000 [13541.495271] 0000000000000000 ffff880036856d20 ffff8800369ede20 0000000000000000 [13541.496814] Call Trace: [13541.497985] [] schedule+0x29/0x70 [13541.499306] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.500858] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [13541.502369] [] schedule_hrtimeout_range+0x13/0x20 [13541.503812] [] ep_poll+0x23e/0x360 [13541.505231] [] ? __fput+0x183/0x260 [13541.506566] [] ? wake_up_state+0x20/0x20 [13541.508025] [] SyS_epoll_wait+0xed/0x120 [13541.509415] [] system_call_fastpath+0x16/0x1b [13541.510888] xinetd S ffff880079eebbb4 0 908 1 0x00000080 [13541.512427] ffff880079eeba30 0000000000000082 ffff880036923ec0 ffff880079eebfd8 [13541.514079] ffff880079eebfd8 ffff880079eebfd8 ffff880036923ec0 0000000000000000 [13541.515668] ffff880079eebc90 00000000fffffffc 0000000000000000 ffff880079eebbb4 [13541.517352] Call Trace: [13541.518536] [] schedule+0x29/0x70 [13541.519991] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.521594] [] ? add_wait_queue+0x3c/0x50 [13541.523079] [] ? add_wait_queue+0x3c/0x50 [13541.524484] [] schedule_hrtimeout_range+0x13/0x20 [13541.526029] [] poll_schedule_timeout+0x55/0xb0 [13541.527519] [] do_sys_poll+0x4cd/0x580 [13541.528956] [] ? __wake_up_common+0x58/0x90 [13541.530417] [] ? __wake_up_sync_key+0x4f/0x60 [13541.531823] [] ? sock_def_readable+0x3a/0x70 [13541.533297] [] ? unix_dgram_sendmsg+0x3a8/0x720 [13541.534774] [] ? poll_select_copy_remaining+0x150/0x150 [13541.536294] [] ? poll_select_copy_remaining+0x150/0x150 [13541.537749] [] ? poll_select_copy_remaining+0x150/0x150 [13541.539248] [] ? remove_wait_queue+0x31/0x40 [13541.540654] [] ? do_wait+0x12b/0x250 [13541.542030] [] ? SyS_wait4+0x8b/0x110 [13541.543403] [] SyS_poll+0x74/0x110 [13541.544693] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.546160] [] system_call_fastpath+0x16/0x1b [13541.547534] munged S ffff88003618be30 0 911 1 0x00000080 [13541.549088] ffff88003618bcc0 0000000000000086 ffff8800368a5e20 ffff88003618bfd8 [13541.550683] ffff88003618bfd8 ffff88003618bfd8 ffff8800368a5e20 ffff88007b130090 [13541.552306] ffff88007b130090 ffff88003618be34 ffff88007b130000 ffff88003618be30 [13541.553854] Call Trace: [13541.555014] [] schedule+0x29/0x70 [13541.556344] [] schedule_timeout+0x239/0x2d0 [13541.557800] [] ? x2apic_send_IPI_mask+0x13/0x20 [13541.559218] [] __skb_recv_datagram+0x3ca/0x460 [13541.560680] [] ? datagram_poll+0x100/0x100 [13541.562057] [] skb_recv_datagram+0x41/0x60 [13541.563481] [] unix_accept+0x97/0x170 [13541.564821] [] SYSC_accept4+0x103/0x220 [13541.566211] [] ? SyS_futex+0x80/0x180 [13541.567569] [] SyS_accept4+0xe/0x10 [13541.568918] [] SyS_accept+0x10/0x20 [13541.570245] [] system_call_fastpath+0x16/0x1b [13541.571593] munged S ffffc90000382a40 0 918 1 0x00000080 [13541.573140] ffff8800369fbcd0 0000000000000086 ffff8800368a0000 ffff8800369fbfd8 [13541.574676] ffff8800369fbfd8 ffff8800369fbfd8 ffff8800368a0000 ffff8800368a0000 [13541.576260] ffff8800368a0000 ffff8800369fbd60 ffff8800369fbde0 ffffc90000382a40 [13541.577796] Call Trace: [13541.578975] [] schedule+0x29/0x70 [13541.580258] [] futex_wait_queue_me+0xc4/0x120 [13541.581678] [] futex_wait+0x179/0x280 [13541.583037] [] ? hrtimer_get_res+0x50/0x50 [13541.584404] [] ? futex_wait_queue_me+0xa2/0x120 [13541.585742] [] do_futex+0xfe/0x5b0 [13541.587065] [] ? mntput+0x24/0x40 [13541.588308] [] ? __fput+0x183/0x260 [13541.589614] [] SyS_futex+0x80/0x180 [13541.590865] [] system_call_fastpath+0x16/0x1b [13541.592237] munged S ffffc9000037d3c0 0 919 1 0x00000080 [13541.593674] ffff880036affcd0 0000000000000086 ffff8800368a6dd0 ffff880036afffd8 [13541.595232] ffff880036afffd8 ffff880036afffd8 ffff8800368a6dd0 ffff8800368a6dd0 [13541.596733] ffff8800368a6dd0 0000000000000000 ffff880036affde0 ffffc9000037d3c0 [13541.598303] Call Trace: [13541.599379] [] schedule+0x29/0x70 [13541.600716] [] futex_wait_queue_me+0xc4/0x120 [13541.602128] [] futex_wait+0x179/0x280 [13541.603485] [] ? fsnotify_clear_marks_by_inode+0xa7/0x140 [13541.605005] [] ? kmem_cache_free+0x1bb/0x1f0 [13541.606441] [] do_futex+0xfe/0x5b0 [13541.607753] [] ? mntput+0x24/0x40 [13541.609115] [] ? __fput+0x183/0x260 [13541.610423] [] SyS_futex+0x80/0x180 [13541.611804] [] ? do_notify_resume+0x92/0xb0 [13541.613220] [] system_call_fastpath+0x16/0x1b [13541.614639] munged S ffffc9000037d3c0 0 920 1 0x00000080 [13541.616175] ffff880079307cd0 0000000000000086 ffff8800368a4e70 ffff880079307fd8 [13541.617702] ffff880079307fd8 ffff880079307fd8 ffff8800368a4e70 ffff8800368a4e70 [13541.619315] ffff8800368a4e70 0000000000000000 ffff880079307de0 ffffc9000037d3c0 [13541.620852] Call Trace: [13541.621979] [] schedule+0x29/0x70 [13541.623278] [] futex_wait_queue_me+0xc4/0x120 [13541.624725] [] futex_wait+0x179/0x280 [13541.626094] [] ? fsnotify_clear_marks_by_inode+0xa7/0x140 [13541.627601] [] ? kmem_cache_free+0x1bb/0x1f0 [13541.628993] [] do_futex+0xfe/0x5b0 [13541.630370] [] ? mntput+0x24/0x40 [13541.631669] [] ? __fput+0x183/0x260 [13541.633038] [] SyS_futex+0x80/0x180 [13541.634402] [] ? do_notify_resume+0x92/0xb0 [13541.635830] [] system_call_fastpath+0x16/0x1b [13541.637270] sshd S 0000000000000005 0 921 1 0x00000080 [13541.638766] ffff88007af6b8e8 0000000000000082 ffff8800369b2f10 ffff88007af6bfd8 [13541.640373] ffff88007af6bfd8 ffff88007af6bfd8 ffff8800369b2f10 0000000000000000 [13541.641926] ffff88007af6bad0 00000000fffffffc 0000000000000005 0000000000000005 [13541.643561] Call Trace: [13541.644675] [] schedule+0x29/0x70 [13541.646052] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.647525] [] ? add_wait_queue+0x3c/0x50 [13541.648966] [] ? __pollwait+0x7f/0xf0 [13541.650311] [] schedule_hrtimeout_range+0x13/0x20 [13541.651818] [] poll_schedule_timeout+0x55/0xb0 [13541.653276] [] do_select+0x6d1/0x7c0 [13541.654675] [] ? __alloc_skb+0x8d/0x2d0 [13541.656059] [] ? poll_select_copy_remaining+0x150/0x150 [13541.657616] [] ? poll_select_copy_remaining+0x150/0x150 [13541.659138] [] ? poll_select_copy_remaining+0x150/0x150 [13541.660650] [] ? __alloc_skb+0xa1/0x2d0 [13541.662034] [] ? __alloc_pages_nodemask+0x176/0x420 [13541.663504] [] ? skb_copy_datagram_from_iovec+0x5f/0x290 [13541.664972] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [13541.666472] [] ? cpumask_any_but+0x39/0x50 [13541.667831] [] ? flush_tlb_page+0x39/0xa0 [13541.669250] [] ? lru_cache_add+0xe/0x10 [13541.670580] [] ? page_add_new_anon_rmap+0x91/0x150 [13541.672041] [] ? wp_page_copy.isra.56+0x2e6/0x470 [13541.673494] [] core_sys_select+0x1db/0x300 [13541.674888] [] ? __d_free+0x35/0x40 [13541.676235] [] ? d_free+0x60/0x70 [13541.677577] [] ? dentry_kill+0x14e/0x1b0 [13541.678930] [] ? mntput_no_expire+0x3e/0x120 [13541.680351] [] ? mntput+0x24/0x40 [13541.681634] [] ? __fput+0x183/0x260 [13541.682981] [] SyS_select+0xba/0x110 [13541.684273] [] ? do_notify_resume+0x92/0xb0 [13541.685653] [] system_call_fastpath+0x16/0x1b [13541.687037] nfsiod S ffff880079e91680 0 925 2 0x00000080 [13541.688489] ffff88003683be08 0000000000000046 ffff880036925e20 ffff88003683bfd8 [13541.690065] ffff88003683bfd8 ffff88003683bfd8 ffff880036925e20 ffff880036925e20 [13541.691566] ffff880079e91680 ffff88003628f280 ffff880079e916b0 ffff880079e91680 [13541.693161] Call Trace: [13541.694234] [] schedule+0x29/0x70 [13541.695562] [] rescuer_thread+0x359/0x460 [13541.696947] [] ? __schedule+0x3b0/0x990 [13541.698256] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13541.699711] [] kthread+0xcf/0xe0 [13541.700990] [] ? kthread_create_on_node+0x140/0x140 [13541.702469] [] ret_from_fork+0x58/0x90 [13541.703841] [] ? kthread_create_on_node+0x140/0x140 [13541.705237] kworker/1:1H S ffff88007fd16840 0 930 2 0x00000080 [13541.706780] ffff8800792dbe50 0000000000000046 ffff880036926dd0 ffff8800792dbfd8 [13541.708377] ffff8800792dbfd8 ffff8800792dbfd8 ffff880036926dd0 ffff88007fd16860 [13541.709952] ffff88007a6a9b30 ffff880036926dd0 ffff88007a6a9b00 ffff88007fd16840 [13541.711494] Call Trace: [13541.712590] [] schedule+0x29/0x70 [13541.713948] [] worker_thread+0x1d9/0x410 [13541.715281] [] ? rescuer_thread+0x460/0x460 [13541.716695] [] kthread+0xcf/0xe0 [13541.718023] [] ? kthread_create_on_node+0x140/0x140 [13541.719474] [] ret_from_fork+0x58/0x90 [13541.720792] [] ? kthread_create_on_node+0x140/0x140 [13541.722269] nfsv4.0-svc S ffff8800798c4238 0 931 2 0x00000080 [13541.723748] ffff88007b5e3d80 0000000000000046 ffff880036920fb0 ffff88007b5e3fd8 [13541.725391] ffff88007b5e3fd8 ffff88007b5e3fd8 ffff880036920fb0 ffff880036920fb0 [13541.726996] ffff8800798c4000 ffff88007a6a9c80 0000000000002000 ffff8800798c4238 [13541.728580] Call Trace: [13541.729733] [] schedule+0x29/0x70 [13541.731107] [] schedule_timeout+0x239/0x2d0 [13541.732530] [] ? svc_xprt_enqueue.part.3+0x17/0x20 [sunrpc] [13541.734131] [] svc_recv+0xa2a/0xb00 [sunrpc] [13541.735555] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [13541.737165] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [13541.738746] [] nfs4_callback_svc+0x33/0x60 [nfsv4] [13541.740247] [] kthread+0xcf/0xe0 [13541.741575] [] ? kthread_create_on_node+0x140/0x140 [13541.743113] [] ret_from_fork+0x58/0x90 [13541.744493] [] ? kthread_create_on_node+0x140/0x140 [13541.746054] crond S 0000000000000000 0 1033 1 0x00000080 [13541.747629] ffff88007a3bbe68 0000000000000086 ffff8800368a0fb0 ffff88007a3bbfd8 [13541.749326] ffff88007a3bbfd8 ffff88007a3bbfd8 ffff8800368a0fb0 ffff8800368a0fb0 [13541.750957] ffff88007a3bbeb8 ffff8800368a0fb0 ffff88007a3bbf58 0000000000000000 [13541.752662] Call Trace: [13541.753874] [] schedule+0x29/0x70 [13541.755345] [] do_nanosleep+0x86/0xf0 [13541.756772] [] hrtimer_nanosleep+0xbb/0x180 [13541.758286] [] ? hrtimer_get_res+0x50/0x50 [13541.759737] [] ? do_nanosleep+0x72/0xf0 [13541.761223] [] SyS_nanosleep+0x7e/0x90 [13541.762615] [] system_call_fastpath+0x16/0x1b [13541.764133] atd S 0000000000000000 0 1034 1 0x00000080 [13541.765692] ffff88007a9bbe68 0000000000000082 ffff880079951f60 ffff88007a9bbfd8 [13541.767370] ffff88007a9bbfd8 ffff88007a9bbfd8 ffff880079951f60 ffff880079951f60 [13541.769068] ffff88007a9bbeb8 ffff880079951f60 ffff88007a9bbf58 0000000000000000 [13541.770677] Call Trace: [13541.771895] [] schedule+0x29/0x70 [13541.773284] [] do_nanosleep+0x86/0xf0 [13541.774757] [] hrtimer_nanosleep+0xbb/0x180 [13541.776194] [] ? hrtimer_get_res+0x50/0x50 [13541.777678] [] ? do_nanosleep+0x72/0xf0 [13541.779137] [] SyS_nanosleep+0x7e/0x90 [13541.780570] [] system_call_fastpath+0x16/0x1b [13541.782052] automount S 0000000000000000 0 1036 1 0x00000080 [13541.783604] ffff88007a9fbd90 0000000000000086 ffff8800368fde20 ffff88007a9fbfd8 [13541.785273] ffff88007a9fbfd8 ffff88007a9fbfd8 ffff8800368fde20 ffff8800368fde20 [13541.786885] 7fffffffffffffff ffff88007a9fbed0 ffff8800368fde20 0000000000000000 [13541.788555] Call Trace: [13541.789730] [] schedule+0x29/0x70 [13541.791178] [] schedule_timeout+0x239/0x2d0 [13541.792650] [] ? unlock_page+0x2b/0x30 [13541.794081] [] ? do_read_fault.isra.42+0xe4/0x130 [13541.795529] [] ? recalc_sigpending+0x1b/0x50 [13541.797004] [] schedule_timeout_interruptible+0x1e/0x20 [13541.798549] [] do_sigtimedwait+0x1c2/0x2a0 [13541.799950] [] SYSC_rt_sigtimedwait+0x85/0xf0 [13541.801427] [] ? __do_page_fault+0x171/0x450 [13541.802848] [] ? do_page_fault+0x35/0x90 [13541.804280] [] SyS_rt_sigtimedwait+0xe/0x10 [13541.805718] [] system_call_fastpath+0x16/0x1b [13541.807119] automount S ffffc90000383c00 0 1038 1 0x00000080 [13541.808700] ffff88007a9ffcd0 0000000000000086 ffff8800368faf10 ffff88007a9fffd8 [13541.810347] ffff88007a9fffd8 ffff88007a9fffd8 ffff8800368faf10 ffff8800368faf10 [13541.811939] ffff8800368faf10 ffff88007a9ffd60 ffff88007a9ffde0 ffffc90000383c00 [13541.813590] Call Trace: [13541.814747] [] schedule+0x29/0x70 [13541.816177] [] futex_wait_queue_me+0xc4/0x120 [13541.817642] [] futex_wait+0x179/0x280 [13541.818999] [] ? hrtimer_get_res+0x50/0x50 [13541.820452] [] ? futex_wait_queue_me+0xa2/0x120 [13541.821879] [] do_futex+0xfe/0x5b0 [13541.823264] [] SyS_futex+0x80/0x180 [13541.824647] [] system_call_fastpath+0x16/0x1b [13541.826092] automount S ffffc9000037f700 0 1039 1 0x00000080 [13541.827617] ffff88007a9cbcd0 0000000000000086 ffff8800368f9f60 ffff88007a9cbfd8 [13541.829282] ffff88007a9cbfd8 ffff88007a9cbfd8 ffff8800368f9f60 ffff8800368f9f60 [13541.830881] ffff8800368f9f60 0000000000000000 ffff88007a9cbde0 ffffc9000037f700 [13541.832530] Call Trace: [13541.833693] [] schedule+0x29/0x70 [13541.835135] [] futex_wait_queue_me+0xc4/0x120 [13541.836609] [] futex_wait+0x179/0x280 [13541.837982] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [13541.839573] [] ? try_to_wake_up+0x1c8/0x320 [13541.841082] [] do_futex+0xfe/0x5b0 [13541.842438] [] ? wake_up_new_task+0x10c/0x170 [13541.843923] [] ? do_fork+0xe8/0x2c0 [13541.845293] [] SyS_futex+0x80/0x180 [13541.846696] [] system_call_fastpath+0x16/0x1b [13541.848117] automount S ffff88007bac3bb4 0 1077 1 0x00000080 [13541.849717] ffff88007bac3a30 0000000000000086 ffff88007b548000 ffff88007bac3fd8 [13541.851381] ffff88007bac3fd8 ffff88007bac3fd8 ffff88007b548000 0000000000000000 [13541.853032] ffff88007bac3c90 00000000fffffffc 0000000000000000 ffff88007bac3bb4 [13541.854619] Call Trace: [13541.855851] [] schedule+0x29/0x70 [13541.857278] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.858845] [] ? wake_up_bit+0x25/0x30 [13541.860282] [] ? __brelse+0x3d/0x50 [13541.861672] [] ? add_wait_queue+0x3c/0x50 [13541.863148] [] schedule_hrtimeout_range+0x13/0x20 [13541.864664] [] poll_schedule_timeout+0x55/0xb0 [13541.866131] [] do_sys_poll+0x4cd/0x580 [13541.867583] [] ? __d_alloc+0x25/0x180 [13541.868969] [] ? terminate_walk+0x51/0x60 [13541.870436] [] ? update_curr+0x104/0x190 [13541.871878] [] ? __enqueue_entity+0x78/0x80 [13541.873342] [] ? enqueue_entity+0x26c/0xb60 [13541.874743] [] ? check_preempt_wakeup+0x148/0x250 [13541.876251] [] ? poll_select_copy_remaining+0x150/0x150 [13541.877793] [] ? poll_select_copy_remaining+0x150/0x150 [13541.879294] [] ? poll_select_copy_remaining+0x150/0x150 [13541.880746] [] ? unlock_page+0x2b/0x30 [13541.882148] [] ? do_read_fault.isra.42+0xe4/0x130 [13541.883597] [] ? handle_mm_fault+0x6b1/0xfe0 [13541.884960] [] ? do_futex+0x172/0x5b0 [13541.886331] [] ? __do_page_fault+0x171/0x450 [13541.887696] [] SyS_poll+0x74/0x110 [13541.889066] [] ? page_fault+0x28/0x30 [13541.890426] [] system_call_fastpath+0x16/0x1b [13541.891807] automount S ffff88007b91fbb4 0 1089 1 0x00000080 [13541.893383] ffff88007b91fa30 0000000000000086 ffff880036184e70 ffff88007b91ffd8 [13541.894971] ffff88007b91ffd8 ffff88007b91ffd8 ffff880036184e70 0000000000000000 [13541.896538] ffff88007b91fc90 00000000fffffffc 0000000000000000 ffff88007b91fbb4 [13541.898118] Call Trace: [13541.899235] [] schedule+0x29/0x70 [13541.900615] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.902142] [] ? wake_up_bit+0x25/0x30 [13541.903526] [] ? zone_statistics+0x89/0xa0 [13541.904889] [] ? add_wait_queue+0x3c/0x50 [13541.906310] [] schedule_hrtimeout_range+0x13/0x20 [13541.907772] [] poll_schedule_timeout+0x55/0xb0 [13541.909161] [] do_sys_poll+0x4cd/0x580 [13541.910564] [] ? __d_lookup+0x120/0x160 [13541.911917] [] ? mntput+0x24/0x40 [13541.913268] [] ? terminate_walk+0x51/0x60 [13541.914669] [] ? update_curr+0x104/0x190 [13541.916073] [] ? __enqueue_entity+0x78/0x80 [13541.917437] [] ? enqueue_entity+0x26c/0xb60 [13541.918835] [] ? kvm_clock_read+0x1f/0x30 [13541.920198] [] ? check_preempt_wakeup+0x148/0x250 [13541.921623] [] ? poll_select_copy_remaining+0x150/0x150 [13541.923029] [] ? poll_select_copy_remaining+0x150/0x150 [13541.924488] [] ? poll_select_copy_remaining+0x150/0x150 [13541.925916] [] ? wake_futex+0x66/0x80 [13541.927162] [] ? futex_wake_op+0x332/0x570 [13541.928526] [] ? do_futex+0x172/0x5b0 [13541.929825] [] ? autofs_dev_ioctl+0x13/0x20 [13541.931108] [] ? do_vfs_ioctl+0x2d5/0x4b0 [13541.932448] [] SyS_poll+0x74/0x110 [13541.933713] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.935049] [] system_call_fastpath+0x16/0x1b [13541.936408] automount S ffff88007b383bb4 0 1093 1 0x00000080 [13541.937848] ffff88007b383a30 0000000000000086 ffff880036180000 ffff88007b383fd8 [13541.939395] ffff88007b383fd8 ffff88007b383fd8 ffff880036180000 0000000000000000 [13541.940935] ffff88007b383c90 00000000fffffffc 0000000000000000 ffff88007b383bb4 [13541.942449] Call Trace: [13541.943534] [] schedule+0x29/0x70 [13541.944906] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.946431] [] ? add_wait_queue+0x3c/0x50 [13541.947781] [] schedule_hrtimeout_range+0x13/0x20 [13541.949276] [] poll_schedule_timeout+0x55/0xb0 [13541.950708] [] do_sys_poll+0x4cd/0x580 [13541.952044] [] ? __d_lookup+0x120/0x160 [13541.953454] [] ? mntput+0x24/0x40 [13541.954792] [] ? terminate_walk+0x51/0x60 [13541.956146] [] ? update_curr+0x104/0x190 [13541.957545] [] ? __enqueue_entity+0x78/0x80 [13541.958951] [] ? enqueue_entity+0x26c/0xb60 [13541.960287] [] ? kvm_clock_read+0x1f/0x30 [13541.961664] [] ? check_preempt_wakeup+0x148/0x250 [13541.963035] [] ? poll_select_copy_remaining+0x150/0x150 [13541.964512] [] ? poll_select_copy_remaining+0x150/0x150 [13541.965941] [] ? poll_select_copy_remaining+0x150/0x150 [13541.967315] [] ? wake_futex+0x66/0x80 [13541.968619] [] ? futex_wake_op+0x332/0x570 [13541.969943] [] ? do_futex+0x172/0x5b0 [13541.971185] [] ? mntput+0x24/0x40 [13541.972464] [] ? __fput+0x183/0x260 [13541.973724] [] SyS_poll+0x74/0x110 [13541.974971] [] ? __audit_syscall_exit+0x1e6/0x280 [13541.976273] [] system_call_fastpath+0x16/0x1b [13541.977612] automount S ffff88007a3b7bb4 0 1109 1 0x00000080 [13541.979074] ffff88007a3b7a30 0000000000000086 ffff88007ae8de20 ffff88007a3b7fd8 [13541.980545] ffff88007a3b7fd8 ffff88007a3b7fd8 ffff88007ae8de20 0000000000000000 [13541.982099] ffff88007a3b7c90 00000000fffffffc 0000000000000000 ffff88007a3b7bb4 [13541.983594] Call Trace: [13541.984685] [] schedule+0x29/0x70 [13541.985949] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13541.987432] [] ? add_wait_queue+0x3c/0x50 [13541.988829] [] schedule_hrtimeout_range+0x13/0x20 [13541.990233] [] poll_schedule_timeout+0x55/0xb0 [13541.991669] [] do_sys_poll+0x4cd/0x580 [13541.993060] [] ? kvm_clock_read+0x1f/0x30 [13541.994467] [] ? dequeue_task_fair+0x41e/0x660 [13541.995852] [] ? set_next_entity+0x3c/0xe0 [13541.997268] [] ? __switch_to+0x15c/0x4c0 [13541.998666] [] ? timerqueue_del+0x24/0x70 [13542.000017] [] ? poll_select_copy_remaining+0x150/0x150 [13542.001518] [] ? poll_select_copy_remaining+0x150/0x150 [13542.002945] [] ? poll_select_copy_remaining+0x150/0x150 [13542.004396] [] ? hrtimer_get_res+0x50/0x50 [13542.005758] [] ? get_futex_key+0x1c8/0x2c0 [13542.007114] [] ? futex_wake+0x80/0x160 [13542.008417] [] ? do_futex+0x122/0x5b0 [13542.009706] [] ? wake_up_new_task+0x10c/0x170 [13542.011012] [] SyS_poll+0x74/0x110 [13542.012290] [] ? __audit_syscall_exit+0x1e6/0x280 [13542.013615] [] system_call_fastpath+0x16/0x1b [13542.014978] master S ffff88007b957ea8 0 1076 1 0x00000080 [13542.016430] ffff88007b957db8 0000000000000086 ffff88007b54ce70 ffff88007b957fd8 [13542.017952] ffff88007b957fd8 ffff88007b957fd8 ffff88007b54ce70 ffff88007b957ea8 [13542.019444] 00000000039386ff 0000000000000000 ffff88007b54ce70 ffff88007b957ea8 [13542.020961] Call Trace: [13542.022018] [] schedule+0x29/0x70 [13542.023343] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13542.024756] [] ? hrtimer_get_res+0x50/0x50 [13542.026147] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13542.027642] [] schedule_hrtimeout_range+0x13/0x20 [13542.029025] [] ep_poll+0x23e/0x360 [13542.030375] [] ? wake_up_state+0x20/0x20 [13542.031735] [] SyS_epoll_wait+0xed/0x120 [13542.033157] [] system_call_fastpath+0x16/0x1b [13542.034534] agetty S ffff88007977b400 0 1080 1 0x00000080 [13542.036097] ffff88007b967cf0 0000000000000082 ffff88007ab06dd0 ffff88007b967fd8 [13542.037732] ffff88007b967fd8 ffff88007b967fd8 ffff88007ab06dd0 ffff880079a6cf00 [13542.039378] 7fffffffffffffff ffff880079001800 ffff880079001800 ffff88007977b400 [13542.040990] Call Trace: [13542.042143] [] schedule+0x29/0x70 [13542.043550] [] schedule_timeout+0x239/0x2d0 [13542.045014] [] ? flush_work+0x15b/0x1c0 [13542.046417] [] ? _raw_spin_unlock_irqrestore+0x1b/0x40 [13542.047942] [] ? set_room+0x68/0xc0 [13542.049329] [] n_tty_read+0x27c/0x990 [13542.050665] [] ? wake_up_state+0x20/0x20 [13542.052079] [] tty_read+0x8d/0x100 [13542.053430] [] vfs_read+0x9e/0x170 [13542.054726] [] SyS_read+0x7f/0xe0 [13542.056058] [] system_call_fastpath+0x16/0x1b [13542.057406] agetty S ffff8800367bbc00 0 1081 1 0x00000080 [13542.058928] ffff88007a9c7cf0 0000000000000086 ffff88007ab00000 ffff88007a9c7fd8 [13542.060524] ffff88007a9c7fd8 ffff88007a9c7fd8 ffff88007ab00000 ffff880036859000 [13542.062099] 7fffffffffffffff ffff8800367bb000 ffff8800367bb000 ffff8800367bbc00 [13542.063637] Call Trace: [13542.064794] [] schedule+0x29/0x70 [13542.066153] [] schedule_timeout+0x239/0x2d0 [13542.067516] [] ? notifier_call_chain+0x11/0x70 [13542.068991] [] ? flush_work+0x15b/0x1c0 [13542.070406] [] ? set_room+0x68/0xc0 [13542.071787] [] n_tty_read+0x27c/0x990 [13542.073150] [] ? wake_up_state+0x20/0x20 [13542.074596] [] tty_read+0x8d/0x100 [13542.075930] [] vfs_read+0x9e/0x170 [13542.077319] [] SyS_read+0x7f/0xe0 [13542.078639] [] system_call_fastpath+0x16/0x1b [13542.080094] qmgr S ffff88007af23ea8 0 1104 1076 0x00000080 [13542.081597] ffff88007af23db8 0000000000000082 ffff88007ae8af10 ffff88007af23fd8 [13542.083227] ffff88007af23fd8 ffff88007af23fd8 ffff88007ae8af10 ffff88007af23ea8 [13542.084765] 0000000005f5e100 0000000000000000 ffff88007ae8af10 ffff88007af23ea8 [13542.086396] Call Trace: [13542.087522] [] schedule+0x29/0x70 [13542.088895] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13542.090399] [] ? hrtimer_get_res+0x50/0x50 [13542.091778] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13542.093316] [] schedule_hrtimeout_range+0x13/0x20 [13542.094727] [] ep_poll+0x23e/0x360 [13542.096105] [] ? do_vfs_ioctl+0x2d5/0x4b0 [13542.097462] [] ? wake_up_state+0x20/0x20 [13542.098880] [] SyS_epoll_wait+0xed/0x120 [13542.100236] [] system_call_fastpath+0x16/0x1b [13542.101683] kdmflush S ffff88007b5ab580 0 2532 2 0x00000080 [13542.103222] ffff880079fb7e08 0000000000000046 ffff8800368fedd0 ffff880079fb7fd8 [13542.104775] ffff880079fb7fd8 ffff880079fb7fd8 ffff8800368fedd0 ffff8800368fedd0 [13542.106359] ffff88007b5ab580 ffff880036a87e80 ffff88007b5ab5b0 ffff88007b5ab580 [13542.107908] Call Trace: [13542.109088] [] schedule+0x29/0x70 [13542.110433] [] rescuer_thread+0x359/0x460 [13542.111845] [] ? __schedule+0x3b0/0x990 [13542.113190] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.114681] [] kthread+0xcf/0xe0 [13542.115981] [] ? kthread_create_on_node+0x140/0x140 [13542.117466] [] ret_from_fork+0x58/0x90 [13542.118875] [] ? kthread_create_on_node+0x140/0x140 [13542.120311] bioset S ffff88007b3e0300 0 2533 2 0x00000080 [13542.121901] ffff880079903e08 0000000000000046 ffff880079fc2f10 ffff880079903fd8 [13542.123570] ffff880079903fd8 ffff880079903fd8 ffff880079fc2f10 ffff880079fc2f10 [13542.125230] ffff88007b3e0300 ffff880036a86c80 ffff88007b3e0330 ffff88007b3e0300 [13542.126838] Call Trace: [13542.128020] [] schedule+0x29/0x70 [13542.129364] [] rescuer_thread+0x359/0x460 [13542.130837] [] ? __schedule+0x3b0/0x990 [13542.132209] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.133717] [] kthread+0xcf/0xe0 [13542.135107] [] ? kthread_create_on_node+0x140/0x140 [13542.136616] [] ret_from_fork+0x58/0x90 [13542.138042] [] ? kthread_create_on_node+0x140/0x140 [13542.139517] kdmflush S ffff88007b3e0380 0 2582 2 0x00000080 [13542.141121] ffff88007b5efe08 0000000000000046 ffff880036921f60 ffff88007b5effd8 [13542.142778] ffff88007b5effd8 ffff88007b5effd8 ffff880036921f60 ffff880036921f60 [13542.144460] ffff88007b3e0380 ffff880036a87880 ffff88007b3e03b0 ffff88007b3e0380 [13542.146137] Call Trace: [13542.147344] [] schedule+0x29/0x70 [13542.148815] [] rescuer_thread+0x359/0x460 [13542.150316] [] ? __schedule+0x3b0/0x990 [13542.151729] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.153306] [] kthread+0xcf/0xe0 [13542.154690] [] ? kthread_create_on_node+0x140/0x140 [13542.156252] [] ret_from_fork+0x58/0x90 [13542.157674] [] ? kthread_create_on_node+0x140/0x140 [13542.159237] bioset S ffff88007b968280 0 2584 2 0x00000080 [13542.160823] ffff880079f6fe08 0000000000000046 ffff880079d61f60 ffff880079f6ffd8 [13542.162526] ffff880079f6ffd8 ffff880079f6ffd8 ffff880079d61f60 ffff880079d61f60 [13542.164218] ffff88007b968280 ffff880036a87680 ffff88007b9682b0 ffff88007b968280 [13542.165844] Call Trace: [13542.167090] [] schedule+0x29/0x70 [13542.168541] [] rescuer_thread+0x359/0x460 [13542.170051] [] ? __schedule+0x3b0/0x990 [13542.171542] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.173061] [] kthread+0xcf/0xe0 [13542.174489] [] ? kthread_create_on_node+0x140/0x140 [13542.176045] [] ret_from_fork+0x58/0x90 [13542.177526] [] ? kthread_create_on_node+0x140/0x140 [13542.179026] kdmflush S ffff88007c343180 0 2631 2 0x00000080 [13542.180664] ffff88007b96fe08 0000000000000046 ffff880079d66dd0 ffff88007b96ffd8 [13542.182367] ffff88007b96ffd8 ffff88007b96ffd8 ffff880079d66dd0 ffff880079d66dd0 [13542.184062] ffff88007c343180 ffff88003696c680 ffff88007c3431b0 ffff88007c343180 [13542.185693] Call Trace: [13542.186934] [] schedule+0x29/0x70 [13542.188362] [] rescuer_thread+0x359/0x460 [13542.189877] [] ? __schedule+0x3b0/0x990 [13542.191315] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.192899] [] kthread+0xcf/0xe0 [13542.194270] [] ? kthread_create_on_node+0x140/0x140 [13542.195835] [] ret_from_fork+0x58/0x90 [13542.197269] [] ? kthread_create_on_node+0x140/0x140 [13542.198829] bioset S ffff88007a6a9300 0 2633 2 0x00000080 [13542.200433] ffff88007913fe08 0000000000000046 ffff88007ae8edd0 ffff88007913ffd8 [13542.202129] ffff88007913ffd8 ffff88007913ffd8 ffff88007ae8edd0 ffff88007ae8edd0 [13542.203767] ffff88007a6a9300 ffff88003696c480 ffff88007a6a9330 ffff88007a6a9300 [13542.205463] Call Trace: [13542.206674] [] schedule+0x29/0x70 [13542.208160] [] rescuer_thread+0x359/0x460 [13542.209668] [] ? __schedule+0x3b0/0x990 [13542.211154] [] ? manage_workers.isra.24+0x2d0/0x2d0 [13542.212709] [] kthread+0xcf/0xe0 [13542.214147] [] ? kthread_create_on_node+0x140/0x140 [13542.215700] [] ret_from_fork+0x58/0x90 [13542.217189] [] ? kthread_create_on_node+0x140/0x140 [13542.218736] spl_kmem_cache S ffff88007ab4f800 0 3868 2 0x00000080 [13542.220376] ffff88007aff3e08 0000000000000046 ffff88007b54af10 ffff88007aff3fd8 [13542.222068] ffff88007aff3fd8 ffff88007aff3fd8 ffff88007b54af10 ffff8800361470c0 [13542.223714] ffff88007ab4f890 ffff880046ae0ec0 ffff88007ab4f8b0 ffff88007ab4f800 [13542.225423] Call Trace: [13542.226635] [] schedule+0x29/0x70 [13542.228122] [] taskq_thread+0x3fa/0x440 [spl] [13542.229663] [] ? wake_up_state+0x20/0x20 [13542.231121] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.232710] [] kthread+0xcf/0xe0 [13542.234155] [] ? kthread_create_on_node+0x140/0x140 [13542.235667] [] ret_from_fork+0x58/0x90 [13542.237169] [] ? kthread_create_on_node+0x140/0x140 [13542.238729] spl_kmem_cache S ffff88007ab4f800 0 3869 2 0x00000080 [13542.240380] ffff88007c03be08 0000000000000046 ffff88007b54edd0 ffff88007c03bfd8 [13542.242067] ffff88007c03bfd8 ffff88007c03bfd8 ffff88007b54edd0 ffff8800361478a0 [13542.243719] ffff88007ab4f890 ffff8800620ae980 ffff88007ab4f8b0 ffff88007ab4f800 [13542.245431] Call Trace: [13542.246650] [] schedule+0x29/0x70 [13542.248138] [] taskq_thread+0x3fa/0x440 [spl] [13542.249695] [] ? wake_up_state+0x20/0x20 [13542.251160] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.252748] [] kthread+0xcf/0xe0 [13542.254184] [] ? kthread_create_on_node+0x140/0x140 [13542.255758] [] ret_from_fork+0x58/0x90 [13542.257243] [] ? kthread_create_on_node+0x140/0x140 [13542.258762] spl_kmem_cache S ffff88007ab4f800 0 3870 2 0x00000080 [13542.260417] ffff88007b907e08 0000000000000046 ffff88007b549f60 ffff88007b907fd8 [13542.262124] ffff88007b907fd8 ffff88007b907fd8 ffff88007b549f60 ffff880036147000 [13542.263787] ffff88007ab4f890 ffff88007a0dc2c0 ffff88007ab4f8b0 ffff88007ab4f800 [13542.265496] Call Trace: [13542.266722] [] schedule+0x29/0x70 [13542.268206] [] taskq_thread+0x3fa/0x440 [spl] [13542.269754] [] ? wake_up_state+0x20/0x20 [13542.271235] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.272831] [] kthread+0xcf/0xe0 [13542.274275] [] ? kthread_create_on_node+0x140/0x140 [13542.275850] [] ret_from_fork+0x58/0x90 [13542.277348] [] ? kthread_create_on_node+0x140/0x140 [13542.278925] spl_kmem_cache S ffff88007ab4f800 0 3871 2 0x00000080 [13542.280534] ffff88007b923e08 0000000000000046 ffff88007c075e20 ffff88007b923fd8 [13542.282274] ffff88007b923fd8 ffff88007b923fd8 ffff88007c075e20 ffff880036147d80 [13542.283940] ffff88007ab4f890 ffff8800661b35c0 ffff88007ab4f8b0 ffff88007ab4f800 [13542.285659] Call Trace: [13542.286897] [] schedule+0x29/0x70 [13542.288396] [] taskq_thread+0x3fa/0x440 [spl] [13542.289976] [] ? wake_up_state+0x20/0x20 [13542.291500] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.293091] [] kthread+0xcf/0xe0 [13542.294535] [] ? kthread_create_on_node+0x140/0x140 [13542.296082] [] ret_from_fork+0x58/0x90 [13542.297600] [] ? kthread_create_on_node+0x140/0x140 [13542.299139] spl_system_task S ffff88007ab4fe00 0 3872 2 0x00000080 [13542.300785] ffff88007943fe08 0000000000000046 ffff8800792fde20 ffff88007943ffd8 [13542.302534] ffff88007943ffd8 ffff88007943ffd8 ffff8800792fde20 ffff880036147240 [13542.304269] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.305943] Call Trace: [13542.307210] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.308829] [] schedule+0x29/0x70 [13542.310270] [] taskq_thread+0x3fa/0x440 [spl] [13542.311857] [] ? wake_up_state+0x20/0x20 [13542.313380] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.314931] [] kthread+0xcf/0xe0 [13542.316414] [] ? kthread_create_on_node+0x140/0x140 [13542.318021] [] ret_from_fork+0x58/0x90 [13542.319487] [] ? kthread_create_on_node+0x140/0x140 [13542.321104] spl_system_task S ffff88007ab4fe00 0 3873 2 0x00000080 [13542.322723] ffff88007a91fe08 0000000000000046 ffff88007c7b2f10 ffff88007a91ffd8 [13542.324472] ffff88007a91ffd8 ffff88007a91ffd8 ffff88007c7b2f10 ffff8800798aa6c0 [13542.326213] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.327923] Call Trace: [13542.329199] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.330838] [] schedule+0x29/0x70 [13542.332343] [] taskq_thread+0x3fa/0x440 [spl] [13542.333912] [] ? wake_up_state+0x20/0x20 [13542.335397] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.337019] [] kthread+0xcf/0xe0 [13542.338460] [] ? kthread_create_on_node+0x140/0x140 [13542.340068] [] ret_from_fork+0x58/0x90 [13542.341530] [] ? kthread_create_on_node+0x140/0x140 [13542.343153] spl_system_task S ffff88007ab4fe00 0 3874 2 0x00000080 [13542.344781] ffff88007b273e08 0000000000000046 ffff880079218fb0 ffff88007b273fd8 [13542.346543] ffff88007b273fd8 ffff88007b273fd8 ffff880079218fb0 ffff8800798aaf60 [13542.348269] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.349963] Call Trace: [13542.351248] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.352880] [] schedule+0x29/0x70 [13542.354331] [] taskq_thread+0x3fa/0x440 [spl] [13542.355922] [] ? wake_up_state+0x20/0x20 [13542.357468] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.359046] [] kthread+0xcf/0xe0 [13542.360534] [] ? kthread_create_on_node+0x140/0x140 [13542.362149] [] ret_from_fork+0x58/0x90 [13542.363628] [] ? kthread_create_on_node+0x140/0x140 [13542.365257] spl_system_task S ffff88007ab4fe00 0 3875 2 0x00000080 [13542.366911] ffff8800793fbe08 0000000000000046 ffff88007ab01f60 ffff8800793fbfd8 [13542.368668] ffff8800793fbfd8 ffff8800793fbfd8 ffff88007ab01f60 ffff8800360ed0c0 [13542.370429] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.372176] Call Trace: [13542.373442] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.375227] [] schedule+0x29/0x70 [13542.376680] [] taskq_thread+0x3fa/0x440 [spl] [13542.378292] [] ? wake_up_state+0x20/0x20 [13542.379806] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.381428] [] kthread+0xcf/0xe0 [13542.382924] [] ? kthread_create_on_node+0x140/0x140 [13542.384478] [] ret_from_fork+0x58/0x90 [13542.386027] [] ? kthread_create_on_node+0x140/0x140 [13542.387608] spl_system_task S ffff88007ab4fe00 0 3876 2 0x00000080 [13542.389344] ffff88007b92be08 0000000000000046 ffff880079fc0000 ffff88007b92bfd8 [13542.391062] ffff88007b92bfd8 ffff88007b92bfd8 ffff880079fc0000 ffff8800360ede40 [13542.392803] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.394561] Call Trace: [13542.395832] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.397482] [] schedule+0x29/0x70 [13542.398955] [] taskq_thread+0x3fa/0x440 [spl] [13542.400546] [] ? wake_up_state+0x20/0x20 [13542.402058] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.403702] [] kthread+0xcf/0xe0 [13542.405185] [] ? kthread_create_on_node+0x140/0x140 [13542.406787] [] ret_from_fork+0x58/0x90 [13542.408326] [] ? kthread_create_on_node+0x140/0x140 [13542.409888] spl_system_task S ffff88007ab4fe00 0 3877 2 0x00000080 [13542.411575] ffff88007b9efe08 0000000000000046 ffff880079fc5e20 ffff88007b9effd8 [13542.413400] ffff88007b9effd8 ffff88007b9effd8 ffff880079fc5e20 ffff8800360edb40 [13542.415140] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.416843] Call Trace: [13542.418161] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.419794] [] schedule+0x29/0x70 [13542.421247] [] taskq_thread+0x3fa/0x440 [spl] [13542.422860] [] ? wake_up_state+0x20/0x20 [13542.424426] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.425996] [] kthread+0xcf/0xe0 [13542.427494] [] ? kthread_create_on_node+0x140/0x140 [13542.429101] [] ret_from_fork+0x58/0x90 [13542.430579] [] ? kthread_create_on_node+0x140/0x140 [13542.432202] spl_system_task S ffff88007ab4fe00 0 3878 2 0x00000080 [13542.433861] ffff88007b6b3e08 0000000000000046 ffff8800369ebec0 ffff88007b6b3fd8 [13542.435626] ffff88007b6b3fd8 ffff88007b6b3fd8 ffff8800369ebec0 ffff8800360edf00 [13542.437374] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.439118] Call Trace: [13542.440387] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.442036] [] schedule+0x29/0x70 [13542.443496] [] taskq_thread+0x3fa/0x440 [spl] [13542.445092] [] ? wake_up_state+0x20/0x20 [13542.446594] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.448237] [] kthread+0xcf/0xe0 [13542.449715] [] ? kthread_create_on_node+0x140/0x140 [13542.451329] [] ret_from_fork+0x58/0x90 [13542.452852] [] ? kthread_create_on_node+0x140/0x140 [13542.454407] spl_system_task S ffff88007ab4fe00 0 3879 2 0x00000080 [13542.456107] ffff88007b18fe08 0000000000000046 ffff880079d60fb0 ffff88007b18ffd8 [13542.457856] ffff88007b18ffd8 ffff88007b18ffd8 ffff880079d60fb0 ffff8800360ed120 [13542.459618] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.461353] Call Trace: [13542.462628] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.464276] [] schedule+0x29/0x70 [13542.465774] [] taskq_thread+0x3fa/0x440 [spl] [13542.467369] [] ? wake_up_state+0x20/0x20 [13542.468929] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.470534] [] kthread+0xcf/0xe0 [13542.472028] [] ? kthread_create_on_node+0x140/0x140 [13542.473585] [] ret_from_fork+0x58/0x90 [13542.475114] [] ? kthread_create_on_node+0x140/0x140 [13542.476724] spl_system_task S ffff88007ab4fe00 0 3880 2 0x00000080 [13542.478419] ffff88007910be08 0000000000000046 ffff8800368f8fb0 ffff88007910bfd8 [13542.480174] ffff88007910bfd8 ffff88007910bfd8 ffff8800368f8fb0 ffff8800360ed7e0 [13542.481868] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.483615] Call Trace: [13542.484888] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.486534] [] schedule+0x29/0x70 [13542.488003] [] taskq_thread+0x3fa/0x440 [spl] [13542.489603] [] ? wake_up_state+0x20/0x20 [13542.491111] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.492754] [] kthread+0xcf/0xe0 [13542.494274] [] ? kthread_create_on_node+0x140/0x140 [13542.495828] [] ret_from_fork+0x58/0x90 [13542.497360] [] ? kthread_create_on_node+0x140/0x140 [13542.498918] spl_system_task S ffff88007ab4fe00 0 3881 2 0x00000080 [13542.500619] ffff88007b8a3e08 0000000000000046 ffff8800368fbec0 ffff88007b8a3fd8 [13542.502368] ffff88007b8a3fd8 ffff88007b8a3fd8 ffff8800368fbec0 ffff8800360ed1e0 [13542.504117] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.505809] Call Trace: [13542.507121] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.508756] [] schedule+0x29/0x70 [13542.510255] [] taskq_thread+0x3fa/0x440 [spl] [13542.511788] [] ? wake_up_state+0x20/0x20 [13542.513365] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.514933] [] kthread+0xcf/0xe0 [13542.516422] [] ? kthread_create_on_node+0x140/0x140 [13542.518046] [] ret_from_fork+0x58/0x90 [13542.519561] [] ? kthread_create_on_node+0x140/0x140 [13542.521121] spl_system_task S ffff88007ab4fe00 0 3882 2 0x00000080 [13542.522818] ffff88007afefe08 0000000000000046 ffff8800368f8000 ffff88007afeffd8 [13542.524592] ffff88007afeffd8 ffff88007afeffd8 ffff8800368f8000 ffff8800360edc60 [13542.526354] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.528050] Call Trace: [13542.529356] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.530938] [] schedule+0x29/0x70 [13542.532447] [] taskq_thread+0x3fa/0x440 [spl] [13542.534044] [] ? wake_up_state+0x20/0x20 [13542.535547] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.537184] [] kthread+0xcf/0xe0 [13542.538668] [] ? kthread_create_on_node+0x140/0x140 [13542.540280] [] ret_from_fork+0x58/0x90 [13542.541751] [] ? kthread_create_on_node+0x140/0x140 [13542.543380] spl_system_task S ffff88007ab4fe00 0 3883 2 0x00000080 [13542.545021] ffff88007b94fe08 0000000000000046 ffff8800368fce70 ffff88007b94ffd8 [13542.546771] ffff88007b94ffd8 ffff88007b94ffd8 ffff8800368fce70 ffff8800360edcc0 [13542.548510] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.550274] Call Trace: [13542.551543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.553209] [] schedule+0x29/0x70 [13542.554660] [] taskq_thread+0x3fa/0x440 [spl] [13542.556258] [] ? wake_up_state+0x20/0x20 [13542.557773] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.559411] [] kthread+0xcf/0xe0 [13542.560904] [] ? kthread_create_on_node+0x140/0x140 [13542.562469] [] ret_from_fork+0x58/0x90 [13542.564012] [] ? kthread_create_on_node+0x140/0x140 [13542.565617] spl_system_task S ffff88007ab4fe00 0 3884 2 0x00000080 [13542.567310] ffff8800368ebe08 0000000000000046 ffff88007c369f60 ffff8800368ebfd8 [13542.569013] ffff8800368ebfd8 ffff8800368ebfd8 ffff88007c369f60 ffff8800360eda20 [13542.570757] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.572507] Call Trace: [13542.573775] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.575434] [] schedule+0x29/0x70 [13542.576885] [] taskq_thread+0x3fa/0x440 [spl] [13542.578483] [] ? wake_up_state+0x20/0x20 [13542.579988] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.581614] [] kthread+0xcf/0xe0 [13542.583102] [] ? kthread_create_on_node+0x140/0x140 [13542.584655] [] ret_from_fork+0x58/0x90 [13542.586185] [] ? kthread_create_on_node+0x140/0x140 [13542.587806] spl_system_task S ffff88007ab4fe00 0 3885 2 0x00000080 [13542.589452] ffff8800793bfe08 0000000000000046 ffff88007ae89f60 ffff8800793bffd8 [13542.591202] ffff8800793bffd8 ffff8800793bffd8 ffff88007ae89f60 ffff88003615aae0 [13542.592899] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.594644] Call Trace: [13542.595915] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.597566] [] schedule+0x29/0x70 [13542.599068] [] taskq_thread+0x3fa/0x440 [spl] [13542.600644] [] ? wake_up_state+0x20/0x20 [13542.602196] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.603767] [] kthread+0xcf/0xe0 [13542.605266] [] ? kthread_create_on_node+0x140/0x140 [13542.606823] [] ret_from_fork+0x58/0x90 [13542.608350] [] ? kthread_create_on_node+0x140/0x140 [13542.609917] spl_system_task S ffff88007ab4fe00 0 3886 2 0x00000080 [13542.611620] ffff88007b5bbe08 0000000000000046 ffff8800364c8000 ffff88007b5bbfd8 [13542.613362] ffff88007b5bbfd8 ffff88007b5bbfd8 ffff8800364c8000 ffff88003615a300 [13542.615071] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.616830] Call Trace: [13542.618123] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.619748] [] schedule+0x29/0x70 [13542.621261] [] taskq_thread+0x3fa/0x440 [spl] [13542.622795] [] ? wake_up_state+0x20/0x20 [13542.624358] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.625980] [] kthread+0xcf/0xe0 [13542.627462] [] ? kthread_create_on_node+0x140/0x140 [13542.629019] [] ret_from_fork+0x58/0x90 [13542.630551] [] ? kthread_create_on_node+0x140/0x140 [13542.632159] spl_system_task S ffff88007ab4fe00 0 3887 2 0x00000080 [13542.633795] ffff88007b8fbe08 0000000000000046 ffff8800364c8fb0 ffff88007b8fbfd8 [13542.635561] ffff88007b8fbfd8 ffff88007b8fbfd8 ffff8800364c8fb0 ffff88003615acc0 [13542.637301] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.639006] Call Trace: [13542.640300] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.641890] [] schedule+0x29/0x70 [13542.643408] [] taskq_thread+0x3fa/0x440 [spl] [13542.644947] [] ? wake_up_state+0x20/0x20 [13542.646521] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.648092] [] kthread+0xcf/0xe0 [13542.649581] [] ? kthread_create_on_node+0x140/0x140 [13542.651214] [] ret_from_fork+0x58/0x90 [13542.652736] [] ? kthread_create_on_node+0x140/0x140 [13542.654302] spl_system_task S ffff88007ab4fe00 0 3888 2 0x00000080 [13542.656009] ffff88007b553e08 0000000000000046 ffff8800364c9f60 ffff88007b553fd8 [13542.657756] ffff88007b553fd8 ffff88007b553fd8 ffff8800364c9f60 ffff88003615a9c0 [13542.659522] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.661286] Call Trace: [13542.662556] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.664206] [] schedule+0x29/0x70 [13542.665706] [] taskq_thread+0x3fa/0x440 [spl] [13542.667299] [] ? wake_up_state+0x20/0x20 [13542.668856] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.670478] [] kthread+0xcf/0xe0 [13542.671917] [] ? kthread_create_on_node+0x140/0x140 [13542.673529] [] ret_from_fork+0x58/0x90 [13542.675015] [] ? kthread_create_on_node+0x140/0x140 [13542.676639] spl_system_task S ffff88007ab4fe00 0 3889 2 0x00000080 [13542.678323] ffff880079567e08 0000000000000046 ffff8800364caf10 ffff880079567fd8 [13542.680037] ffff880079567fd8 ffff880079567fd8 ffff8800364caf10 ffff88003615a660 [13542.681759] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.683533] Call Trace: [13542.684807] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.686480] [] schedule+0x29/0x70 [13542.687935] [] taskq_thread+0x3fa/0x440 [spl] [13542.689529] [] ? wake_up_state+0x20/0x20 [13542.691097] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.692661] [] kthread+0xcf/0xe0 [13542.694175] [] ? kthread_create_on_node+0x140/0x140 [13542.695787] [] ret_from_fork+0x58/0x90 [13542.697303] [] ? kthread_create_on_node+0x140/0x140 [13542.698914] spl_system_task S ffff88007ab4fe00 0 3890 2 0x00000080 [13542.700584] ffff88007b26fe08 0000000000000046 ffff8800364cbec0 ffff88007b26ffd8 [13542.702333] ffff88007b26ffd8 ffff88007b26ffd8 ffff8800364cbec0 ffff88003615ad80 [13542.704043] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.705799] Call Trace: [13542.707116] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.708696] [] schedule+0x29/0x70 [13542.710221] [] taskq_thread+0x3fa/0x440 [spl] [13542.711757] [] ? wake_up_state+0x20/0x20 [13542.713323] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.714910] [] kthread+0xcf/0xe0 [13542.716401] [] ? kthread_create_on_node+0x140/0x140 [13542.718034] [] ret_from_fork+0x58/0x90 [13542.719557] [] ? kthread_create_on_node+0x140/0x140 [13542.721196] spl_system_task S ffff88007ab4fe00 0 3891 2 0x00000080 [13542.722841] ffff88007b267e08 0000000000000046 ffff8800364cce70 ffff88007b267fd8 [13542.724609] ffff88007b267fd8 ffff88007b267fd8 ffff8800364cce70 ffff88007926d840 [13542.726350] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.728060] Call Trace: [13542.729359] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.730992] [] schedule+0x29/0x70 [13542.732462] [] taskq_thread+0x3fa/0x440 [spl] [13542.734069] [] ? wake_up_state+0x20/0x20 [13542.735585] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.737216] [] kthread+0xcf/0xe0 [13542.738701] [] ? kthread_create_on_node+0x140/0x140 [13542.740319] [] ret_from_fork+0x58/0x90 [13542.741846] [] ? kthread_create_on_node+0x140/0x140 [13542.743467] spl_system_task S ffff88007ab4fe00 0 3892 2 0x00000080 [13542.745112] ffff88007b253e08 0000000000000046 ffff8800364cde20 ffff88007b253fd8 [13542.746878] ffff88007b253fd8 ffff88007b253fd8 ffff8800364cde20 ffff88007926d1e0 [13542.748627] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.750403] Call Trace: [13542.751675] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.753329] [] schedule+0x29/0x70 [13542.754844] [] taskq_thread+0x3fa/0x440 [spl] [13542.756424] [] ? wake_up_state+0x20/0x20 [13542.757989] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.759556] [] kthread+0xcf/0xe0 [13542.761053] [] ? kthread_create_on_node+0x140/0x140 [13542.762672] [] ret_from_fork+0x58/0x90 [13542.764194] [] ? kthread_create_on_node+0x140/0x140 [13542.765802] spl_system_task S ffff88007ab4fe00 0 3893 2 0x00000080 [13542.767462] ffff88007b24be08 0000000000000046 ffff8800364cedd0 ffff88007b24bfd8 [13542.769206] ffff88007b24bfd8 ffff88007b24bfd8 ffff8800364cedd0 ffff88007926df60 [13542.770917] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.772663] Call Trace: [13542.773936] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.775591] [] schedule+0x29/0x70 [13542.777088] [] taskq_thread+0x3fa/0x440 [spl] [13542.778628] [] ? wake_up_state+0x20/0x20 [13542.780202] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.781823] [] kthread+0xcf/0xe0 [13542.783252] [] ? kthread_create_on_node+0x140/0x140 [13542.784871] [] ret_from_fork+0x58/0x90 [13542.786397] [] ? kthread_create_on_node+0x140/0x140 [13542.787954] spl_system_task S ffff88007ab4fe00 0 3894 2 0x00000080 [13542.789658] ffff88007b247e08 0000000000000046 ffff88007b280000 ffff88007b247fd8 [13542.791409] ffff88007b247fd8 ffff88007b247fd8 ffff88007b280000 ffff88007926df00 [13542.793145] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.794861] Call Trace: [13542.796155] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.797776] [] schedule+0x29/0x70 [13542.799275] [] taskq_thread+0x3fa/0x440 [spl] [13542.800824] [] ? wake_up_state+0x20/0x20 [13542.802418] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.804052] [] kthread+0xcf/0xe0 [13542.805480] [] ? kthread_create_on_node+0x140/0x140 [13542.807100] [] ret_from_fork+0x58/0x90 [13542.808578] [] ? kthread_create_on_node+0x140/0x140 [13542.810208] spl_system_task S ffff88007ab4fe00 0 3895 2 0x00000080 [13542.811862] ffff88007b22be08 0000000000000046 ffff88007b280fb0 ffff88007b22bfd8 [13542.813618] ffff88007b22bfd8 ffff88007b22bfd8 ffff88007b280fb0 ffff88007926dea0 [13542.815397] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.817127] Call Trace: [13542.818393] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.820042] [] schedule+0x29/0x70 [13542.821544] [] taskq_thread+0x3fa/0x440 [spl] [13542.823075] [] ? wake_up_state+0x20/0x20 [13542.824635] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.826257] [] kthread+0xcf/0xe0 [13542.827685] [] ? kthread_create_on_node+0x140/0x140 [13542.829301] [] ret_from_fork+0x58/0x90 [13542.830828] [] ? kthread_create_on_node+0x140/0x140 [13542.832426] spl_system_task S ffff88007ab4fe00 0 3896 2 0x00000080 [13542.834071] ffff88007b223e08 0000000000000046 ffff88007b281f60 ffff88007b223fd8 [13542.835821] ffff88007b223fd8 ffff88007b223fd8 ffff88007b281f60 ffff88007926de40 [13542.837554] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.839292] Call Trace: [13542.840555] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.842203] [] schedule+0x29/0x70 [13542.843698] [] taskq_thread+0x3fa/0x440 [spl] [13542.845228] [] ? wake_up_state+0x20/0x20 [13542.846807] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.848412] [] kthread+0xcf/0xe0 [13542.849888] [] ? kthread_create_on_node+0x140/0x140 [13542.851441] [] ret_from_fork+0x58/0x90 [13542.852976] [] ? kthread_create_on_node+0x140/0x140 [13542.854542] spl_system_task S ffff88007ab4fe00 0 3897 2 0x00000080 [13542.856236] ffff88007b21fe08 0000000000000046 ffff88007b282f10 ffff88007b21ffd8 [13542.857983] ffff88007b21ffd8 ffff88007b21ffd8 ffff88007b282f10 ffff88007926dde0 [13542.859717] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.861477] Call Trace: [13542.862746] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.864407] [] schedule+0x29/0x70 [13542.865864] [] taskq_thread+0x3fa/0x440 [spl] [13542.867459] [] ? wake_up_state+0x20/0x20 [13542.869017] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.870635] [] kthread+0xcf/0xe0 [13542.872118] [] ? kthread_create_on_node+0x140/0x140 [13542.873681] [] ret_from_fork+0x58/0x90 [13542.875210] [] ? kthread_create_on_node+0x140/0x140 [13542.876767] spl_system_task S ffff88007ab4fe00 0 3898 2 0x00000080 [13542.878474] ffff88007b217e08 0000000000000046 ffff88007b283ec0 ffff88007b217fd8 [13542.880218] ffff88007b217fd8 ffff88007b217fd8 ffff88007b283ec0 ffff88007926dd80 [13542.881922] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.883666] Call Trace: [13542.884936] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.886579] [] schedule+0x29/0x70 [13542.888040] [] taskq_thread+0x3fa/0x440 [spl] [13542.889633] [] ? wake_up_state+0x20/0x20 [13542.891188] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.892752] [] kthread+0xcf/0xe0 [13542.894255] [] ? kthread_create_on_node+0x140/0x140 [13542.895810] [] ret_from_fork+0x58/0x90 [13542.897343] [] ? kthread_create_on_node+0x140/0x140 [13542.898950] spl_system_task S ffff88007ab4fe00 0 3899 2 0x00000080 [13542.900607] ffff88007b203e08 0000000000000046 ffff88007b284e70 ffff88007b203fd8 [13542.902383] ffff88007b203fd8 ffff88007b203fd8 ffff88007b284e70 ffff88007926dd20 [13542.904171] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.905872] Call Trace: [13542.907169] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.908744] [] schedule+0x29/0x70 [13542.910273] [] taskq_thread+0x3fa/0x440 [spl] [13542.911859] [] ? wake_up_state+0x20/0x20 [13542.913410] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.914986] [] kthread+0xcf/0xe0 [13542.916471] [] ? kthread_create_on_node+0x140/0x140 [13542.918079] [] ret_from_fork+0x58/0x90 [13542.919606] [] ? kthread_create_on_node+0x140/0x140 [13542.921166] spl_system_task S ffff88007ab4fe00 0 3900 2 0x00000080 [13542.922861] ffff88007ba77e08 0000000000000046 ffff88007b285e20 ffff88007ba77fd8 [13542.924614] ffff88007ba77fd8 ffff88007ba77fd8 ffff88007b285e20 ffff88007926dcc0 [13542.926362] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.928052] Call Trace: [13542.929366] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.930958] [] schedule+0x29/0x70 [13542.932462] [] taskq_thread+0x3fa/0x440 [spl] [13542.934053] [] ? wake_up_state+0x20/0x20 [13542.935555] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.937197] [] kthread+0xcf/0xe0 [13542.938627] [] ? kthread_create_on_node+0x140/0x140 [13542.940245] [] ret_from_fork+0x58/0x90 [13542.941727] [] ? kthread_create_on_node+0x140/0x140 [13542.943349] spl_system_task S ffff88007ab4fe00 0 3901 2 0x00000080 [13542.945008] ffff88007b2d7e08 0000000000000046 ffff88007b286dd0 ffff88007b2d7fd8 [13542.946747] ffff88007b2d7fd8 ffff88007b2d7fd8 ffff88007b286dd0 ffff88007926dc60 [13542.948511] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.950212] Call Trace: [13542.951476] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.953079] [] schedule+0x29/0x70 [13542.954531] [] taskq_thread+0x3fa/0x440 [spl] [13542.956066] [] ? wake_up_state+0x20/0x20 [13542.957634] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.959198] [] kthread+0xcf/0xe0 [13542.960633] [] ? kthread_create_on_node+0x140/0x140 [13542.962198] [] ret_from_fork+0x58/0x90 [13542.963663] [] ? kthread_create_on_node+0x140/0x140 [13542.965240] spl_system_task S ffff88007ab4fe00 0 3902 2 0x00000080 [13542.966882] ffff88007b2c3e08 0000000000000046 ffff88007b198000 ffff88007b2c3fd8 [13542.968600] ffff88007b2c3fd8 ffff88007b2c3fd8 ffff88007b198000 ffff88007926dc00 [13542.970306] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.972003] Call Trace: [13542.973266] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.974841] [] schedule+0x29/0x70 [13542.976287] [] taskq_thread+0x3fa/0x440 [spl] [13542.977839] [] ? wake_up_state+0x20/0x20 [13542.979339] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.980916] [] kthread+0xcf/0xe0 [13542.982344] [] ? kthread_create_on_node+0x140/0x140 [13542.983897] [] ret_from_fork+0x58/0x90 [13542.985377] [] ? kthread_create_on_node+0x140/0x140 [13542.986932] spl_system_task S ffff88007ab4fe00 0 3903 2 0x00000080 [13542.988589] ffff88007b2bbe08 0000000000000046 ffff88007b198fb0 ffff88007b2bbfd8 [13542.990299] ffff88007b2bbfd8 ffff88007b2bbfd8 ffff88007b198fb0 ffff88007926db40 [13542.992009] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13542.993710] Call Trace: [13542.994977] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13542.996548] [] schedule+0x29/0x70 [13542.998020] [] taskq_thread+0x3fa/0x440 [spl] [13542.999545] [] ? wake_up_state+0x20/0x20 [13543.001049] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.002621] [] kthread+0xcf/0xe0 [13543.004089] [] ? kthread_create_on_node+0x140/0x140 [13543.005712] [] ret_from_fork+0x58/0x90 [13543.007177] [] ? kthread_create_on_node+0x140/0x140 [13543.008728] spl_system_task S ffff88007ab4fe00 0 3904 2 0x00000080 [13543.010408] ffff88007b2b7e08 0000000000000046 ffff88007b199f60 ffff88007b2b7fd8 [13543.012135] ffff88007b2b7fd8 ffff88007b2b7fd8 ffff88007b199f60 ffff88007b204000 [13543.013839] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.015585] Call Trace: [13543.016882] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.018457] [] schedule+0x29/0x70 [13543.019921] [] taskq_thread+0x3fa/0x440 [spl] [13543.021452] [] ? wake_up_state+0x20/0x20 [13543.022955] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.024533] [] kthread+0xcf/0xe0 [13543.025963] [] ? kthread_create_on_node+0x140/0x140 [13543.027529] [] ret_from_fork+0x58/0x90 [13543.028997] [] ? kthread_create_on_node+0x140/0x140 [13543.030553] spl_system_task S ffff88007ab4fe00 0 3905 2 0x00000080 [13543.032208] ffff88007b29be08 0000000000000046 ffff88007b19af10 ffff88007b29bfd8 [13543.033909] ffff88007b29bfd8 ffff88007b29bfd8 ffff88007b19af10 ffff88007b204060 [13543.035627] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.037336] Call Trace: [13543.038595] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.040171] [] schedule+0x29/0x70 [13543.041622] [] taskq_thread+0x3fa/0x440 [spl] [13543.043158] [] ? wake_up_state+0x20/0x20 [13543.044656] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.046231] [] kthread+0xcf/0xe0 [13543.047659] [] ? kthread_create_on_node+0x140/0x140 [13543.049226] [] ret_from_fork+0x58/0x90 [13543.050691] [] ? kthread_create_on_node+0x140/0x140 [13543.052246] spl_system_task S ffff88007ab4fe00 0 3906 2 0x00000080 [13543.053891] ffff88007b297e08 0000000000000046 ffff88007b19bec0 ffff88007b297fd8 [13543.055593] ffff88007b297fd8 ffff88007b297fd8 ffff88007b19bec0 ffff88007b2040c0 [13543.057310] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.059006] Call Trace: [13543.060272] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.061849] [] schedule+0x29/0x70 [13543.063307] [] taskq_thread+0x3fa/0x440 [spl] [13543.064840] [] ? wake_up_state+0x20/0x20 [13543.066340] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.067910] [] kthread+0xcf/0xe0 [13543.069334] [] ? kthread_create_on_node+0x140/0x140 [13543.070895] [] ret_from_fork+0x58/0x90 [13543.072364] [] ? kthread_create_on_node+0x140/0x140 [13543.073925] spl_system_task S ffff88007ab4fe00 0 3907 2 0x00000080 [13543.075577] ffff88007b27be08 0000000000000046 ffff88007b19ce70 ffff88007b27bfd8 [13543.077285] ffff88007b27bfd8 ffff88007b27bfd8 ffff88007b19ce70 ffff88007b204120 [13543.078981] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.080672] Call Trace: [13543.081945] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.083516] [] schedule+0x29/0x70 [13543.084962] [] taskq_thread+0x3fa/0x440 [spl] [13543.086497] [] ? wake_up_state+0x20/0x20 [13543.087997] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.089561] [] kthread+0xcf/0xe0 [13543.090998] [] ? kthread_create_on_node+0x140/0x140 [13543.092546] [] ret_from_fork+0x58/0x90 [13543.094013] [] ? kthread_create_on_node+0x140/0x140 [13543.095568] spl_system_task S ffff88007ab4fe00 0 3908 2 0x00000080 [13543.097202] ffff88007b2afe08 0000000000000046 ffff88007b19de20 ffff88007b2affd8 [13543.098909] ffff88007b2affd8 ffff88007b2affd8 ffff88007b19de20 ffff88007b204180 [13543.100614] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.102311] Call Trace: [13543.103570] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.105142] [] schedule+0x29/0x70 [13543.106591] [] taskq_thread+0x3fa/0x440 [spl] [13543.108121] [] ? wake_up_state+0x20/0x20 [13543.109615] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.111184] [] kthread+0xcf/0xe0 [13543.112608] [] ? kthread_create_on_node+0x140/0x140 [13543.114165] [] ret_from_fork+0x58/0x90 [13543.115634] [] ? kthread_create_on_node+0x140/0x140 [13543.117189] spl_system_task S ffff88007ab4fe00 0 3909 2 0x00000080 [13543.118829] ffff88007b23fe08 0000000000000046 ffff88007b19edd0 ffff88007b23ffd8 [13543.120529] ffff88007b23ffd8 ffff88007b23ffd8 ffff88007b19edd0 ffff88007b2041e0 [13543.122233] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.123937] Call Trace: [13543.125193] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.126769] [] schedule+0x29/0x70 [13543.128211] [] taskq_thread+0x3fa/0x440 [spl] [13543.129736] [] ? wake_up_state+0x20/0x20 [13543.131236] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.132803] [] kthread+0xcf/0xe0 [13543.134224] [] ? kthread_create_on_node+0x140/0x140 [13543.135773] [] ret_from_fork+0x58/0x90 [13543.137240] [] ? kthread_create_on_node+0x140/0x140 [13543.138789] spl_system_task S ffff88007ab4fe00 0 3910 2 0x00000080 [13543.140450] ffff88007bac7e08 0000000000000046 ffff88007b698000 ffff88007bac7fd8 [13543.142156] ffff88007bac7fd8 ffff88007bac7fd8 ffff88007b698000 ffff88007b204240 [13543.143858] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.145547] Call Trace: [13543.146811] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.148390] [] schedule+0x29/0x70 [13543.149841] [] taskq_thread+0x3fa/0x440 [spl] [13543.151368] [] ? wake_up_state+0x20/0x20 [13543.152870] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.154444] [] kthread+0xcf/0xe0 [13543.155872] [] ? kthread_create_on_node+0x140/0x140 [13543.157425] [] ret_from_fork+0x58/0x90 [13543.158912] [] ? kthread_create_on_node+0x140/0x140 [13543.160462] spl_system_task S ffff88007ab4fe00 0 3911 2 0x00000080 [13543.162114] ffff88007afe3e08 0000000000000046 ffff88007b698fb0 ffff88007afe3fd8 [13543.163810] ffff88007afe3fd8 ffff88007afe3fd8 ffff88007b698fb0 ffff88007b2042a0 [13543.165521] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.167221] Call Trace: [13543.168481] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.170069] [] schedule+0x29/0x70 [13543.171534] [] taskq_thread+0x3fa/0x440 [spl] [13543.173068] [] ? wake_up_state+0x20/0x20 [13543.174573] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.176149] [] kthread+0xcf/0xe0 [13543.177573] [] ? kthread_create_on_node+0x140/0x140 [13543.179125] [] ret_from_fork+0x58/0x90 [13543.180607] [] ? kthread_create_on_node+0x140/0x140 [13543.182163] spl_system_task S ffff88007ab4fe00 0 3912 2 0x00000080 [13543.183803] ffff88007afe7e08 0000000000000046 ffff88007b699f60 ffff88007afe7fd8 [13543.185503] ffff88007afe7fd8 ffff88007afe7fd8 ffff88007b699f60 ffff88007b204300 [13543.187218] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.188905] Call Trace: [13543.190163] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.191732] [] schedule+0x29/0x70 [13543.193199] [] taskq_thread+0x3fa/0x440 [spl] [13543.194725] [] ? wake_up_state+0x20/0x20 [13543.196227] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.197802] [] kthread+0xcf/0xe0 [13543.199227] [] ? kthread_create_on_node+0x140/0x140 [13543.200778] [] ret_from_fork+0x58/0x90 [13543.202249] [] ? kthread_create_on_node+0x140/0x140 [13543.203811] spl_system_task S ffff88007ab4fe00 0 3913 2 0x00000080 [13543.205452] ffff88007b25be08 0000000000000046 ffff88007b69af10 ffff88007b25bfd8 [13543.207175] ffff88007b25bfd8 ffff88007b25bfd8 ffff88007b69af10 ffff88007b204360 [13543.208862] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.210569] Call Trace: [13543.211835] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.213406] [] schedule+0x29/0x70 [13543.214855] [] taskq_thread+0x3fa/0x440 [spl] [13543.216399] [] ? wake_up_state+0x20/0x20 [13543.217903] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.219460] [] kthread+0xcf/0xe0 [13543.220908] [] ? kthread_create_on_node+0x140/0x140 [13543.222460] [] ret_from_fork+0x58/0x90 [13543.223928] [] ? kthread_create_on_node+0x140/0x140 [13543.225496] spl_system_task S ffff88007ab4fe00 0 3914 2 0x00000080 [13543.227134] ffff88007b25fe08 0000000000000046 ffff88007b69bec0 ffff88007b25ffd8 [13543.228845] ffff88007b25ffd8 ffff88007b25ffd8 ffff88007b69bec0 ffff88007b2043c0 [13543.230541] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.232263] Call Trace: [13543.233521] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.235094] [] schedule+0x29/0x70 [13543.236535] [] taskq_thread+0x3fa/0x440 [spl] [13543.238075] [] ? wake_up_state+0x20/0x20 [13543.239570] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.241129] [] kthread+0xcf/0xe0 [13543.242567] [] ? kthread_create_on_node+0x140/0x140 [13543.244137] [] ret_from_fork+0x58/0x90 [13543.245600] [] ? kthread_create_on_node+0x140/0x140 [13543.247169] spl_system_task S ffff88007ab4fe00 0 3915 2 0x00000080 [13543.248818] ffff88007b233e08 0000000000000046 ffff88007b69ce70 ffff88007b233fd8 [13543.250519] ffff88007b233fd8 ffff88007b233fd8 ffff88007b69ce70 ffff88007b204420 [13543.252243] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.253929] Call Trace: [13543.255195] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.256920] [] schedule+0x29/0x70 [13543.258370] [] taskq_thread+0x3fa/0x440 [spl] [13543.259921] [] ? wake_up_state+0x20/0x20 [13543.261423] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.262989] [] kthread+0xcf/0xe0 [13543.264434] [] ? kthread_create_on_node+0x140/0x140 [13543.265992] [] ret_from_fork+0x58/0x90 [13543.267460] [] ? kthread_create_on_node+0x140/0x140 [13543.269037] spl_system_task S ffff88007ab4fe00 0 3916 2 0x00000080 [13543.270669] ffff88007b237e08 0000000000000046 ffff88007b69de20 ffff88007b237fd8 [13543.272383] ffff88007b237fd8 ffff88007b237fd8 ffff88007b69de20 ffff88007b204480 [13543.274106] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.275790] Call Trace: [13543.277063] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.278650] [] schedule+0x29/0x70 [13543.280097] [] taskq_thread+0x3fa/0x440 [spl] [13543.281636] [] ? wake_up_state+0x20/0x20 [13543.283155] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.284710] [] kthread+0xcf/0xe0 [13543.286141] [] ? kthread_create_on_node+0x140/0x140 [13543.287702] [] ret_from_fork+0x58/0x90 [13543.289173] [] ? kthread_create_on_node+0x140/0x140 [13543.290731] spl_system_task S ffff88007ab4fe00 0 3917 2 0x00000080 [13543.292390] ffff88007b20be08 0000000000000046 ffff88007b69edd0 ffff88007b20bfd8 [13543.294094] ffff88007b20bfd8 ffff88007b20bfd8 ffff88007b69edd0 ffff88007b2044e0 [13543.295802] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.297496] Call Trace: [13543.298776] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.300348] [] schedule+0x29/0x70 [13543.301796] [] taskq_thread+0x3fa/0x440 [spl] [13543.303325] [] ? wake_up_state+0x20/0x20 [13543.304827] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.306404] [] kthread+0xcf/0xe0 [13543.307836] [] ? kthread_create_on_node+0x140/0x140 [13543.309386] [] ret_from_fork+0x58/0x90 [13543.310852] [] ? kthread_create_on_node+0x140/0x140 [13543.312421] spl_system_task S ffff88007ab4fe00 0 3918 2 0x00000080 [13543.314056] ffff88007b20fe08 0000000000000046 ffff88007b2c8000 ffff88007b20ffd8 [13543.315759] ffff88007b20ffd8 ffff88007b20ffd8 ffff88007b2c8000 ffff88007b204540 [13543.317454] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.319168] Call Trace: [13543.320428] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.322010] [] schedule+0x29/0x70 [13543.323463] [] taskq_thread+0x3fa/0x440 [spl] [13543.325003] [] ? wake_up_state+0x20/0x20 [13543.326503] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.328074] [] kthread+0xcf/0xe0 [13543.329502] [] ? kthread_create_on_node+0x140/0x140 [13543.331056] [] ret_from_fork+0x58/0x90 [13543.332523] [] ? kthread_create_on_node+0x140/0x140 [13543.334091] spl_system_task S ffff88007ab4fe00 0 3919 2 0x00000080 [13543.335721] ffff88007b2a7e08 0000000000000046 ffff88007b2c8fb0 ffff88007b2a7fd8 [13543.337480] ffff88007b2a7fd8 ffff88007b2a7fd8 ffff88007b2c8fb0 ffff88007b2045a0 [13543.339823] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.342442] Call Trace: [13543.343798] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.345430] [] schedule+0x29/0x70 [13543.346934] [] taskq_thread+0x3fa/0x440 [spl] [13543.348505] [] ? wake_up_state+0x20/0x20 [13543.350061] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.351663] [] kthread+0xcf/0xe0 [13543.353161] [] ? kthread_create_on_node+0x140/0x140 [13543.354772] [] ret_from_fork+0x58/0x90 [13543.356231] [] ? kthread_create_on_node+0x140/0x140 [13543.357860] spl_system_task S ffff88007ab4fe00 0 3920 2 0x00000080 [13543.359500] ffff88007a883e08 0000000000000046 ffff88007b2c9f60 ffff88007a883fd8 [13543.361248] ffff88007a883fd8 ffff88007a883fd8 ffff88007b2c9f60 ffff88007b204600 [13543.362932] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.364693] Call Trace: [13543.365957] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.367595] [] schedule+0x29/0x70 [13543.369040] [] taskq_thread+0x3fa/0x440 [spl] [13543.370638] [] ? wake_up_state+0x20/0x20 [13543.372184] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.373801] [] kthread+0xcf/0xe0 [13543.375222] [] ? kthread_create_on_node+0x140/0x140 [13543.376847] [] ret_from_fork+0x58/0x90 [13543.378311] [] ? kthread_create_on_node+0x140/0x140 [13543.379921] spl_system_task S ffff88007ab4fe00 0 3921 2 0x00000080 [13543.381577] ffff88007a887e08 0000000000000046 ffff88007b2caf10 ffff88007a887fd8 [13543.383309] ffff88007a887fd8 ffff88007a887fd8 ffff88007b2caf10 ffff88007b204660 [13543.385013] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.386771] Call Trace: [13543.388285] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.389891] [] schedule+0x29/0x70 [13543.391405] [] taskq_thread+0x3fa/0x440 [spl] [13543.392981] [] ? wake_up_state+0x20/0x20 [13543.394531] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.396089] [] kthread+0xcf/0xe0 [13543.397574] [] ? kthread_create_on_node+0x140/0x140 [13543.399192] [] ret_from_fork+0x58/0x90 [13543.400646] [] ? kthread_create_on_node+0x140/0x140 [13543.402272] spl_system_task S ffff88007ab4fe00 0 3922 2 0x00000080 [13543.403899] ffff88007a88be08 0000000000000046 ffff88007b2cbec0 ffff88007a88bfd8 [13543.405666] ffff88007a88bfd8 ffff88007a88bfd8 ffff88007b2cbec0 ffff88007b2046c0 [13543.407412] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.409159] Call Trace: [13543.410421] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.412101] [] schedule+0x29/0x70 [13543.413560] [] taskq_thread+0x3fa/0x440 [spl] [13543.415160] [] ? wake_up_state+0x20/0x20 [13543.416718] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.418327] [] kthread+0xcf/0xe0 [13543.419831] [] ? kthread_create_on_node+0x140/0x140 [13543.421386] [] ret_from_fork+0x58/0x90 [13543.422921] [] ? kthread_create_on_node+0x140/0x140 [13543.424546] spl_system_task S ffff88007ab4fe00 0 3923 2 0x00000080 [13543.426186] ffff88007b6a3e08 0000000000000046 ffff88007b2cce70 ffff88007b6a3fd8 [13543.427941] ffff88007b6a3fd8 ffff88007b6a3fd8 ffff88007b2cce70 ffff88007b204720 [13543.429698] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.431452] Call Trace: [13543.432712] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.434383] [] schedule+0x29/0x70 [13543.435827] [] taskq_thread+0x3fa/0x440 [spl] [13543.437429] [] ? wake_up_state+0x20/0x20 [13543.438976] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.440583] [] kthread+0xcf/0xe0 [13543.442009] [] ? kthread_create_on_node+0x140/0x140 [13543.443623] [] ret_from_fork+0x58/0x90 [13543.445094] [] ? kthread_create_on_node+0x140/0x140 [13543.446704] spl_system_task S ffff88007ab4fe00 0 3924 2 0x00000080 [13543.448384] ffff88007b6a7e08 0000000000000046 ffff88007b2cde20 ffff88007b6a7fd8 [13543.450119] ffff88007b6a7fd8 ffff88007b6a7fd8 ffff88007b2cde20 ffff88007b204780 [13543.451801] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.453540] Call Trace: [13543.454795] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.456446] [] schedule+0x29/0x70 [13543.457882] [] taskq_thread+0x3fa/0x440 [spl] [13543.459474] [] ? wake_up_state+0x20/0x20 [13543.461026] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.462575] [] kthread+0xcf/0xe0 [13543.464061] [] ? kthread_create_on_node+0x140/0x140 [13543.465605] [] ret_from_fork+0x58/0x90 [13543.467132] [] ? kthread_create_on_node+0x140/0x140 [13543.468729] spl_system_task S ffff88007ab4fe00 0 3925 2 0x00000080 [13543.470434] ffff88007b6afe08 0000000000000046 ffff88007b2cedd0 ffff88007b6affd8 [13543.472174] ffff88007b6affd8 ffff88007b6affd8 ffff88007b2cedd0 ffff88007b2047e0 [13543.473858] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.475621] Call Trace: [13543.476889] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.478533] [] schedule+0x29/0x70 [13543.480040] [] taskq_thread+0x3fa/0x440 [spl] [13543.481562] [] ? wake_up_state+0x20/0x20 [13543.483144] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.484717] [] kthread+0xcf/0xe0 [13543.486220] [] ? kthread_create_on_node+0x140/0x140 [13543.487795] [] ret_from_fork+0x58/0x90 [13543.489334] [] ? kthread_create_on_node+0x140/0x140 [13543.490956] spl_system_task S ffff88007ab4fe00 0 3926 2 0x00000080 [13543.492598] ffff88007baebe08 0000000000000046 ffff88007bae0000 ffff88007baebfd8 [13543.494357] ffff88007baebfd8 ffff88007baebfd8 ffff88007bae0000 ffff88007b204840 [13543.496041] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.497787] Call Trace: [13543.499111] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.500688] [] schedule+0x29/0x70 [13543.502202] [] taskq_thread+0x3fa/0x440 [spl] [13543.503783] [] ? wake_up_state+0x20/0x20 [13543.505337] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.506898] [] kthread+0xcf/0xe0 [13543.508392] [] ? kthread_create_on_node+0x140/0x140 [13543.509949] [] ret_from_fork+0x58/0x90 [13543.511483] [] ? kthread_create_on_node+0x140/0x140 [13543.513034] spl_system_task S ffff88007ab4fe00 0 3927 2 0x00000080 [13543.514724] ffff88007baefe08 0000000000000046 ffff88007bae0fb0 ffff88007baeffd8 [13543.516473] ffff88007baeffd8 ffff88007baeffd8 ffff88007bae0fb0 ffff88007b2048a0 [13543.518208] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.519886] Call Trace: [13543.521221] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.522841] [] schedule+0x29/0x70 [13543.524344] [] taskq_thread+0x3fa/0x440 [spl] [13543.525916] [] ? wake_up_state+0x20/0x20 [13543.527464] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.529040] [] kthread+0xcf/0xe0 [13543.530530] [] ? kthread_create_on_node+0x140/0x140 [13543.532139] [] ret_from_fork+0x58/0x90 [13543.533650] [] ? kthread_create_on_node+0x140/0x140 [13543.535254] spl_system_task S ffff88007ab4fe00 0 3928 2 0x00000080 [13543.536889] ffff88007ba6be08 0000000000000046 ffff88007bae1f60 ffff88007ba6bfd8 [13543.538650] ffff88007ba6bfd8 ffff88007ba6bfd8 ffff88007bae1f60 ffff88007b204900 [13543.540396] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.542131] Call Trace: [13543.543388] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.545032] [] schedule+0x29/0x70 [13543.546521] [] taskq_thread+0x3fa/0x440 [spl] [13543.548055] [] ? wake_up_state+0x20/0x20 [13543.549623] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.551182] [] kthread+0xcf/0xe0 [13543.552668] [] ? kthread_create_on_node+0x140/0x140 [13543.554268] [] ret_from_fork+0x58/0x90 [13543.555775] [] ? kthread_create_on_node+0x140/0x140 [13543.557379] spl_system_task S ffff88007ab4fe00 0 3929 2 0x00000080 [13543.559017] ffff88007ba6fe08 0000000000000046 ffff88007bae2f10 ffff88007ba6ffd8 [13543.560769] ffff88007ba6ffd8 ffff88007ba6ffd8 ffff88007bae2f10 ffff88007b204960 [13543.562554] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.564335] Call Trace: [13543.565592] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.567238] [] schedule+0x29/0x70 [13543.568681] [] taskq_thread+0x3fa/0x440 [spl] [13543.570273] [] ? wake_up_state+0x20/0x20 [13543.571823] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.573436] [] kthread+0xcf/0xe0 [13543.574906] [] ? kthread_create_on_node+0x140/0x140 [13543.576516] [] ret_from_fork+0x58/0x90 [13543.578039] [] ? kthread_create_on_node+0x140/0x140 [13543.579637] spl_system_task S ffff88007ab4fe00 0 3930 2 0x00000080 [13543.581335] ffff88007b6d3e08 0000000000000046 ffff88007bae3ec0 ffff88007b6d3fd8 [13543.583026] ffff88007b6d3fd8 ffff88007b6d3fd8 ffff88007bae3ec0 ffff88007b2049c0 [13543.584763] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.586499] Call Trace: [13543.587755] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.589404] [] schedule+0x29/0x70 [13543.590905] [] taskq_thread+0x3fa/0x440 [spl] [13543.592436] [] ? wake_up_state+0x20/0x20 [13543.594022] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.595580] [] kthread+0xcf/0xe0 [13543.597075] [] ? kthread_create_on_node+0x140/0x140 [13543.598684] [] ret_from_fork+0x58/0x90 [13543.600154] [] ? kthread_create_on_node+0x140/0x140 [13543.601785] spl_system_task S ffff88007ab4fe00 0 3931 2 0x00000080 [13543.603437] ffff88007b6d7e08 0000000000000046 ffff88007bae4e70 ffff88007b6d7fd8 [13543.605206] ffff88007b6d7fd8 ffff88007b6d7fd8 ffff88007bae4e70 ffff88007b204a20 [13543.606898] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.608648] Call Trace: [13543.609914] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.611574] [] schedule+0x29/0x70 [13543.613064] [] taskq_thread+0x3fa/0x440 [spl] [13543.614605] [] ? wake_up_state+0x20/0x20 [13543.616166] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.617721] [] kthread+0xcf/0xe0 [13543.619212] [] ? kthread_create_on_node+0x140/0x140 [13543.620758] [] ret_from_fork+0x58/0x90 [13543.622285] [] ? kthread_create_on_node+0x140/0x140 [13543.623892] spl_system_task S ffff88007ab4fe00 0 3932 2 0x00000080 [13543.625540] ffff88007b6dbe08 0000000000000046 ffff88007bae5e20 ffff88007b6dbfd8 [13543.627307] ffff88007b6dbfd8 ffff88007b6dbfd8 ffff88007bae5e20 ffff88007b204a80 [13543.629009] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.630752] Call Trace: [13543.632072] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.633654] [] schedule+0x29/0x70 [13543.635179] [] taskq_thread+0x3fa/0x440 [spl] [13543.636732] [] ? wake_up_state+0x20/0x20 [13543.638313] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.639945] [] kthread+0xcf/0xe0 [13543.641420] [] ? kthread_create_on_node+0x140/0x140 [13543.643035] [] ret_from_fork+0x58/0x90 [13543.644557] [] ? kthread_create_on_node+0x140/0x140 [13543.646158] spl_system_task S ffff88007ab4fe00 0 3933 2 0x00000080 [13543.647784] ffff88007b6dfe08 0000000000000046 ffff88007bae6dd0 ffff88007b6dffd8 [13543.649541] ffff88007b6dffd8 ffff88007b6dffd8 ffff88007bae6dd0 ffff88007b204ae0 [13543.651290] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.652980] Call Trace: [13543.654273] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.655838] [] schedule+0x29/0x70 [13543.657358] [] taskq_thread+0x3fa/0x440 [spl] [13543.658881] [] ? wake_up_state+0x20/0x20 [13543.660442] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.662012] [] kthread+0xcf/0xe0 [13543.663497] [] ? kthread_create_on_node+0x140/0x140 [13543.665055] [] ret_from_fork+0x58/0x90 [13543.666579] [] ? kthread_create_on_node+0x140/0x140 [13543.668180] spl_system_task S ffff88007ab4fe00 0 3934 2 0x00000080 [13543.669824] ffff88007b57be08 0000000000000046 ffff88007b570000 ffff88007b57bfd8 [13543.671585] ffff88007b57bfd8 ffff88007b57bfd8 ffff88007b570000 ffff88007b204b40 [13543.673315] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.675006] Call Trace: [13543.676297] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.677914] [] schedule+0x29/0x70 [13543.679410] [] taskq_thread+0x3fa/0x440 [spl] [13543.680936] [] ? wake_up_state+0x20/0x20 [13543.682492] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.684117] [] kthread+0xcf/0xe0 [13543.685586] [] ? kthread_create_on_node+0x140/0x140 [13543.687190] [] ret_from_fork+0x58/0x90 [13543.688649] [] ? kthread_create_on_node+0x140/0x140 [13543.690273] spl_system_task S ffff88007ab4fe00 0 3935 2 0x00000080 [13543.691906] ffff88007b57fe08 0000000000000046 ffff88007b570fb0 ffff88007b57ffd8 [13543.693662] ffff88007b57ffd8 ffff88007b57ffd8 ffff88007b570fb0 ffff88007b204ba0 [13543.695416] ffff88007ab4fe90 ffffffffa06624b0 ffff88007ab4feb0 ffff88007ab4fe00 [13543.697149] Call Trace: [13543.698407] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.700045] [] schedule+0x29/0x70 [13543.701492] [] taskq_thread+0x3fa/0x440 [spl] [13543.703086] [] ? wake_up_state+0x20/0x20 [13543.704581] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.706213] [] kthread+0xcf/0xe0 [13543.707682] [] ? kthread_create_on_node+0x140/0x140 [13543.709245] [] ret_from_fork+0x58/0x90 [13543.710768] [] ? kthread_create_on_node+0x140/0x140 [13543.712333] spl_dynamic_tas S ffff88007ab4fd00 0 3936 2 0x00000080 [13543.714015] ffff88007b583e08 0000000000000046 ffff88007b571f60 ffff88007b583fd8 [13543.715758] ffff88007b583fd8 ffff88007b583fd8 ffff88007b571f60 ffff88007b204c00 [13543.717505] ffff88007ab4fd90 ffffffffa06624b0 ffff88007ab4fdb0 ffff88007ab4fd00 [13543.719233] Call Trace: [13543.720489] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.722142] [] schedule+0x29/0x70 [13543.723639] [] taskq_thread+0x3fa/0x440 [spl] [13543.725168] [] ? wake_up_state+0x20/0x20 [13543.726739] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.728652] [] kthread+0xcf/0xe0 [13543.730136] [] ? kthread_create_on_node+0x140/0x140 [13543.731684] [] ret_from_fork+0x58/0x90 [13543.733149] [] ? kthread_create_on_node+0x140/0x140 [13543.734704] dbu_evict S ffff88007b3d8b00 0 3948 2 0x00000080 [13543.736353] ffff88007ba0fe08 0000000000000046 ffff88007a635e20 ffff88007ba0ffd8 [13543.738057] ffff88007ba0ffd8 ffff88007ba0ffd8 ffff88007a635e20 ffff88007ae0a960 [13543.739750] ffff88007b3d8b90 ffff88004fe75670 ffff88007b3d8bb0 ffff88007b3d8b00 [13543.741442] Call Trace: [13543.742703] [] schedule+0x29/0x70 [13543.744158] [] taskq_thread+0x3fa/0x440 [spl] [13543.745673] [] ? wake_up_state+0x20/0x20 [13543.747171] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.748727] [] kthread+0xcf/0xe0 [13543.750167] [] ? kthread_create_on_node+0x140/0x140 [13543.751723] [] ret_from_fork+0x58/0x90 [13543.753194] [] ? kthread_create_on_node+0x140/0x140 [13543.754742] arc_prune S ffff88007ab96d00 0 3949 2 0x00000080 [13543.756400] ffff88007ba13e08 0000000000000046 ffff88007a636dd0 ffff88007ba13fd8 [13543.758122] ffff88007ba13fd8 ffff88007ba13fd8 ffff88007a636dd0 ffff88007ae0a9c0 [13543.759812] ffff88007ab96d90 ffffffffa06624b0 ffff88007ab96db0 ffff88007ab96d00 [13543.761541] Call Trace: [13543.762797] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.764375] [] schedule+0x29/0x70 [13543.765830] [] taskq_thread+0x3fa/0x440 [spl] [13543.767344] [] ? wake_up_state+0x20/0x20 [13543.768834] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.770396] [] kthread+0xcf/0xe0 [13543.771886] [] ? kthread_create_on_node+0x140/0x140 [13543.773475] [] ret_from_fork+0x58/0x90 [13543.774953] [] ? kthread_create_on_node+0x140/0x140 [13543.776665] arc_prune S ffff88007ab96d00 0 3950 2 0x00000080 [13543.778354] ffff88007ba17e08 0000000000000046 ffff88007ba18000 ffff88007ba17fd8 [13543.780037] ffff88007ba17fd8 ffff88007ba17fd8 ffff88007ba18000 ffff88007ae0aa20 [13543.781756] ffff88007ab96d90 ffffffffa06624b0 ffff88007ab96db0 ffff88007ab96d00 [13543.783472] Call Trace: [13543.784724] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.786380] [] schedule+0x29/0x70 [13543.787868] [] taskq_thread+0x3fa/0x440 [spl] [13543.789434] [] ? wake_up_state+0x20/0x20 [13543.790916] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.792538] [] kthread+0xcf/0xe0 [13543.794017] [] ? kthread_create_on_node+0x140/0x140 [13543.795605] [] ret_from_fork+0x58/0x90 [13543.797059] [] ? kthread_create_on_node+0x140/0x140 [13543.798663] arc_reclaim S 000000000000f888 0 3951 2 0x00000080 [13543.800347] ffff88007ba23d30 0000000000000046 ffff88007ba18fb0 ffff88007ba23fd8 [13543.802038] ffff88007ba23fd8 ffff88007ba23fd8 ffff88007ba18fb0 ffffffff81e380c0 [13543.803765] ffff88007ba23d60 0000000100ca18b9 ffffffff81e380c0 000000000000f888 [13543.805510] Call Trace: [13543.806761] [] schedule+0x29/0x70 [13543.808288] [] schedule_timeout+0x175/0x2d0 [13543.809836] [] ? internal_add_timer+0x70/0x70 [13543.811345] [] ? prepare_to_wait_exclusive+0x56/0x90 [13543.812990] [] __cv_timedwait_common+0xba/0x160 [spl] [13543.814578] [] ? wake_up_atomic_t+0x30/0x30 [13543.816152] [] __cv_timedwait_sig+0x13/0x20 [spl] [13543.817726] [] arc_reclaim_thread+0x109/0x210 [zfs] [13543.819359] [] ? arc_shrink+0xc0/0xc0 [zfs] [13543.820914] [] thread_generic_wrapper+0x71/0x80 [spl] [13543.822522] [] ? __thread_exit+0x20/0x20 [spl] [13543.824094] [] kthread+0xcf/0xe0 [13543.825508] [] ? kthread_create_on_node+0x140/0x140 [13543.827115] [] ret_from_fork+0x58/0x90 [13543.828623] [] ? kthread_create_on_node+0x140/0x140 [13543.830161] arc_user_evicts S 000000000000f888 0 3952 2 0x00000080 [13543.831810] ffff88007ba27d40 0000000000000046 ffff88007ba19f60 ffff88007ba27fd8 [13543.833535] ffff88007ba27fd8 ffff88007ba27fd8 ffff88007ba19f60 ffffffff81e380c0 [13543.835272] ffff88007ba27d70 0000000100ca18d4 ffffffff81e380c0 000000000000f888 [13543.836933] Call Trace: [13543.838202] [] schedule+0x29/0x70 [13543.839672] [] schedule_timeout+0x175/0x2d0 [13543.841157] [] ? internal_add_timer+0x70/0x70 [13543.842722] [] ? prepare_to_wait_exclusive+0x56/0x90 [13543.844337] [] __cv_timedwait_common+0xba/0x160 [spl] [13543.845941] [] ? wake_up_atomic_t+0x30/0x30 [13543.847495] [] ? hdr_recl+0x50/0x50 [zfs] [13543.848971] [] __cv_timedwait_sig+0x13/0x20 [spl] [13543.850574] [] arc_user_evicts_thread+0xad/0x110 [zfs] [13543.852180] [] thread_generic_wrapper+0x71/0x80 [spl] [13543.853765] [] ? __thread_exit+0x20/0x20 [spl] [13543.855261] [] kthread+0xcf/0xe0 [13543.856727] [] ? kthread_create_on_node+0x140/0x140 [13543.858301] [] ret_from_fork+0x58/0x90 [13543.859718] [] ? kthread_create_on_node+0x140/0x140 [13543.861289] l2arc_feed S 000000000000f888 0 3953 2 0x00000080 [13543.862875] ffff88007ba2fc70 0000000000000046 ffff88007ba1af10 ffff88007ba2ffd8 [13543.864571] ffff88007ba2ffd8 ffff88007ba2ffd8 ffff88007ba1af10 ffffffff81e380c0 [13543.866276] ffff88007ba2fca0 0000000100ca18fa ffffffff81e380c0 000000000000f888 [13543.867924] Call Trace: [13543.869195] [] schedule+0x29/0x70 [13543.870653] [] schedule_timeout+0x175/0x2d0 [13543.872137] [] ? internal_add_timer+0x70/0x70 [13543.873679] [] ? prepare_to_wait_exclusive+0x56/0x90 [13543.875267] [] __cv_timedwait_common+0xba/0x160 [spl] [13543.876840] [] ? wake_up_atomic_t+0x30/0x30 [13543.878384] [] ? l2arc_evict+0x280/0x280 [zfs] [13543.879856] [] __cv_timedwait_sig+0x13/0x20 [spl] [13543.881429] [] l2arc_feed_thread+0x78/0xbe0 [zfs] [13543.882978] [] ? dequeue_task_fair+0x41e/0x660 [13543.884506] [] ? __switch_to+0xd9/0x4c0 [13543.885991] [] ? __schedule+0x3b0/0x990 [13543.887464] [] ? l2arc_evict+0x280/0x280 [zfs] [13543.888934] [] thread_generic_wrapper+0x71/0x80 [spl] [13543.890522] [] ? __thread_exit+0x20/0x20 [spl] [13543.892036] [] kthread+0xcf/0xe0 [13543.893459] [] ? kthread_create_on_node+0x140/0x140 [13543.894957] [] ret_from_fork+0x58/0x90 [13543.896428] [] ? kthread_create_on_node+0x140/0x140 [13543.897935] z_unmount S ffff880036879600 0 3954 2 0x00000080 [13543.899575] ffff88007ba33e08 0000000000000046 ffff88007ba1bec0 ffff88007ba33fd8 [13543.901275] ffff88007ba33fd8 ffff88007ba33fd8 ffff88007ba1bec0 ffff88007ae0aa80 [13543.902908] ffff880036879690 ffffffffa06624b0 ffff8800368796b0 ffff880036879600 [13543.904645] Call Trace: [13543.905873] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.907473] [] schedule+0x29/0x70 [13543.908934] [] taskq_thread+0x3fa/0x440 [spl] [13543.910442] [] ? wake_up_state+0x20/0x20 [13543.911873] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13543.913451] [] kthread+0xcf/0xe0 [13543.914888] [] ? kthread_create_on_node+0x140/0x140 [13543.916398] [] ret_from_fork+0x58/0x90 [13543.917898] [] ? kthread_create_on_node+0x140/0x140 [13543.919454] cfs_rh_00 S ffff88007bbfa0b0 0 5899 2 0x00000080 [13543.921042] ffff88007a0fbe28 0000000000000046 ffff88007a120fb0 ffff88007a0fbfd8 [13543.922736] ffff88007a0fbfd8 ffff88007a0fbfd8 ffff88007a120fb0 ffff88007bbfa080 [13543.924435] ffff88007bbfa098 ffff88007bbfa090 0000000000000000 ffff88007bbfa0b0 [13543.926127] Call Trace: [13543.927337] [] schedule+0x29/0x70 [13543.928829] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [13543.930432] [] ? wake_up_atomic_t+0x30/0x30 [13543.931972] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [13543.933559] [] kthread+0xcf/0xe0 [13543.934965] [] ? kthread_create_on_node+0x140/0x140 [13543.936555] [] ret_from_fork+0x58/0x90 [13543.938042] [] ? kthread_create_on_node+0x140/0x140 [13543.939549] cfs_rh_01 S ffff88007bbfa0b0 0 5900 2 0x00000080 [13543.941361] ffff88007a0ffe28 0000000000000046 ffff88007a124e70 ffff88007a0fffd8 [13543.943039] ffff88007a0fffd8 ffff88007a0fffd8 ffff88007a124e70 ffff88007bbfa080 [13543.944727] ffff88007bbfa098 ffff88007bbfa090 0000000000000000 ffff88007bbfa0b0 [13543.946426] Call Trace: [13543.947635] [] schedule+0x29/0x70 [13543.949126] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [13543.950656] [] ? wake_up_atomic_t+0x30/0x30 [13543.952198] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [13543.953757] [] kthread+0xcf/0xe0 [13543.955234] [] ? kthread_create_on_node+0x140/0x140 [13543.956761] [] ret_from_fork+0x58/0x90 [13543.958275] [] ? kthread_create_on_node+0x140/0x140 [13543.959822] obd_zombid S 0000000000000000 0 5905 2 0x00000080 [13543.961504] ffff88007a0c7e40 0000000000000046 ffff88007a126dd0 ffff88007a0c7fd8 [13543.963270] ffff88007a0c7fd8 ffff88007a0c7fd8 ffff88007a126dd0 ffff88007a126dd0 [13543.964974] ffff88007a126dd0 ffff88007a126dd0 0000000000000000 0000000000000000 [13543.966726] Call Trace: [13543.967985] [] schedule+0x29/0x70 [13543.969533] [] obd_zombie_impexp_thread+0xd5/0x1c0 [obdclass] [13543.971185] [] ? wake_up_state+0x20/0x20 [13543.972751] [] ? obd_zombie_impexp_cull+0x930/0x930 [obdclass] [13543.974441] [] kthread+0xcf/0xe0 [13543.975878] [] ? kthread_create_on_node+0x140/0x140 [13543.977524] [] ret_from_fork+0x58/0x90 [13543.978994] [] ? kthread_create_on_node+0x140/0x140 [13543.980614] ptlrpc_hr00_000 S 00000000ffffffff 0 5910 2 0x00000080 [13543.982249] ffff88006a277dc0 0000000000000046 ffff88007a121f60 ffff88006a277fd8 [13543.984017] ffff88006a277fd8 ffff88006a277fd8 ffff88007a121f60 ffff88007b927288 [13543.985755] 0000000000000000 ffff88007a121f60 ffff88007b927280 00000000ffffffff [13543.987502] Call Trace: [13543.988760] [] schedule+0x29/0x70 [13543.990335] [] ptlrpc_hr_main+0x555/0x790 [ptlrpc] [13543.991955] [] ? wake_up_state+0x20/0x20 [13543.993540] [] ? ptlrpc_server_request_get+0x240/0x240 [ptlrpc] [13543.995247] [] kthread+0xcf/0xe0 [13543.996691] [] ? kthread_create_on_node+0x140/0x140 [13543.998341] [] ret_from_fork+0x58/0x90 [13543.999885] [] ? kthread_create_on_node+0x140/0x140 [13544.001461] ptlrpc_hr00_001 S 00000000ffffffff 0 5911 2 0x00000080 [13544.003166] ffff88006a28fdc0 0000000000000046 ffff88007a125e20 ffff88006a28ffd8 [13544.004932] ffff88006a28ffd8 ffff88006a28ffd8 ffff88007a125e20 ffff88007b9272c0 [13544.006688] 0000000000000000 ffff88007a125e20 ffff88007b9272b8 00000000ffffffff [13544.008459] Call Trace: [13544.009734] [] schedule+0x29/0x70 [13544.011325] [] ptlrpc_hr_main+0x555/0x790 [ptlrpc] [13544.012969] [] ? sched_clock_cpu+0x98/0xc0 [13544.014551] [] ? wake_up_state+0x20/0x20 [13544.016150] [] ? ptlrpc_server_request_get+0x240/0x240 [ptlrpc] [13544.017837] [] kthread+0xcf/0xe0 [13544.019374] [] ? kthread_create_on_node+0x140/0x140 [13544.020974] [] ret_from_fork+0x58/0x90 [13544.022535] [] ? kthread_create_on_node+0x140/0x140 [13544.024124] socknal_cd00 S ffffffffa09c4348 0 5914 2 0x00000080 [13544.025832] ffff88006a043d50 0000000000000046 ffff88007a122f10 ffff88006a043fd8 [13544.027657] ffff88006a043fd8 ffff88006a043fd8 ffff88007a122f10 7fffffffffffffff [13544.029506] ffffffffa09c4348 0000000058e52d6c ffff88007a122f10 ffffffffa09c4348 [13544.031281] Call Trace: [13544.032606] [] schedule+0x29/0x70 [13544.034185] [] schedule_timeout+0x239/0x2d0 [13544.035736] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [13544.037397] [] ? wake_up_state+0x20/0x20 [13544.038962] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [13544.040602] [] kthread+0xcf/0xe0 [13544.042144] [] ? kthread_create_on_node+0x140/0x140 [13544.043783] [] ret_from_fork+0x58/0x90 [13544.045336] [] ? kthread_create_on_node+0x140/0x140 [13544.046969] socknal_cd01 S ffffffffa09c4348 0 5915 2 0x00000080 [13544.048637] ffff88006a047d50 0000000000000046 ffff88007b1f1f60 ffff88006a047fd8 [13544.050415] ffff88006a047fd8 ffff88006a047fd8 ffff88007b1f1f60 7fffffffffffffff [13544.052173] ffffffffa09c4348 0000000058e52d6c ffff88007b1f1f60 ffffffffa09c4348 [13544.053875] Call Trace: [13544.055194] [] schedule+0x29/0x70 [13544.056719] [] schedule_timeout+0x239/0x2d0 [13544.058304] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [13544.059881] [] ? wake_up_state+0x20/0x20 [13544.061457] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [13544.063128] [] kthread+0xcf/0xe0 [13544.064887] [] ? kthread_create_on_node+0x140/0x140 [13544.066543] [] ret_from_fork+0x58/0x90 [13544.068090] [] ? kthread_create_on_node+0x140/0x140 [13544.069721] socknal_cd02 S ffffffffa09c4348 0 5916 2 0x00000080 [13544.071426] ffff88006a263d50 0000000000000046 ffff88007b1f2f10 ffff88006a263fd8 [13544.073197] ffff88006a263fd8 ffff88006a263fd8 ffff88007b1f2f10 7fffffffffffffff [13544.074908] ffffffffa09c4348 0000000058e52d6c ffff88007b1f2f10 ffffffffa09c4348 [13544.076665] Call Trace: [13544.077949] [] schedule+0x29/0x70 [13544.079495] [] schedule_timeout+0x239/0x2d0 [13544.081075] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [13544.082712] [] ? wake_up_state+0x20/0x20 [13544.084227] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [13544.085910] [] kthread+0xcf/0xe0 [13544.087404] [] ? kthread_create_on_node+0x140/0x140 [13544.088970] [] ret_from_fork+0x58/0x90 [13544.090508] [] ? kthread_create_on_node+0x140/0x140 [13544.092128] socknal_cd03 S ffffffffa09c4348 0 5917 2 0x00000080 [13544.093763] ffff88006a267d50 0000000000000046 ffff88007a04af10 ffff88006a267fd8 [13544.095528] ffff88006a267fd8 ffff88006a267fd8 ffff88007a04af10 7fffffffffffffff [13544.097269] ffffffffa09c4348 0000000058e524dc ffff88007a04af10 ffffffffa09c4348 [13544.098966] Call Trace: [13544.100283] [] schedule+0x29/0x70 [13544.101789] [] schedule_timeout+0x239/0x2d0 [13544.103438] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [13544.105013] [] ? wake_up_state+0x20/0x20 [13544.106571] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [13544.108177] [] kthread+0xcf/0xe0 [13544.109669] [] ? kthread_create_on_node+0x140/0x140 [13544.111287] [] ret_from_fork+0x58/0x90 [13544.112760] [] ? kthread_create_on_node+0x140/0x140 [13544.114385] socknal_reaper S 000000000000f888 0 5918 2 0x00000080 [13544.116027] ffff88006909bd70 0000000000000046 ffff88007a04de20 ffff88006909bfd8 [13544.117763] ffff88006909bfd8 ffff88006909bfd8 ffff88007a04de20 ffffffff81e380c0 [13544.119509] ffff88006909bda0 0000000100ca1959 ffffffff81e380c0 000000000000f888 [13544.121244] Call Trace: [13544.122508] [] schedule+0x29/0x70 [13544.124031] [] schedule_timeout+0x175/0x2d0 [13544.125582] [] ? internal_add_timer+0x70/0x70 [13544.127100] [] ksocknal_reaper+0x36e/0x3b0 [ksocklnd] [13544.128740] [] ? wake_up_state+0x20/0x20 [13544.130270] [] ? ksocknal_recv_hello+0x830/0x830 [ksocklnd] [13544.131881] [] kthread+0xcf/0xe0 [13544.133383] [] ? kthread_create_on_node+0x140/0x140 [13544.135013] [] ret_from_fork+0x58/0x90 [13544.136533] [] ? kthread_create_on_node+0x140/0x140 [13544.138093] socknal_sd00_00 S ffffc90000bf0008 0 5919 2 0x00000080 [13544.139757] ffff88006909fe18 0000000000000046 ffff88007a048000 ffff88006909ffd8 [13544.141516] ffff88006909ffd8 ffff88006909ffd8 ffff88007a048000 ffffc90000bf0000 [13544.143252] ffffc90000bf0038 ffff88006909fe68 ffffc90000bf0008 ffffc90000bf0008 [13544.144935] Call Trace: [13544.146220] [] schedule+0x29/0x70 [13544.147653] [] ksocknal_scheduler+0x617/0x670 [ksocklnd] [13544.149315] [] ? wake_up_atomic_t+0x30/0x30 [13544.150824] [] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] [13544.152470] [] kthread+0xcf/0xe0 [13544.153903] [] ? kthread_create_on_node+0x140/0x140 [13544.155529] [] ret_from_fork+0x58/0x90 [13544.157010] [] ? kthread_create_on_node+0x140/0x140 [13544.158637] socknal_sd00_01 S ffffc90000bf1868 0 5920 2 0x00000080 [13544.160352] ffff88006a297e18 0000000000000046 ffff88006a298000 ffff88006a297fd8 [13544.162046] ffff88006a297fd8 ffff88006a297fd8 ffff88006a298000 ffffc90000bf1860 [13544.163766] ffffc90000bf1898 ffff88006a297e68 ffffc90000bf1868 ffffc90000bf1868 [13544.165506] Call Trace: [13544.166770] [] schedule+0x29/0x70 [13544.168285] [] ksocknal_scheduler+0x617/0x670 [ksocklnd] [13544.169926] [] ? wake_up_atomic_t+0x30/0x30 [13544.171429] [] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] [13544.173077] [] kthread+0xcf/0xe0 [13544.174548] [] ? kthread_create_on_node+0x140/0x140 [13544.176098] [] ret_from_fork+0x58/0x90 [13544.177631] [] ? kthread_create_on_node+0x140/0x140 [13544.179250] acceptor_000 S 0000000000000000 0 5921 2 0x00000080 [13544.180877] ffff88006a2a3db8 0000000000000046 ffff88006a298fb0 ffff88006a2a3fd8 [13544.182607] ffff88006a2a3fd8 ffff88006a2a3fd8 ffff88006a298fb0 ffff88007068a500 [13544.184296] ffff88006a298fb0 ffff88006a2a3e70 ffff88007bb9ee00 0000000000000000 [13544.185968] Call Trace: [13544.187209] [] schedule+0x29/0x70 [13544.188656] [] lnet_sock_accept+0x15d/0x190 [lnet] [13544.190202] [] ? wake_up_state+0x20/0x20 [13544.191676] [] lnet_acceptor+0x1be/0xc10 [lnet] [13544.193210] [] ? lnet_connect+0x170/0x170 [lnet] [13544.194733] [] kthread+0xcf/0xe0 [13544.196141] [] ? kthread_create_on_node+0x140/0x140 [13544.197675] [] ret_from_fork+0x58/0x90 [13544.199133] [] ? kthread_create_on_node+0x140/0x140 [13544.200673] router_checker S 0000000000000000 0 5922 2 0x00000080 [13544.202303] ffff88006a2a7dd0 0000000000000046 ffff88006a299f60 ffff88006a2a7fd8 [13544.203979] ffff88006a2a7fd8 ffff88006a2a7fd8 ffff88006a299f60 ffff88006a299f60 [13544.205647] 0000000000000000 ffffffffa0a94c80 0000000000000000 0000000000000000 [13544.207327] Call Trace: [13544.208573] [] ? lnet_notify_locked+0x240/0x240 [lnet] [13544.210160] [] schedule+0x29/0x70 [13544.211616] [] lnet_router_checker+0x6a2/0x890 [lnet] [13544.213198] [] ? __switch_to+0xd9/0x4c0 [13544.214679] [] ? __schedule+0x3b0/0x990 [13544.216156] [] ? wake_up_atomic_t+0x30/0x30 [13544.217660] [] ? lnet_notify_locked+0x240/0x240 [lnet] [13544.219244] [] kthread+0xcf/0xe0 [13544.220669] [] ? kthread_create_on_node+0x140/0x140 [13544.222218] [] ret_from_fork+0x58/0x90 [13544.223672] [] ? kthread_create_on_node+0x140/0x140 [13544.225220] ptlrpcd_rcv S 000000000000f888 0 5923 2 0x00000080 [13544.226836] ffff88006a2abcf0 0000000000000046 ffff88006a29af10 ffff88006a2abfd8 [13544.228529] ffff88006a2abfd8 ffff88006a2abfd8 ffff88006a29af10 ffffffff81e380c0 [13544.230214] ffff88006a2abd20 0000000100ca179b ffffffff81e380c0 000000000000f888 [13544.231899] Call Trace: [13544.233141] [] schedule+0x29/0x70 [13544.234567] [] schedule_timeout+0x175/0x2d0 [13544.236119] [] ? internal_add_timer+0x70/0x70 [13544.237686] [] ptlrpcd+0x298/0x560 [ptlrpc] [13544.239179] [] ? wake_up_state+0x20/0x20 [13544.240665] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [13544.242203] [] kthread+0xcf/0xe0 [13544.243607] [] ? kthread_create_on_node+0x140/0x140 [13544.245142] [] ret_from_fork+0x58/0x90 [13544.246584] [] ? kthread_create_on_node+0x140/0x140 [13544.248114] ptlrpcd_00_00 S 000000000000f888 0 5924 2 0x00000080 [13544.249722] ffff88006a2b3cf0 0000000000000046 ffff88006a29bec0 ffff88006a2b3fd8 [13544.251403] ffff88006a2b3fd8 ffff88006a2b3fd8 ffff88006a29bec0 ffffffff81e380c0 [13544.253067] ffff88006a2b3d20 0000000100ca2ece ffffffff81e380c0 000000000000f888 [13544.254709] Call Trace: [13544.255941] [] schedule+0x29/0x70 [13544.257371] [] schedule_timeout+0x175/0x2d0 [13544.258891] [] ? internal_add_timer+0x70/0x70 [13544.260449] [] ptlrpcd+0x298/0x560 [ptlrpc] [13544.261928] [] ? wake_up_state+0x20/0x20 [13544.263404] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [13544.264937] [] kthread+0xcf/0xe0 [13544.266338] [] ? kthread_create_on_node+0x140/0x140 [13544.267858] [] ret_from_fork+0x58/0x90 [13544.269312] [] ? kthread_create_on_node+0x140/0x140 [13544.270838] ptlrpcd_00_01 R running task 0 5925 2 0x00000080 [13544.272453] ffff88006a2b7cf0 0000000000000046 ffff88006a29ce70 ffffffffa0d575c8 [13544.274132] ffff88006a2b7fd8 ffff88006a2b7fd8 ffff88006a29ce70 ffffffff81e380c0 [13544.275784] ffff88006a2b7d20 ffff88004a082bd0 ffffffff81e380c0 000000000000f888 [13544.277454] Call Trace: [13544.278676] [] schedule+0x29/0x70 [13544.280086] [] schedule_timeout+0x175/0x2d0 [13544.281574] [] ? ptlrpc_request_cache_free+0x90/0x1e0 [ptlrpc] [13544.283184] [] ? internal_add_timer+0x70/0x70 [13544.284701] [] ptlrpcd+0x298/0x560 [ptlrpc] [13544.286175] [] ? wake_up_state+0x20/0x20 [13544.287653] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [13544.289193] [] kthread+0xcf/0xe0 [13544.290586] [] ? kthread_create_on_node+0x140/0x140 [13544.292121] [] ret_from_fork+0x58/0x90 [13544.293552] [] ? kthread_create_on_node+0x140/0x140 [13544.295071] ll_ping S 000000000000f888 0 5926 2 0x00000080 [13544.296668] ffff88006a223d20 0000000000000046 ffff88006a29de20 ffff88006a223fd8 [13544.298325] ffff88006a223fd8 ffff88006a223fd8 ffff88006a29de20 ffffffff81e380c0 [13544.299963] ffff88006a223d50 0000000100ca1791 ffffffff81e380c0 000000000000f888 [13544.301613] Call Trace: [13544.302828] [] schedule+0x29/0x70 [13544.304233] [] schedule_timeout+0x175/0x2d0 [13544.305689] [] ? internal_add_timer+0x70/0x70 [13544.307201] [] ptlrpc_pinger_main+0x92c/0xbb0 [ptlrpc] [13544.308757] [] ? wake_up_state+0x20/0x20 [13544.310242] [] ? ptlrpc_obd_ping+0x130/0x130 [ptlrpc] [13544.311779] [] kthread+0xcf/0xe0 [13544.313187] [] ? kthread_create_on_node+0x140/0x140 [13544.314718] [] ret_from_fork+0x58/0x90 [13544.316164] [] ? kthread_create_on_node+0x140/0x140 [13544.317695] sptlrpc_gc S 000000000000f888 0 5927 2 0x00000080 [13544.319317] ffff88006a227d70 0000000000000046 ffff88006a29edd0 ffff88006a227fd8 [13544.320974] ffff88006a227fd8 ffff88006a227fd8 ffff88006a29edd0 ffffffff81e380c0 [13544.322630] ffff88006a227da0 0000000100df379c ffffffff81e380c0 000000000000f888 [13544.324302] Call Trace: [13544.325530] [] schedule+0x29/0x70 [13544.326946] [] schedule_timeout+0x175/0x2d0 [13544.328420] [] ? internal_add_timer+0x70/0x70 [13544.329927] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [13544.331507] [] sec_gc_main+0x2e5/0x420 [ptlrpc] [13544.333023] [] ? wake_up_state+0x20/0x20 [13544.334510] [] ? sptlrpc_gc_add_ctx+0x130/0x130 [ptlrpc] [13544.336087] [] kthread+0xcf/0xe0 [13544.337502] [] ? kthread_create_on_node+0x140/0x140 [13544.339053] [] ret_from_fork+0x58/0x90 [13544.340511] [] ? kthread_create_on_node+0x140/0x140 [13544.342055] kworker/u4:2 S ffff88007c060800 0 17953 2 0x00000080 [13544.343669] ffff880060be7e50 0000000000000046 ffff88005f6d0fb0 ffff880060be7fd8 [13544.345346] ffff880060be7fd8 ffff880060be7fd8 ffff88005f6d0fb0 ffff88007c060820 [13544.347011] ffff880062a67330 ffff88005f6d0fb0 ffff880062a67300 ffff88007c060800 [13544.348665] Call Trace: [13544.349888] [] schedule+0x29/0x70 [13544.351313] [] worker_thread+0x1d9/0x410 [13544.352774] [] ? rescuer_thread+0x460/0x460 [13544.354255] [] kthread+0xcf/0xe0 [13544.355646] [] ? kthread_create_on_node+0x140/0x140 [13544.357170] [] ret_from_fork+0x58/0x90 [13544.358604] [] ? kthread_create_on_node+0x140/0x140 [13544.360133] kworker/1:3 R running task 0 5031 2 0x00000080 [13544.361739] Workqueue: events key_garbage_collector [13544.363530] ffff880066effc40 0000000000000046 ffff880079b3af10 ffff880066efffd8 [13544.365835] ffff880066efffd8 ffff880066efffd8 ffff880079b3af10 ffff880066effd80 [13544.367568] ffff880066effd88 7fffffffffffffff ffff880079b3af10 0000000000000000 [13544.369268] Call Trace: [13544.370503] [] schedule+0x29/0x70 [13544.372003] [] schedule_timeout+0x239/0x2d0 [13544.373534] [] ? __slab_free+0x81/0x2f0 [13544.374979] [] ? insert_work+0x4c/0xa0 [13544.376462] [] ? __slab_free+0x81/0x2f0 [13544.377938] [] wait_for_completion+0x116/0x170 [13544.379427] [] ? wake_up_state+0x20/0x20 [13544.380925] [] ? __call_rcu+0x2c0/0x2c0 [13544.382390] [] wait_rcu_gp+0x5b/0x80 [13544.383826] [] ? ftrace_raw_output_rcu_utilization+0x50/0x50 [13544.385434] [] synchronize_sched+0x3b/0x50 [13544.386930] [] key_garbage_collector+0x1cd/0x390 [13544.388497] [] process_one_work+0x17b/0x470 [13544.389946] [] worker_thread+0x126/0x410 [13544.391416] [] ? rescuer_thread+0x460/0x460 [13544.392890] [] kthread+0xcf/0xe0 [13544.394230] [] ? kthread_create_on_node+0x140/0x140 [13544.395760] [] ret_from_fork+0x58/0x90 [13544.397131] [] ? kthread_create_on_node+0x140/0x140 [13544.398644] kworker/0:2 S ffff88007fc16480 0 9350 2 0x00000080 [13544.400245] ffff880060f2fe50 0000000000000046 ffff880060819f60 ffff880060f2ffd8 [13544.401846] ffff880060f2ffd8 ffff880060f2ffd8 ffff880060819f60 ffff88007fc164a0 [13544.403507] ffff88005e053230 ffff880060819f60 ffff88005e053200 ffff88007fc16480 [13544.405142] Call Trace: [13544.406304] [] schedule+0x29/0x70 [13544.407737] [] worker_thread+0x1d9/0x410 [13544.409184] [] ? rescuer_thread+0x460/0x460 [13544.410650] [] kthread+0xcf/0xe0 [13544.411994] [] ? kthread_create_on_node+0x140/0x140 [13544.413552] [] ret_from_fork+0x58/0x90 [13544.414918] [] ? kthread_create_on_node+0x140/0x140 [13544.416422] z_null_iss S ffff880067849100 0 19724 2 0x00000080 [13544.417956] ffff88005efbbe08 0000000000000046 ffff880067b1de20 ffff88005efbbfd8 [13544.419613] ffff88005efbbfd8 ffff88005efbbfd8 ffff880067b1de20 ffff88005b8d7ea0 [13544.421244] ffff880067849190 ffff8800679d9580 ffff8800678491b0 ffff880067849100 [13544.422815] Call Trace: [13544.424014] [] schedule+0x29/0x70 [13544.425374] [] taskq_thread+0x3fa/0x440 [spl] [13544.426886] [] ? wake_up_state+0x20/0x20 [13544.428338] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.429863] [] kthread+0xcf/0xe0 [13544.431260] [] ? kthread_create_on_node+0x140/0x140 [13544.432720] [] ret_from_fork+0x58/0x90 [13544.434151] [] ? kthread_create_on_node+0x140/0x140 [13544.435593] z_null_int S ffff880067849b00 0 19725 2 0x00000080 [13544.437233] ffff88007c373e08 0000000000000046 ffff880067b19f60 ffff88007c373fd8 [13544.438827] ffff88007c373fd8 ffff88007c373fd8 ffff880067b19f60 ffff88005b8d7780 [13544.440471] ffff880067849b90 ffff8800679d9580 ffff880067849bb0 ffff880067849b00 [13544.442096] Call Trace: [13544.443255] [] schedule+0x29/0x70 [13544.444681] [] taskq_thread+0x3fa/0x440 [spl] [13544.446129] [] ? wake_up_state+0x20/0x20 [13544.447612] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.449167] [] kthread+0xcf/0xe0 [13544.450532] [] ? kthread_create_on_node+0x140/0x140 [13544.452099] [] ret_from_fork+0x58/0x90 [13544.453563] [] ? kthread_create_on_node+0x140/0x140 [13544.455074] z_rd_iss S ffff880067849e00 0 19726 2 0x00000080 [13544.456701] ffff880067c4fe08 0000000000000046 ffff880067b1ce70 ffff880067c4ffd8 [13544.458403] ffff880067c4ffd8 ffff880067c4ffd8 ffff880067b1ce70 ffff88005b8d7360 [13544.460086] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.461714] Call Trace: [13544.462969] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.464507] [] schedule+0x29/0x70 [13544.465993] [] taskq_thread+0x3fa/0x440 [spl] [13544.467494] [] ? wake_up_state+0x20/0x20 [13544.469033] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.470610] [] kthread+0xcf/0xe0 [13544.472004] [] ? kthread_create_on_node+0x140/0x140 [13544.473588] [] ret_from_fork+0x58/0x90 [13544.475034] [] ? kthread_create_on_node+0x140/0x140 [13544.476613] z_rd_iss S ffff880067849e00 0 19727 2 0x00000080 [13544.478225] ffff88005a5a7e08 0000000000000046 ffff880067b1af10 ffff88005a5a7fd8 [13544.479953] ffff88005a5a7fd8 ffff88005a5a7fd8 ffff880067b1af10 ffff88005b8d7d20 [13544.481674] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.483396] Call Trace: [13544.484641] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.486284] [] schedule+0x29/0x70 [13544.487735] [] taskq_thread+0x3fa/0x440 [spl] [13544.489319] [] ? wake_up_state+0x20/0x20 [13544.490782] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.492394] [] kthread+0xcf/0xe0 [13544.493813] [] ? kthread_create_on_node+0x140/0x140 [13544.495418] [] ret_from_fork+0x58/0x90 [13544.496919] [] ? kthread_create_on_node+0x140/0x140 [13544.498459] z_rd_iss S ffff880067849e00 0 19728 2 0x00000080 [13544.500132] ffff88005f64fe08 0000000000000046 ffff880067b1edd0 ffff88005f64ffd8 [13544.501867] ffff88005f64ffd8 ffff88005f64ffd8 ffff880067b1edd0 ffff88005b8d78a0 [13544.503602] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.505321] Call Trace: [13544.506570] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.508227] [] schedule+0x29/0x70 [13544.509717] [] taskq_thread+0x3fa/0x440 [spl] [13544.511281] [] ? wake_up_state+0x20/0x20 [13544.512768] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.514395] [] kthread+0xcf/0xe0 [13544.515816] [] ? kthread_create_on_node+0x140/0x140 [13544.517432] [] ret_from_fork+0x58/0x90 [13544.518894] [] ? kthread_create_on_node+0x140/0x140 [13544.520510] z_rd_iss S ffff880067849e00 0 19729 2 0x00000080 [13544.522147] ffff880049f8be08 0000000000000046 ffff880067b1bec0 ffff880049f8bfd8 [13544.523889] ffff880049f8bfd8 ffff880049f8bfd8 ffff880067b1bec0 ffff88005b8d72a0 [13544.525631] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.527379] Call Trace: [13544.528637] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.530286] [] schedule+0x29/0x70 [13544.531794] [] taskq_thread+0x3fa/0x440 [spl] [13544.533372] [] ? wake_up_state+0x20/0x20 [13544.534868] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.536488] [] kthread+0xcf/0xe0 [13544.537977] [] ? kthread_create_on_node+0x140/0x140 [13544.539585] [] ret_from_fork+0x58/0x90 [13544.541065] [] ? kthread_create_on_node+0x140/0x140 [13544.542681] z_rd_iss S ffff880067849e00 0 19730 2 0x00000080 [13544.544377] ffff880062843e08 0000000000000046 ffff8800666ebec0 ffff880062843fd8 [13544.546127] ffff880062843fd8 ffff880062843fd8 ffff8800666ebec0 ffff88005b8d7c00 [13544.547820] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.549544] Call Trace: [13544.550806] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.552455] [] schedule+0x29/0x70 [13544.553962] [] taskq_thread+0x3fa/0x440 [spl] [13544.555493] [] ? wake_up_state+0x20/0x20 [13544.557063] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.558678] [] kthread+0xcf/0xe0 [13544.560101] [] ? kthread_create_on_node+0x140/0x140 [13544.561723] [] ret_from_fork+0x58/0x90 [13544.563188] [] ? kthread_create_on_node+0x140/0x140 [13544.564814] z_rd_iss S ffff880067849e00 0 19731 2 0x00000080 [13544.566458] ffff880065abbe08 0000000000000046 ffff8800666e9f60 ffff880065abbfd8 [13544.568205] ffff880065abbfd8 ffff880065abbfd8 ffff8800666e9f60 ffff88005b8d7480 [13544.569889] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.571635] Call Trace: [13544.572905] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.574553] [] schedule+0x29/0x70 [13544.576047] [] taskq_thread+0x3fa/0x440 [spl] [13544.577569] [] ? wake_up_state+0x20/0x20 [13544.579133] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.580687] [] kthread+0xcf/0xe0 [13544.582176] [] ? kthread_create_on_node+0x140/0x140 [13544.583725] [] ret_from_fork+0x58/0x90 [13544.585265] [] ? kthread_create_on_node+0x140/0x140 [13544.586814] z_rd_iss S ffff880067849e00 0 19732 2 0x00000080 [13544.588487] ffff8800681b3e08 0000000000000046 ffff8800666e8fb0 ffff8800681b3fd8 [13544.590229] ffff8800681b3fd8 ffff8800681b3fd8 ffff8800666e8fb0 ffff88005b8d7900 [13544.591914] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.593655] Call Trace: [13544.594918] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.596562] [] schedule+0x29/0x70 [13544.598011] [] taskq_thread+0x3fa/0x440 [spl] [13544.599603] [] ? wake_up_state+0x20/0x20 [13544.601150] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.602711] [] kthread+0xcf/0xe0 [13544.604201] [] ? kthread_create_on_node+0x140/0x140 [13544.605798] [] ret_from_fork+0x58/0x90 [13544.607311] [] ? kthread_create_on_node+0x140/0x140 [13544.608910] z_rd_iss S ffff880067849e00 0 19733 2 0x00000080 [13544.610548] ffff880065d57e08 0000000000000046 ffff8800666ece70 ffff880065d57fd8 [13544.612300] ffff880065d57fd8 ffff880065d57fd8 ffff8800666ece70 ffff88005b8d77e0 [13544.613988] ffff880067849e90 ffffffffa06624b0 ffff880067849eb0 ffff880067849e00 [13544.615724] Call Trace: [13544.616988] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.618630] [] schedule+0x29/0x70 [13544.620088] [] taskq_thread+0x3fa/0x440 [spl] [13544.621676] [] ? wake_up_state+0x20/0x20 [13544.623225] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.624828] [] kthread+0xcf/0xe0 [13544.626257] [] ? kthread_create_on_node+0x140/0x140 [13544.627882] [] ret_from_fork+0x58/0x90 [13544.629391] [] ? kthread_create_on_node+0x140/0x140 [13544.630943] z_rd_int_0 S ffff880067849c00 0 19734 2 0x00000080 [13544.632617] ffff88006161be08 0000000000000046 ffff8800666eaf10 ffff88006161bfd8 [13544.634362] ffff88006161bfd8 ffff88006161bfd8 ffff8800666eaf10 ffff88005b8d7420 [13544.636050] ffff880067849c90 ffff880048da59d8 ffff880067849cb0 ffff880067849c00 [13544.637783] Call Trace: [13544.639105] [] schedule+0x29/0x70 [13544.640601] [] taskq_thread+0x3fa/0x440 [spl] [13544.642165] [] ? wake_up_state+0x20/0x20 [13544.643698] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.645313] [] kthread+0xcf/0xe0 [13544.646732] [] ? kthread_create_on_node+0x140/0x140 [13544.648349] [] ret_from_fork+0x58/0x90 [13544.649799] [] ? kthread_create_on_node+0x140/0x140 [13544.651411] z_rd_int_0 S ffff880067849c00 0 19735 2 0x00000080 [13544.653035] ffff880060673e08 0000000000000046 ffff8800666e8000 ffff880060673fd8 [13544.654766] ffff880060673fd8 ffff880060673fd8 ffff8800666e8000 ffff88005b8d7120 [13544.656504] ffff880067849c90 ffff880048da6b38 ffff880067849cb0 ffff880067849c00 [13544.658226] Call Trace: [13544.659471] [] schedule+0x29/0x70 [13544.660983] [] taskq_thread+0x3fa/0x440 [spl] [13544.662498] [] ? wake_up_state+0x20/0x20 [13544.664036] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.665632] [] kthread+0xcf/0xe0 [13544.667064] [] ? kthread_create_on_node+0x140/0x140 [13544.668667] [] ret_from_fork+0x58/0x90 [13544.670116] [] ? kthread_create_on_node+0x140/0x140 [13544.671712] z_rd_int_0 S ffff880067849c00 0 19736 2 0x00000080 [13544.673368] ffff88005f70fe08 0000000000000046 ffff880067546dd0 ffff88005f70ffd8 [13544.675044] ffff88005f70ffd8 ffff88005f70ffd8 ffff880067546dd0 ffff88005b8d7180 [13544.676758] ffff880067849c90 ffff880048da6288 ffff880067849cb0 ffff880067849c00 [13544.678476] Call Trace: [13544.679711] [] schedule+0x29/0x70 [13544.681224] [] taskq_thread+0x3fa/0x440 [spl] [13544.682779] [] ? wake_up_state+0x20/0x20 [13544.684308] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.685892] [] kthread+0xcf/0xe0 [13544.687304] [] ? kthread_create_on_node+0x140/0x140 [13544.688916] [] ret_from_fork+0x58/0x90 [13544.690364] [] ? kthread_create_on_node+0x140/0x140 [13544.691965] z_rd_int_0 S ffff880067849c00 0 19737 2 0x00000080 [13544.693580] ffff88005e71fe08 0000000000000046 ffff880067545e20 ffff88005e71ffd8 [13544.695293] ffff88005e71ffd8 ffff88005e71ffd8 ffff880067545e20 ffff88005b8d7ae0 [13544.696949] ffff880067849c90 ffff880048da6288 ffff880067849cb0 ffff880067849c00 [13544.698680] Call Trace: [13544.699909] [] schedule+0x29/0x70 [13544.701410] [] taskq_thread+0x3fa/0x440 [spl] [13544.702916] [] ? wake_up_state+0x20/0x20 [13544.704448] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.706037] [] kthread+0xcf/0xe0 [13544.707477] [] ? kthread_create_on_node+0x140/0x140 [13544.709052] [] ret_from_fork+0x58/0x90 [13544.710502] [] ? kthread_create_on_node+0x140/0x140 [13544.712087] z_rd_int_0 S ffff880067849c00 0 19738 2 0x00000080 [13544.713693] ffff880060ebfe08 0000000000000046 ffff880067542f10 ffff880060ebffd8 [13544.715414] ffff880060ebffd8 ffff880060ebffd8 ffff880067542f10 ffff88005b8d79c0 [13544.717117] ffff880067849c90 ffff880048da5580 ffff880067849cb0 ffff880067849c00 [13544.718762] Call Trace: [13544.720062] [] schedule+0x29/0x70 [13544.721549] [] taskq_thread+0x3fa/0x440 [spl] [13544.723050] [] ? wake_up_state+0x20/0x20 [13544.724595] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.726187] [] kthread+0xcf/0xe0 [13544.727621] [] ? kthread_create_on_node+0x140/0x140 [13544.729200] [] ret_from_fork+0x58/0x90 [13544.730639] [] ? kthread_create_on_node+0x140/0x140 [13544.732238] z_rd_int_0 S ffff880067849c00 0 19739 2 0x00000080 [13544.733832] ffff88005b367e08 0000000000000046 ffff880067541f60 ffff88005b367fd8 [13544.735559] ffff88005b367fd8 ffff88005b367fd8 ffff880067541f60 ffff88005b8d7600 [13544.737269] ffff880067849c90 ffff880068a933e8 ffff880067849cb0 ffff880067849c00 [13544.738925] Call Trace: [13544.740217] [] schedule+0x29/0x70 [13544.741635] [] taskq_thread+0x3fa/0x440 [spl] [13544.743201] [] ? wake_up_state+0x20/0x20 [13544.744660] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.746246] [] kthread+0xcf/0xe0 [13544.747638] [] ? kthread_create_on_node+0x140/0x140 [13544.749229] [] ret_from_fork+0x58/0x90 [13544.750715] [] ? kthread_create_on_node+0x140/0x140 [13544.752239] z_rd_int_0 S ffff880067849c00 0 19740 2 0x00000080 [13544.753884] ffff880060b5be08 0000000000000046 ffff880067540fb0 ffff880060b5bfd8 [13544.755598] ffff880060b5bfd8 ffff880060b5bfd8 ffff880067540fb0 ffff88005b8d7f60 [13544.757317] ffff880067849c90 ffff88005f9019d8 ffff880067849cb0 ffff880067849c00 [13544.758964] Call Trace: [13544.760218] [] schedule+0x29/0x70 [13544.761635] [] taskq_thread+0x3fa/0x440 [spl] [13544.763208] [] ? wake_up_state+0x20/0x20 [13544.764667] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.766269] [] kthread+0xcf/0xe0 [13544.767705] [] ? kthread_create_on_node+0x140/0x140 [13544.769280] [] ret_from_fork+0x58/0x90 [13544.770715] [] ? kthread_create_on_node+0x140/0x140 [13544.772292] z_rd_int_0 S ffff880067849c00 0 19741 2 0x00000080 [13544.773890] ffff88005f303e08 0000000000000046 ffff880067544e70 ffff88005f303fd8 [13544.775601] ffff88005f303fd8 ffff88005f303fd8 ffff880067544e70 ffff88005b8d7f00 [13544.777297] ffff880067849c90 ffff880048d5f3e8 ffff880067849cb0 ffff880067849c00 [13544.778953] Call Trace: [13544.780204] [] schedule+0x29/0x70 [13544.781618] [] taskq_thread+0x3fa/0x440 [spl] [13544.783177] [] ? wake_up_state+0x20/0x20 [13544.784689] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.786262] [] kthread+0xcf/0xe0 [13544.787710] [] ? kthread_create_on_node+0x140/0x140 [13544.789233] [] ret_from_fork+0x58/0x90 [13544.790728] [] ? kthread_create_on_node+0x140/0x140 [13544.792255] z_rd_int_0 S ffff880067849c00 0 19742 2 0x00000080 [13544.793895] ffff88006343be08 0000000000000046 ffff880067543ec0 ffff88006343bfd8 [13544.795610] ffff88006343bfd8 ffff88006343bfd8 ffff880067543ec0 ffff88005b8d76c0 [13544.797324] ffff880067849c90 ffff880048d5d580 ffff880067849cb0 ffff880067849c00 [13544.798979] Call Trace: [13544.800264] [] schedule+0x29/0x70 [13544.801683] [] taskq_thread+0x3fa/0x440 [spl] [13544.803256] [] ? wake_up_state+0x20/0x20 [13544.804715] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.806310] [] kthread+0xcf/0xe0 [13544.807696] [] ? kthread_create_on_node+0x140/0x140 [13544.809283] [] ret_from_fork+0x58/0x90 [13544.810770] [] ? kthread_create_on_node+0x140/0x140 [13544.812290] z_rd_int_0 S ffff880067849c00 0 19743 2 0x00000080 [13544.813944] ffff88006173fe08 0000000000000046 ffff880067540000 ffff88006173ffd8 [13544.815661] ffff88006173ffd8 ffff88006173ffd8 ffff880067540000 ffff88005b8d7840 [13544.817375] ffff880067849c90 ffff880048da59d8 ffff880067849cb0 ffff880067849c00 [13544.819072] Call Trace: [13544.820294] [] schedule+0x29/0x70 [13544.821781] [] taskq_thread+0x3fa/0x440 [spl] [13544.823283] [] ? wake_up_state+0x20/0x20 [13544.824816] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.826400] [] kthread+0xcf/0xe0 [13544.827787] [] ? kthread_create_on_node+0x140/0x140 [13544.829380] [] ret_from_fork+0x58/0x90 [13544.830822] [] ? kthread_create_on_node+0x140/0x140 [13544.832415] z_rd_int_0 S ffff880067849c00 0 19744 2 0x00000080 [13544.834027] ffff88005a647e08 0000000000000046 ffff880060f7edd0 ffff88005a647fd8 [13544.835733] ffff88005a647fd8 ffff88005a647fd8 ffff880060f7edd0 ffff88005b8d7540 [13544.837457] ffff880067849c90 ffff880048da59d8 ffff880067849cb0 ffff880067849c00 [13544.839172] Call Trace: [13544.840405] [] schedule+0x29/0x70 [13544.841899] [] taskq_thread+0x3fa/0x440 [spl] [13544.843411] [] ? wake_up_state+0x20/0x20 [13544.844965] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.846551] [] kthread+0xcf/0xe0 [13544.848004] [] ? kthread_create_on_node+0x140/0x140 [13544.849580] [] ret_from_fork+0x58/0x90 [13544.851028] [] ? kthread_create_on_node+0x140/0x140 [13544.852608] z_rd_int_0 S ffff880067849c00 0 19745 2 0x00000080 [13544.854223] ffff88005e59be08 0000000000000046 ffff880060f7de20 ffff88005e59bfd8 [13544.855935] ffff88005e59bfd8 ffff88005e59bfd8 ffff880060f7de20 ffff88005b8d7000 [13544.857654] ffff880067849c90 ffff880048da6b38 ffff880067849cb0 ffff880067849c00 [13544.859370] Call Trace: [13544.860594] [] schedule+0x29/0x70 [13544.862097] [] taskq_thread+0x3fa/0x440 [spl] [13544.863643] [] ? wake_up_state+0x20/0x20 [13544.865165] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.866738] [] kthread+0xcf/0xe0 [13544.868132] [] ? kthread_create_on_node+0x140/0x140 [13544.869717] [] ret_from_fork+0x58/0x90 [13544.871210] [] ? kthread_create_on_node+0x140/0x140 [13544.872737] z_rd_int_1 S ffff880067849600 0 19746 2 0x00000080 [13544.874379] ffff880045d3fe08 0000000000000046 ffff880060f7ce70 ffff880045d3ffd8 [13544.876090] ffff880045d3ffd8 ffff880045d3ffd8 ffff880060f7ce70 ffff88005b8d7960 [13544.877742] ffff880067849690 ffff880048da7840 ffff8800678496b0 ffff880067849600 [13544.879458] Call Trace: [13544.880680] [] schedule+0x29/0x70 [13544.882180] [] taskq_thread+0x3fa/0x440 [spl] [13544.883730] [] ? wake_up_state+0x20/0x20 [13544.885242] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.886818] [] kthread+0xcf/0xe0 [13544.888258] [] ? kthread_create_on_node+0x140/0x140 [13544.889777] [] ret_from_fork+0x58/0x90 [13544.891295] [] ? kthread_create_on_node+0x140/0x140 [13544.892806] z_rd_int_1 S ffff880067849600 0 19747 2 0x00000080 [13544.894470] ffff8800683f7e08 0000000000000046 ffff880060f7bec0 ffff8800683f7fd8 [13544.896185] ffff8800683f7fd8 ffff8800683f7fd8 ffff880060f7bec0 ffff88005b8d7a80 [13544.897840] ffff880067849690 ffff880068a94878 ffff8800678496b0 ffff880067849600 [13544.899532] Call Trace: [13544.900754] [] schedule+0x29/0x70 [13544.902251] [] taskq_thread+0x3fa/0x440 [spl] [13544.903746] [] ? wake_up_state+0x20/0x20 [13544.905276] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.906853] [] kthread+0xcf/0xe0 [13544.908248] [] ? kthread_create_on_node+0x140/0x140 [13544.909848] [] ret_from_fork+0x58/0x90 [13544.911294] [] ? kthread_create_on_node+0x140/0x140 [13544.912877] z_rd_int_1 S ffff880067849600 0 19748 2 0x00000080 [13544.914484] ffff88004651be08 0000000000000046 ffff880060f7af10 ffff88004651bfd8 [13544.916193] ffff88004651bfd8 ffff88004651bfd8 ffff880060f7af10 ffff88005b8d7cc0 [13544.917848] ffff880067849690 ffff88005b348420 ffff8800678496b0 ffff880067849600 [13544.919558] Call Trace: [13544.920780] [] schedule+0x29/0x70 [13544.922296] [] taskq_thread+0x3fa/0x440 [spl] [13544.923795] [] ? wake_up_state+0x20/0x20 [13544.925335] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.926860] [] kthread+0xcf/0xe0 [13544.928313] [] ? kthread_create_on_node+0x140/0x140 [13544.929849] [] ret_from_fork+0x58/0x90 [13544.931357] [] ? kthread_create_on_node+0x140/0x140 [13544.932948] z_rd_int_1 S ffff880067849600 0 19749 2 0x00000080 [13544.934773] ffff88005f3e7e08 0000000000000046 ffff880060f79f60 ffff88005f3e7fd8 [13544.936525] ffff88005f3e7fd8 ffff88005f3e7fd8 ffff880060f79f60 ffff88005b8d7240 [13544.938248] ffff880067849690 ffff880048da6288 ffff8800678496b0 ffff880067849600 [13544.939907] Call Trace: [13544.941165] [] schedule+0x29/0x70 [13544.942647] [] taskq_thread+0x3fa/0x440 [spl] [13544.944197] [] ? wake_up_state+0x20/0x20 [13544.945710] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.947240] [] kthread+0xcf/0xe0 [13544.948700] [] ? kthread_create_on_node+0x140/0x140 [13544.950227] [] ret_from_fork+0x58/0x90 [13544.951731] [] ? kthread_create_on_node+0x140/0x140 [13544.953301] z_rd_int_1 S ffff880067849600 0 19750 2 0x00000080 [13544.954897] ffff88005ea8be08 0000000000000046 ffff880060f78fb0 ffff88005ea8bfd8 [13544.956615] ffff88005ea8bfd8 ffff88005ea8bfd8 ffff880060f78fb0 ffff88005b8d7de0 [13544.958319] ffff880067849690 ffff880048d5ef90 ffff8800678496b0 ffff880067849600 [13544.959978] Call Trace: [13544.961239] [] schedule+0x29/0x70 [13544.962669] [] taskq_thread+0x3fa/0x440 [spl] [13544.964237] [] ? wake_up_state+0x20/0x20 [13544.965752] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.967330] [] kthread+0xcf/0xe0 [13544.968722] [] ? finish_task_switch+0x53/0x180 [13544.970278] [] ? kthread_create_on_node+0x140/0x140 [13544.971812] [] ret_from_fork+0x58/0x90 [13544.973310] [] ? kthread_create_on_node+0x140/0x140 [13544.974876] z_rd_int_1 S ffff880067849600 0 19751 2 0x00000080 [13544.976485] ffff880052823e08 0000000000000046 ffff880060f78000 ffff880052823fd8 [13544.978197] ffff880052823fd8 ffff880052823fd8 ffff880060f78000 ffff88007ba97840 [13544.979870] ffff880067849690 ffff880048da6288 ffff8800678496b0 ffff880067849600 [13544.981574] Call Trace: [13544.982799] [] schedule+0x29/0x70 [13544.984301] [] taskq_thread+0x3fa/0x440 [spl] [13544.985844] [] ? wake_up_state+0x20/0x20 [13544.987359] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13544.988927] [] kthread+0xcf/0xe0 [13544.990368] [] ? kthread_create_on_node+0x140/0x140 [13544.991945] [] ret_from_fork+0x58/0x90 [13544.993422] [] ? kthread_create_on_node+0x140/0x140 [13544.994940] z_rd_int_1 S ffff880067849600 0 19752 2 0x00000080 [13544.996593] ffff88005d8f7e08 0000000000000046 ffff880049b1edd0 ffff88005d8f7fd8 [13544.998304] ffff88005d8f7fd8 ffff88005d8f7fd8 ffff880049b1edd0 ffff88007ba97b40 [13544.999964] ffff880067849690 ffff880048da6f90 ffff8800678496b0 ffff880067849600 [13545.001713] Call Trace: [13545.002950] [] schedule+0x29/0x70 [13545.004464] [] taskq_thread+0x3fa/0x440 [spl] [13545.006027] [] ? wake_up_state+0x20/0x20 [13545.007543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.009070] [] kthread+0xcf/0xe0 [13545.010551] [] ? kthread_create_on_node+0x140/0x140 [13545.012111] [] ret_from_fork+0x58/0x90 [13545.013620] [] ? kthread_create_on_node+0x140/0x140 [13545.015209] z_rd_int_1 S ffff880067849600 0 19753 2 0x00000080 [13545.016903] ffff880066dabe08 0000000000000046 ffff880049b1de20 ffff880066dabfd8 [13545.018630] ffff880066dabfd8 ffff880066dabfd8 ffff880049b1de20 ffff88007ba97c00 [13545.020352] ffff880067849690 ffff880048da6f90 ffff8800678496b0 ffff880067849600 [13545.022014] Call Trace: [13545.023277] [] schedule+0x29/0x70 [13545.024749] [] taskq_thread+0x3fa/0x440 [spl] [13545.026254] [] ? wake_up_state+0x20/0x20 [13545.027774] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.029343] [] kthread+0xcf/0xe0 [13545.030742] [] ? kthread_create_on_node+0x140/0x140 [13545.032341] [] ret_from_fork+0x58/0x90 [13545.033785] [] ? kthread_create_on_node+0x140/0x140 [13545.035371] z_rd_int_1 S ffff880067849600 0 19754 2 0x00000080 [13545.036976] ffff88005d0c3e08 0000000000000046 ffff880049b1ce70 ffff88005d0c3fd8 [13545.038693] ffff88005d0c3fd8 ffff88005d0c3fd8 ffff880049b1ce70 ffff88007ba976c0 [13545.040414] ffff880067849690 ffff880048da6f90 ffff8800678496b0 ffff880067849600 [13545.042118] Call Trace: [13545.043346] [] schedule+0x29/0x70 [13545.044839] [] taskq_thread+0x3fa/0x440 [spl] [13545.046391] [] ? wake_up_state+0x20/0x20 [13545.047865] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.049466] [] kthread+0xcf/0xe0 [13545.050914] [] ? kthread_create_on_node+0x140/0x140 [13545.052447] [] ret_from_fork+0x58/0x90 [13545.053957] [] ? kthread_create_on_node+0x140/0x140 [13545.055528] z_rd_int_1 S ffff880067849600 0 19755 2 0x00000080 [13545.057134] ffff880067117e08 0000000000000046 ffff880049b1bec0 ffff880067117fd8 [13545.058842] ffff880067117fd8 ffff880067117fd8 ffff880049b1bec0 ffff88007ba97ba0 [13545.060547] ffff880067849690 ffff880048da6288 ffff8800678496b0 ffff880067849600 [13545.062259] Call Trace: [13545.063486] [] schedule+0x29/0x70 [13545.064990] [] taskq_thread+0x3fa/0x440 [spl] [13545.066507] [] ? wake_up_state+0x20/0x20 [13545.068044] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.069569] [] kthread+0xcf/0xe0 [13545.071037] [] ? kthread_create_on_node+0x140/0x140 [13545.072613] [] ret_from_fork+0x58/0x90 [13545.074113] [] ? kthread_create_on_node+0x140/0x140 [13545.075677] z_rd_int_1 S ffff880067849600 0 19756 2 0x00000080 [13545.077344] ffff88005e05be08 0000000000000046 ffff880049b1af10 ffff88005e05bfd8 [13545.079006] ffff88005e05bfd8 ffff88005e05bfd8 ffff880049b1af10 ffff88007ba97900 [13545.080711] ffff880067849690 ffff880048da6f90 ffff8800678496b0 ffff880067849600 [13545.082419] Call Trace: [13545.083645] [] schedule+0x29/0x70 [13545.085153] [] taskq_thread+0x3fa/0x440 [spl] [13545.086700] [] ? wake_up_state+0x20/0x20 [13545.088176] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.089767] [] kthread+0xcf/0xe0 [13545.091213] [] ? kthread_create_on_node+0x140/0x140 [13545.092791] [] ret_from_fork+0x58/0x90 [13545.094240] [] ? kthread_create_on_node+0x140/0x140 [13545.095835] z_rd_int_1 S ffff880067849600 0 19757 2 0x00000080 [13545.097447] ffff880059a37e08 0000000000000046 ffff880049b19f60 ffff880059a37fd8 [13545.099166] ffff880059a37fd8 ffff880059a37fd8 ffff880049b19f60 ffff88007ba975a0 [13545.100818] ffff880067849690 ffff880068a97c98 ffff8800678496b0 ffff880067849600 [13545.102527] Call Trace: [13545.103754] [] schedule+0x29/0x70 [13545.105263] [] taskq_thread+0x3fa/0x440 [spl] [13545.106769] [] ? wake_up_state+0x20/0x20 [13545.108306] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.109888] [] kthread+0xcf/0xe0 [13545.111358] [] ? kthread_create_on_node+0x140/0x140 [13545.112945] [] ret_from_fork+0x58/0x90 [13545.114435] [] ? kthread_create_on_node+0x140/0x140 [13545.115957] z_rd_int_2 S ffff880067849300 0 19758 2 0x00000080 [13545.117616] ffff8800637a3e08 0000000000000046 ffff880049b18fb0 ffff8800637a3fd8 [13545.119326] ffff8800637a3fd8 ffff8800637a3fd8 ffff880049b18fb0 ffff88007ba97660 [13545.120991] ffff880067849390 ffff880068a966e0 ffff8800678493b0 ffff880067849300 [13545.122720] Call Trace: [13545.123964] [] schedule+0x29/0x70 [13545.125462] [] taskq_thread+0x3fa/0x440 [spl] [13545.127034] [] ? wake_up_state+0x20/0x20 [13545.128560] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.130153] [] kthread+0xcf/0xe0 [13545.131554] [] ? kthread_create_on_node+0x140/0x140 [13545.133164] [] ret_from_fork+0x58/0x90 [13545.134667] [] ? kthread_create_on_node+0x140/0x140 [13545.136187] z_rd_int_2 S ffff880067849300 0 19759 2 0x00000080 [13545.137836] ffff880060603e08 0000000000000046 ffff880049b18000 ffff880060603fd8 [13545.139552] ffff880060603fd8 ffff880060603fd8 ffff880049b18000 ffff88007ba97960 [13545.141289] ffff880067849390 ffff880068a94cd0 ffff8800678493b0 ffff880067849300 [13545.142945] Call Trace: [13545.144222] [] schedule+0x29/0x70 [13545.145695] [] taskq_thread+0x3fa/0x440 [spl] [13545.147247] [] ? wake_up_state+0x20/0x20 [13545.148764] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.150344] [] kthread+0xcf/0xe0 [13545.151789] [] ? kthread_create_on_node+0x140/0x140 [13545.153317] [] ret_from_fork+0x58/0x90 [13545.154830] [] ? kthread_create_on_node+0x140/0x140 [13545.156353] z_rd_int_2 S ffff880067849300 0 19760 2 0x00000080 [13545.158014] ffff88005babfe08 0000000000000046 ffff88005cdaedd0 ffff88005babffd8 [13545.159760] ffff88005babffd8 ffff88005babffd8 ffff88005cdaedd0 ffff88007ba978a0 [13545.161494] ffff880067849390 ffff88005b348420 ffff8800678493b0 ffff880067849300 [13545.163196] Call Trace: [13545.164422] [] schedule+0x29/0x70 [13545.165941] [] taskq_thread+0x3fa/0x440 [spl] [13545.167505] [] ? wake_up_state+0x20/0x20 [13545.169044] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.170638] [] kthread+0xcf/0xe0 [13545.172050] [] ? kthread_create_on_node+0x140/0x140 [13545.173642] [] ret_from_fork+0x58/0x90 [13545.175138] [] ? kthread_create_on_node+0x140/0x140 [13545.176708] z_rd_int_2 S ffff880067849300 0 19761 2 0x00000080 [13545.178351] ffff88006817fe08 0000000000000046 ffff88005cdade20 ffff88006817ffd8 [13545.180017] ffff88006817ffd8 ffff88006817ffd8 ffff88005cdade20 ffff8800360b1000 [13545.181722] ffff880067849390 ffff88005b349e30 ffff8800678493b0 ffff880067849300 [13545.183440] Call Trace: [13545.184665] [] schedule+0x29/0x70 [13545.186176] [] taskq_thread+0x3fa/0x440 [spl] [13545.187739] [] ? wake_up_state+0x20/0x20 [13545.189211] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.190809] [] kthread+0xcf/0xe0 [13545.192215] [] ? kthread_create_on_node+0x140/0x140 [13545.193805] [] ret_from_fork+0x58/0x90 [13545.195300] [] ? kthread_create_on_node+0x140/0x140 [13545.196868] z_rd_int_2 S ffff880067849300 0 19762 2 0x00000080 [13545.198472] ffff880066be7e08 0000000000000046 ffff88005cdace70 ffff880066be7fd8 [13545.200184] ffff880066be7fd8 ffff880066be7fd8 ffff88005cdace70 ffff8800360b17e0 [13545.201839] ffff880067849390 ffff880048da59d8 ffff8800678493b0 ffff880067849300 [13545.203555] Call Trace: [13545.204781] [] schedule+0x29/0x70 [13545.206288] [] taskq_thread+0x3fa/0x440 [spl] [13545.207791] [] ? wake_up_state+0x20/0x20 [13545.209329] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.210910] [] kthread+0xcf/0xe0 [13545.212354] [] ? kthread_create_on_node+0x140/0x140 [13545.213936] [] ret_from_fork+0x58/0x90 [13545.215381] [] ? kthread_create_on_node+0x140/0x140 [13545.216971] z_rd_int_2 S ffff880067849300 0 19763 2 0x00000080 [13545.218582] ffff88004fedbe08 0000000000000046 ffff88005cdabec0 ffff88004fedbfd8 [13545.220291] ffff88004fedbfd8 ffff88004fedbfd8 ffff88005cdabec0 ffff8800360b1480 [13545.221959] ffff880067849390 ffff88005b348420 ffff8800678493b0 ffff880067849300 [13545.223695] Call Trace: [13545.224933] [] schedule+0x29/0x70 [13545.226436] [] taskq_thread+0x3fa/0x440 [spl] [13545.227943] [] ? wake_up_state+0x20/0x20 [13545.229482] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.231066] [] kthread+0xcf/0xe0 [13545.232460] [] ? kthread_create_on_node+0x140/0x140 [13545.234059] [] ret_from_fork+0x58/0x90 [13545.235504] [] ? kthread_create_on_node+0x140/0x140 [13545.237109] z_rd_int_2 S ffff880067849300 0 19764 2 0x00000080 [13545.238713] ffff88005fb7fe08 0000000000000046 ffff88005cdaaf10 ffff88005fb7ffd8 [13545.240442] ffff88005fb7ffd8 ffff88005fb7ffd8 ffff88005cdaaf10 ffff8800360b1e40 [13545.242149] ffff880067849390 ffff880048da6f90 ffff8800678493b0 ffff880067849300 [13545.243804] Call Trace: [13545.245072] [] schedule+0x29/0x70 [13545.246550] [] taskq_thread+0x3fa/0x440 [spl] [13545.248058] [] ? wake_up_state+0x20/0x20 [13545.249604] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.251190] [] kthread+0xcf/0xe0 [13545.252583] [] ? finish_task_switch+0x53/0x180 [13545.254150] [] ? kthread_create_on_node+0x140/0x140 [13545.255739] [] ret_from_fork+0x58/0x90 [13545.257180] [] ? kthread_create_on_node+0x140/0x140 [13545.258768] z_rd_int_2 S ffff880067849300 0 19765 2 0x00000080 [13545.260384] ffff88007922be08 0000000000000046 ffff88005cda9f60 ffff88007922bfd8 [13545.262114] ffff88007922bfd8 ffff88007922bfd8 ffff88005cda9f60 ffff8800360b1600 [13545.263770] ffff880067849390 ffff880048da6f90 ffff8800678493b0 ffff880067849300 [13545.265497] Call Trace: [13545.266727] [] schedule+0x29/0x70 [13545.268245] [] taskq_thread+0x3fa/0x440 [spl] [13545.269799] [] ? wake_up_state+0x20/0x20 [13545.271262] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.272852] [] kthread+0xcf/0xe0 [13545.274302] [] ? finish_task_switch+0x53/0x180 [13545.275796] [] ? kthread_create_on_node+0x140/0x140 [13545.277389] [] ret_from_fork+0x58/0x90 [13545.278821] [] ? kthread_create_on_node+0x140/0x140 [13545.280419] z_rd_int_2 S ffff880067849300 0 19766 2 0x00000080 [13545.282022] ffff88005f8ffe08 0000000000000046 ffff88005cda8fb0 ffff88005f8fffd8 [13545.283731] ffff88005f8fffd8 ffff88005f8fffd8 ffff88005cda8fb0 ffff8800360b1840 [13545.285447] ffff880067849390 ffff880048da73e8 ffff8800678493b0 ffff880067849300 [13545.287148] Call Trace: [13545.288387] [] schedule+0x29/0x70 [13545.289875] [] taskq_thread+0x3fa/0x440 [spl] [13545.291420] [] ? wake_up_state+0x20/0x20 [13545.292921] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.294498] [] kthread+0xcf/0xe0 [13545.295909] [] ? kthread_create_on_node+0x140/0x140 [13545.297489] [] ret_from_fork+0x58/0x90 [13545.298923] [] ? kthread_create_on_node+0x140/0x140 [13545.300506] z_rd_int_2 S ffff880067849300 0 19767 2 0x00000080 [13545.302112] ffff88006605fe08 0000000000000046 ffff88005cda8000 ffff88006605ffd8 [13545.303830] ffff88006605ffd8 ffff88006605ffd8 ffff88005cda8000 ffff8800360b1de0 [13545.305549] ffff880067849390 ffff880048da73e8 ffff8800678493b0 ffff880067849300 [13545.307248] Call Trace: [13545.308475] [] schedule+0x29/0x70 [13545.309980] [] taskq_thread+0x3fa/0x440 [spl] [13545.311518] [] ? wake_up_state+0x20/0x20 [13545.312967] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.314552] [] kthread+0xcf/0xe0 [13545.316024] [] ? kthread_create_on_node+0x140/0x140 [13545.317589] [] ret_from_fork+0x58/0x90 [13545.319075] [] ? kthread_create_on_node+0x140/0x140 [13545.320593] z_rd_int_2 S ffff880067849300 0 19768 2 0x00000080 [13545.322287] ffff880061767e08 0000000000000046 ffff880063ae6dd0 ffff880061767fd8 [13545.323954] ffff880061767fd8 ffff880061767fd8 ffff880063ae6dd0 ffff8800360b1a80 [13545.325678] ffff880067849390 ffff880048da5580 ffff8800678493b0 ffff880067849300 [13545.327401] Call Trace: [13545.328632] [] schedule+0x29/0x70 [13545.330144] [] taskq_thread+0x3fa/0x440 [spl] [13545.331693] [] ? wake_up_state+0x20/0x20 [13545.333202] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.334773] [] kthread+0xcf/0xe0 [13545.336236] [] ? kthread_create_on_node+0x140/0x140 [13545.337813] [] ret_from_fork+0x58/0x90 [13545.339293] [] ? kthread_create_on_node+0x140/0x140 [13545.340876] z_rd_int_2 S ffff880067849300 0 19769 2 0x00000080 [13545.342485] ffff880069013e08 0000000000000046 ffff880063ae3ec0 ffff880069013fd8 [13545.344201] ffff880069013fd8 ffff880069013fd8 ffff880063ae3ec0 ffff8800360b1900 [13545.345862] ffff880067849390 ffff880068a91580 ffff8800678493b0 ffff880067849300 [13545.347570] Call Trace: [13545.348802] [] schedule+0x29/0x70 [13545.350313] [] taskq_thread+0x3fa/0x440 [spl] [13545.351872] [] ? wake_up_state+0x20/0x20 [13545.353348] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.354944] [] kthread+0xcf/0xe0 [13545.356384] [] ? kthread_create_on_node+0x140/0x140 [13545.357967] [] ret_from_fork+0x58/0x90 [13545.359461] [] ? kthread_create_on_node+0x140/0x140 [13545.360986] z_rd_int_3 S ffff880067849a00 0 19770 2 0x00000080 [13545.362625] ffff88004e363e08 0000000000000046 ffff880063ae0fb0 ffff88004e363fd8 [13545.364339] ffff88004e363fd8 ffff88004e363fd8 ffff880063ae0fb0 ffff8800360b1ea0 [13545.366006] ffff880067849a90 ffff88005b349e30 ffff880067849ab0 ffff880067849a00 [13545.367722] Call Trace: [13545.368988] [] schedule+0x29/0x70 [13545.370413] [] taskq_thread+0x3fa/0x440 [spl] [13545.371984] [] ? wake_up_state+0x20/0x20 [13545.373509] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.375093] [] kthread+0xcf/0xe0 [13545.376490] [] ? kthread_create_on_node+0x140/0x140 [13545.378084] [] ret_from_fork+0x58/0x90 [13545.379528] [] ? kthread_create_on_node+0x140/0x140 [13545.381120] z_rd_int_3 S ffff880067849a00 0 19771 2 0x00000080 [13545.382719] ffff88005e24fe08 0000000000000046 ffff880063ae0000 ffff88005e24ffd8 [13545.384447] ffff88005e24ffd8 ffff88005e24ffd8 ffff880063ae0000 ffff8800360b18a0 [13545.386156] ffff880067849a90 ffff880048da59d8 ffff880067849ab0 ffff880067849a00 [13545.387812] Call Trace: [13545.389107] [] schedule+0x29/0x70 [13545.390590] [] taskq_thread+0x3fa/0x440 [spl] [13545.392096] [] ? wake_up_state+0x20/0x20 [13545.393631] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.395222] [] kthread+0xcf/0xe0 [13545.396615] [] ? kthread_create_on_node+0x140/0x140 [13545.398212] [] ret_from_fork+0x58/0x90 [13545.399654] [] ? kthread_create_on_node+0x140/0x140 [13545.401248] z_rd_int_3 S ffff880067849a00 0 19772 2 0x00000080 [13545.402851] ffff88005ceafe08 0000000000000046 ffff880063ae1f60 ffff88005ceaffd8 [13545.404571] ffff88005ceaffd8 ffff88005ceaffd8 ffff880063ae1f60 ffff8800360b1c00 [13545.406276] ffff880067849a90 ffff880048da6f90 ffff880067849ab0 ffff880067849a00 [13545.407932] Call Trace: [13545.409196] [] schedule+0x29/0x70 [13545.410617] [] taskq_thread+0x3fa/0x440 [spl] [13545.412188] [] ? wake_up_state+0x20/0x20 [13545.413730] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.415262] [] kthread+0xcf/0xe0 [13545.416717] [] ? kthread_create_on_node+0x140/0x140 [13545.418310] [] ret_from_fork+0x58/0x90 [13545.419807] [] ? kthread_create_on_node+0x140/0x140 [13545.421392] z_rd_int_3 S ffff880067849a00 0 19773 2 0x00000080 [13545.423034] ffff8800690a3e08 0000000000000046 ffff880065a8de20 ffff8800690a3fd8 [13545.424755] ffff8800690a3fd8 ffff8800690a3fd8 ffff880065a8de20 ffff8800360b1060 [13545.426492] ffff880067849a90 ffff880048da6288 ffff880067849ab0 ffff880067849a00 [13545.428210] Call Trace: [13545.429435] [] schedule+0x29/0x70 [13545.430934] [] taskq_thread+0x3fa/0x440 [spl] [13545.432490] [] ? wake_up_state+0x20/0x20 [13545.433972] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.435574] [] kthread+0xcf/0xe0 [13545.436988] [] ? kthread_create_on_node+0x140/0x140 [13545.438595] [] ret_from_fork+0x58/0x90 [13545.440092] [] ? kthread_create_on_node+0x140/0x140 [13545.441614] z_rd_int_3 S ffff880067849a00 0 19774 2 0x00000080 [13545.443328] ffff88005a463e08 0000000000000046 ffff880065a8af10 ffff88005a463fd8 [13545.445015] ffff88005a463fd8 ffff88005a463fd8 ffff880065a8af10 ffff8800360b1f60 [13545.446717] ffff880067849a90 ffff880048da6f90 ffff880067849ab0 ffff880067849a00 [13545.448436] Call Trace: [13545.449661] [] schedule+0x29/0x70 [13545.451163] [] taskq_thread+0x3fa/0x440 [spl] [13545.452665] [] ? wake_up_state+0x20/0x20 [13545.454195] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.455772] [] kthread+0xcf/0xe0 [13545.457252] [] ? kthread_create_on_node+0x140/0x140 [13545.458842] [] ret_from_fork+0x58/0x90 [13545.460286] [] ? kthread_create_on_node+0x140/0x140 [13545.461869] z_rd_int_3 S ffff880067849a00 0 19775 2 0x00000080 [13545.463477] ffff880066d17e08 0000000000000046 ffff880065a8edd0 ffff880066d17fd8 [13545.465184] ffff880066d17fd8 ffff880066d17fd8 ffff880065a8edd0 ffff8800360b1ba0 [13545.466841] ffff880067849a90 ffff880048da6f90 ffff880067849ab0 ffff880067849a00 [13545.468562] Call Trace: [13545.469789] [] schedule+0x29/0x70 [13545.471293] [] taskq_thread+0x3fa/0x440 [spl] [13545.472844] [] ? wake_up_state+0x20/0x20 [13545.474311] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.475908] [] kthread+0xcf/0xe0 [13545.477300] [] ? kthread_create_on_node+0x140/0x140 [13545.478896] [] ret_from_fork+0x58/0x90 [13545.480386] [] ? kthread_create_on_node+0x140/0x140 [13545.481904] z_rd_int_3 S ffff880067849a00 0 19776 2 0x00000080 [13545.483552] ffff8800590a3e08 0000000000000046 ffff880065a88fb0 ffff8800590a3fd8 [13545.485274] ffff8800590a3fd8 ffff8800590a3fd8 ffff880065a88fb0 ffff8800360b14e0 [13545.486938] ffff880067849a90 ffff880048da6f90 ffff880067849ab0 ffff880067849a00 [13545.488672] Call Trace: [13545.489903] [] schedule+0x29/0x70 [13545.491401] [] taskq_thread+0x3fa/0x440 [spl] [13545.492908] [] ? wake_up_state+0x20/0x20 [13545.494436] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.495965] [] kthread+0xcf/0xe0 [13545.497435] [] ? kthread_create_on_node+0x140/0x140 [13545.499033] [] ret_from_fork+0x58/0x90 [13545.500506] [] ? kthread_create_on_node+0x140/0x140 [13545.502110] z_rd_int_3 S ffff880067849a00 0 19777 2 0x00000080 [13545.503724] ffff88005e2d7e08 0000000000000046 ffff880065a8ce70 ffff88005e2d7fd8 [13545.505475] ffff88005e2d7fd8 ffff88005e2d7fd8 ffff880065a8ce70 ffff8800360b1a20 [13545.507196] ffff880067849a90 ffff880048da7840 ffff880067849ab0 ffff880067849a00 [13545.508848] Call Trace: [13545.510147] [] schedule+0x29/0x70 [13545.511573] [] taskq_thread+0x3fa/0x440 [spl] [13545.513145] [] ? wake_up_state+0x20/0x20 [13545.514665] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.516198] [] kthread+0xcf/0xe0 [13545.517654] [] ? kthread_create_on_node+0x140/0x140 [13545.519254] [] ret_from_fork+0x58/0x90 [13545.520696] [] ? kthread_create_on_node+0x140/0x140 [13545.522285] z_rd_int_3 S ffff880067849a00 0 19778 2 0x00000080 [13545.523888] ffff880060437e08 0000000000000046 ffff880065a8bec0 ffff880060437fd8 [13545.525612] ffff880060437fd8 ffff880060437fd8 ffff880065a8bec0 ffff8800360b1540 [13545.527312] ffff880067849a90 ffff88005f901580 ffff880067849ab0 ffff880067849a00 [13545.528973] Call Trace: [13545.530263] [] schedule+0x29/0x70 [13545.531738] [] taskq_thread+0x3fa/0x440 [spl] [13545.533298] [] ? wake_up_state+0x20/0x20 [13545.534812] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.536395] [] kthread+0xcf/0xe0 [13545.537838] [] ? kthread_create_on_node+0x140/0x140 [13545.539410] [] ret_from_fork+0x58/0x90 [13545.540861] [] ? kthread_create_on_node+0x140/0x140 [13545.542447] z_rd_int_3 S ffff880067849a00 0 19779 2 0x00000080 [13545.544047] ffff88006085fe08 0000000000000046 ffff880065a89f60 ffff88006085ffd8 [13545.545751] ffff88006085ffd8 ffff88006085ffd8 ffff880065a89f60 ffff8800360b1300 [13545.547462] ffff880067849a90 ffff88005e6c19d8 ffff880067849ab0 ffff880067849a00 [13545.549162] Call Trace: [13545.550382] [] schedule+0x29/0x70 [13545.551871] [] taskq_thread+0x3fa/0x440 [spl] [13545.553368] [] ? wake_up_state+0x20/0x20 [13545.554902] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.556426] [] kthread+0xcf/0xe0 [13545.557879] [] ? kthread_create_on_node+0x140/0x140 [13545.559447] [] ret_from_fork+0x58/0x90 [13545.560944] [] ? kthread_create_on_node+0x140/0x140 [13545.562516] z_rd_int_3 S ffff880067849a00 0 19780 2 0x00000080 [13545.564116] ffff880060943e08 0000000000000046 ffff880065a88000 ffff880060943fd8 [13545.565823] ffff880060943fd8 ffff880060943fd8 ffff880065a88000 ffff8800360b10c0 [13545.567536] ffff880067849a90 ffff88005e6c19d8 ffff880067849ab0 ffff880067849a00 [13545.569230] Call Trace: [13545.570450] [] schedule+0x29/0x70 [13545.571943] [] taskq_thread+0x3fa/0x440 [spl] [13545.573447] [] ? wake_up_state+0x20/0x20 [13545.574976] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.576497] [] kthread+0xcf/0xe0 [13545.577954] [] ? kthread_create_on_node+0x140/0x140 [13545.579474] [] ret_from_fork+0x58/0x90 [13545.580981] [] ? kthread_create_on_node+0x140/0x140 [13545.582495] z_rd_int_3 S ffff880067849a00 0 19781 2 0x00000080 [13545.584139] ffff880066befe08 0000000000000046 ffff880060616dd0 ffff880066beffd8 [13545.585843] ffff880066beffd8 ffff880066beffd8 ffff880060616dd0 ffff8800360b1ae0 [13545.587546] ffff880067849a90 ffff88005e6c19d8 ffff880067849ab0 ffff880067849a00 [13545.589244] Call Trace: [13545.590462] [] schedule+0x29/0x70 [13545.591964] [] taskq_thread+0x3fa/0x440 [spl] [13545.593521] [] ? wake_up_state+0x20/0x20 [13545.595045] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.596623] [] kthread+0xcf/0xe0 [13545.598064] [] ? kthread_create_on_node+0x140/0x140 [13545.599580] [] ret_from_fork+0x58/0x90 [13545.601090] [] ? kthread_create_on_node+0x140/0x140 [13545.602602] z_rd_int_4 S ffff880067849700 0 19782 2 0x00000080 [13545.604317] ffff880045db3e08 0000000000000046 ffff880060614e70 ffff880045db3fd8 [13545.605976] ffff880045db3fd8 ffff880045db3fd8 ffff880060614e70 ffff8800360b19c0 [13545.607684] ffff880067849790 ffff880068a97c98 ffff8800678497b0 ffff880067849700 [13545.609395] Call Trace: [13545.610624] [] schedule+0x29/0x70 [13545.612092] [] taskq_thread+0x3fa/0x440 [spl] [13545.613592] [] ? wake_up_state+0x20/0x20 [13545.615129] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.616652] [] kthread+0xcf/0xe0 [13545.618117] [] ? kthread_create_on_node+0x140/0x140 [13545.619647] [] ret_from_fork+0x58/0x90 [13545.621156] [] ? kthread_create_on_node+0x140/0x140 [13545.622677] z_rd_int_4 S ffff880067849700 0 19783 2 0x00000080 [13545.624391] ffff88007b48be08 0000000000000046 ffff880060612f10 ffff88007b48bfd8 [13545.626113] ffff88007b48bfd8 ffff88007b48bfd8 ffff880060612f10 ffff8800360b1180 [13545.627772] ffff880067849790 ffff880048da59d8 ffff8800678497b0 ffff880067849700 [13545.629488] Call Trace: [13545.630720] [] schedule+0x29/0x70 [13545.632232] [] taskq_thread+0x3fa/0x440 [spl] [13545.633738] [] ? wake_up_state+0x20/0x20 [13545.635272] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.636806] [] kthread+0xcf/0xe0 [13545.638264] [] ? kthread_create_on_node+0x140/0x140 [13545.639786] [] ret_from_fork+0x58/0x90 [13545.641299] [] ? kthread_create_on_node+0x140/0x140 [13545.642868] z_rd_int_4 S ffff880067849700 0 19784 2 0x00000080 [13545.644474] ffff88005b41fe08 0000000000000046 ffff880060610fb0 ffff88005b41ffd8 [13545.646188] ffff88005b41ffd8 ffff88005b41ffd8 ffff880060610fb0 ffff8800360b1120 [13545.647881] ffff880067849790 ffff880048da59d8 ffff8800678497b0 ffff880067849700 [13545.649574] Call Trace: [13545.650799] [] schedule+0x29/0x70 [13545.652311] [] taskq_thread+0x3fa/0x440 [spl] [13545.653871] [] ? wake_up_state+0x20/0x20 [13545.655339] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.656955] [] kthread+0xcf/0xe0 [13545.658402] [] ? kthread_create_on_node+0x140/0x140 [13545.659946] [] ret_from_fork+0x58/0x90 [13545.661473] [] ? kthread_create_on_node+0x140/0x140 [13545.663015] z_rd_int_4 S ffff880067849700 0 19785 2 0x00000080 [13545.664691] ffff880048c13e08 0000000000000046 ffff880060615e20 ffff880048c13fd8 [13545.666413] ffff880048c13fd8 ffff880048c13fd8 ffff880060615e20 ffff8800360b1b40 [13545.668119] ffff880067849790 ffff880048da6b38 ffff8800678497b0 ffff880067849700 [13545.669783] Call Trace: [13545.671056] [] schedule+0x29/0x70 [13545.672482] [] taskq_thread+0x3fa/0x440 [spl] [13545.674057] [] ? wake_up_state+0x20/0x20 [13545.675573] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.677163] [] kthread+0xcf/0xe0 [13545.678602] [] ? kthread_create_on_node+0x140/0x140 [13545.680187] [] ret_from_fork+0x58/0x90 [13545.681630] [] ? kthread_create_on_node+0x140/0x140 [13545.683216] z_rd_int_4 S ffff880067849700 0 19786 2 0x00000080 [13545.684814] ffff88006bf23e08 0000000000000046 ffff880060613ec0 ffff88006bf23fd8 [13545.686537] ffff88006bf23fd8 ffff88006bf23fd8 ffff880060613ec0 ffff8800360b1420 [13545.688243] ffff880067849790 ffff880048da6288 ffff8800678497b0 ffff880067849700 [13545.689908] Call Trace: [13545.691170] [] schedule+0x29/0x70 [13545.692598] [] taskq_thread+0x3fa/0x440 [spl] [13545.694163] [] ? wake_up_state+0x20/0x20 [13545.695631] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.697238] [] kthread+0xcf/0xe0 [13545.698630] [] ? finish_task_switch+0x53/0x180 [13545.700196] [] ? kthread_create_on_node+0x140/0x140 [13545.701722] [] ret_from_fork+0x58/0x90 [13545.703226] [] ? kthread_create_on_node+0x140/0x140 [13545.704742] z_rd_int_4 S ffff880067849700 0 19787 2 0x00000080 [13545.706441] ffff880063ac3e08 0000000000000046 ffff880060611f60 ffff880063ac3fd8 [13545.708164] ffff880063ac3fd8 ffff880063ac3fd8 ffff880060611f60 ffff8800360b1c60 [13545.709823] ffff880067849790 ffff880048da6f90 ffff8800678497b0 ffff880067849700 [13545.711543] Call Trace: [13545.712769] [] schedule+0x29/0x70 [13545.714281] [] taskq_thread+0x3fa/0x440 [spl] [13545.715841] [] ? wake_up_state+0x20/0x20 [13545.717297] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.718886] [] kthread+0xcf/0xe0 [13545.720289] [] ? kthread_create_on_node+0x140/0x140 [13545.721885] [] ret_from_fork+0x58/0x90 [13545.723317] [] ? kthread_create_on_node+0x140/0x140 [13545.724905] z_rd_int_4 S ffff880067849700 0 19788 2 0x00000080 [13545.726510] ffff88005ab83e08 0000000000000046 ffff880060610000 ffff88005ab83fd8 [13545.728218] ffff88005ab83fd8 ffff88005ab83fd8 ffff880060610000 ffff8800360b16c0 [13545.729889] ffff880067849790 ffff880068a95580 ffff8800678497b0 ffff880067849700 [13545.731609] Call Trace: [13545.732839] [] schedule+0x29/0x70 [13545.734332] [] taskq_thread+0x3fa/0x440 [spl] [13545.735898] [] ? wake_up_state+0x20/0x20 [13545.737361] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.738946] [] kthread+0xcf/0xe0 [13545.740394] [] ? kthread_create_on_node+0x140/0x140 [13545.741986] [] ret_from_fork+0x58/0x90 [13545.743474] [] ? kthread_create_on_node+0x140/0x140 [13545.744995] z_rd_int_4 S ffff880067849700 0 19789 2 0x00000080 [13545.746661] ffff88005f8dbe08 0000000000000046 ffff88005a8fedd0 ffff88005f8dbfd8 [13545.748385] ffff88005f8dbfd8 ffff88005f8dbfd8 ffff88005a8fedd0 ffff8800360b12a0 [13545.750096] ffff880067849790 ffff880067590cd0 ffff8800678497b0 ffff880067849700 [13545.751744] Call Trace: [13545.753025] [] schedule+0x29/0x70 [13545.754508] [] taskq_thread+0x3fa/0x440 [spl] [13545.756013] [] ? wake_up_state+0x20/0x20 [13545.757537] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.759112] [] kthread+0xcf/0xe0 [13545.760558] [] ? kthread_create_on_node+0x140/0x140 [13545.762090] [] ret_from_fork+0x58/0x90 [13545.763592] [] ? kthread_create_on_node+0x140/0x140 [13545.765113] z_rd_int_4 S ffff880067849700 0 19790 2 0x00000080 [13545.766762] ffff8800492e3e08 0000000000000046 ffff88005a8fce70 ffff8800492e3fd8 [13545.768485] ffff8800492e3fd8 ffff8800492e3fd8 ffff88005a8fce70 ffff8800360b1660 [13545.770197] ffff880067849790 ffff880068a959d8 ffff8800678497b0 ffff880067849700 [13545.771851] Call Trace: [13545.773120] [] schedule+0x29/0x70 [13545.774602] [] taskq_thread+0x3fa/0x440 [spl] [13545.776158] [] ? wake_up_state+0x20/0x20 [13545.777644] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.779252] [] kthread+0xcf/0xe0 [13545.780660] [] ? kthread_create_on_node+0x140/0x140 [13545.782264] [] ret_from_fork+0x58/0x90 [13545.783773] [] ? kthread_create_on_node+0x140/0x140 [13545.785356] z_rd_int_4 S ffff880067849700 0 19791 2 0x00000080 [13545.786963] ffff88005a6e3e08 0000000000000046 ffff88005a8faf10 ffff88005a6e3fd8 [13545.788687] ffff88005a6e3fd8 ffff88005a6e3fd8 ffff88005a8faf10 ffff8800360b1cc0 [13545.790369] ffff880067849790 ffff880068a95e30 ffff8800678497b0 ffff880067849700 [13545.792090] Call Trace: [13545.793380] [] schedule+0x29/0x70 [13545.794804] [] taskq_thread+0x3fa/0x440 [spl] [13545.796311] [] ? wake_up_state+0x20/0x20 [13545.797778] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.799325] [] kthread+0xcf/0xe0 [13545.800722] [] ? finish_task_switch+0x53/0x180 [13545.802224] [] ? kthread_create_on_node+0x140/0x140 [13545.803770] [] ret_from_fork+0x58/0x90 [13545.805214] [] ? kthread_create_on_node+0x140/0x140 [13545.806732] z_rd_int_4 S ffff880067849700 0 19792 2 0x00000080 [13545.808346] ffff88005f15be08 0000000000000046 ffff88005a8f8fb0 ffff88005f15bfd8 [13545.810017] ffff88005f15bfd8 ffff88005f15bfd8 ffff88005a8f8fb0 ffff8800360b13c0 [13545.811675] ffff880067849790 ffff880068a94878 ffff8800678497b0 ffff880067849700 [13545.813338] Call Trace: [13545.814563] [] schedule+0x29/0x70 [13545.815996] [] taskq_thread+0x3fa/0x440 [spl] [13545.817505] [] ? wake_up_state+0x20/0x20 [13545.818967] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.820490] [] kthread+0xcf/0xe0 [13545.821901] [] ? kthread_create_on_node+0x140/0x140 [13545.823436] [] ret_from_fork+0x58/0x90 [13545.824869] [] ? kthread_create_on_node+0x140/0x140 [13545.826388] z_rd_int_4 S ffff880067849700 0 19793 2 0x00000080 [13545.827992] ffff88005f7a3e08 0000000000000046 ffff88005a8fde20 ffff88005f7a3fd8 [13545.829660] ffff88005f7a3fd8 ffff88005f7a3fd8 ffff88005a8fde20 ffff8800360b11e0 [13545.831340] ffff880067849790 ffff880068a966e0 ffff8800678497b0 ffff880067849700 [13545.832993] Call Trace: [13545.834222] [] schedule+0x29/0x70 [13545.835644] [] taskq_thread+0x3fa/0x440 [spl] [13545.837148] [] ? wake_up_state+0x20/0x20 [13545.838617] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.840140] [] kthread+0xcf/0xe0 [13545.841542] [] ? kthread_create_on_node+0x140/0x140 [13545.843072] [] ret_from_fork+0x58/0x90 [13545.844502] [] ? kthread_create_on_node+0x140/0x140 [13545.846024] z_rd_int_5 S ffff88005da7fa00 0 19794 2 0x00000080 [13545.847619] ffff88006339fe08 0000000000000046 ffff88005a8fbec0 ffff88006339ffd8 [13545.849299] ffff88006339ffd8 ffff88006339ffd8 ffff88005a8fbec0 ffff8800360b1240 [13545.850956] ffff88005da7fa90 ffff880048da66e0 ffff88005da7fab0 ffff88005da7fa00 [13545.852615] Call Trace: [13545.853844] [] schedule+0x29/0x70 [13545.855269] [] taskq_thread+0x3fa/0x440 [spl] [13545.856772] [] ? wake_up_state+0x20/0x20 [13545.858233] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.859748] [] kthread+0xcf/0xe0 [13545.861157] [] ? kthread_create_on_node+0x140/0x140 [13545.862684] [] ret_from_fork+0x58/0x90 [13545.864116] [] ? kthread_create_on_node+0x140/0x140 [13545.865637] z_rd_int_5 S ffff88005da7fa00 0 19795 2 0x00000080 [13545.867247] ffff880067bdbe08 0000000000000046 ffff88005a8f9f60 ffff880067bdbfd8 [13545.868912] ffff880067bdbfd8 ffff880067bdbfd8 ffff88005a8f9f60 ffff8800360b1780 [13545.870576] ffff88005da7fa90 ffff880048da6f90 ffff88005da7fab0 ffff88005da7fa00 [13545.872238] Call Trace: [13545.873464] [] schedule+0x29/0x70 [13545.874892] [] taskq_thread+0x3fa/0x440 [spl] [13545.876398] [] ? wake_up_state+0x20/0x20 [13545.877869] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.879404] [] kthread+0xcf/0xe0 [13545.880796] [] ? kthread_create_on_node+0x140/0x140 [13545.882326] [] ret_from_fork+0x58/0x90 [13545.883772] [] ? kthread_create_on_node+0x140/0x140 [13545.885300] z_rd_int_5 S ffff88005da7fa00 0 19796 2 0x00000080 [13545.886904] ffff88004797fe08 0000000000000046 ffff88005a8f8000 ffff88004797ffd8 [13545.888577] ffff88004797ffd8 ffff88004797ffd8 ffff88005a8f8000 ffff8800360b15a0 [13545.890249] ffff88005da7fa90 ffff880048da6f90 ffff88005da7fab0 ffff88005da7fa00 [13545.891905] Call Trace: [13545.893131] [] schedule+0x29/0x70 [13545.894555] [] taskq_thread+0x3fa/0x440 [spl] [13545.896059] [] ? wake_up_state+0x20/0x20 [13545.897524] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.899059] [] kthread+0xcf/0xe0 [13545.900451] [] ? kthread_create_on_node+0x140/0x140 [13545.901980] [] ret_from_fork+0x58/0x90 [13545.903426] [] ? kthread_create_on_node+0x140/0x140 [13545.904944] z_rd_int_5 S ffff88005da7fa00 0 19797 2 0x00000080 [13545.906558] ffff88005e717e08 0000000000000046 ffff880068a5de20 ffff88005e717fd8 [13545.908229] ffff88005e717fd8 ffff88005e717fd8 ffff880068a5de20 ffff8800360b1f00 [13545.909887] ffff88005da7fa90 ffff880048da6288 ffff88005da7fab0 ffff88005da7fa00 [13545.911553] Call Trace: [13545.912777] [] schedule+0x29/0x70 [13545.914212] [] taskq_thread+0x3fa/0x440 [spl] [13545.915722] [] ? wake_up_state+0x20/0x20 [13545.917205] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.918738] [] kthread+0xcf/0xe0 [13545.920142] [] ? kthread_create_on_node+0x140/0x140 [13545.921669] [] ret_from_fork+0x58/0x90 [13545.923116] [] ? kthread_create_on_node+0x140/0x140 [13545.924653] z_rd_int_5 S ffff88005da7fa00 0 19798 2 0x00000080 [13545.926282] ffff880060463e08 0000000000000046 ffff880068a5af10 ffff880060463fd8 [13545.927949] ffff880060463fd8 ffff880060463fd8 ffff880068a5af10 ffff8800360b1720 [13545.929621] ffff88005da7fa90 ffff880048da6288 ffff88005da7fab0 ffff88005da7fa00 [13545.931294] Call Trace: [13545.932527] [] schedule+0x29/0x70 [13545.933961] [] taskq_thread+0x3fa/0x440 [spl] [13545.935458] [] ? wake_up_state+0x20/0x20 [13545.936937] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.938464] [] kthread+0xcf/0xe0 [13545.939861] [] ? kthread_create_on_node+0x140/0x140 [13545.941391] [] ret_from_fork+0x58/0x90 [13545.942830] [] ? kthread_create_on_node+0x140/0x140 [13545.944346] z_rd_int_5 S ffff88005da7fa00 0 19799 2 0x00000080 [13545.945949] ffff880063abfe08 0000000000000046 ffff880068a59f60 ffff880063abffd8 [13545.947615] ffff880063abffd8 ffff880063abffd8 ffff880068a59f60 ffff8800360b1960 [13545.949292] ffff88005da7fa90 ffff880048da6f90 ffff88005da7fab0 ffff88005da7fa00 [13545.950945] Call Trace: [13545.952169] [] schedule+0x29/0x70 [13545.953595] [] taskq_thread+0x3fa/0x440 [spl] [13545.955097] [] ? wake_up_state+0x20/0x20 [13545.956562] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.958091] [] kthread+0xcf/0xe0 [13545.959483] [] ? kthread_create_on_node+0x140/0x140 [13545.961014] [] ret_from_fork+0x58/0x90 [13545.962452] [] ? kthread_create_on_node+0x140/0x140 [13545.963980] z_rd_int_5 S ffff88005da7fa00 0 19800 2 0x00000080 [13545.965586] ffff8800690b3e08 0000000000000046 ffff880068a5edd0 ffff8800690b3fd8 [13545.967272] ffff8800690b3fd8 ffff8800690b3fd8 ffff880068a5edd0 ffff8800360b1d20 [13545.969009] ffff88005da7fa90 ffff880048da6f90 ffff88005da7fab0 ffff88005da7fa00 [13545.971527] Call Trace: [13545.972753] [] schedule+0x29/0x70 [13545.974247] [] taskq_thread+0x3fa/0x440 [spl] [13545.975798] [] ? wake_up_state+0x20/0x20 [13545.977318] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.978853] [] kthread+0xcf/0xe0 [13545.980314] [] ? kthread_create_on_node+0x140/0x140 [13545.981891] [] ret_from_fork+0x58/0x90 [13545.983395] [] ? kthread_create_on_node+0x140/0x140 [13545.984914] z_rd_int_5 S ffff88005da7fa00 0 19801 2 0x00000080 [13545.986570] ffff88006bc03e08 0000000000000046 ffff880068a5ce70 ffff88006bc03fd8 [13545.988276] ffff88006bc03fd8 ffff88006bc03fd8 ffff880068a5ce70 ffff8800360b1d80 [13545.989972] ffff88005da7fa90 ffff880048da6f90 ffff88005da7fab0 ffff88005da7fa00 [13545.991698] Call Trace: [13545.992940] [] schedule+0x29/0x70 [13545.994439] [] taskq_thread+0x3fa/0x440 [spl] [13545.995991] [] ? wake_up_state+0x20/0x20 [13545.997513] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13545.999097] [] kthread+0xcf/0xe0 [13546.000545] [] ? kthread_create_on_node+0x140/0x140 [13546.002079] [] ret_from_fork+0x58/0x90 [13546.003595] [] ? kthread_create_on_node+0x140/0x140 [13546.005169] z_rd_int_5 S ffff88005da7fa00 0 19802 2 0x00000080 [13546.006766] ffff8800626dbe08 0000000000000046 ffff880068a5bec0 ffff8800626dbfd8 [13546.008495] ffff8800626dbfd8 ffff8800626dbfd8 ffff880068a5bec0 ffff880065916660 [13546.010200] ffff88005da7fa90 ffff880048da6288 ffff88005da7fab0 ffff88005da7fa00 [13546.011857] Call Trace: [13546.013118] [] schedule+0x29/0x70 [13546.014605] [] taskq_thread+0x3fa/0x440 [spl] [13546.016122] [] ? wake_up_state+0x20/0x20 [13546.017660] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.019245] [] kthread+0xcf/0xe0 [13546.020689] [] ? kthread_create_on_node+0x140/0x140 [13546.022267] [] ret_from_fork+0x58/0x90 [13546.023715] [] ? kthread_create_on_node+0x140/0x140 [13546.025304] z_rd_int_5 S ffff88005da7fa00 0 19803 2 0x00000080 [13546.026917] ffff88005e6f3e08 0000000000000046 ffff880068a58fb0 ffff88005e6f3fd8 [13546.028637] ffff88005e6f3fd8 ffff88005e6f3fd8 ffff880068a58fb0 ffff880065916a20 [13546.030357] ffff88005da7fa90 ffff880048da7840 ffff88005da7fab0 ffff88005da7fa00 [13546.032012] Call Trace: [13546.033316] [] schedule+0x29/0x70 [13546.034791] [] taskq_thread+0x3fa/0x440 [spl] [13546.036295] [] ? wake_up_state+0x20/0x20 [13546.037843] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.039373] [] kthread+0xcf/0xe0 [13546.040835] [] ? kthread_create_on_node+0x140/0x140 [13546.042418] [] ret_from_fork+0x58/0x90 [13546.043919] [] ? kthread_create_on_node+0x140/0x140 [13546.045452] z_rd_int_5 S ffff88005da7fa00 0 19804 2 0x00000080 [13546.047110] ffff88006141be08 0000000000000046 ffff880068a58000 ffff88006141bfd8 [13546.048817] ffff88006141bfd8 ffff88006141bfd8 ffff880068a58000 ffff880065916f60 [13546.050526] ffff88005da7fa90 ffff880048da6b38 ffff88005da7fab0 ffff88005da7fa00 [13546.052243] Call Trace: [13546.053470] [] schedule+0x29/0x70 [13546.054990] [] taskq_thread+0x3fa/0x440 [spl] [13546.056490] [] ? wake_up_state+0x20/0x20 [13546.058033] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.059562] [] kthread+0xcf/0xe0 [13546.061024] [] ? finish_task_switch+0x53/0x180 [13546.062571] [] ? kthread_create_on_node+0x140/0x140 [13546.064153] [] ret_from_fork+0x58/0x90 [13546.065634] [] ? kthread_create_on_node+0x140/0x140 [13546.067156] z_rd_int_5 S ffff88005da7fa00 0 19805 2 0x00000080 [13546.068805] ffff88005f97be08 0000000000000046 ffff88006050edd0 ffff88005f97bfd8 [13546.070522] ffff88005f97bfd8 ffff88005f97bfd8 ffff88006050edd0 ffff880065916b40 [13546.072226] ffff88005da7fa90 ffff880048da6b38 ffff88005da7fab0 ffff88005da7fa00 [13546.073879] Call Trace: [13546.075135] [] schedule+0x29/0x70 [13546.076557] [] taskq_thread+0x3fa/0x440 [spl] [13546.078118] [] ? wake_up_state+0x20/0x20 [13546.079619] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.081143] [] kthread+0xcf/0xe0 [13546.082599] [] ? kthread_create_on_node+0x140/0x140 [13546.084185] [] ret_from_fork+0x58/0x90 [13546.085659] [] ? kthread_create_on_node+0x140/0x140 [13546.087226] z_rd_int_6 S ffff88005da7fd00 0 19806 2 0x00000080 [13546.088821] ffff880060eebe08 0000000000000046 ffff88006050ce70 ffff880060eebfd8 [13546.090537] ffff880060eebfd8 ffff880060eebfd8 ffff88006050ce70 ffff880065916240 [13546.092250] ffff88005da7fd90 ffff880048d5d580 ffff88005da7fdb0 ffff88005da7fd00 [13546.093899] Call Trace: [13546.095184] [] schedule+0x29/0x70 [13546.096601] [] taskq_thread+0x3fa/0x440 [spl] [13546.098171] [] ? wake_up_state+0x20/0x20 [13546.099626] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.101210] [] kthread+0xcf/0xe0 [13546.102605] [] ? kthread_create_on_node+0x140/0x140 [13546.104205] [] ret_from_fork+0x58/0x90 [13546.105681] [] ? kthread_create_on_node+0x140/0x140 [13546.107207] z_rd_int_6 S ffff88005da7fd00 0 19807 2 0x00000080 [13546.108857] ffff88005e043e08 0000000000000046 ffff88006050bec0 ffff88005e043fd8 [13546.110585] ffff88005e043fd8 ffff88005e043fd8 ffff88006050bec0 ffff880065916540 [13546.112256] ffff88005da7fd90 ffff880048da6288 ffff88005da7fdb0 ffff88005da7fd00 [13546.113918] Call Trace: [13546.115147] [] schedule+0x29/0x70 [13546.116580] [] taskq_thread+0x3fa/0x440 [spl] [13546.118089] [] ? wake_up_state+0x20/0x20 [13546.119550] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.121078] [] kthread+0xcf/0xe0 [13546.122470] [] ? kthread_create_on_node+0x140/0x140 [13546.123995] [] ret_from_fork+0x58/0x90 [13546.125430] [] ? kthread_create_on_node+0x140/0x140 [13546.126947] z_rd_int_6 S ffff88005da7fd00 0 19808 2 0x00000080 [13546.128558] ffff88006a68be08 0000000000000046 ffff880060509f60 ffff88006a68bfd8 [13546.130233] ffff88006a68bfd8 ffff88006a68bfd8 ffff880060509f60 ffff880065916de0 [13546.131906] ffff88005da7fd90 ffff880048da6288 ffff88005da7fdb0 ffff88005da7fd00 [13546.133561] Call Trace: [13546.134791] [] schedule+0x29/0x70 [13546.136226] [] taskq_thread+0x3fa/0x440 [spl] [13546.137720] [] ? wake_up_state+0x20/0x20 [13546.139196] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.140725] [] kthread+0xcf/0xe0 [13546.142123] [] ? kthread_create_on_node+0x140/0x140 [13546.143652] [] ret_from_fork+0x58/0x90 [13546.145093] [] ? kthread_create_on_node+0x140/0x140 [13546.146602] z_rd_int_6 S ffff88005da7fd00 0 19809 2 0x00000080 [13546.148208] ffff88005e253e08 0000000000000046 ffff880060508000 ffff88005e253fd8 [13546.149869] ffff88005e253fd8 ffff88005e253fd8 ffff880060508000 ffff880065916e40 [13546.151534] ffff88005da7fd90 ffff880048da6288 ffff88005da7fdb0 ffff88005da7fd00 [13546.153199] Call Trace: [13546.154423] [] schedule+0x29/0x70 [13546.155846] [] taskq_thread+0x3fa/0x440 [spl] [13546.157345] [] ? wake_up_state+0x20/0x20 [13546.158826] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.160351] [] kthread+0xcf/0xe0 [13546.161738] [] ? kthread_create_on_node+0x140/0x140 [13546.163269] [] ret_from_fork+0x58/0x90 [13546.164707] [] ? kthread_create_on_node+0x140/0x140 [13546.166227] z_rd_int_6 S ffff88005da7fd00 0 19810 2 0x00000080 [13546.167826] ffff880063a57e08 0000000000000046 ffff88006050de20 ffff880063a57fd8 [13546.169493] ffff880063a57fd8 ffff880063a57fd8 ffff88006050de20 ffff880065916420 [13546.171160] ffff88005da7fd90 ffff880048da5580 ffff88005da7fdb0 ffff88005da7fd00 [13546.172813] Call Trace: [13546.174048] [] schedule+0x29/0x70 [13546.175467] [] taskq_thread+0x3fa/0x440 [spl] [13546.176970] [] ? wake_up_state+0x20/0x20 [13546.178438] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.179968] [] kthread+0xcf/0xe0 [13546.181358] [] ? kthread_create_on_node+0x140/0x140 [13546.182889] [] ret_from_fork+0x58/0x90 [13546.184326] [] ? kthread_create_on_node+0x140/0x140 [13546.185846] z_rd_int_6 S ffff88005da7fd00 0 19811 2 0x00000080 [13546.187447] ffff88005ebd3e08 0000000000000046 ffff88006050af10 ffff88005ebd3fd8 [13546.189123] ffff88005ebd3fd8 ffff88005ebd3fd8 ffff88006050af10 ffff8800659161e0 [13546.190771] ffff88005da7fd90 ffff880048da7840 ffff88005da7fdb0 ffff88005da7fd00 [13546.192438] Call Trace: [13546.193661] [] schedule+0x29/0x70 [13546.195089] [] taskq_thread+0x3fa/0x440 [spl] [13546.196587] [] ? wake_up_state+0x20/0x20 [13546.198058] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.199582] [] kthread+0xcf/0xe0 [13546.200977] [] ? kthread_create_on_node+0x140/0x140 [13546.202502] [] ret_from_fork+0x58/0x90 [13546.203945] [] ? kthread_create_on_node+0x140/0x140 [13546.205465] z_rd_int_6 S ffff88005da7fd00 0 19812 2 0x00000080 [13546.207068] ffff880060b1fe08 0000000000000046 ffff880060508fb0 ffff880060b1ffd8 [13546.208731] ffff880060b1ffd8 ffff880060b1ffd8 ffff880060508fb0 ffff880065916ea0 [13546.210400] ffff88005da7fd90 ffff880067591580 ffff88005da7fdb0 ffff88005da7fd00 [13546.212070] Call Trace: [13546.213292] [] schedule+0x29/0x70 [13546.214709] [] taskq_thread+0x3fa/0x440 [spl] [13546.216209] [] ? wake_up_state+0x20/0x20 [13546.217678] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.219206] [] kthread+0xcf/0xe0 [13546.220592] [] ? kthread_create_on_node+0x140/0x140 [13546.222123] [] ret_from_fork+0x58/0x90 [13546.223561] [] ? kthread_create_on_node+0x140/0x140 [13546.225127] z_rd_int_6 S ffff88005da7fd00 0 19813 2 0x00000080 [13546.226778] ffff8800682abe08 0000000000000046 ffff880063ae5e20 ffff8800682abfd8 [13546.228477] ffff8800682abfd8 ffff8800682abfd8 ffff880063ae5e20 ffff8800659168a0 [13546.230154] ffff88005da7fd90 ffff880068a96f90 ffff88005da7fdb0 ffff88005da7fd00 [13546.231830] Call Trace: [13546.233051] [] schedule+0x29/0x70 [13546.234475] [] taskq_thread+0x3fa/0x440 [spl] [13546.235986] [] ? wake_up_state+0x20/0x20 [13546.237452] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.238990] [] kthread+0xcf/0xe0 [13546.240394] [] ? kthread_create_on_node+0x140/0x140 [13546.241924] [] ret_from_fork+0x58/0x90 [13546.243363] [] ? kthread_create_on_node+0x140/0x140 [13546.244889] z_rd_int_6 S ffff88005da7fd00 0 19814 2 0x00000080 [13546.246490] ffff88005ec6fe08 0000000000000046 ffff880063ae4e70 ffff88005ec6ffd8 [13546.248161] ffff88005ec6ffd8 ffff88005ec6ffd8 ffff880063ae4e70 ffff880065916900 [13546.249822] ffff88005da7fd90 ffff880068a95e30 ffff88005da7fdb0 ffff88005da7fd00 [13546.251489] Call Trace: [13546.252710] [] schedule+0x29/0x70 [13546.254133] [] taskq_thread+0x3fa/0x440 [spl] [13546.255629] [] ? wake_up_state+0x20/0x20 [13546.257097] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.258632] [] kthread+0xcf/0xe0 [13546.260022] [] ? kthread_create_on_node+0x140/0x140 [13546.261552] [] ret_from_fork+0x58/0x90 [13546.262995] [] ? kthread_create_on_node+0x140/0x140 [13546.264508] z_rd_int_6 S ffff88005da7fd00 0 19815 2 0x00000080 [13546.266175] ffff88005fb4be08 0000000000000046 ffff880063ae2f10 ffff88005fb4bfd8 [13546.267838] ffff88005fb4bfd8 ffff88005fb4bfd8 ffff880063ae2f10 ffff880065916120 [13546.269499] ffff88005da7fd90 ffff880068a966e0 ffff88005da7fdb0 ffff88005da7fd00 [13546.271161] Call Trace: [13546.272397] [] schedule+0x29/0x70 [13546.273820] [] taskq_thread+0x3fa/0x440 [spl] [13546.275315] [] ? wake_up_state+0x20/0x20 [13546.276783] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.278309] [] kthread+0xcf/0xe0 [13546.279697] [] ? finish_task_switch+0x53/0x180 [13546.281188] [] ? kthread_create_on_node+0x140/0x140 [13546.282721] [] ret_from_fork+0x58/0x90 [13546.284153] [] ? kthread_create_on_node+0x140/0x140 [13546.285667] z_rd_int_6 S ffff88005da7fd00 0 19816 2 0x00000080 [13546.287279] ffff88005e2ffe08 0000000000000046 ffff880061c5edd0 ffff88005e2fffd8 [13546.288947] ffff88005e2fffd8 ffff88005e2fffd8 ffff880061c5edd0 ffff880065916c00 [13546.291270] ffff88005da7fd90 ffff880068a959d8 ffff88005da7fdb0 ffff88005da7fd00 [13546.293360] Call Trace: [13546.294586] [] schedule+0x29/0x70 [13546.296084] [] taskq_thread+0x3fa/0x440 [spl] [13546.297642] [] ? wake_up_state+0x20/0x20 [13546.299151] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.300719] [] kthread+0xcf/0xe0 [13546.302124] [] ? kthread_create_on_node+0x140/0x140 [13546.303728] [] ret_from_fork+0x58/0x90 [13546.305214] [] ? kthread_create_on_node+0x140/0x140 [13546.306779] z_rd_int_6 S ffff88005da7fd00 0 19817 2 0x00000080 [13546.308389] ffff88005f8d3e08 0000000000000046 ffff880061c5de20 ffff88005f8d3fd8 [13546.310095] ffff88005f8d3fd8 ffff88005f8d3fd8 ffff880061c5de20 ffff880065916d20 [13546.311750] ffff88005da7fd90 ffff880068a95580 ffff88005da7fdb0 ffff88005da7fd00 [13546.313460] Call Trace: [13546.314686] [] schedule+0x29/0x70 [13546.316193] [] taskq_thread+0x3fa/0x440 [spl] [13546.317743] [] ? wake_up_state+0x20/0x20 [13546.319257] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.320769] [] kthread+0xcf/0xe0 [13546.322234] [] ? kthread_create_on_node+0x140/0x140 [13546.323757] [] ret_from_fork+0x58/0x90 [13546.325263] [] ? kthread_create_on_node+0x140/0x140 [13546.326831] z_rd_int_7 S ffff88005da7f000 0 19818 2 0x00000080 [13546.328443] ffff880066103e08 0000000000000046 ffff880061c5ce70 ffff880066103fd8 [13546.330155] ffff880066103fd8 ffff880066103fd8 ffff880061c5ce70 ffff880065916600 [13546.331810] ffff88005da7f090 ffff880048da6288 ffff88005da7f0b0 ffff88005da7f000 [13546.333518] Call Trace: [13546.334743] [] schedule+0x29/0x70 [13546.336252] [] taskq_thread+0x3fa/0x440 [spl] [13546.337749] [] ? wake_up_state+0x20/0x20 [13546.339280] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.340856] [] kthread+0xcf/0xe0 [13546.342299] [] ? kthread_create_on_node+0x140/0x140 [13546.343876] [] ret_from_fork+0x58/0x90 [13546.345365] [] ? kthread_create_on_node+0x140/0x140 [13546.346932] z_rd_int_7 S ffff88005da7f000 0 19819 2 0x00000080 [13546.348553] ffff88004e36be08 0000000000000046 ffff880061c5bec0 ffff88004e36bfd8 [13546.350260] ffff88004e36bfd8 ffff88004e36bfd8 ffff880061c5bec0 ffff880065916780 [13546.351936] ffff88005da7f090 ffff880048da6288 ffff88005da7f0b0 ffff88005da7f000 [13546.353660] Call Trace: [13546.354897] [] schedule+0x29/0x70 [13546.356408] [] taskq_thread+0x3fa/0x440 [spl] [13546.357960] [] ? wake_up_state+0x20/0x20 [13546.359425] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.361028] [] kthread+0xcf/0xe0 [13546.362474] [] ? finish_task_switch+0x53/0x180 [13546.364029] [] ? kthread_create_on_node+0x140/0x140 [13546.365604] [] ret_from_fork+0x58/0x90 [13546.367091] [] ? kthread_create_on_node+0x140/0x140 [13546.368662] z_rd_int_7 S ffff88005da7f000 0 19820 2 0x00000080 [13546.370324] ffff880051963e08 0000000000000046 ffff880061c5af10 ffff880051963fd8 [13546.371992] ffff880051963fd8 ffff880051963fd8 ffff880061c5af10 ffff8800659163c0 [13546.373706] ffff88005da7f090 ffff880048da6288 ffff88005da7f0b0 ffff88005da7f000 [13546.375420] Call Trace: [13546.376645] [] schedule+0x29/0x70 [13546.378147] [] taskq_thread+0x3fa/0x440 [spl] [13546.379692] [] ? wake_up_state+0x20/0x20 [13546.381205] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.382779] [] kthread+0xcf/0xe0 [13546.384180] [] ? kthread_create_on_node+0x140/0x140 [13546.385778] [] ret_from_fork+0x58/0x90 [13546.387245] [] ? kthread_create_on_node+0x140/0x140 [13546.388828] z_rd_int_7 S ffff88005da7f000 0 19821 2 0x00000080 [13546.390441] ffff88007ba8be08 0000000000000046 ffff880061c59f60 ffff88007ba8bfd8 [13546.392150] ffff88007ba8bfd8 ffff88007ba8bfd8 ffff880061c59f60 ffff880065916720 [13546.393824] ffff88005da7f090 ffff880048da4cd0 ffff88005da7f0b0 ffff88005da7f000 [13546.395555] Call Trace: [13546.396792] [] schedule+0x29/0x70 [13546.398288] [] taskq_thread+0x3fa/0x440 [spl] [13546.399788] [] ? wake_up_state+0x20/0x20 [13546.401315] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.402834] [] kthread+0xcf/0xe0 [13546.404301] [] ? finish_task_switch+0x53/0x180 [13546.405851] [] ? kthread_create_on_node+0x140/0x140 [13546.407415] [] ret_from_fork+0x58/0x90 [13546.408848] [] ? kthread_create_on_node+0x140/0x140 [13546.410426] z_rd_int_7 S ffff88005da7f000 0 19822 2 0x00000080 [13546.412029] ffff88005cd67e08 0000000000000046 ffff880061c58fb0 ffff88005cd67fd8 [13546.413753] ffff88005cd67fd8 ffff88005cd67fd8 ffff880061c58fb0 ffff880065916f00 [13546.415478] ffff88005da7f090 ffff880048da4cd0 ffff88005da7f0b0 ffff88005da7f000 [13546.417174] Call Trace: [13546.418401] [] schedule+0x29/0x70 [13546.419886] [] taskq_thread+0x3fa/0x440 [spl] [13546.421374] [] ? wake_up_state+0x20/0x20 [13546.422889] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.424410] [] kthread+0xcf/0xe0 [13546.425879] [] ? kthread_create_on_node+0x140/0x140 [13546.427396] [] ret_from_fork+0x58/0x90 [13546.428891] [] ? kthread_create_on_node+0x140/0x140 [13546.430413] z_rd_int_7 S ffff88005da7f000 0 19823 2 0x00000080 [13546.432062] ffff8800607fbe08 0000000000000046 ffff880061c58000 ffff8800607fbfd8 [13546.433777] ffff8800607fbfd8 ffff8800607fbfd8 ffff880061c58000 ffff880065916a80 [13546.435492] ffff88005da7f090 ffff880048da5580 ffff88005da7f0b0 ffff88005da7f000 [13546.437196] Call Trace: [13546.438422] [] schedule+0x29/0x70 [13546.439910] [] taskq_thread+0x3fa/0x440 [spl] [13546.441423] [] ? wake_up_state+0x20/0x20 [13546.442947] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.444467] [] kthread+0xcf/0xe0 [13546.445927] [] ? kthread_create_on_node+0x140/0x140 [13546.447501] [] ret_from_fork+0x58/0x90 [13546.448987] [] ? kthread_create_on_node+0x140/0x140 [13546.450551] z_rd_int_7 S ffff88005da7f000 0 19824 2 0x00000080 [13546.452157] ffff88005d85fe08 0000000000000046 ffff8800368a1f60 ffff88005d85ffd8 [13546.453851] ffff88005d85ffd8 ffff88005d85ffd8 ffff8800368a1f60 ffff8800659169c0 [13546.455559] ffff88005da7f090 ffff880068a96b38 ffff88005da7f0b0 ffff88005da7f000 [13546.457261] Call Trace: [13546.458482] [] schedule+0x29/0x70 [13546.459993] [] taskq_thread+0x3fa/0x440 [spl] [13546.461540] [] ? wake_up_state+0x20/0x20 [13546.463047] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.464616] [] kthread+0xcf/0xe0 [13546.466064] [] ? finish_task_switch+0x53/0x180 [13546.467602] [] ? kthread_create_on_node+0x140/0x140 [13546.469128] [] ret_from_fork+0x58/0x90 [13546.470621] [] ? kthread_create_on_node+0x140/0x140 [13546.472200] z_rd_int_7 S ffff88005da7f000 0 19825 2 0x00000080 [13546.473793] ffff88005e4cfe08 0000000000000046 ffff8800368a2f10 ffff88005e4cffd8 [13546.475513] ffff88005e4cffd8 ffff88005e4cffd8 ffff8800368a2f10 ffff880065916cc0 [13546.477206] ffff88005da7f090 ffff880068a91580 ffff88005da7f0b0 ffff88005da7f000 [13546.478851] Call Trace: [13546.480108] [] schedule+0x29/0x70 [13546.481523] [] taskq_thread+0x3fa/0x440 [spl] [13546.483083] [] ? wake_up_state+0x20/0x20 [13546.484546] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.486122] [] kthread+0xcf/0xe0 [13546.487563] [] ? kthread_create_on_node+0x140/0x140 [13546.489142] [] ret_from_fork+0x58/0x90 [13546.490575] [] ? kthread_create_on_node+0x140/0x140 [13546.492160] z_rd_int_7 S ffff88005da7f000 0 19826 2 0x00000080 [13546.493754] ffff88007a103e08 0000000000000046 ffff880059a94e70 ffff88007a103fd8 [13546.495478] ffff88007a103fd8 ffff88007a103fd8 ffff880059a94e70 ffff8800659165a0 [13546.497181] ffff88005da7f090 ffff880068a94cd0 ffff88005da7f0b0 ffff88005da7f000 [13546.498833] Call Trace: [13546.500117] [] schedule+0x29/0x70 [13546.501536] [] taskq_thread+0x3fa/0x440 [spl] [13546.503102] [] ? wake_up_state+0x20/0x20 [13546.504607] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.506125] [] kthread+0xcf/0xe0 [13546.507583] [] ? kthread_create_on_node+0x140/0x140 [13546.509164] [] ret_from_fork+0x58/0x90 [13546.510597] [] ? kthread_create_on_node+0x140/0x140 [13546.512185] z_rd_int_7 S ffff88005da7f000 0 19827 2 0x00000080 [13546.513792] ffff88007c0d3e08 0000000000000046 ffff880059a90fb0 ffff88007c0d3fd8 [13546.515513] ffff88007c0d3fd8 ffff88007c0d3fd8 ffff880059a90fb0 ffff880065916480 [13546.517232] ffff88005da7f090 ffff88005e6c0878 ffff88005da7f0b0 ffff88005da7f000 [13546.518883] Call Trace: [13546.520140] [] schedule+0x29/0x70 [13546.521568] [] taskq_thread+0x3fa/0x440 [spl] [13546.523140] [] ? wake_up_state+0x20/0x20 [13546.524646] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.526226] [] kthread+0xcf/0xe0 [13546.527676] [] ? kthread_create_on_node+0x140/0x140 [13546.529219] [] ret_from_fork+0x58/0x90 [13546.530713] [] ? kthread_create_on_node+0x140/0x140 [13546.532280] z_rd_int_7 S ffff88005da7f000 0 19828 2 0x00000080 [13546.533884] ffff88005ec43e08 0000000000000046 ffff880059a96dd0 ffff88005ec43fd8 [13546.535608] ffff88005ec43fd8 ffff88005ec43fd8 ffff880059a96dd0 ffff880065916c60 [13546.537309] ffff88005da7f090 ffff880048d5f3e8 ffff88005da7f0b0 ffff88005da7f000 [13546.538965] Call Trace: [13546.540264] [] schedule+0x29/0x70 [13546.541694] [] taskq_thread+0x3fa/0x440 [spl] [13546.543263] [] ? wake_up_state+0x20/0x20 [13546.544774] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.546353] [] kthread+0xcf/0xe0 [13546.547741] [] ? kthread_create_on_node+0x140/0x140 [13546.549326] [] ret_from_fork+0x58/0x90 [13546.550767] [] ? kthread_create_on_node+0x140/0x140 [13546.552385] z_rd_int_7 S ffff88005da7f000 0 19829 2 0x00000080 [13546.553988] ffff880065c97e08 0000000000000046 ffff880059a91f60 ffff880065c97fd8 [13546.555737] ffff880065c97fd8 ffff880065c97fd8 ffff880059a91f60 ffff880065916840 [13546.557495] ffff88005da7f090 ffff880048da6b38 ffff88005da7f0b0 ffff88005da7f000 [13546.559191] Call Trace: [13546.560415] [] schedule+0x29/0x70 [13546.561902] [] taskq_thread+0x3fa/0x440 [spl] [13546.563459] [] ? wake_up_state+0x20/0x20 [13546.564926] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.566516] [] kthread+0xcf/0xe0 [13546.567957] [] ? kthread_create_on_node+0x140/0x140 [13546.569477] [] ret_from_fork+0x58/0x90 [13546.570987] [] ? kthread_create_on_node+0x140/0x140 [13546.572548] z_wr_iss S ffff88005da7f500 0 19830 2 0x00000080 [13546.574149] ffff88005d8ffe08 0000000000000046 ffff880059a92f10 ffff88005d8fffd8 [13546.575862] ffff88005d8fffd8 ffff88005d8fffd8 ffff880059a92f10 ffff8800659164e0 [13546.577583] ffff88005da7f590 ffff8800479bccd0 ffff88005da7f5b0 ffff88005da7f500 [13546.579295] Call Trace: [13546.580516] [] schedule+0x29/0x70 [13546.581974] [] taskq_thread+0x3fa/0x440 [spl] [13546.583518] [] ? wake_up_state+0x20/0x20 [13546.585040] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.586614] [] kthread+0xcf/0xe0 [13546.588009] [] ? kthread_create_on_node+0x140/0x140 [13546.589593] [] ret_from_fork+0x58/0x90 [13546.591042] [] ? kthread_create_on_node+0x140/0x140 [13546.592620] z_wr_iss_h S ffff88005da7ff00 0 19831 2 0x00000080 [13546.594218] ffff88005e883e08 0000000000000046 ffff880059a90000 ffff88005e883fd8 [13546.595932] ffff88005e883fd8 ffff88005e883fd8 ffff880059a90000 ffff880065916ba0 [13546.597648] ffff88005da7ff90 ffffffffa06624b0 ffff88005da7ffb0 ffff88005da7ff00 [13546.599364] Call Trace: [13546.600590] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.602217] [] schedule+0x29/0x70 [13546.603648] [] taskq_thread+0x3fa/0x440 [spl] [13546.605225] [] ? wake_up_state+0x20/0x20 [13546.606748] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.608323] [] kthread+0xcf/0xe0 [13546.609726] [] ? kthread_create_on_node+0x140/0x140 [13546.611338] [] ret_from_fork+0x58/0x90 [13546.612829] [] ? kthread_create_on_node+0x140/0x140 [13546.614406] z_wr_iss_h S ffff88005da7ff00 0 19832 2 0x00000080 [13546.616016] ffff88005eac7e08 0000000000000046 ffff88007a04edd0 ffff88005eac7fd8 [13546.617736] ffff88005eac7fd8 ffff88005eac7fd8 ffff88007a04edd0 ffff8800659167e0 [13546.619460] ffff88005da7ff90 ffffffffa06624b0 ffff88005da7ffb0 ffff88005da7ff00 [13546.621176] Call Trace: [13546.622415] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.624047] [] schedule+0x29/0x70 [13546.625540] [] taskq_thread+0x3fa/0x440 [spl] [13546.627058] [] ? wake_up_state+0x20/0x20 [13546.628587] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.630133] [] kthread+0xcf/0xe0 [13546.631623] [] ? kthread_create_on_node+0x140/0x140 [13546.633229] [] ret_from_fork+0x58/0x90 [13546.634677] [] ? kthread_create_on_node+0x140/0x140 [13546.636283] z_wr_iss_h S ffff88005da7ff00 0 19833 2 0x00000080 [13546.637914] ffff88005f65be08 0000000000000046 ffff88007a04ce70 ffff88005f65bfd8 [13546.639651] ffff88005f65bfd8 ffff88005f65bfd8 ffff88007a04ce70 ffff8800659160c0 [13546.641396] ffff88005da7ff90 ffffffffa06624b0 ffff88005da7ffb0 ffff88005da7ff00 [13546.643118] Call Trace: [13546.644380] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.646027] [] schedule+0x29/0x70 [13546.647522] [] taskq_thread+0x3fa/0x440 [spl] [13546.649092] [] ? wake_up_state+0x20/0x20 [13546.650574] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.652208] [] kthread+0xcf/0xe0 [13546.653635] [] ? kthread_create_on_node+0x140/0x140 [13546.655250] [] ret_from_fork+0x58/0x90 [13546.656763] [] ? kthread_create_on_node+0x140/0x140 [13546.658368] z_wr_iss_h S ffff88005da7ff00 0 19834 2 0x00000080 [13546.660006] ffff880050e7fe08 0000000000000046 ffff88007a049f60 ffff880050e7ffd8 [13546.661745] ffff880050e7ffd8 ffff880050e7ffd8 ffff88007a049f60 ffff8800659162a0 [13546.663497] ffff88005da7ff90 ffffffffa06624b0 ffff88005da7ffb0 ffff88005da7ff00 [13546.665227] Call Trace: [13546.666484] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.668128] [] schedule+0x29/0x70 [13546.669632] [] taskq_thread+0x3fa/0x440 [spl] [13546.671220] [] ? wake_up_state+0x20/0x20 [13546.672729] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.674352] [] kthread+0xcf/0xe0 [13546.675777] [] ? kthread_create_on_node+0x140/0x140 [13546.677398] [] ret_from_fork+0x58/0x90 [13546.678918] [] ? kthread_create_on_node+0x140/0x140 [13546.680515] z_wr_iss_h S ffff88005da7ff00 0 19835 2 0x00000080 [13546.682152] ffff88005f42be08 0000000000000046 ffff88007a04bec0 ffff88005f42bfd8 [13546.683891] ffff88005f42bfd8 ffff88005f42bfd8 ffff88007a04bec0 ffff880065916060 [13546.685636] ffff88005da7ff90 ffffffffa06624b0 ffff88005da7ffb0 ffff88005da7ff00 [13546.687387] Call Trace: [13546.688646] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.690291] [] schedule+0x29/0x70 [13546.691794] [] taskq_thread+0x3fa/0x440 [spl] [13546.693367] [] ? wake_up_state+0x20/0x20 [13546.694867] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.696485] [] kthread+0xcf/0xe0 [13546.697960] [] ? kthread_create_on_node+0x140/0x140 [13546.699518] [] ret_from_fork+0x58/0x90 [13546.701053] [] ? kthread_create_on_node+0x140/0x140 [13546.702601] z_wr_int_0 S ffff88005da7f700 0 19836 2 0x00000080 [13546.704340] ffff88006610be08 0000000000000046 ffff88007a048fb0 ffff88006610bfd8 [13546.706040] ffff88006610bfd8 ffff88006610bfd8 ffff88007a048fb0 ffff880065916360 [13546.707765] ffff88005da7f790 ffff88005ab7af90 ffff88005da7f7b0 ffff88005da7f700 [13546.709509] Call Trace: [13546.710763] [] schedule+0x29/0x70 [13546.712296] [] taskq_thread+0x3fa/0x440 [spl] [13546.713861] [] ? wake_up_state+0x20/0x20 [13546.715347] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.716981] [] kthread+0xcf/0xe0 [13546.718403] [] ? kthread_create_on_node+0x140/0x140 [13546.720029] [] ret_from_fork+0x58/0x90 [13546.721535] [] ? kthread_create_on_node+0x140/0x140 [13546.723126] z_wr_int_0 S ffff88005da7f700 0 19837 2 0x00000080 [13546.724750] ffff88005f90fe08 0000000000000046 ffff880045f78fb0 ffff88005f90ffd8 [13546.726496] ffff88005f90ffd8 ffff88005f90ffd8 ffff880045f78fb0 ffff8800659166c0 [13546.728225] ffff88005da7f790 ffff88005ab7b3e8 ffff88005da7f7b0 ffff88005da7f700 [13546.729899] Call Trace: [13546.731196] [] schedule+0x29/0x70 [13546.732684] [] taskq_thread+0x3fa/0x440 [spl] [13546.734196] [] ? wake_up_state+0x20/0x20 [13546.735729] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.737320] [] kthread+0xcf/0xe0 [13546.738738] [] ? kthread_create_on_node+0x140/0x140 [13546.740347] [] ret_from_fork+0x58/0x90 [13546.741799] [] ? kthread_create_on_node+0x140/0x140 [13546.743393] z_wr_int_0 S ffff88005da7f700 0 19838 2 0x00000080 [13546.745009] ffff88005f33be08 0000000000000046 ffff880045f7de20 ffff88005f33bfd8 [13546.746725] ffff88005f33bfd8 ffff88005f33bfd8 ffff880045f7de20 ffff880065916d80 [13546.748454] ffff88005da7f790 ffff880048d5fc98 ffff88005da7f7b0 ffff88005da7f700 [13546.750168] Call Trace: [13546.751403] [] schedule+0x29/0x70 [13546.752913] [] taskq_thread+0x3fa/0x440 [spl] [13546.754424] [] ? wake_up_state+0x20/0x20 [13546.755963] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.757540] [] kthread+0xcf/0xe0 [13546.759013] [] ? kthread_create_on_node+0x140/0x140 [13546.760599] [] ret_from_fork+0x58/0x90 [13546.762090] [] ? kthread_create_on_node+0x140/0x140 [13546.763665] z_wr_int_0 S ffff88005da7f700 0 19839 2 0x00000080 [13546.765322] ffff88005efafe08 0000000000000046 ffff880045f7edd0 ffff88005efaffd8 [13546.766993] ffff88005efaffd8 ffff88005efaffd8 ffff880045f7edd0 ffff880065916300 [13546.768706] ffff88005da7f790 ffff880066f10878 ffff88005da7f7b0 ffff88005da7f700 [13546.770434] Call Trace: [13546.771658] [] schedule+0x29/0x70 [13546.773160] [] taskq_thread+0x3fa/0x440 [spl] [13546.774658] [] ? wake_up_state+0x20/0x20 [13546.776198] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.777719] [] kthread+0xcf/0xe0 [13546.779177] [] ? kthread_create_on_node+0x140/0x140 [13546.780713] [] ret_from_fork+0x58/0x90 [13546.782222] [] ? kthread_create_on_node+0x140/0x140 [13546.783796] z_wr_int_0 S ffff88005da7f700 0 19840 2 0x00000080 [13546.785403] ffff88005edf3e08 0000000000000046 ffff880045f7ce70 ffff88005edf3fd8 [13546.787116] ffff88005edf3fd8 ffff88005edf3fd8 ffff880045f7ce70 ffff880065916ae0 [13546.788770] ffff88005da7f790 ffff880048d5ef90 ffff88005da7f7b0 ffff88005da7f700 [13546.790485] Call Trace: [13546.791708] [] schedule+0x29/0x70 [13546.793219] [] taskq_thread+0x3fa/0x440 [spl] [13546.794777] [] ? wake_up_state+0x20/0x20 [13546.796291] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.797873] [] kthread+0xcf/0xe0 [13546.799323] [] ? kthread_create_on_node+0x140/0x140 [13546.800901] [] ret_from_fork+0x58/0x90 [13546.802345] [] ? kthread_create_on_node+0x140/0x140 [13546.803946] z_wr_int_0 S ffff88005da7f700 0 19841 2 0x00000080 [13546.805561] ffff88005e22fe08 0000000000000046 ffff880045f7af10 ffff88005e22ffd8 [13546.807282] ffff88005e22ffd8 ffff88005e22ffd8 ffff880045f7af10 ffff880065916000 [13546.808941] ffff88005da7f790 ffff880048d5e288 ffff88005da7f7b0 ffff88005da7f700 [13546.810665] Call Trace: [13546.811896] [] schedule+0x29/0x70 [13546.813394] [] taskq_thread+0x3fa/0x440 [spl] [13546.814900] [] ? wake_up_state+0x20/0x20 [13546.816431] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.817967] [] kthread+0xcf/0xe0 [13546.819424] [] ? kthread_create_on_node+0x140/0x140 [13546.821019] [] ret_from_fork+0x58/0x90 [13546.822526] [] ? kthread_create_on_node+0x140/0x140 [13546.824058] z_wr_int_0 S ffff88005da7f700 0 19842 2 0x00000080 [13546.825711] ffff88005b6afe08 0000000000000046 ffff880045f7bec0 ffff88005b6affd8 [13546.827445] ffff88005b6affd8 ffff88005b6affd8 ffff880045f7bec0 ffff880065916960 [13546.829157] ffff88005da7f790 ffff8800679d99d8 ffff88005da7f7b0 ffff88005da7f700 [13546.830812] Call Trace: [13546.832079] [] schedule+0x29/0x70 [13546.833515] [] taskq_thread+0x3fa/0x440 [spl] [13546.835095] [] ? wake_up_state+0x20/0x20 [13546.836576] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.838176] [] kthread+0xcf/0xe0 [13546.839624] [] ? kthread_create_on_node+0x140/0x140 [13546.841153] [] ret_from_fork+0x58/0x90 [13546.842666] [] ? kthread_create_on_node+0x140/0x140 [13546.844188] z_wr_int_0 S ffff88005da7f700 0 19843 2 0x00000080 [13546.845839] ffff88005f81fe08 0000000000000046 ffff880045f78000 ffff88005f81ffd8 [13546.847561] ffff88005f81ffd8 ffff88005f81ffd8 ffff880045f78000 ffff880065916180 [13546.849266] ffff88005da7f790 ffff880048d5d9d8 ffff88005da7f7b0 ffff88005da7f700 [13546.850923] Call Trace: [13546.852182] [] schedule+0x29/0x70 [13546.853653] [] taskq_thread+0x3fa/0x440 [spl] [13546.855155] [] ? wake_up_state+0x20/0x20 [13546.856685] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.858213] [] kthread+0xcf/0xe0 [13546.859667] [] ? kthread_create_on_node+0x140/0x140 [13546.861194] [] ret_from_fork+0x58/0x90 [13546.862701] [] ? kthread_create_on_node+0x140/0x140 [13546.864272] z_wr_int_0 S ffff88005da7f700 0 19844 2 0x00000080 [13546.865869] ffff880067e17e08 0000000000000046 ffff88007b1f5e20 ffff880067e17fd8 [13546.867588] ffff880067e17fd8 ffff880067e17fd8 ffff88007b1f5e20 ffff880060f950c0 [13546.869288] ffff88005da7f790 ffff880048d5fc98 ffff88005da7f7b0 ffff88005da7f700 [13546.870940] Call Trace: [13546.872199] [] schedule+0x29/0x70 [13546.873620] [] taskq_thread+0x3fa/0x440 [spl] [13546.875190] [] ? wake_up_state+0x20/0x20 [13546.876655] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.878254] [] kthread+0xcf/0xe0 [13546.879641] [] ? kthread_create_on_node+0x140/0x140 [13546.881233] [] ret_from_fork+0x58/0x90 [13546.882724] [] ? kthread_create_on_node+0x140/0x140 [13546.884244] z_wr_int_0 S ffff88005da7f700 0 19845 2 0x00000080 [13546.885894] ffff880048723e08 0000000000000046 ffff88007b1f4e70 ffff880048723fd8 [13546.887602] ffff880048723fd8 ffff880048723fd8 ffff88007b1f4e70 ffff880060f95720 [13546.889310] ffff88005da7f790 ffff880048d5e288 ffff88005da7f7b0 ffff88005da7f700 [13546.890966] Call Trace: [13546.892225] [] schedule+0x29/0x70 [13546.893696] [] taskq_thread+0x3fa/0x440 [spl] [13546.895212] [] ? wake_up_state+0x20/0x20 [13546.896748] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.898275] [] kthread+0xcf/0xe0 [13546.899738] [] ? kthread_create_on_node+0x140/0x140 [13546.901313] [] ret_from_fork+0x58/0x90 [13546.902810] [] ? kthread_create_on_node+0x140/0x140 [13546.904327] z_wr_int_0 S ffff88005da7f700 0 19846 2 0x00000080 [13546.905978] ffff88005ecd7e08 0000000000000046 ffff88007b1f3ec0 ffff88005ecd7fd8 [13546.907699] ffff88005ecd7fd8 ffff88005ecd7fd8 ffff88007b1f3ec0 ffff880060f95060 [13546.909423] ffff88005da7f790 ffff88004919a288 ffff88005da7f7b0 ffff88005da7f700 [13546.911125] Call Trace: [13546.912351] [] schedule+0x29/0x70 [13546.913844] [] taskq_thread+0x3fa/0x440 [spl] [13546.915397] [] ? wake_up_state+0x20/0x20 [13546.916915] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.918493] [] kthread+0xcf/0xe0 [13546.919939] [] ? kthread_create_on_node+0x140/0x140 [13546.921512] [] ret_from_fork+0x58/0x90 [13546.923023] [] ? kthread_create_on_node+0x140/0x140 [13546.924539] z_wr_int_0 S ffff88005da7f700 0 19847 2 0x00000080 [13546.926195] ffff88005e7b7e08 0000000000000046 ffff88007b1f0fb0 ffff88005e7b7fd8 [13546.927910] ffff88005e7b7fd8 ffff88005e7b7fd8 ffff88007b1f0fb0 ffff880060f952a0 [13546.929633] ffff88005da7f790 ffff88005ab7bc98 ffff88005da7f7b0 ffff88005da7f700 [13546.931349] Call Trace: [13546.932581] [] schedule+0x29/0x70 [13546.934101] [] taskq_thread+0x3fa/0x440 [spl] [13546.935602] [] ? wake_up_state+0x20/0x20 [13546.937139] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.938670] [] kthread+0xcf/0xe0 [13546.940128] [] ? kthread_create_on_node+0x140/0x140 [13546.941708] [] ret_from_fork+0x58/0x90 [13546.943208] [] ? kthread_create_on_node+0x140/0x140 [13546.944783] z_wr_int_1 S ffff88005da7fb00 0 19848 2 0x00000080 [13546.946405] ffff88005cd73e08 0000000000000046 ffff88007b1f6dd0 ffff88005cd73fd8 [13546.948114] ffff88005cd73fd8 ffff88005cd73fd8 ffff88007b1f6dd0 ffff880060f95ea0 [13546.949784] ffff88005da7fb90 ffff880066f11128 ffff88005da7fbb0 ffff88005da7fb00 [13546.951503] Call Trace: [13546.952729] [] schedule+0x29/0x70 [13546.954237] [] taskq_thread+0x3fa/0x440 [spl] [13546.955791] [] ? wake_up_state+0x20/0x20 [13546.957265] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.958860] [] kthread+0xcf/0xe0 [13546.960253] [] ? kthread_create_on_node+0x140/0x140 [13546.961851] [] ret_from_fork+0x58/0x90 [13546.963341] [] ? kthread_create_on_node+0x140/0x140 [13546.964865] z_wr_int_1 S ffff88005da7fb00 0 19849 2 0x00000080 [13546.966521] ffff88004fed3e08 0000000000000046 ffff88007b1f0000 ffff88004fed3fd8 [13546.968237] ffff88004fed3fd8 ffff88004fed3fd8 ffff88007b1f0000 ffff880060f957e0 [13546.969893] ffff88005da7fb90 ffff880048d5e6e0 ffff88005da7fbb0 ffff88005da7fb00 [13546.971631] Call Trace: [13546.972871] [] schedule+0x29/0x70 [13546.974379] [] taskq_thread+0x3fa/0x440 [spl] [13546.975944] [] ? wake_up_state+0x20/0x20 [13546.977418] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.979032] [] kthread+0xcf/0xe0 [13546.980426] [] ? kthread_create_on_node+0x140/0x140 [13546.982031] [] ret_from_fork+0x58/0x90 [13546.983533] [] ? kthread_create_on_node+0x140/0x140 [13546.985107] z_wr_int_1 S ffff88005da7fb00 0 19850 2 0x00000080 [13546.986712] ffff88005f323e08 0000000000000046 ffff88007a123ec0 ffff88005f323fd8 [13546.988457] ffff88005f323fd8 ffff88005f323fd8 ffff88007a123ec0 ffff880060f95ba0 [13546.990170] ffff88005da7fb90 ffff8800679da6e0 ffff88005da7fbb0 ffff88005da7fb00 [13546.991846] Call Trace: [13546.993105] [] schedule+0x29/0x70 [13546.994527] [] taskq_thread+0x3fa/0x440 [spl] [13546.996096] [] ? wake_up_state+0x20/0x20 [13546.997613] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13546.999192] [] kthread+0xcf/0xe0 [13547.000632] [] ? kthread_create_on_node+0x140/0x140 [13547.002211] [] ret_from_fork+0x58/0x90 [13547.003654] [] ? kthread_create_on_node+0x140/0x140 [13547.005244] z_wr_int_1 S ffff88005da7fb00 0 19851 2 0x00000080 [13547.006855] ffff88007929be08 0000000000000046 ffff88007a120000 ffff88007929bfd8 [13547.008573] ffff88007929bfd8 ffff88007929bfd8 ffff88007a120000 ffff880060f95600 [13547.010279] ffff88005da7fb90 ffff8800679d9128 ffff88005da7fbb0 ffff88005da7fb00 [13547.011935] Call Trace: [13547.013198] [] schedule+0x29/0x70 [13547.014623] [] taskq_thread+0x3fa/0x440 [spl] [13547.016189] [] ? wake_up_state+0x20/0x20 [13547.017711] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.019296] [] kthread+0xcf/0xe0 [13547.020689] [] ? kthread_create_on_node+0x140/0x140 [13547.022291] [] ret_from_fork+0x58/0x90 [13547.023791] [] ? kthread_create_on_node+0x140/0x140 [13547.025321] z_wr_int_1 S ffff88005da7fb00 0 19852 2 0x00000080 [13547.026977] ffff880061617e08 0000000000000046 ffff880036186dd0 ffff880061617fd8 [13547.028703] ffff880061617fd8 ffff880061617fd8 ffff880036186dd0 ffff880060f95360 [13547.030429] ffff88005da7fb90 ffff88005ab7af90 ffff88005da7fbb0 ffff88005da7fb00 [13547.032140] Call Trace: [13547.033368] [] schedule+0x29/0x70 [13547.034860] [] taskq_thread+0x3fa/0x440 [spl] [13547.036423] [] ? wake_up_state+0x20/0x20 [13547.037946] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.039534] [] kthread+0xcf/0xe0 [13547.040981] [] ? kthread_create_on_node+0x140/0x140 [13547.042568] [] ret_from_fork+0x58/0x90 [13547.044065] [] ? kthread_create_on_node+0x140/0x140 [13547.045583] z_wr_int_1 S ffff88005da7fb00 0 19853 2 0x00000080 [13547.047277] ffff88004ae2fe08 0000000000000046 ffff880036180fb0 ffff88004ae2ffd8 [13547.048942] ffff88004ae2ffd8 ffff88004ae2ffd8 ffff880036180fb0 ffff880060f95120 [13547.050659] ffff88005da7fb90 ffff88005ab7af90 ffff88005da7fbb0 ffff88005da7fb00 [13547.052382] Call Trace: [13547.053611] [] schedule+0x29/0x70 [13547.055114] [] taskq_thread+0x3fa/0x440 [spl] [13547.056668] [] ? wake_up_state+0x20/0x20 [13547.058146] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.059741] [] kthread+0xcf/0xe0 [13547.061138] [] ? kthread_create_on_node+0x140/0x140 [13547.062736] [] ret_from_fork+0x58/0x90 [13547.064232] [] ? kthread_create_on_node+0x140/0x140 [13547.065808] z_wr_int_1 S ffff88005da7fb00 0 19854 2 0x00000080 [13547.067416] ffff880066f87e08 0000000000000046 ffff88006081bec0 ffff880066f87fd8 [13547.069135] ffff880066f87fd8 ffff880066f87fd8 ffff88006081bec0 ffff880060f95780 [13547.070795] ffff88005da7fb90 ffff88005ab79580 ffff88005da7fbb0 ffff88005da7fb00 [13547.072507] Call Trace: [13547.073735] [] schedule+0x29/0x70 [13547.075235] [] taskq_thread+0x3fa/0x440 [spl] [13547.076736] [] ? wake_up_state+0x20/0x20 [13547.078270] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.079800] [] kthread+0xcf/0xe0 [13547.081255] [] ? kthread_create_on_node+0x140/0x140 [13547.082780] [] ret_from_fork+0x58/0x90 [13547.084287] [] ? kthread_create_on_node+0x140/0x140 [13547.085809] z_wr_int_1 S ffff88005da7fb00 0 19855 2 0x00000080 [13547.087461] ffff880045d33e08 0000000000000046 ffff88006081edd0 ffff880045d33fd8 [13547.089176] ffff880045d33fd8 ffff880045d33fd8 ffff88006081edd0 ffff880060f95c60 [13547.090833] ffff88005da7fb90 ffff8800679d99d8 ffff88005da7fbb0 ffff88005da7fb00 [13547.092543] Call Trace: [13547.093785] [] schedule+0x29/0x70 [13547.095292] [] taskq_thread+0x3fa/0x440 [spl] [13547.096845] [] ? wake_up_state+0x20/0x20 [13547.098368] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.099953] [] kthread+0xcf/0xe0 [13547.101394] [] ? kthread_create_on_node+0x140/0x140 [13547.102982] [] ret_from_fork+0x58/0x90 [13547.104423] [] ? kthread_create_on_node+0x140/0x140 [13547.106028] z_wr_int_1 S ffff88005da7fb00 0 19856 2 0x00000080 [13547.107624] ffff88006812fe08 0000000000000046 ffff880060818000 ffff88006812ffd8 [13547.109333] ffff88006812ffd8 ffff88006812ffd8 ffff880060818000 ffff880060f95ae0 [13547.110992] ffff88005da7fb90 ffff8800679db840 ffff88005da7fbb0 ffff88005da7fb00 [13547.112764] Call Trace: [13547.113990] [] schedule+0x29/0x70 [13547.115487] [] taskq_thread+0x3fa/0x440 [spl] [13547.117053] [] ? wake_up_state+0x20/0x20 [13547.118520] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.120120] [] kthread+0xcf/0xe0 [13547.121516] [] ? kthread_create_on_node+0x140/0x140 [13547.123113] [] ret_from_fork+0x58/0x90 [13547.124562] [] ? kthread_create_on_node+0x140/0x140 [13547.126155] z_wr_int_1 S ffff88005da7fb00 0 19857 2 0x00000080 [13547.127764] ffff8800682a7e08 0000000000000046 ffff880060818fb0 ffff8800682a7fd8 [13547.129484] ffff8800682a7fd8 ffff8800682a7fd8 ffff880060818fb0 ffff880060f95de0 [13547.131194] ffff88005da7fb90 ffff880066f11580 ffff88005da7fbb0 ffff88005da7fb00 [13547.132850] Call Trace: [13547.134115] [] schedule+0x29/0x70 [13547.135597] [] taskq_thread+0x3fa/0x440 [spl] [13547.137107] [] ? wake_up_state+0x20/0x20 [13547.138646] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.140181] [] kthread+0xcf/0xe0 [13547.141638] [] ? kthread_create_on_node+0x140/0x140 [13547.143227] [] ret_from_fork+0x58/0x90 [13547.144717] [] ? kthread_create_on_node+0x140/0x140 [13547.146289] z_wr_int_1 S ffff88005da7fb00 0 19858 2 0x00000080 [13547.147889] ffff88005fa6be08 0000000000000046 ffff88006081ce70 ffff88005fa6bfd8 [13547.149599] ffff88005fa6bfd8 ffff88005fa6bfd8 ffff88006081ce70 ffff880060f95420 [13547.151316] ffff88005da7fb90 ffff880066f11580 ffff88005da7fbb0 ffff88005da7fb00 [13547.152973] Call Trace: [13547.154231] [] schedule+0x29/0x70 [13547.155651] [] taskq_thread+0x3fa/0x440 [spl] [13547.157230] [] ? wake_up_state+0x20/0x20 [13547.158695] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.160315] [] kthread+0xcf/0xe0 [13547.161715] [] ? kthread_create_on_node+0x140/0x140 [13547.163301] [] ret_from_fork+0x58/0x90 [13547.164737] [] ? kthread_create_on_node+0x140/0x140 [13547.166327] z_wr_int_1 S ffff88005da7fb00 0 19859 2 0x00000080 [13547.167926] ffff8800684ffe08 0000000000000046 ffff88006081de20 ffff8800684fffd8 [13547.169631] ffff8800684fffd8 ffff8800684fffd8 ffff88006081de20 ffff880060f95cc0 [13547.171347] ffff88005da7fb90 ffff880066f119d8 ffff88005da7fbb0 ffff88005da7fb00 [13547.172999] Call Trace: [13547.174257] [] schedule+0x29/0x70 [13547.175726] [] taskq_thread+0x3fa/0x440 [spl] [13547.177225] [] ? wake_up_state+0x20/0x20 [13547.178761] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.180334] [] kthread+0xcf/0xe0 [13547.181719] [] ? kthread_create_on_node+0x140/0x140 [13547.183305] [] ret_from_fork+0x58/0x90 [13547.184791] [] ? kthread_create_on_node+0x140/0x140 [13547.186303] z_wr_int_2 S ffff88005da7f400 0 19860 2 0x00000080 [13547.187958] ffff88005f653e08 0000000000000046 ffff8800792fedd0 ffff88005f653fd8 [13547.189670] ffff88005f653fd8 ffff88005f653fd8 ffff8800792fedd0 ffff880060f95540 [13547.191394] ffff88005da7f490 ffff88005ab78cd0 ffff88005da7f4b0 ffff88005da7f400 [13547.193101] Call Trace: [13547.194338] [] schedule+0x29/0x70 [13547.195826] [] taskq_thread+0x3fa/0x440 [spl] [13547.197376] [] ? wake_up_state+0x20/0x20 [13547.198907] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.200430] [] kthread+0xcf/0xe0 [13547.201890] [] ? kthread_create_on_node+0x140/0x140 [13547.203471] [] ret_from_fork+0x58/0x90 [13547.204913] [] ? kthread_create_on_node+0x140/0x140 [13547.206497] z_wr_int_2 S ffff88005da7f400 0 19861 2 0x00000080 [13547.208105] ffff880066317e08 0000000000000046 ffff8800792fbec0 ffff880066317fd8 [13547.209811] ffff880066317fd8 ffff880066317fd8 ffff8800792fbec0 ffff880060f95660 [13547.211520] ffff88005da7f490 ffff880048d5c420 ffff88005da7f4b0 ffff88005da7f400 [13547.213216] Call Trace: [13547.214443] [] schedule+0x29/0x70 [13547.215941] [] taskq_thread+0x3fa/0x440 [spl] [13547.217487] [] ? wake_up_state+0x20/0x20 [13547.218959] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.220547] [] kthread+0xcf/0xe0 [13547.221992] [] ? kthread_create_on_node+0x140/0x140 [13547.223522] [] ret_from_fork+0x58/0x90 [13547.225035] [] ? kthread_create_on_node+0x140/0x140 [13547.226613] z_wr_int_2 S ffff88005da7f400 0 19862 2 0x00000080 [13547.228214] ffff88006677fe08 0000000000000046 ffff88007921edd0 ffff88006677ffd8 [13547.229926] ffff88006677ffd8 ffff88006677ffd8 ffff88007921edd0 ffff880060f95e40 [13547.231642] ffff88005da7f490 ffff8800679db840 ffff88005da7f4b0 ffff88005da7f400 [13547.233361] Call Trace: [13547.234586] [] schedule+0x29/0x70 [13547.236066] [] taskq_thread+0x3fa/0x440 [spl] [13547.237620] [] ? wake_up_state+0x20/0x20 [13547.239143] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.240726] [] kthread+0xcf/0xe0 [13547.242121] [] ? kthread_create_on_node+0x140/0x140 [13547.243712] [] ret_from_fork+0x58/0x90 [13547.245213] [] ? kthread_create_on_node+0x140/0x140 [13547.246731] z_wr_int_2 S ffff88005da7f400 0 19863 2 0x00000080 [13547.248403] ffff880063017e08 0000000000000046 ffff88007921af10 ffff880063017fd8 [13547.250111] ffff880063017fd8 ffff880063017fd8 ffff88007921af10 ffff880060f958a0 [13547.251774] ffff88005da7f490 ffff880048d5f3e8 ffff88005da7f4b0 ffff88005da7f400 [13547.253470] Call Trace: [13547.254696] [] schedule+0x29/0x70 [13547.256208] [] taskq_thread+0x3fa/0x440 [spl] [13547.257759] [] ? wake_up_state+0x20/0x20 [13547.259275] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.260864] [] kthread+0xcf/0xe0 [13547.262304] [] ? kthread_create_on_node+0x140/0x140 [13547.263885] [] ret_from_fork+0x58/0x90 [13547.265376] [] ? kthread_create_on_node+0x140/0x140 [13547.266897] z_wr_int_2 S ffff88005da7f400 0 19864 2 0x00000080 [13547.268540] ffff88004b57fe08 0000000000000046 ffff88005f6d5e20 ffff88004b57ffd8 [13547.270255] ffff88004b57ffd8 ffff88004b57ffd8 ffff88005f6d5e20 ffff880060f95d20 [13547.271929] ffff88005da7f490 ffff8800679d8cd0 ffff88005da7f4b0 ffff88005da7f400 [13547.273663] Call Trace: [13547.274903] [] schedule+0x29/0x70 [13547.276404] [] taskq_thread+0x3fa/0x440 [spl] [13547.277971] [] ? wake_up_state+0x20/0x20 [13547.279487] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.281078] [] kthread+0xcf/0xe0 [13547.282519] [] ? kthread_create_on_node+0x140/0x140 [13547.284108] [] ret_from_fork+0x58/0x90 [13547.285571] [] ? kthread_create_on_node+0x140/0x140 [13547.287152] z_wr_int_2 S ffff88005da7f400 0 19865 2 0x00000080 [13547.288757] ffff880065c8fe08 0000000000000046 ffff88005f6d2f10 ffff880065c8ffd8 [13547.290484] ffff880065c8ffd8 ffff880065c8ffd8 ffff88005f6d2f10 ffff880060f95a80 [13547.292194] ffff88005da7f490 ffff8800679da6e0 ffff88005da7f4b0 ffff88005da7f400 [13547.293861] Call Trace: [13547.295120] [] schedule+0x29/0x70 [13547.296601] [] taskq_thread+0x3fa/0x440 [spl] [13547.298103] [] ? wake_up_state+0x20/0x20 [13547.299633] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.301230] [] kthread+0xcf/0xe0 [13547.302670] [] ? kthread_create_on_node+0x140/0x140 [13547.304250] [] ret_from_fork+0x58/0x90 [13547.305689] [] ? kthread_create_on_node+0x140/0x140 [13547.307278] z_wr_int_2 S ffff88005da7f400 0 19866 2 0x00000080 [13547.308872] ffff880062807e08 0000000000000046 ffff88005f6d0000 ffff880062807fd8 [13547.310597] ffff880062807fd8 ffff880062807fd8 ffff88005f6d0000 ffff880060f956c0 [13547.312297] ffff88005da7f490 ffff88005b1d6288 ffff88005da7f4b0 ffff88005da7f400 [13547.313957] Call Trace: [13547.315212] [] schedule+0x29/0x70 [13547.316680] [] taskq_thread+0x3fa/0x440 [spl] [13547.318229] [] ? wake_up_state+0x20/0x20 [13547.319744] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.321321] [] kthread+0xcf/0xe0 [13547.322761] [] ? finish_task_switch+0x53/0x180 [13547.324248] [] ? kthread_create_on_node+0x140/0x140 [13547.325844] [] ret_from_fork+0x58/0x90 [13547.327269] [] ? kthread_create_on_node+0x140/0x140 [13547.328856] z_wr_int_2 S ffff88005da7f400 0 19867 2 0x00000080 [13547.330459] ffff880062a2be08 0000000000000046 ffff88005f6d3ec0 ffff880062a2bfd8 [13547.332173] ffff880062a2bfd8 ffff880062a2bfd8 ffff88005f6d3ec0 ffff880060f95960 [13547.333826] ffff88005da7f490 ffff88005b1d7c98 ffff88005da7f4b0 ffff88005da7f400 [13547.335528] Call Trace: [13547.336752] [] schedule+0x29/0x70 [13547.338243] [] taskq_thread+0x3fa/0x440 [spl] [13547.339735] [] ? wake_up_state+0x20/0x20 [13547.341251] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.342763] [] kthread+0xcf/0xe0 [13547.344218] [] ? kthread_create_on_node+0x140/0x140 [13547.345742] [] ret_from_fork+0x58/0x90 [13547.347230] [] ? kthread_create_on_node+0x140/0x140 [13547.348743] z_wr_int_2 S ffff88005da7f400 0 19868 2 0x00000080 [13547.350402] ffff880060d13e08 0000000000000046 ffff88005f6d1f60 ffff880060d13fd8 [13547.352104] ffff880060d13fd8 ffff880060d13fd8 ffff88005f6d1f60 ffff880060f955a0 [13547.353755] ffff88005da7f490 ffff8800630b1e30 ffff88005da7f4b0 ffff88005da7f400 [13547.355454] Call Trace: [13547.356674] [] schedule+0x29/0x70 [13547.358181] [] taskq_thread+0x3fa/0x440 [spl] [13547.359671] [] ? wake_up_state+0x20/0x20 [13547.361189] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.362698] [] kthread+0xcf/0xe0 [13547.364156] [] ? kthread_create_on_node+0x140/0x140 [13547.365689] [] ret_from_fork+0x58/0x90 [13547.367190] [] ? kthread_create_on_node+0x140/0x140 [13547.368751] z_wr_int_2 S ffff88005da7f400 0 19869 2 0x00000080 [13547.370399] ffff88007a707e08 0000000000000046 ffff88005f6d6dd0 ffff88007a707fd8 [13547.372102] ffff88007a707fd8 ffff88007a707fd8 ffff88005f6d6dd0 ffff880060f95180 [13547.373765] ffff88005da7f490 ffff8800679da288 ffff88005da7f4b0 ffff88005da7f400 [13547.375465] Call Trace: [13547.376684] [] schedule+0x29/0x70 [13547.378182] [] taskq_thread+0x3fa/0x440 [spl] [13547.379730] [] ? wake_up_state+0x20/0x20 [13547.381230] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.382744] [] kthread+0xcf/0xe0 [13547.384204] [] ? kthread_create_on_node+0x140/0x140 [13547.385725] [] ret_from_fork+0x58/0x90 [13547.387216] [] ? kthread_create_on_node+0x140/0x140 [13547.388732] z_wr_int_2 S ffff88005da7f400 0 19870 2 0x00000080 [13547.390386] ffff8800615c3e08 0000000000000046 ffff88005f6d4e70 ffff8800615c3fd8 [13547.392202] ffff8800615c3fd8 ffff8800615c3fd8 ffff88005f6d4e70 ffff880060f954e0 [13547.393856] ffff88005da7f490 ffff880048d5f840 ffff88005da7f4b0 ffff88005da7f400 [13547.395560] Call Trace: [13547.396800] [] schedule+0x29/0x70 [13547.398303] [] taskq_thread+0x3fa/0x440 [spl] [13547.399857] [] ? wake_up_state+0x20/0x20 [13547.401335] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.402929] [] kthread+0xcf/0xe0 [13547.404320] [] ? kthread_create_on_node+0x140/0x140 [13547.405916] [] ret_from_fork+0x58/0x90 [13547.407403] [] ? kthread_create_on_node+0x140/0x140 [13547.408934] z_wr_int_2 S ffff88005da7f400 0 19871 2 0x00000080 [13547.410595] ffff8800537a7e08 0000000000000046 ffff88005f141f60 ffff8800537a7fd8 [13547.412302] ffff8800537a7fd8 ffff8800537a7fd8 ffff88005f141f60 ffff880060f95840 [13547.413964] ffff88005da7f490 ffff88005ab7bc98 ffff88005da7f4b0 ffff88005da7f400 [13547.415684] Call Trace: [13547.416910] [] schedule+0x29/0x70 [13547.418421] [] taskq_thread+0x3fa/0x440 [spl] [13547.419923] [] ? wake_up_state+0x20/0x20 [13547.421460] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.423013] [] kthread+0xcf/0xe0 [13547.424469] [] ? kthread_create_on_node+0x140/0x140 [13547.426051] [] ret_from_fork+0x58/0x90 [13547.427489] [] ? kthread_create_on_node+0x140/0x140 [13547.429086] z_wr_int_3 S ffff88005da7f100 0 19872 2 0x00000080 [13547.430677] ffff88006a417e08 0000000000000046 ffff88005f140000 ffff88006a417fd8 [13547.432410] ffff88006a417fd8 ffff88006a417fd8 ffff88005f140000 ffff880060f95c00 [13547.434108] ffff88005da7f190 ffff880066f11e30 ffff88005da7f1b0 ffff88005da7f100 [13547.435775] Call Trace: [13547.437058] [] schedule+0x29/0x70 [13547.438478] [] taskq_thread+0x3fa/0x440 [spl] [13547.440046] [] ? wake_up_state+0x20/0x20 [13547.441511] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.443116] [] kthread+0xcf/0xe0 [13547.444504] [] ? kthread_create_on_node+0x140/0x140 [13547.446094] [] ret_from_fork+0x58/0x90 [13547.447591] [] ? kthread_create_on_node+0x140/0x140 [13547.449116] z_wr_int_3 S ffff88005da7f100 0 19873 2 0x00000080 [13547.450770] ffff880067b4fe08 0000000000000046 ffff88005f140fb0 ffff880067b4ffd8 [13547.452502] ffff880067b4ffd8 ffff880067b4ffd8 ffff88005f140fb0 ffff880060f95b40 [13547.454229] ffff88005da7f190 ffff880048d5d128 ffff88005da7f1b0 ffff88005da7f100 [13547.455886] Call Trace: [13547.457166] [] schedule+0x29/0x70 [13547.458683] [] taskq_thread+0x3fa/0x440 [spl] [13547.460262] [] ? wake_up_state+0x20/0x20 [13547.461748] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.463406] [] kthread+0xcf/0xe0 [13547.464852] [] ? kthread_create_on_node+0x140/0x140 [13547.466391] [] ret_from_fork+0x58/0x90 [13547.467902] [] ? kthread_create_on_node+0x140/0x140 [13547.469473] z_wr_int_3 S ffff88005da7f100 0 19874 2 0x00000080 [13547.471076] ffff880079cabe08 0000000000000046 ffff88005f143ec0 ffff880079cabfd8 [13547.472786] ffff880079cabfd8 ffff880079cabfd8 ffff88005f143ec0 ffff880060f951e0 [13547.474495] ffff88005da7f190 ffff8800679da288 ffff88005da7f1b0 ffff88005da7f100 [13547.476200] Call Trace: [13547.477424] [] schedule+0x29/0x70 [13547.478925] [] taskq_thread+0x3fa/0x440 [spl] [13547.480473] [] ? wake_up_state+0x20/0x20 [13547.481942] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.483532] [] kthread+0xcf/0xe0 [13547.484930] [] ? kthread_create_on_node+0x140/0x140 [13547.486520] [] ret_from_fork+0x58/0x90 [13547.488024] [] ? kthread_create_on_node+0x140/0x140 [13547.489604] z_wr_int_3 S ffff88005da7f100 0 19875 2 0x00000080 [13547.491202] ffff88005b36fe08 0000000000000046 ffff88005f145e20 ffff88005b36ffd8 [13547.492911] ffff88005b36ffd8 ffff88005b36ffd8 ffff88005f145e20 ffff880060f95240 [13547.494626] ffff88005da7f190 ffff880048d5d580 ffff88005da7f1b0 ffff88005da7f100 [13547.496339] Call Trace: [13547.497561] [] schedule+0x29/0x70 [13547.499062] [] taskq_thread+0x3fa/0x440 [spl] [13547.500605] [] ? wake_up_state+0x20/0x20 [13547.502124] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.503700] [] kthread+0xcf/0xe0 [13547.505189] [] ? kthread_create_on_node+0x140/0x140 [13547.506760] [] ret_from_fork+0x58/0x90 [13547.508260] [] ? kthread_create_on_node+0x140/0x140 [13547.509776] z_wr_int_3 S ffff88005da7f100 0 19876 2 0x00000080 [13547.511430] ffff88006376fe08 0000000000000046 ffff88005f144e70 ffff88006376ffd8 [13547.513134] ffff88006376ffd8 ffff88006376ffd8 ffff88005f144e70 ffff880060f95480 [13547.514790] ffff88005da7f190 ffff880066f11128 ffff88005da7f1b0 ffff88005da7f100 [13547.516478] Call Trace: [13547.517702] [] schedule+0x29/0x70 [13547.519199] [] taskq_thread+0x3fa/0x440 [spl] [13547.520751] [] ? wake_up_state+0x20/0x20 [13547.522216] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.523809] [] kthread+0xcf/0xe0 [13547.525246] [] ? kthread_create_on_node+0x140/0x140 [13547.526772] [] ret_from_fork+0x58/0x90 [13547.528289] [] ? kthread_create_on_node+0x140/0x140 [13547.529811] z_wr_int_3 S ffff88005da7f100 0 19877 2 0x00000080 [13547.531462] ffff880048dafe08 0000000000000046 ffff88005f146dd0 ffff880048daffd8 [13547.533172] ffff880048daffd8 ffff880048daffd8 ffff88005f146dd0 ffff880060f95d80 [13547.534829] ffff88005da7f190 ffff8800679d9580 ffff88005da7f1b0 ffff88005da7f100 [13547.536542] Call Trace: [13547.537769] [] schedule+0x29/0x70 [13547.539266] [] taskq_thread+0x3fa/0x440 [spl] [13547.540820] [] ? wake_up_state+0x20/0x20 [13547.542286] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.543884] [] kthread+0xcf/0xe0 [13547.545275] [] ? kthread_create_on_node+0x140/0x140 [13547.546871] [] ret_from_fork+0x58/0x90 [13547.548362] [] ? kthread_create_on_node+0x140/0x140 [13547.549880] z_wr_int_3 S ffff88005da7f100 0 19878 2 0x00000080 [13547.551533] ffff88005f883e08 0000000000000046 ffff880079b3edd0 ffff88005f883fd8 [13547.553258] ffff88005f883fd8 ffff88005f883fd8 ffff880079b3edd0 ffff880060f95f00 [13547.554916] ffff88005da7f190 ffff880066f12288 ffff88005da7f1b0 ffff88005da7f100 [13547.556635] Call Trace: [13547.557866] [] schedule+0x29/0x70 [13547.559339] [] taskq_thread+0x3fa/0x440 [spl] [13547.560892] [] ? wake_up_state+0x20/0x20 [13547.562364] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.563964] [] kthread+0xcf/0xe0 [13547.565365] [] ? kthread_create_on_node+0x140/0x140 [13547.566955] [] ret_from_fork+0x58/0x90 [13547.568398] [] ? kthread_create_on_node+0x140/0x140 [13547.569984] z_wr_int_3 S ffff88005da7f100 0 19879 2 0x00000080 [13547.571591] ffff88005196be08 0000000000000046 ffff880079b3ce70 ffff88005196bfd8 [13547.573314] ffff88005196bfd8 ffff88005196bfd8 ffff880079b3ce70 ffff880060f953c0 [13547.574983] ffff88005da7f190 ffff880048d5d580 ffff88005da7f1b0 ffff88005da7f100 [13547.576691] Call Trace: [13547.577920] [] schedule+0x29/0x70 [13547.579426] [] taskq_thread+0x3fa/0x440 [spl] [13547.580983] [] ? wake_up_state+0x20/0x20 [13547.582459] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.584055] [] kthread+0xcf/0xe0 [13547.585500] [] ? kthread_create_on_node+0x140/0x140 [13547.587037] [] ret_from_fork+0x58/0x90 [13547.588548] [] ? kthread_create_on_node+0x140/0x140 [13547.590127] z_wr_int_3 S ffff88005da7f100 0 19880 2 0x00000080 [13547.591723] ffff880060c97e08 0000000000000046 ffff880079b38fb0 ffff880060c97fd8 [13547.593447] ffff880060c97fd8 ffff880060c97fd8 ffff880079b38fb0 ffff880060f95300 [13547.595150] ffff88005da7f190 ffff88004919bc98 ffff88005da7f1b0 ffff88005da7f100 [13547.596804] Call Trace: [13547.598064] [] schedule+0x29/0x70 [13547.599490] [] taskq_thread+0x3fa/0x440 [spl] [13547.601054] [] ? wake_up_state+0x20/0x20 [13547.602569] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.604161] [] kthread+0xcf/0xe0 [13547.605550] [] ? kthread_create_on_node+0x140/0x140 [13547.607142] [] ret_from_fork+0x58/0x90 [13547.608585] [] ? kthread_create_on_node+0x140/0x140 [13547.610167] z_wr_int_3 S ffff88005da7f100 0 19881 2 0x00000080 [13547.611773] ffff88007aa87e08 0000000000000046 ffff880079b3bec0 ffff88007aa87fd8 [13547.613493] ffff88007aa87fd8 ffff88007aa87fd8 ffff880079b3bec0 ffff880060f95a20 [13547.615203] ffff88005da7f190 ffff88005b1d59d8 ffff88005da7f1b0 ffff88005da7f100 [13547.616858] Call Trace: [13547.618142] [] schedule+0x29/0x70 [13547.619559] [] taskq_thread+0x3fa/0x440 [spl] [13547.621127] [] ? wake_up_state+0x20/0x20 [13547.622640] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.624225] [] kthread+0xcf/0xe0 [13547.625614] [] ? kthread_create_on_node+0x140/0x140 [13547.627208] [] ret_from_fork+0x58/0x90 [13547.628656] [] ? kthread_create_on_node+0x140/0x140 [13547.630238] z_wr_int_3 S ffff88005da7f100 0 19882 2 0x00000080 [13547.631853] ffff88007b3cbe08 0000000000000046 ffff880079b3de20 ffff88007b3cbfd8 [13547.633574] ffff88007b3cbfd8 ffff88007b3cbfd8 ffff880079b3de20 ffff880060f95000 [13547.635289] ffff88005da7f190 ffff88005b1d6b38 ffff88005da7f1b0 ffff88005da7f100 [13547.636952] Call Trace: [13547.638212] [] schedule+0x29/0x70 [13547.639683] [] taskq_thread+0x3fa/0x440 [spl] [13547.641244] [] ? wake_up_state+0x20/0x20 [13547.642763] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.644339] [] kthread+0xcf/0xe0 [13547.645788] [] ? kthread_create_on_node+0x140/0x140 [13547.647359] [] ret_from_fork+0x58/0x90 [13547.648812] [] ? kthread_create_on_node+0x140/0x140 [13547.650394] z_wr_int_3 S ffff88005da7f100 0 19883 2 0x00000080 [13547.651995] ffff8800661f3e08 0000000000000046 ffff880079b38000 ffff8800661f3fd8 [13547.653717] ffff8800661f3fd8 ffff8800661f3fd8 ffff880079b38000 ffff880060f95f60 [13547.655442] ffff88005da7f190 ffff8800679d99d8 ffff88005da7f1b0 ffff88005da7f100 [13547.657141] Call Trace: [13547.658372] [] schedule+0x29/0x70 [13547.659865] [] taskq_thread+0x3fa/0x440 [spl] [13547.661367] [] ? wake_up_state+0x20/0x20 [13547.662901] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.664476] [] kthread+0xcf/0xe0 [13547.665870] [] ? kthread_create_on_node+0x140/0x140 [13547.667479] [] ret_from_fork+0x58/0x90 [13547.668981] [] ? kthread_create_on_node+0x140/0x140 [13547.670547] z_wr_int_4 S ffff88005da7f900 0 19884 2 0x00000080 [13547.672152] ffff88005f23be08 0000000000000046 ffff880079b39f60 ffff88005f23bfd8 [13547.673874] ffff88005f23bfd8 ffff88005f23bfd8 ffff880079b39f60 ffff88004b55fea0 [13547.675594] ffff88005da7f990 ffff8800679dab38 ffff88005da7f9b0 ffff88005da7f900 [13547.677294] Call Trace: [13547.678521] [] schedule+0x29/0x70 [13547.680037] [] taskq_thread+0x3fa/0x440 [spl] [13547.681541] [] ? wake_up_state+0x20/0x20 [13547.683076] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.684660] [] kthread+0xcf/0xe0 [13547.686062] [] ? kthread_create_on_node+0x140/0x140 [13547.687667] [] ret_from_fork+0x58/0x90 [13547.689168] [] ? kthread_create_on_node+0x140/0x140 [13547.690751] z_wr_int_4 S ffff88005da7f900 0 19885 2 0x00000080 [13547.692401] ffff88005e8c3e08 0000000000000046 ffff880079fbedd0 ffff88005e8c3fd8 [13547.694121] ffff88005e8c3fd8 ffff88005e8c3fd8 ffff880079fbedd0 ffff88004b55f960 [13547.695779] ffff88005da7f990 ffff880066f11128 ffff88005da7f9b0 ffff88005da7f900 [13547.697509] Call Trace: [13547.698744] [] schedule+0x29/0x70 [13547.700243] [] taskq_thread+0x3fa/0x440 [spl] [13547.701792] [] ? wake_up_state+0x20/0x20 [13547.703260] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.704863] [] kthread+0xcf/0xe0 [13547.706252] [] ? kthread_create_on_node+0x140/0x140 [13547.707847] [] ret_from_fork+0x58/0x90 [13547.709335] [] ? kthread_create_on_node+0x140/0x140 [13547.710851] z_wr_int_4 S ffff88005da7f900 0 19886 2 0x00000080 [13547.712505] ffff88005cd7fe08 0000000000000046 ffff880079fb8fb0 ffff88005cd7ffd8 [13547.714210] ffff88005cd7ffd8 ffff88005cd7ffd8 ffff880079fb8fb0 ffff88004b55f300 [13547.715864] ffff88005da7f990 ffff88005ab7bc98 ffff88005da7f9b0 ffff88005da7f900 [13547.717575] Call Trace: [13547.718805] [] schedule+0x29/0x70 [13547.720294] [] taskq_thread+0x3fa/0x440 [spl] [13547.721843] [] ? wake_up_state+0x20/0x20 [13547.723311] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.724902] [] kthread+0xcf/0xe0 [13547.726335] [] ? kthread_create_on_node+0x140/0x140 [13547.727873] [] ret_from_fork+0x58/0x90 [13547.729378] [] ? kthread_create_on_node+0x140/0x140 [13547.730907] z_wr_int_4 S ffff88005da7f900 0 19887 2 0x00000080 [13547.732571] ffff88005ec63e08 0000000000000046 ffff880079fbce70 ffff88005ec63fd8 [13547.734283] ffff88005ec63fd8 ffff88005ec63fd8 ffff880079fbce70 ffff88004b55f360 [13547.735938] ffff88005da7f990 ffff880048d5c420 ffff88005da7f9b0 ffff88005da7f900 [13547.737663] Call Trace: [13547.738888] [] schedule+0x29/0x70 [13547.740383] [] taskq_thread+0x3fa/0x440 [spl] [13547.741943] [] ? wake_up_state+0x20/0x20 [13547.743408] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.745012] [] kthread+0xcf/0xe0 [13547.746445] [] ? kthread_create_on_node+0x140/0x140 [13547.747977] [] ret_from_fork+0x58/0x90 [13547.749481] [] ? kthread_create_on_node+0x140/0x140 [13547.750999] z_wr_int_4 S ffff88005da7f900 0 19888 2 0x00000080 [13547.752659] ffff88006358be08 0000000000000046 ffff88007c368fb0 ffff88006358bfd8 [13547.754388] ffff88006358bfd8 ffff88006358bfd8 ffff88007c368fb0 ffff88004b55f600 [13547.756103] ffff88005da7f990 ffff8800679da6e0 ffff88005da7f9b0 ffff88005da7f900 [13547.757758] Call Trace: [13547.759030] [] schedule+0x29/0x70 [13547.760499] [] taskq_thread+0x3fa/0x440 [spl] [13547.762011] [] ? wake_up_state+0x20/0x20 [13547.763538] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.765068] [] kthread+0xcf/0xe0 [13547.766521] [] ? kthread_create_on_node+0x140/0x140 [13547.768112] [] ret_from_fork+0x58/0x90 [13547.769598] [] ? kthread_create_on_node+0x140/0x140 [13547.771118] z_wr_int_4 S ffff88005da7f900 0 19889 2 0x00000080 [13547.772764] ffff8800622ebe08 0000000000000046 ffff88007c368000 ffff8800622ebfd8 [13547.774481] ffff8800622ebfd8 ffff8800622ebfd8 ffff88007c368000 ffff88004b55f2a0 [13547.776184] ffff88005da7f990 ffff880066f13c98 ffff88005da7f9b0 ffff88005da7f900 [13547.777850] Call Trace: [13547.779108] [] schedule+0x29/0x70 [13547.780585] [] taskq_thread+0x3fa/0x440 [spl] [13547.782090] [] ? wake_up_state+0x20/0x20 [13547.783629] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.785159] [] kthread+0xcf/0xe0 [13547.786615] [] ? kthread_create_on_node+0x140/0x140 [13547.788197] [] ret_from_fork+0x58/0x90 [13547.789638] [] ? kthread_create_on_node+0x140/0x140 [13547.791225] z_wr_int_4 S ffff88005da7f900 0 19890 2 0x00000080 [13547.792826] ffff88005e0c3e08 0000000000000046 ffff8800369b1f60 ffff88005e0c3fd8 [13547.794548] ffff88005e0c3fd8 ffff88005e0c3fd8 ffff8800369b1f60 ffff88004b55f000 [13547.796268] ffff88005da7f990 ffff8800679db840 ffff88005da7f9b0 ffff88005da7f900 [13547.797924] Call Trace: [13547.799191] [] schedule+0x29/0x70 [13547.800663] [] taskq_thread+0x3fa/0x440 [spl] [13547.802166] [] ? wake_up_state+0x20/0x20 [13547.803705] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.805282] [] kthread+0xcf/0xe0 [13547.806674] [] ? kthread_create_on_node+0x140/0x140 [13547.808278] [] ret_from_fork+0x58/0x90 [13547.809780] [] ? kthread_create_on_node+0x140/0x140 [13547.811351] z_wr_int_4 S ffff88005da7f900 0 19891 2 0x00000080 [13547.812950] ffff8800667e3e08 0000000000000046 ffff8800369b0000 ffff8800667e3fd8 [13547.814679] ffff8800667e3fd8 ffff8800667e3fd8 ffff8800369b0000 ffff88004b55fc60 [13547.816398] ffff88005da7f990 ffff880048d5ef90 ffff88005da7f9b0 ffff88005da7f900 [13547.818106] Call Trace: [13547.819333] [] schedule+0x29/0x70 [13547.820824] [] taskq_thread+0x3fa/0x440 [spl] [13547.822325] [] ? wake_up_state+0x20/0x20 [13547.823863] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.825441] [] kthread+0xcf/0xe0 [13547.826888] [] ? finish_task_switch+0x53/0x180 [13547.828436] [] ? kthread_create_on_node+0x140/0x140 [13547.829967] [] ret_from_fork+0x58/0x90 [13547.831461] [] ? kthread_create_on_node+0x140/0x140 [13547.833050] z_wr_int_4 S ffff88005da7f900 0 19892 2 0x00000080 [13547.834648] ffff88005b087e08 0000000000000046 ffff8800369b4e70 ffff88005b087fd8 [13547.836374] ffff88005b087fd8 ffff88005b087fd8 ffff8800369b4e70 ffff88004b55fcc0 [13547.838087] ffff88005da7f990 ffff88005b1d73e8 ffff88005da7f9b0 ffff88005da7f900 [13547.839738] Call Trace: [13547.841033] [] schedule+0x29/0x70 [13547.842523] [] taskq_thread+0x3fa/0x440 [spl] [13547.844041] [] ? wake_up_state+0x20/0x20 [13547.845564] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.847146] [] kthread+0xcf/0xe0 [13547.848556] [] ? kthread_create_on_node+0x140/0x140 [13547.850154] [] ret_from_fork+0x58/0x90 [13547.851585] [] ? kthread_create_on_node+0x140/0x140 [13547.853175] z_wr_int_4 S ffff88005da7f900 0 19893 2 0x00000080 [13547.854783] ffff88005952fe08 0000000000000046 ffff8800369b3ec0 ffff88005952ffd8 [13547.856498] ffff88005952ffd8 ffff88005952ffd8 ffff8800369b3ec0 ffff88004b55f9c0 [13547.858211] ffff88005da7f990 ffff88005e6833e8 ffff88005da7f9b0 ffff88005da7f900 [13547.859900] Call Trace: [13547.861223] [] schedule+0x29/0x70 [13547.862658] [] taskq_thread+0x3fa/0x440 [spl] [13547.864222] [] ? wake_up_state+0x20/0x20 [13547.865739] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.867305] [] kthread+0xcf/0xe0 [13547.868696] [] ? kthread_create_on_node+0x140/0x140 [13547.870296] [] ret_from_fork+0x58/0x90 [13547.871777] [] ? kthread_create_on_node+0x140/0x140 [13547.873301] z_wr_int_4 S ffff88005da7f900 0 19894 2 0x00000080 [13547.874955] ffff880061563e08 0000000000000046 ffff8800369b5e20 ffff880061563fd8 [13547.876676] ffff880061563fd8 ffff880061563fd8 ffff8800369b5e20 ffff88004b55f480 [13547.878395] ffff88005da7f990 ffff880066f11580 ffff88005da7f9b0 ffff88005da7f900 [13547.880089] Call Trace: [13547.881313] [] schedule+0x29/0x70 [13547.882803] [] taskq_thread+0x3fa/0x440 [spl] [13547.884349] [] ? wake_up_state+0x20/0x20 [13547.885808] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.887383] [] kthread+0xcf/0xe0 [13547.888829] [] ? kthread_create_on_node+0x140/0x140 [13547.890400] [] ret_from_fork+0x58/0x90 [13547.891830] [] ? kthread_create_on_node+0x140/0x140 [13547.893411] z_wr_int_4 S ffff88005da7f900 0 19895 2 0x00000080 [13547.895011] ffff880065b03e08 0000000000000046 ffff88007b573ec0 ffff880065b03fd8 [13547.896717] ffff880065b03fd8 ffff880065b03fd8 ffff88007b573ec0 ffff88004b55fa20 [13547.898428] ffff88005da7f990 ffff880066f133e8 ffff88005da7f9b0 ffff88005da7f900 [13547.900129] Call Trace: [13547.901350] [] schedule+0x29/0x70 [13547.902836] [] taskq_thread+0x3fa/0x440 [spl] [13547.904380] [] ? wake_up_state+0x20/0x20 [13547.905901] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.907469] [] kthread+0xcf/0xe0 [13547.908914] [] ? kthread_create_on_node+0x140/0x140 [13547.910481] [] ret_from_fork+0x58/0x90 [13547.911923] [] ? kthread_create_on_node+0x140/0x140 [13547.913509] z_wr_int_5 S ffff88005da7f300 0 19896 2 0x00000080 [13547.915107] ffff88005ec4be08 0000000000000046 ffff88007b574e70 ffff88005ec4bfd8 [13547.916818] ffff88005ec4bfd8 ffff88005ec4bfd8 ffff88007b574e70 ffff88004b55f7e0 [13547.918532] ffff88005da7f390 ffff880066f11580 ffff88005da7f3b0 ffff88005da7f300 [13547.920242] Call Trace: [13547.921468] [] schedule+0x29/0x70 [13547.922966] [] taskq_thread+0x3fa/0x440 [spl] [13547.924518] [] ? wake_up_state+0x20/0x20 [13547.925984] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.927572] [] kthread+0xcf/0xe0 [13547.929036] [] ? kthread_create_on_node+0x140/0x140 [13547.930606] [] ret_from_fork+0x58/0x90 [13547.932061] [] ? kthread_create_on_node+0x140/0x140 [13547.933650] z_wr_int_5 S ffff88005da7f300 0 19897 2 0x00000080 [13547.935309] ffff88005dd03e08 0000000000000046 ffff88007b576dd0 ffff88005dd03fd8 [13547.936975] ffff88005dd03fd8 ffff88005dd03fd8 ffff88007b576dd0 ffff88004b55fd20 [13547.938685] ffff88005da7f390 ffff880048d5f840 ffff88005da7f3b0 ffff88005da7f300 [13547.940402] Call Trace: [13547.941625] [] schedule+0x29/0x70 [13547.943123] [] taskq_thread+0x3fa/0x440 [spl] [13547.944678] [] ? wake_up_state+0x20/0x20 [13547.946146] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.947740] [] kthread+0xcf/0xe0 [13547.949176] [] ? kthread_create_on_node+0x140/0x140 [13547.950748] [] ret_from_fork+0x58/0x90 [13547.952236] [] ? kthread_create_on_node+0x140/0x140 [13547.953751] z_wr_int_5 S ffff88005da7f300 0 19898 2 0x00000080 [13547.955411] ffff8800494cfe08 0000000000000046 ffff88007b572f10 ffff8800494cffd8 [13547.957117] ffff8800494cffd8 ffff8800494cffd8 ffff88007b572f10 ffff88004b55f240 [13547.958783] ffff88005da7f390 ffff880066f13840 ffff88005da7f3b0 ffff88005da7f300 [13547.960493] Call Trace: [13547.961723] [] schedule+0x29/0x70 [13547.963205] [] taskq_thread+0x3fa/0x440 [spl] [13547.964697] [] ? wake_up_state+0x20/0x20 [13547.966232] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.967814] [] kthread+0xcf/0xe0 [13547.969203] [] ? kthread_create_on_node+0x140/0x140 [13547.970794] [] ret_from_fork+0x58/0x90 [13547.972244] [] ? kthread_create_on_node+0x140/0x140 [13547.973831] z_wr_int_5 S ffff88005da7f300 0 19899 2 0x00000080 [13547.975436] ffff880068a67e08 0000000000000046 ffff88007b575e20 ffff880068a67fd8 [13547.977150] ffff880068a67fd8 ffff880068a67fd8 ffff88007b575e20 ffff88004b55f0c0 [13547.978868] ffff88005da7f390 ffff880048d5c878 ffff88005da7f3b0 ffff88005da7f300 [13547.980584] Call Trace: [13547.981822] [] schedule+0x29/0x70 [13547.983328] [] taskq_thread+0x3fa/0x440 [spl] [13547.984881] [] ? wake_up_state+0x20/0x20 [13547.986359] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13547.987953] [] kthread+0xcf/0xe0 [13547.989396] [] ? kthread_create_on_node+0x140/0x140 [13547.990975] [] ret_from_fork+0x58/0x90 [13547.992462] [] ? kthread_create_on_node+0x140/0x140 [13547.993995] z_wr_int_5 S ffff88005da7f300 0 19900 2 0x00000080 [13547.995659] ffff880066b53e08 0000000000000046 ffff88007ba1de20 ffff880066b53fd8 [13547.997383] ffff880066b53fd8 ffff880066b53fd8 ffff88007ba1de20 ffff88004b55f060 [13547.999096] ffff88005da7f390 ffff880066f10420 ffff88005da7f3b0 ffff88005da7f300 [13548.000764] Call Trace: [13548.002039] [] schedule+0x29/0x70 [13548.003472] [] taskq_thread+0x3fa/0x440 [spl] [13548.005053] [] ? wake_up_state+0x20/0x20 [13548.006519] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.008132] [] kthread+0xcf/0xe0 [13548.009576] [] ? kthread_create_on_node+0x140/0x140 [13548.011151] [] ret_from_fork+0x58/0x90 [13548.012642] [] ? kthread_create_on_node+0x140/0x140 [13548.014168] z_wr_int_5 S ffff88005da7f300 0 19901 2 0x00000080 [13548.015818] ffff880066057e08 0000000000000046 ffff88007ba1ce70 ffff880066057fd8 [13548.017526] ffff880066057fd8 ffff880066057fd8 ffff88007ba1ce70 ffff88004b55f1e0 [13548.019251] ffff88005da7f390 ffff880048d5d9d8 ffff88005da7f3b0 ffff88005da7f300 [13548.020949] Call Trace: [13548.022209] [] schedule+0x29/0x70 [13548.023680] [] taskq_thread+0x3fa/0x440 [spl] [13548.025245] [] ? wake_up_state+0x20/0x20 [13548.026714] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.028310] [] kthread+0xcf/0xe0 [13548.029714] [] ? kthread_create_on_node+0x140/0x140 [13548.031301] [] ret_from_fork+0x58/0x90 [13548.032800] [] ? kthread_create_on_node+0x140/0x140 [13548.034372] z_wr_int_5 S ffff88005da7f300 0 19902 2 0x00000080 [13548.035975] ffff880065c87e08 0000000000000046 ffff88007ba1edd0 ffff880065c87fd8 [13548.037734] ffff880065c87fd8 ffff880065c87fd8 ffff88007ba1edd0 ffff88004b55f660 [13548.039477] ffff88005da7f390 ffff8800679db3e8 ffff88005da7f3b0 ffff88005da7f300 [13548.041181] Call Trace: [13548.042408] [] schedule+0x29/0x70 [13548.043927] [] taskq_thread+0x3fa/0x440 [spl] [13548.045504] [] ? wake_up_state+0x20/0x20 [13548.046977] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.048586] [] kthread+0xcf/0xe0 [13548.050038] [] ? kthread_create_on_node+0x140/0x140 [13548.051613] [] ret_from_fork+0x58/0x90 [13548.053124] [] ? kthread_create_on_node+0x140/0x140 [13548.054637] z_wr_int_5 S ffff88005da7f300 0 19903 2 0x00000080 [13548.056337] ffff88005b74fe08 0000000000000046 ffff88007a632f10 ffff88005b74ffd8 [13548.057999] ffff88005b74ffd8 ffff88005b74ffd8 ffff88007a632f10 ffff88004b55f3c0 [13548.059721] ffff88005da7f390 ffff88005ab78878 ffff88005da7f3b0 ffff88005da7f300 [13548.061432] Call Trace: [13548.062657] [] schedule+0x29/0x70 [13548.064163] [] taskq_thread+0x3fa/0x440 [spl] [13548.065746] [] ? wake_up_state+0x20/0x20 [13548.067288] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.068868] [] kthread+0xcf/0xe0 [13548.070305] [] ? kthread_create_on_node+0x140/0x140 [13548.071890] [] ret_from_fork+0x58/0x90 [13548.073386] [] ? kthread_create_on_node+0x140/0x140 [13548.074906] z_wr_int_5 R running task 0 19904 2 0x00000080 [13548.076564] ffff880079c43e08 0000000000000046 ffff88007a631f60 ffff880079c43fd8 [13548.078269] ffff880079c43fd8 ffff880079c43fd8 ffff88007a631f60 ffff88004b55fb40 [13548.079933] ffff88005da7f390 ffff880066f10420 ffff88005da7f3b0 ffff88005da7f300 [13548.081644] Call Trace: [13548.082872] [] schedule+0x29/0x70 [13548.084375] [] taskq_thread+0x3fa/0x440 [spl] [13548.085878] [] ? wake_up_state+0x20/0x20 [13548.087423] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.089012] [] kthread+0xcf/0xe0 [13548.090449] [] ? kthread_create_on_node+0x140/0x140 [13548.091976] [] ret_from_fork+0x58/0x90 [13548.093485] [] ? kthread_create_on_node+0x140/0x140 [13548.095064] z_wr_int_5 S ffff88005da7f300 0 19905 2 0x00000080 [13548.096657] ffff88007a1afe08 0000000000000046 ffff88007a634e70 ffff88007a1affd8 [13548.098384] ffff88007a1affd8 ffff88007a1affd8 ffff88007a634e70 ffff88005d7ad660 [13548.100093] ffff88005da7f390 ffff8800679d99d8 ffff88005da7f3b0 ffff88005da7f300 [13548.101750] Call Trace: [13548.103016] [] schedule+0x29/0x70 [13548.104438] [] taskq_thread+0x3fa/0x440 [spl] [13548.106018] [] ? wake_up_state+0x20/0x20 [13548.107541] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.109127] [] kthread+0xcf/0xe0 [13548.110567] [] ? kthread_create_on_node+0x140/0x140 [13548.112149] [] ret_from_fork+0x58/0x90 [13548.113617] [] ? kthread_create_on_node+0x140/0x140 [13548.115200] z_wr_int_5 S ffff88005da7f300 0 19906 2 0x00000080 [13548.116801] ffff88005f74be08 0000000000000046 ffff88007a630000 ffff88005f74bfd8 [13548.118515] ffff88005f74bfd8 ffff88005f74bfd8 ffff88007a630000 ffff88005d7add80 [13548.120222] ffff88005da7f390 ffff880066f12b38 ffff88005da7f3b0 ffff88005da7f300 [13548.121874] Call Trace: [13548.123170] [] schedule+0x29/0x70 [13548.124644] [] taskq_thread+0x3fa/0x440 [spl] [13548.126144] [] ? wake_up_state+0x20/0x20 [13548.127681] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.129210] [] kthread+0xcf/0xe0 [13548.130667] [] ? kthread_create_on_node+0x140/0x140 [13548.132194] [] ret_from_fork+0x58/0x90 [13548.133709] [] ? kthread_create_on_node+0x140/0x140 [13548.135235] z_wr_int_5 S ffff88005da7f300 0 19907 2 0x00000080 [13548.136897] ffff880066a97e08 0000000000000046 ffff88007a630fb0 ffff880066a97fd8 [13548.138627] ffff880066a97fd8 ffff880066a97fd8 ffff88007a630fb0 ffff88005d7ad4e0 [13548.140333] ffff88005da7f390 ffff8800679d8878 ffff88005da7f3b0 ffff88005da7f300 [13548.141989] Call Trace: [13548.143246] [] schedule+0x29/0x70 [13548.144665] [] taskq_thread+0x3fa/0x440 [spl] [13548.146230] [] ? wake_up_state+0x20/0x20 [13548.147747] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.149323] [] kthread+0xcf/0xe0 [13548.150769] [] ? kthread_create_on_node+0x140/0x140 [13548.152351] [] ret_from_fork+0x58/0x90 [13548.153843] [] ? kthread_create_on_node+0x140/0x140 [13548.155357] z_wr_int_6 S ffff88006182e400 0 19908 2 0x00000080 [13548.157012] ffff88005f643e08 0000000000000046 ffff88007a633ec0 ffff88005f643fd8 [13548.158731] ffff88005f643fd8 ffff88005f643fd8 ffff88007a633ec0 ffff88005d7adc00 [13548.160457] ffff88006182e490 ffff88005e680878 ffff88006182e4b0 ffff88006182e400 [13548.162175] Call Trace: [13548.163397] [] schedule+0x29/0x70 [13548.164888] [] taskq_thread+0x3fa/0x440 [spl] [13548.166439] [] ? wake_up_state+0x20/0x20 [13548.167953] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.169530] [] kthread+0xcf/0xe0 [13548.170979] [] ? finish_task_switch+0x53/0x180 [13548.172513] [] ? kthread_create_on_node+0x140/0x140 [13548.174088] [] ret_from_fork+0x58/0x90 [13548.175519] [] ? kthread_create_on_node+0x140/0x140 [13548.177106] z_wr_int_6 S ffff88006182e400 0 19909 2 0x00000080 [13548.178707] ffff88005f747e08 0000000000000046 ffff8800666eedd0 ffff88005f747fd8 [13548.180492] ffff88005f747fd8 ffff88005f747fd8 ffff8800666eedd0 ffff88005d7ad9c0 [13548.182197] ffff88006182e490 ffff88005ab7a6e0 ffff88006182e4b0 ffff88006182e400 [13548.183848] Call Trace: [13548.185113] [] schedule+0x29/0x70 [13548.186581] [] taskq_thread+0x3fa/0x440 [spl] [13548.188079] [] ? wake_up_state+0x20/0x20 [13548.189621] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.191191] [] kthread+0xcf/0xe0 [13548.192634] [] ? kthread_create_on_node+0x140/0x140 [13548.194225] [] ret_from_fork+0x58/0x90 [13548.195652] [] ? kthread_create_on_node+0x140/0x140 [13548.197254] z_wr_int_6 S ffff88006182e400 0 19910 2 0x00000080 [13548.198872] ffff88007a70be08 0000000000000046 ffff8800666ede20 ffff88007a70bfd8 [13548.200601] ffff88007a70bfd8 ffff88007a70bfd8 ffff8800666ede20 ffff88005d7ad840 [13548.202302] ffff88006182e490 ffff880048d5d9d8 ffff88006182e4b0 ffff88006182e400 [13548.203962] Call Trace: [13548.205221] [] schedule+0x29/0x70 [13548.206643] [] taskq_thread+0x3fa/0x440 [spl] [13548.208217] [] ? wake_up_state+0x20/0x20 [13548.209726] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.211239] [] kthread+0xcf/0xe0 [13548.212711] [] ? kthread_create_on_node+0x140/0x140 [13548.214284] [] ret_from_fork+0x58/0x90 [13548.215772] [] ? kthread_create_on_node+0x140/0x140 [13548.217288] z_wr_int_6 S ffff88006182e400 0 19911 2 0x00000080 [13548.218944] ffff880060d3fe08 0000000000000046 ffff880059183ec0 ffff880060d3ffd8 [13548.220660] ffff880060d3ffd8 ffff880060d3ffd8 ffff880059183ec0 ffff88005d7ad7e0 [13548.222386] ffff88006182e490 ffff880048d5d580 ffff88006182e4b0 ffff88006182e400 [13548.224082] Call Trace: [13548.225307] [] schedule+0x29/0x70 [13548.226807] [] taskq_thread+0x3fa/0x440 [spl] [13548.228357] [] ? wake_up_state+0x20/0x20 [13548.229816] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.231399] [] kthread+0xcf/0xe0 [13548.232845] [] ? kthread_create_on_node+0x140/0x140 [13548.234419] [] ret_from_fork+0x58/0x90 [13548.235901] [] ? kthread_create_on_node+0x140/0x140 [13548.237414] z_wr_int_6 S ffff88006182e400 0 19912 2 0x00000080 [13548.239044] ffff88005b417e08 0000000000000046 ffff880059185e20 ffff88005b417fd8 [13548.240750] ffff88005b417fd8 ffff88005b417fd8 ffff880059185e20 ffff88005d7adcc0 [13548.242461] ffff88006182e490 ffff8800679d99d8 ffff88006182e4b0 ffff88006182e400 [13548.244156] Call Trace: [13548.245380] [] schedule+0x29/0x70 [13548.246868] [] taskq_thread+0x3fa/0x440 [spl] [13548.248413] [] ? wake_up_state+0x20/0x20 [13548.249892] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.251479] [] kthread+0xcf/0xe0 [13548.252922] [] ? kthread_create_on_node+0x140/0x140 [13548.254452] [] ret_from_fork+0x58/0x90 [13548.255963] [] ? kthread_create_on_node+0x140/0x140 [13548.257486] z_wr_int_6 S ffff88006182e400 0 19913 2 0x00000080 [13548.259139] ffff880065fcbe08 0000000000000046 ffff880059180fb0 ffff880065fcbfd8 [13548.260859] ffff880065fcbfd8 ffff880065fcbfd8 ffff880059180fb0 ffff88005d7adde0 [13548.262565] ffff88006182e490 ffff8800679daf90 ffff88006182e4b0 ffff88006182e400 [13548.264284] Call Trace: [13548.265508] [] schedule+0x29/0x70 [13548.267016] [] taskq_thread+0x3fa/0x440 [spl] [13548.268564] [] ? wake_up_state+0x20/0x20 [13548.270083] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.271668] [] kthread+0xcf/0xe0 [13548.273109] [] ? kthread_create_on_node+0x140/0x140 [13548.274698] [] ret_from_fork+0x58/0x90 [13548.276190] [] ? kthread_create_on_node+0x140/0x140 [13548.277708] z_wr_int_6 S ffff88006182e400 0 19914 2 0x00000080 [13548.279403] ffff88006bca3e08 0000000000000046 ffff880059180000 ffff88006bca3fd8 [13548.281113] ffff88006bca3fd8 ffff88006bca3fd8 ffff880059180000 ffff88005d7ad780 [13548.282777] ffff88006182e490 ffff8800679da6e0 ffff88006182e4b0 ffff88006182e400 [13548.284491] Call Trace: [13548.285719] [] schedule+0x29/0x70 [13548.287217] [] taskq_thread+0x3fa/0x440 [spl] [13548.288781] [] ? wake_up_state+0x20/0x20 [13548.290245] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.291842] [] kthread+0xcf/0xe0 [13548.293281] [] ? kthread_create_on_node+0x140/0x140 [13548.294819] [] ret_from_fork+0x58/0x90 [13548.296325] [] ? kthread_create_on_node+0x140/0x140 [13548.297900] z_wr_int_6 S ffff88006182e400 0 19915 2 0x00000080 [13548.299504] ffff88006a733e08 0000000000000046 ffff880059186dd0 ffff88006a733fd8 [13548.301221] ffff88006a733fd8 ffff88006a733fd8 ffff880059186dd0 ffff88005d7ada20 [13548.302879] ffff88006182e490 ffff880066f126e0 ffff88006182e4b0 ffff88006182e400 [13548.304606] Call Trace: [13548.305837] [] schedule+0x29/0x70 [13548.307344] [] taskq_thread+0x3fa/0x440 [spl] [13548.308896] [] ? wake_up_state+0x20/0x20 [13548.310420] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.312018] [] kthread+0xcf/0xe0 [13548.313461] [] ? kthread_create_on_node+0x140/0x140 [13548.315045] [] ret_from_fork+0x58/0x90 [13548.316534] [] ? kthread_create_on_node+0x140/0x140 [13548.318052] z_wr_int_6 S ffff88006182e400 0 19916 2 0x00000080 [13548.319711] ffff88004ef63e08 0000000000000046 ffff880059182f10 ffff88004ef63fd8 [13548.321431] ffff88004ef63fd8 ffff88004ef63fd8 ffff880059182f10 ffff88005d7adb40 [13548.323143] ffff88006182e490 ffff880048d5d128 ffff88006182e4b0 ffff88006182e400 [13548.324797] Call Trace: [13548.326092] [] schedule+0x29/0x70 [13548.327513] [] taskq_thread+0x3fa/0x440 [spl] [13548.329081] [] ? wake_up_state+0x20/0x20 [13548.330548] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.332151] [] kthread+0xcf/0xe0 [13548.333593] [] ? kthread_create_on_node+0x140/0x140 [13548.335129] [] ret_from_fork+0x58/0x90 [13548.336636] [] ? kthread_create_on_node+0x140/0x140 [13548.338158] z_wr_int_6 S ffff88006182e400 0 19917 2 0x00000080 [13548.339815] ffff880062837e08 0000000000000046 ffff880059184e70 ffff880062837fd8 [13548.341543] ffff880062837fd8 ffff880062837fd8 ffff880059184e70 ffff88005d7adc60 [13548.343245] ffff88006182e490 ffff880066f11e30 ffff88006182e4b0 ffff88006182e400 [13548.344906] Call Trace: [13548.346166] [] schedule+0x29/0x70 [13548.347638] [] taskq_thread+0x3fa/0x440 [spl] [13548.349195] [] ? wake_up_state+0x20/0x20 [13548.350661] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.352260] [] kthread+0xcf/0xe0 [13548.353651] [] ? kthread_create_on_node+0x140/0x140 [13548.355245] [] ret_from_fork+0x58/0x90 [13548.356746] [] ? kthread_create_on_node+0x140/0x140 [13548.358308] z_wr_int_6 S ffff88006182e400 0 19918 2 0x00000080 [13548.359916] ffff88006a5fbe08 0000000000000046 ffff880079309f60 ffff88006a5fbfd8 [13548.361633] ffff88006a5fbfd8 ffff88006a5fbfd8 ffff880079309f60 ffff88005d7ad600 [13548.363360] ffff88006182e490 ffff8800679d9128 ffff88006182e4b0 ffff88006182e400 [13548.365014] Call Trace: [13548.366274] [] schedule+0x29/0x70 [13548.367754] [] taskq_thread+0x3fa/0x440 [spl] [13548.369253] [] ? wake_up_state+0x20/0x20 [13548.370789] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.372380] [] kthread+0xcf/0xe0 [13548.373824] [] ? kthread_create_on_node+0x140/0x140 [13548.375404] [] ret_from_fork+0x58/0x90 [13548.376897] [] ? kthread_create_on_node+0x140/0x140 [13548.378470] z_wr_int_6 S ffff88006182e400 0 19919 2 0x00000080 [13548.380070] ffff88005f72fe08 0000000000000046 ffff880079308000 ffff88005f72ffd8 [13548.381784] ffff88005f72ffd8 ffff88005f72ffd8 ffff880079308000 ffff88005d7adea0 [13548.383501] ffff88006182e490 ffff88005e685128 ffff88006182e4b0 ffff88006182e400 [13548.385215] Call Trace: [13548.386436] [] schedule+0x29/0x70 [13548.387934] [] taskq_thread+0x3fa/0x440 [spl] [13548.389438] [] ? wake_up_state+0x20/0x20 [13548.390973] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.392535] [] kthread+0xcf/0xe0 [13548.393994] [] ? kthread_create_on_node+0x140/0x140 [13548.395514] [] ret_from_fork+0x58/0x90 [13548.397031] [] ? kthread_create_on_node+0x140/0x140 [13548.398595] z_wr_int_7 S ffff88006182eb00 0 19920 2 0x00000080 [13548.400192] ffff88005f757e08 0000000000000046 ffff88007930af10 ffff88005f757fd8 [13548.401914] ffff88005f757fd8 ffff88005f757fd8 ffff88007930af10 ffff88005d7add20 [13548.403663] ffff88006182eb90 ffff8800679d99d8 ffff88006182ebb0 ffff88006182eb00 [13548.405398] Call Trace: [13548.406621] [] schedule+0x29/0x70 [13548.408121] [] taskq_thread+0x3fa/0x440 [spl] [13548.409619] [] ? wake_up_state+0x20/0x20 [13548.411147] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.412678] [] kthread+0xcf/0xe0 [13548.414135] [] ? kthread_create_on_node+0x140/0x140 [13548.415724] [] ret_from_fork+0x58/0x90 [13548.417161] [] ? kthread_create_on_node+0x140/0x140 [13548.418745] z_wr_int_7 S ffff88006182eb00 0 19921 2 0x00000080 [13548.420397] ffff88005b603e08 0000000000000046 ffff88007ae88000 ffff88005b603fd8 [13548.422102] ffff88005b603fd8 ffff88005b603fd8 ffff88007ae88000 ffff88005d7ad960 [13548.423758] ffff88006182eb90 ffff880048d5c878 ffff88006182ebb0 ffff88006182eb00 [13548.425466] Call Trace: [13548.426692] [] schedule+0x29/0x70 [13548.428183] [] taskq_thread+0x3fa/0x440 [spl] [13548.429681] [] ? wake_up_state+0x20/0x20 [13548.431211] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.432739] [] kthread+0xcf/0xe0 [13548.434191] [] ? kthread_create_on_node+0x140/0x140 [13548.435764] [] ret_from_fork+0x58/0x90 [13548.437254] [] ? kthread_create_on_node+0x140/0x140 [13548.438769] z_wr_int_7 S ffff88006182eb00 0 19922 2 0x00000080 [13548.440420] ffff880046ef3e08 0000000000000046 ffff88007bb50000 ffff880046ef3fd8 [13548.442128] ffff880046ef3fd8 ffff880046ef3fd8 ffff88007bb50000 ffff88005d7ad720 [13548.443788] ffff88006182eb90 ffff8800679da6e0 ffff88006182ebb0 ffff88006182eb00 [13548.445503] Call Trace: [13548.446731] [] schedule+0x29/0x70 [13548.448224] [] taskq_thread+0x3fa/0x440 [spl] [13548.449737] [] ? wake_up_state+0x20/0x20 [13548.451302] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.452841] [] kthread+0xcf/0xe0 [13548.454295] [] ? kthread_create_on_node+0x140/0x140 [13548.455873] [] ret_from_fork+0x58/0x90 [13548.457325] [] ? kthread_create_on_node+0x140/0x140 [13548.458917] z_wr_int_7 S ffff88006182eb00 0 19923 2 0x00000080 [13548.460535] ffff88005f9dbe08 0000000000000046 ffff88007bb50fb0 ffff88005f9dbfd8 [13548.462257] ffff88005f9dbfd8 ffff88005f9dbfd8 ffff88007bb50fb0 ffff88005d7adba0 [13548.463919] ffff88006182eb90 ffff8800679d9128 ffff88006182ebb0 ffff88006182eb00 [13548.465643] Call Trace: [13548.466878] [] schedule+0x29/0x70 [13548.468376] [] taskq_thread+0x3fa/0x440 [spl] [13548.469881] [] ? wake_up_state+0x20/0x20 [13548.471422] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.473012] [] kthread+0xcf/0xe0 [13548.474402] [] ? kthread_create_on_node+0x140/0x140 [13548.476017] [] ret_from_fork+0x58/0x90 [13548.477507] [] ? kthread_create_on_node+0x140/0x140 [13548.479033] z_wr_int_7 S ffff88006182eb00 0 19924 2 0x00000080 [13548.480662] ffff88005ee8be08 0000000000000046 ffff88007bb51f60 ffff88005ee8bfd8 [13548.482399] ffff88005ee8bfd8 ffff88005ee8bfd8 ffff88007bb51f60 ffff88005d7ad5a0 [13548.484106] ffff88006182eb90 ffff8800630b3840 ffff88006182ebb0 ffff88006182eb00 [13548.485773] Call Trace: [13548.487034] [] schedule+0x29/0x70 [13548.488513] [] taskq_thread+0x3fa/0x440 [spl] [13548.490072] [] ? wake_up_state+0x20/0x20 [13548.491542] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.493151] [] kthread+0xcf/0xe0 [13548.494590] [] ? kthread_create_on_node+0x140/0x140 [13548.496120] [] ret_from_fork+0x58/0x90 [13548.497640] [] ? kthread_create_on_node+0x140/0x140 [13548.499166] z_wr_int_7 S ffff88006182eb00 0 19925 2 0x00000080 [13548.500836] ffff880065fd7e08 0000000000000046 ffff88007bb52f10 ffff880065fd7fd8 [13548.502557] ffff880065fd7fd8 ffff880065fd7fd8 ffff88007bb52f10 ffff88005d7ad000 [13548.504265] ffff88006182eb90 ffff880066f126e0 ffff88006182ebb0 ffff88006182eb00 [13548.505922] Call Trace: [13548.507184] [] schedule+0x29/0x70 [13548.508615] [] taskq_thread+0x3fa/0x440 [spl] [13548.510184] [] ? wake_up_state+0x20/0x20 [13548.511725] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.513309] [] kthread+0xcf/0xe0 [13548.514702] [] ? finish_task_switch+0x53/0x180 [13548.516258] [] ? kthread_create_on_node+0x140/0x140 [13548.517838] [] ret_from_fork+0x58/0x90 [13548.519274] [] ? kthread_create_on_node+0x140/0x140 [13548.520861] z_wr_int_7 S ffff88006182eb00 0 19926 2 0x00000080 [13548.522469] ffff880046bcfe08 0000000000000046 ffff88007bb53ec0 ffff880046bcffd8 [13548.524182] ffff880046bcffd8 ffff880046bcffd8 ffff88007bb53ec0 ffff88005d7ade40 [13548.525837] ffff88006182eb90 ffff8800679da288 ffff88006182ebb0 ffff88006182eb00 [13548.527542] Call Trace: [13548.528772] [] schedule+0x29/0x70 [13548.530269] [] taskq_thread+0x3fa/0x440 [spl] [13548.531842] [] ? wake_up_state+0x20/0x20 [13548.533346] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.534865] [] kthread+0xcf/0xe0 [13548.536326] [] ? kthread_create_on_node+0x140/0x140 [13548.537868] [] ret_from_fork+0x58/0x90 [13548.539361] [] ? kthread_create_on_node+0x140/0x140 [13548.540896] z_wr_int_7 S ffff88006182eb00 0 19927 2 0x00000080 [13548.542555] ffff88005329be08 0000000000000046 ffff88007bb54e70 ffff88005329bfd8 [13548.544285] ffff88005329bfd8 ffff88005329bfd8 ffff88007bb54e70 ffff88005d7ad2a0 [13548.545941] ffff88006182eb90 ffff88005ab7a288 ffff88006182ebb0 ffff88006182eb00 [13548.547672] Call Trace: [13548.548900] [] schedule+0x29/0x70 [13548.550399] [] taskq_thread+0x3fa/0x440 [spl] [13548.551961] [] ? wake_up_state+0x20/0x20 [13548.553468] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.554996] [] kthread+0xcf/0xe0 [13548.556459] [] ? kthread_create_on_node+0x140/0x140 [13548.557993] [] ret_from_fork+0x58/0x90 [13548.559498] [] ? kthread_create_on_node+0x140/0x140 [13548.561066] z_wr_int_7 S ffff88006182eb00 0 19928 2 0x00000080 [13548.562661] ffff880051637e08 0000000000000046 ffff88007bb55e20 ffff880051637fd8 [13548.564389] ffff880051637fd8 ffff880051637fd8 ffff88007bb55e20 ffff88005d7ad120 [13548.566098] ffff88006182eb90 ffff8800630b2b38 ffff88006182ebb0 ffff88006182eb00 [13548.567746] Call Trace: [13548.569034] [] schedule+0x29/0x70 [13548.570503] [] taskq_thread+0x3fa/0x440 [spl] [13548.572011] [] ? wake_up_state+0x20/0x20 [13548.573533] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.575053] [] kthread+0xcf/0xe0 [13548.576517] [] ? kthread_create_on_node+0x140/0x140 [13548.578097] [] ret_from_fork+0x58/0x90 [13548.579572] [] ? kthread_create_on_node+0x140/0x140 [13548.581146] z_wr_int_7 S ffff88006182eb00 0 19929 2 0x00000080 [13548.582743] ffff880065c33e08 0000000000000046 ffff88007bb56dd0 ffff880065c33fd8 [13548.584458] ffff880065c33fd8 ffff880065c33fd8 ffff88007bb56dd0 ffff88005d7ad540 [13548.586164] ffff88006182eb90 ffff8800630b0420 ffff88006182ebb0 ffff88006182eb00 [13548.587817] Call Trace: [13548.589075] [] schedule+0x29/0x70 [13548.590551] [] taskq_thread+0x3fa/0x440 [spl] [13548.592061] [] ? wake_up_state+0x20/0x20 [13548.593591] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.595179] [] kthread+0xcf/0xe0 [13548.596568] [] ? kthread_create_on_node+0x140/0x140 [13548.598160] [] ret_from_fork+0x58/0x90 [13548.599603] [] ? kthread_create_on_node+0x140/0x140 [13548.601189] z_wr_int_7 S ffff88006182eb00 0 19930 2 0x00000080 [13548.602796] ffff880047973e08 0000000000000046 ffff88005d7f0000 ffff880047973fd8 [13548.604512] ffff880047973fd8 ffff880047973fd8 ffff88005d7f0000 ffff88005d7ad180 [13548.606226] ffff88006182eb90 ffff8800630b1580 ffff88006182ebb0 ffff88006182eb00 [13548.607881] Call Trace: [13548.609142] [] schedule+0x29/0x70 [13548.610573] [] taskq_thread+0x3fa/0x440 [spl] [13548.612140] [] ? wake_up_state+0x20/0x20 [13548.613660] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.615199] [] kthread+0xcf/0xe0 [13548.616654] [] ? kthread_create_on_node+0x140/0x140 [13548.618190] [] ret_from_fork+0x58/0x90 [13548.619703] [] ? kthread_create_on_node+0x140/0x140 [13548.621225] z_wr_int_7 S ffff88006182eb00 0 19931 2 0x00000080 [13548.622884] ffff88006bcafe08 0000000000000046 ffff88005d7f0fb0 ffff88006bcaffd8 [13548.624610] ffff88006bcaffd8 ffff88006bcaffd8 ffff88005d7f0fb0 ffff88005d7ad3c0 [13548.626316] ffff88006182eb90 ffff88005ab799d8 ffff88006182ebb0 ffff88006182eb00 [13548.627974] Call Trace: [13548.629236] [] schedule+0x29/0x70 [13548.630661] [] taskq_thread+0x3fa/0x440 [spl] [13548.632244] [] ? wake_up_state+0x20/0x20 [13548.633770] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.635355] [] kthread+0xcf/0xe0 [13548.636754] [] ? kthread_create_on_node+0x140/0x140 [13548.638352] [] ret_from_fork+0x58/0x90 [13548.639810] [] ? kthread_create_on_node+0x140/0x140 [13548.641391] z_wr_int_h S ffff88006182e000 0 19932 2 0x00000080 [13548.643007] ffff880060d2fe08 0000000000000046 ffff88005d7f1f60 ffff880060d2ffd8 [13548.644719] ffff880060d2ffd8 ffff880060d2ffd8 ffff88005d7f1f60 ffff88005d7ad6c0 [13548.646447] ffff88006182e090 ffffffffa06624b0 ffff88006182e0b0 ffff88006182e000 [13548.648173] Call Trace: [13548.649410] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.651038] [] schedule+0x29/0x70 [13548.652530] [] taskq_thread+0x3fa/0x440 [spl] [13548.654045] [] ? wake_up_state+0x20/0x20 [13548.655581] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.657169] [] kthread+0xcf/0xe0 [13548.658574] [] ? kthread_create_on_node+0x140/0x140 [13548.660188] [] ret_from_fork+0x58/0x90 [13548.661682] [] ? kthread_create_on_node+0x140/0x140 [13548.663209] z_wr_int_h S ffff88006182e000 0 19933 2 0x00000080 [13548.664877] ffff880066da7e08 0000000000000046 ffff88005d7f2f10 ffff880066da7fd8 [13548.666606] ffff880066da7fd8 ffff880066da7fd8 ffff88005d7f2f10 ffff88005d7ad900 [13548.668329] ffff88006182e090 ffffffffa06624b0 ffff88006182e0b0 ffff88006182e000 [13548.670009] Call Trace: [13548.671287] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.672903] [] schedule+0x29/0x70 [13548.674350] [] taskq_thread+0x3fa/0x440 [spl] [13548.675935] [] ? wake_up_state+0x20/0x20 [13548.677451] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.679057] [] kthread+0xcf/0xe0 [13548.680477] [] ? kthread_create_on_node+0x140/0x140 [13548.682089] [] ret_from_fork+0x58/0x90 [13548.683588] [] ? kthread_create_on_node+0x140/0x140 [13548.685181] z_wr_int_h S ffff88006182e000 0 19934 2 0x00000080 [13548.686801] ffff88004650be08 0000000000000046 ffff88005d7f3ec0 ffff88004650bfd8 [13548.688523] ffff88004650bfd8 ffff88004650bfd8 ffff88005d7f3ec0 ffff88005d7adf60 [13548.690240] ffff88006182e090 ffffffffa06624b0 ffff88006182e0b0 ffff88006182e000 [13548.691914] Call Trace: [13548.693199] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.694824] [] schedule+0x29/0x70 [13548.696266] [] taskq_thread+0x3fa/0x440 [spl] [13548.697843] [] ? wake_up_state+0x20/0x20 [13548.699335] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.700966] [] kthread+0xcf/0xe0 [13548.702436] [] ? kthread_create_on_node+0x140/0x140 [13548.704038] [] ret_from_fork+0x58/0x90 [13548.705549] [] ? kthread_create_on_node+0x140/0x140 [13548.707152] z_wr_int_h S ffff88006182e000 0 19935 2 0x00000080 [13548.708783] ffff880067337e08 0000000000000046 ffff88005d7f4e70 ffff880067337fd8 [13548.710534] ffff880067337fd8 ffff880067337fd8 ffff88005d7f4e70 ffff88005d7ad240 [13548.712279] ffff88006182e090 ffffffffa06624b0 ffff88006182e0b0 ffff88006182e000 [13548.713980] Call Trace: [13548.715274] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.716847] [] schedule+0x29/0x70 [13548.718353] [] taskq_thread+0x3fa/0x440 [spl] [13548.719934] [] ? wake_up_state+0x20/0x20 [13548.721429] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.723056] [] kthread+0xcf/0xe0 [13548.724526] [] ? kthread_create_on_node+0x140/0x140 [13548.726073] [] ret_from_fork+0x58/0x90 [13548.727600] [] ? kthread_create_on_node+0x140/0x140 [13548.729157] z_wr_int_h S ffff88006182e000 0 19936 2 0x00000080 [13548.730977] ffff880060e23e08 0000000000000046 ffff88005d7f5e20 ffff880060e23fd8 [13548.732711] ffff880060e23fd8 ffff880060e23fd8 ffff88005d7f5e20 ffff88005d7ad1e0 [13548.734435] ffff88006182e090 ffffffffa06624b0 ffff88006182e0b0 ffff88006182e000 [13548.736125] Call Trace: [13548.737380] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.738964] [] schedule+0x29/0x70 [13548.740402] [] taskq_thread+0x3fa/0x440 [spl] [13548.741929] [] ? wake_up_state+0x20/0x20 [13548.743423] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.744978] [] kthread+0xcf/0xe0 [13548.746407] [] ? kthread_create_on_node+0x140/0x140 [13548.747956] [] ret_from_fork+0x58/0x90 [13548.749415] [] ? kthread_create_on_node+0x140/0x140 [13548.750970] z_fr_iss_0 S ffff88006182e100 0 19937 2 0x00000080 [13548.752606] ffff88006733be08 0000000000000046 ffff88005d7f6dd0 ffff88006733bfd8 [13548.754315] ffff88006733bfd8 ffff88006733bfd8 ffff88005d7f6dd0 ffff88005d7ad420 [13548.756020] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.757704] Call Trace: [13548.758962] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.760536] [] schedule+0x29/0x70 [13548.761982] [] taskq_thread+0x3fa/0x440 [spl] [13548.763511] [] ? wake_up_state+0x20/0x20 [13548.765016] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.766567] [] kthread+0xcf/0xe0 [13548.767995] [] ? kthread_create_on_node+0x140/0x140 [13548.769549] [] ret_from_fork+0x58/0x90 [13548.771019] [] ? kthread_create_on_node+0x140/0x140 [13548.772577] z_fr_iss_0 S ffff88006182e100 0 19938 2 0x00000080 [13548.774224] ffff88005b08fe08 0000000000000046 ffff880066860000 ffff88005b08ffd8 [13548.775932] ffff88005b08ffd8 ffff88005b08ffd8 ffff880066860000 ffff88005d7ad8a0 [13548.777734] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.779427] Call Trace: [13548.780693] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.782277] [] schedule+0x29/0x70 [13548.783727] [] taskq_thread+0x3fa/0x440 [spl] [13548.785258] [] ? wake_up_state+0x20/0x20 [13548.786768] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.788322] [] kthread+0xcf/0xe0 [13548.789745] [] ? kthread_create_on_node+0x140/0x140 [13548.791290] [] ret_from_fork+0x58/0x90 [13548.792753] [] ? kthread_create_on_node+0x140/0x140 [13548.794321] z_fr_iss_0 S ffff88006182e100 0 19939 2 0x00000080 [13548.795960] ffff88007aa4fe08 0000000000000046 ffff880066860fb0 ffff88007aa4ffd8 [13548.797674] ffff88007aa4ffd8 ffff88007aa4ffd8 ffff880066860fb0 ffff88005d7ad060 [13548.799415] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.801116] Call Trace: [13548.802372] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.803943] [] schedule+0x29/0x70 [13548.805388] [] taskq_thread+0x3fa/0x440 [spl] [13548.806916] [] ? wake_up_state+0x20/0x20 [13548.808415] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.809973] [] kthread+0xcf/0xe0 [13548.811396] [] ? kthread_create_on_node+0x140/0x140 [13548.812947] [] ret_from_fork+0x58/0x90 [13548.814410] [] ? kthread_create_on_node+0x140/0x140 [13548.815960] z_fr_iss_0 S ffff88006182e100 0 19940 2 0x00000080 [13548.817599] ffff880063767e08 0000000000000046 ffff880066861f60 ffff880063767fd8 [13548.819304] ffff880063767fd8 ffff880063767fd8 ffff880066861f60 ffff8800497d07e0 [13548.820988] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.822704] Call Trace: [13548.823961] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.825530] [] schedule+0x29/0x70 [13548.826977] [] taskq_thread+0x3fa/0x440 [spl] [13548.828508] [] ? wake_up_state+0x20/0x20 [13548.830011] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.831564] [] kthread+0xcf/0xe0 [13548.832987] [] ? kthread_create_on_node+0x140/0x140 [13548.834531] [] ret_from_fork+0x58/0x90 [13548.835996] [] ? kthread_create_on_node+0x140/0x140 [13548.837543] z_fr_iss_0 S ffff88006182e100 0 19941 2 0x00000080 [13548.839176] ffff88007b1dfe08 0000000000000046 ffff880066862f10 ffff88007b1dffd8 [13548.840868] ffff88007b1dffd8 ffff88007b1dffd8 ffff880066862f10 ffff8800497d0900 [13548.842562] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.844256] Call Trace: [13548.845512] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.847086] [] schedule+0x29/0x70 [13548.848524] [] taskq_thread+0x3fa/0x440 [spl] [13548.850062] [] ? wake_up_state+0x20/0x20 [13548.851558] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.853115] [] kthread+0xcf/0xe0 [13548.854551] [] ? kthread_create_on_node+0x140/0x140 [13548.856100] [] ret_from_fork+0x58/0x90 [13548.857565] [] ? kthread_create_on_node+0x140/0x140 [13548.859120] z_fr_iss_0 S ffff88006182e100 0 19942 2 0x00000080 [13548.860752] ffff88004b557e08 0000000000000046 ffff880066863ec0 ffff88004b557fd8 [13548.862459] ffff88004b557fd8 ffff88004b557fd8 ffff880066863ec0 ffff8800497d0c00 [13548.864162] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.865845] Call Trace: [13548.867101] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.868678] [] schedule+0x29/0x70 [13548.870119] [] taskq_thread+0x3fa/0x440 [spl] [13548.871638] [] ? wake_up_state+0x20/0x20 [13548.873144] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.874701] [] kthread+0xcf/0xe0 [13548.876121] [] ? kthread_create_on_node+0x140/0x140 [13548.877684] [] ret_from_fork+0x58/0x90 [13548.879142] [] ? kthread_create_on_node+0x140/0x140 [13548.880694] z_fr_iss_0 S ffff88006182e100 0 19943 2 0x00000080 [13548.882331] ffff88006264fe08 0000000000000046 ffff880066864e70 ffff88006264ffd8 [13548.884036] ffff88006264ffd8 ffff88006264ffd8 ffff880066864e70 ffff8800497d0180 [13548.885728] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.887416] Call Trace: [13548.888675] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.890245] [] schedule+0x29/0x70 [13548.891688] [] taskq_thread+0x3fa/0x440 [spl] [13548.893213] [] ? wake_up_state+0x20/0x20 [13548.894717] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.896272] [] kthread+0xcf/0xe0 [13548.897698] [] ? kthread_create_on_node+0x140/0x140 [13548.899243] [] ret_from_fork+0x58/0x90 [13548.900705] [] ? kthread_create_on_node+0x140/0x140 [13548.902261] z_fr_iss_0 S ffff88006182e100 0 19944 2 0x00000080 [13548.903894] ffff88005ccffe08 0000000000000046 ffff880066865e20 ffff88005ccfffd8 [13548.905596] ffff88005ccfffd8 ffff88005ccfffd8 ffff880066865e20 ffff8800497d0a80 [13548.907293] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.908987] Call Trace: [13548.910245] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.911821] [] schedule+0x29/0x70 [13548.913264] [] taskq_thread+0x3fa/0x440 [spl] [13548.914791] [] ? wake_up_state+0x20/0x20 [13548.916292] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.917851] [] kthread+0xcf/0xe0 [13548.919275] [] ? kthread_create_on_node+0x140/0x140 [13548.920831] [] ret_from_fork+0x58/0x90 [13548.922292] [] ? kthread_create_on_node+0x140/0x140 [13548.923853] z_fr_iss_0 S ffff88006182e100 0 19945 2 0x00000080 [13548.925491] ffff88006a663e08 0000000000000046 ffff880066866dd0 ffff88006a663fd8 [13548.927199] ffff88006a663fd8 ffff88006a663fd8 ffff880066866dd0 ffff8800497d02a0 [13548.928886] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.930587] Call Trace: [13548.931854] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.933432] [] schedule+0x29/0x70 [13548.934892] [] taskq_thread+0x3fa/0x440 [spl] [13548.936422] [] ? wake_up_state+0x20/0x20 [13548.937924] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.939483] [] kthread+0xcf/0xe0 [13548.940974] [] ? kthread_create_on_node+0x140/0x140 [13548.942590] [] ret_from_fork+0x58/0x90 [13548.944136] [] ? kthread_create_on_node+0x140/0x140 [13548.945741] z_fr_iss_0 S ffff88006182e100 0 19946 2 0x00000080 [13548.947393] ffff880067c47e08 0000000000000046 ffff880068a40000 ffff880067c47fd8 [13548.949103] ffff880067c47fd8 ffff880067c47fd8 ffff880068a40000 ffff8800497d0f60 [13548.950788] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.952482] Call Trace: [13548.953746] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.955312] [] schedule+0x29/0x70 [13548.956760] [] taskq_thread+0x3fa/0x440 [spl] [13548.958283] [] ? wake_up_state+0x20/0x20 [13548.959781] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.961341] [] kthread+0xcf/0xe0 [13548.962764] [] ? kthread_create_on_node+0x140/0x140 [13548.964313] [] ret_from_fork+0x58/0x90 [13548.965776] [] ? kthread_create_on_node+0x140/0x140 [13548.967325] z_fr_iss_0 S ffff88006182e100 0 19947 2 0x00000080 [13548.968956] ffff88007a667e08 0000000000000046 ffff880068a40fb0 ffff88007a667fd8 [13548.970654] ffff88007a667fd8 ffff88007a667fd8 ffff880068a40fb0 ffff8800497d03c0 [13548.972366] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.974047] Call Trace: [13548.975303] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.976879] [] schedule+0x29/0x70 [13548.978317] [] taskq_thread+0x3fa/0x440 [spl] [13548.979844] [] ? wake_up_state+0x20/0x20 [13548.981345] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13548.982904] [] kthread+0xcf/0xe0 [13548.984330] [] ? kthread_create_on_node+0x140/0x140 [13548.985878] [] ret_from_fork+0x58/0x90 [13548.987338] [] ? kthread_create_on_node+0x140/0x140 [13548.988895] z_fr_iss_0 S ffff88006182e100 0 19948 2 0x00000080 [13548.990528] ffff880036513e08 0000000000000046 ffff880068a41f60 ffff880036513fd8 [13548.992232] ffff880036513fd8 ffff880036513fd8 ffff880068a41f60 ffff8800497d0960 [13548.993929] ffff88006182e190 ffffffffa06624b0 ffff88006182e1b0 ffff88006182e100 [13548.995833] Call Trace: [13548.998414] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.001301] [] schedule+0x29/0x70 [13549.002816] [] taskq_thread+0x3fa/0x440 [spl] [13549.004401] [] ? wake_up_state+0x20/0x20 [13549.005970] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.007577] [] kthread+0xcf/0xe0 [13549.009018] [] ? kthread_create_on_node+0x140/0x140 [13549.010632] [] ret_from_fork+0x58/0x90 [13549.012101] [] ? kthread_create_on_node+0x140/0x140 [13549.013728] z_fr_iss_1 S ffff88006182e600 0 19949 2 0x00000080 [13549.015424] ffff88005aa3be08 0000000000000046 ffff880068a42f10 ffff88005aa3bfd8 [13549.017166] ffff88005aa3bfd8 ffff88005aa3bfd8 ffff880068a42f10 ffff8800497d0ae0 [13549.018862] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.020604] Call Trace: [13549.021872] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.023520] [] schedule+0x29/0x70 [13549.025032] [] taskq_thread+0x3fa/0x440 [spl] [13549.026600] [] ? wake_up_state+0x20/0x20 [13549.028254] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.029888] [] kthread+0xcf/0xe0 [13549.031379] [] ? kthread_create_on_node+0x140/0x140 [13549.032934] [] ret_from_fork+0x58/0x90 [13549.034464] [] ? kthread_create_on_node+0x140/0x140 [13549.036022] z_fr_iss_1 S ffff88006182e600 0 19950 2 0x00000080 [13549.037709] ffff8800673cbe08 0000000000000046 ffff880068a43ec0 ffff8800673cbfd8 [13549.039463] ffff8800673cbfd8 ffff8800673cbfd8 ffff880068a43ec0 ffff8800497d0e40 [13549.041204] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.042893] Call Trace: [13549.044191] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.045816] [] schedule+0x29/0x70 [13549.047310] [] taskq_thread+0x3fa/0x440 [spl] [13549.048850] [] ? wake_up_state+0x20/0x20 [13549.050414] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.052038] [] kthread+0xcf/0xe0 [13549.053511] [] ? kthread_create_on_node+0x140/0x140 [13549.055113] [] ret_from_fork+0x58/0x90 [13549.056623] [] ? kthread_create_on_node+0x140/0x140 [13549.058233] z_fr_iss_1 S ffff88006182e600 0 19951 2 0x00000080 [13549.059867] ffff8800509fbe08 0000000000000046 ffff880068a44e70 ffff8800509fbfd8 [13549.061623] ffff8800509fbfd8 ffff8800509fbfd8 ffff880068a44e70 ffff8800497d0d80 [13549.063356] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.065082] Call Trace: [13549.066401] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.067982] [] schedule+0x29/0x70 [13549.069530] [] taskq_thread+0x3fa/0x440 [spl] [13549.071098] [] ? wake_up_state+0x20/0x20 [13549.072667] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.074278] [] kthread+0xcf/0xe0 [13549.075764] [] ? kthread_create_on_node+0x140/0x140 [13549.077312] [] ret_from_fork+0x58/0x90 [13549.078849] [] ? kthread_create_on_node+0x140/0x140 [13549.080464] z_fr_iss_1 S ffff88006182e600 0 19952 2 0x00000080 [13549.082099] ffff880066ef7e08 0000000000000046 ffff880068a45e20 ffff880066ef7fd8 [13549.083869] ffff880066ef7fd8 ffff880066ef7fd8 ffff880068a45e20 ffff8800497d0b40 [13549.085609] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.087355] Call Trace: [13549.088616] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.090272] [] schedule+0x29/0x70 [13549.091768] [] taskq_thread+0x3fa/0x440 [spl] [13549.093355] [] ? wake_up_state+0x20/0x20 [13549.094918] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.096534] [] kthread+0xcf/0xe0 [13549.098017] [] ? kthread_create_on_node+0x140/0x140 [13549.099620] [] ret_from_fork+0x58/0x90 [13549.101141] [] ? kthread_create_on_node+0x140/0x140 [13549.102745] z_fr_iss_1 S ffff88006182e600 0 19953 2 0x00000080 [13549.104435] ffff88005f107e08 0000000000000046 ffff880068a46dd0 ffff88005f107fd8 [13549.106173] ffff88005f107fd8 ffff88005f107fd8 ffff880068a46dd0 ffff8800497d09c0 [13549.107865] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.109607] Call Trace: [13549.110896] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.112544] [] schedule+0x29/0x70 [13549.114052] [] taskq_thread+0x3fa/0x440 [spl] [13549.115629] [] ? wake_up_state+0x20/0x20 [13549.117178] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.118743] [] kthread+0xcf/0xe0 [13549.120227] [] ? kthread_create_on_node+0x140/0x140 [13549.121781] [] ret_from_fork+0x58/0x90 [13549.123308] [] ? kthread_create_on_node+0x140/0x140 [13549.124873] z_fr_iss_1 S ffff88006182e600 0 19954 2 0x00000080 [13549.126546] ffff88005f337e08 0000000000000046 ffff88007a110000 ffff88005f337fd8 [13549.128299] ffff88005f337fd8 ffff88005f337fd8 ffff88007a110000 ffff8800497d0a20 [13549.129991] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.131754] Call Trace: [13549.133017] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.134667] [] schedule+0x29/0x70 [13549.136123] [] taskq_thread+0x3fa/0x440 [spl] [13549.137720] [] ? wake_up_state+0x20/0x20 [13549.139273] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.140832] [] kthread+0xcf/0xe0 [13549.142324] [] ? kthread_create_on_node+0x140/0x140 [13549.143932] [] ret_from_fork+0x58/0x90 [13549.145450] [] ? kthread_create_on_node+0x140/0x140 [13549.147065] z_fr_iss_1 S ffff88006182e600 0 19955 2 0x00000080 [13549.148698] ffff88007b85fe08 0000000000000046 ffff88007a110fb0 ffff88007b85ffd8 [13549.150453] ffff88007b85ffd8 ffff88007b85ffd8 ffff88007a110fb0 ffff8800497d0000 [13549.152191] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.153885] Call Trace: [13549.155212] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.156791] [] schedule+0x29/0x70 [13549.158301] [] taskq_thread+0x3fa/0x440 [spl] [13549.159855] [] ? wake_up_state+0x20/0x20 [13549.161417] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.162983] [] kthread+0xcf/0xe0 [13549.164471] [] ? kthread_create_on_node+0x140/0x140 [13549.166071] [] ret_from_fork+0x58/0x90 [13549.167586] [] ? kthread_create_on_node+0x140/0x140 [13549.169189] z_fr_iss_1 S ffff88006182e600 0 19956 2 0x00000080 [13549.170822] ffff88006060be08 0000000000000046 ffff88007a111f60 ffff88006060bfd8 [13549.172568] ffff88006060bfd8 ffff88006060bfd8 ffff88007a111f60 ffff8800497d0120 [13549.174306] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.175991] Call Trace: [13549.177281] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.178908] [] schedule+0x29/0x70 [13549.180349] [] taskq_thread+0x3fa/0x440 [spl] [13549.181953] [] ? wake_up_state+0x20/0x20 [13549.183497] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.185109] [] kthread+0xcf/0xe0 [13549.186580] [] ? kthread_create_on_node+0x140/0x140 [13549.188198] [] ret_from_fork+0x58/0x90 [13549.189715] [] ? kthread_create_on_node+0x140/0x140 [13549.191270] z_fr_iss_1 S ffff88006182e600 0 19957 2 0x00000080 [13549.192955] ffff880065b53e08 0000000000000046 ffff88007a112f10 ffff880065b53fd8 [13549.194706] ffff880065b53fd8 ffff880065b53fd8 ffff88007a112f10 ffff8800497d0ea0 [13549.196453] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.198192] Call Trace: [13549.199451] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.201089] [] schedule+0x29/0x70 [13549.202532] [] taskq_thread+0x3fa/0x440 [spl] [13549.204127] [] ? wake_up_state+0x20/0x20 [13549.205626] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.207243] [] kthread+0xcf/0xe0 [13549.208724] [] ? kthread_create_on_node+0x140/0x140 [13549.210271] [] ret_from_fork+0x58/0x90 [13549.211807] [] ? kthread_create_on_node+0x140/0x140 [13549.213405] z_fr_iss_1 S ffff88006182e600 0 19958 2 0x00000080 [13549.215046] ffff880079abbe08 0000000000000046 ffff88007a113ec0 ffff880079abbfd8 [13549.216784] ffff880079abbfd8 ffff880079abbfd8 ffff88007a113ec0 ffff8800497d0d20 [13549.218527] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.220262] Call Trace: [13549.221523] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.223182] [] schedule+0x29/0x70 [13549.224675] [] taskq_thread+0x3fa/0x440 [spl] [13549.226254] [] ? wake_up_state+0x20/0x20 [13549.227816] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.229384] [] kthread+0xcf/0xe0 [13549.230883] [] ? kthread_create_on_node+0x140/0x140 [13549.232485] [] ret_from_fork+0x58/0x90 [13549.234012] [] ? kthread_create_on_node+0x140/0x140 [13549.235624] z_fr_iss_1 S ffff88006182e600 0 19959 2 0x00000080 [13549.237320] ffff8800684f7e08 0000000000000046 ffff88007a114e70 ffff8800684f7fd8 [13549.239022] ffff8800684f7fd8 ffff8800684f7fd8 ffff88007a114e70 ffff8800497d0600 [13549.240770] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.242520] Call Trace: [13549.243802] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.245449] [] schedule+0x29/0x70 [13549.246901] [] taskq_thread+0x3fa/0x440 [spl] [13549.248494] [] ? wake_up_state+0x20/0x20 [13549.250059] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.251671] [] kthread+0xcf/0xe0 [13549.253145] [] ? kthread_create_on_node+0x140/0x140 [13549.254708] [] ret_from_fork+0x58/0x90 [13549.256234] [] ? kthread_create_on_node+0x140/0x140 [13549.257842] z_fr_iss_1 S ffff88006182e600 0 19960 2 0x00000080 [13549.259488] ffff880055ab7e08 0000000000000046 ffff88007a115e20 ffff880055ab7fd8 [13549.261231] ffff880055ab7fd8 ffff880055ab7fd8 ffff88007a115e20 ffff8800497d0240 [13549.262925] ffff88006182e690 ffffffffa06624b0 ffff88006182e6b0 ffff88006182e600 [13549.264676] Call Trace: [13549.265940] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.267597] [] schedule+0x29/0x70 [13549.269107] [] taskq_thread+0x3fa/0x440 [spl] [13549.270692] [] ? wake_up_state+0x20/0x20 [13549.272199] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.273825] [] kthread+0xcf/0xe0 [13549.275266] [] ? kthread_create_on_node+0x140/0x140 [13549.276887] [] ret_from_fork+0x58/0x90 [13549.278406] [] ? kthread_create_on_node+0x140/0x140 [13549.280017] z_fr_iss_2 S ffff88006182ee00 0 19961 2 0x00000080 [13549.281666] ffff88005f973e08 0000000000000046 ffff88007a116dd0 ffff88005f973fd8 [13549.283407] ffff88005f973fd8 ffff88005f973fd8 ffff88007a116dd0 ffff8800497d0cc0 [13549.285148] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.286841] Call Trace: [13549.288137] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.289765] [] schedule+0x29/0x70 [13549.291268] [] taskq_thread+0x3fa/0x440 [spl] [13549.292807] [] ? wake_up_state+0x20/0x20 [13549.294369] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.295987] [] kthread+0xcf/0xe0 [13549.297414] [] ? kthread_create_on_node+0x140/0x140 [13549.299036] [] ret_from_fork+0x58/0x90 [13549.300513] [] ? kthread_create_on_node+0x140/0x140 [13549.302135] z_fr_iss_2 S ffff88006182ee00 0 19962 2 0x00000080 [13549.303794] ffff88006815be08 0000000000000046 ffff88005b7e8000 ffff88006815bfd8 [13549.305551] ffff88006815bfd8 ffff88006815bfd8 ffff88005b7e8000 ffff8800497d08a0 [13549.307285] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.308975] Call Trace: [13549.310303] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.311934] [] schedule+0x29/0x70 [13549.313429] [] taskq_thread+0x3fa/0x440 [spl] [13549.315015] [] ? wake_up_state+0x20/0x20 [13549.316512] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.318141] [] kthread+0xcf/0xe0 [13549.319567] [] ? kthread_create_on_node+0x140/0x140 [13549.321188] [] ret_from_fork+0x58/0x90 [13549.322715] [] ? kthread_create_on_node+0x140/0x140 [13549.324317] z_fr_iss_2 S ffff88006182ee00 0 19963 2 0x00000080 [13549.325965] ffff88006184fe08 0000000000000046 ffff88005b7e8fb0 ffff88006184ffd8 [13549.327724] ffff88006184ffd8 ffff88006184ffd8 ffff88005b7e8fb0 ffff8800497d0360 [13549.329474] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.331208] Call Trace: [13549.332470] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.334115] [] schedule+0x29/0x70 [13549.335623] [] taskq_thread+0x3fa/0x440 [spl] [13549.337238] [] ? wake_up_state+0x20/0x20 [13549.338739] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.340359] [] kthread+0xcf/0xe0 [13549.341785] [] ? kthread_create_on_node+0x140/0x140 [13549.343402] [] ret_from_fork+0x58/0x90 [13549.344917] [] ? kthread_create_on_node+0x140/0x140 [13549.346473] z_fr_iss_2 S ffff88006182ee00 0 19964 2 0x00000080 [13549.348156] ffff88004924fe08 0000000000000046 ffff88005b7e9f60 ffff88004924ffd8 [13549.349906] ffff88004924ffd8 ffff88004924ffd8 ffff88005b7e9f60 ffff8800497d0720 [13549.351655] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.353393] Call Trace: [13549.354660] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.356304] [] schedule+0x29/0x70 [13549.357800] [] taskq_thread+0x3fa/0x440 [spl] [13549.359388] [] ? wake_up_state+0x20/0x20 [13549.360890] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.362518] [] kthread+0xcf/0xe0 [13549.363946] [] ? kthread_create_on_node+0x140/0x140 [13549.365572] [] ret_from_fork+0x58/0x90 [13549.367091] [] ? kthread_create_on_node+0x140/0x140 [13549.368698] z_fr_iss_2 S ffff88006182ee00 0 19965 2 0x00000080 [13549.370388] ffff88005e6fbe08 0000000000000046 ffff88005b7eaf10 ffff88005e6fbfd8 [13549.372135] ffff88005e6fbfd8 ffff88005e6fbfd8 ffff88005b7eaf10 ffff8800497d04e0 [13549.373824] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.375552] Call Trace: [13549.376825] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.378474] [] schedule+0x29/0x70 [13549.379975] [] taskq_thread+0x3fa/0x440 [spl] [13549.381512] [] ? wake_up_state+0x20/0x20 [13549.383081] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.384646] [] kthread+0xcf/0xe0 [13549.386129] [] ? kthread_create_on_node+0x140/0x140 [13549.387682] [] ret_from_fork+0x58/0x90 [13549.389209] [] ? kthread_create_on_node+0x140/0x140 [13549.390835] z_fr_iss_2 S ffff88006182ee00 0 19966 2 0x00000080 [13549.392538] ffff880066efbe08 0000000000000046 ffff88005b7ebec0 ffff880066efbfd8 [13549.394306] ffff880066efbfd8 ffff880066efbfd8 ffff88005b7ebec0 ffff8800497d0c60 [13549.396003] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.397754] Call Trace: [13549.399016] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.400681] [] schedule+0x29/0x70 [13549.402177] [] taskq_thread+0x3fa/0x440 [spl] [13549.403705] [] ? wake_up_state+0x20/0x20 [13549.405274] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.406836] [] kthread+0xcf/0xe0 [13549.408333] [] ? kthread_create_on_node+0x140/0x140 [13549.409886] [] ret_from_fork+0x58/0x90 [13549.411415] [] ? kthread_create_on_node+0x140/0x140 [13549.413016] z_fr_iss_2 S ffff88006182ee00 0 19967 2 0x00000080 [13549.414698] ffff88004a14be08 0000000000000046 ffff88005b7ece70 ffff88004a14bfd8 [13549.416459] ffff88004a14bfd8 ffff88004a14bfd8 ffff88005b7ece70 ffff8800497d0ba0 [13549.418199] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.419886] Call Trace: [13549.421183] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.422771] [] schedule+0x29/0x70 [13549.424277] [] taskq_thread+0x3fa/0x440 [spl] [13549.425812] [] ? wake_up_state+0x20/0x20 [13549.427374] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.428935] [] kthread+0xcf/0xe0 [13549.430430] [] ? kthread_create_on_node+0x140/0x140 [13549.432040] [] ret_from_fork+0x58/0x90 [13549.433557] [] ? kthread_create_on_node+0x140/0x140 [13549.435158] z_fr_iss_2 S ffff88006182ee00 0 19968 2 0x00000080 [13549.436796] ffff880054bfbe08 0000000000000046 ffff88005b7ede20 ffff880054bfbfd8 [13549.438543] ffff880054bfbfd8 ffff880054bfbfd8 ffff88005b7ede20 ffff8800497d0300 [13549.440279] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.441965] Call Trace: [13549.443260] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.444837] [] schedule+0x29/0x70 [13549.446346] [] taskq_thread+0x3fa/0x440 [spl] [13549.447932] [] ? wake_up_state+0x20/0x20 [13549.449431] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.451066] [] kthread+0xcf/0xe0 [13549.452541] [] ? kthread_create_on_node+0x140/0x140 [13549.454096] [] ret_from_fork+0x58/0x90 [13549.455628] [] ? kthread_create_on_node+0x140/0x140 [13549.457185] z_fr_iss_2 S ffff88006182ee00 0 19969 2 0x00000080 [13549.458871] ffff88005f037e08 0000000000000046 ffff88005b7eedd0 ffff88005f037fd8 [13549.460613] ffff88005f037fd8 ffff88005f037fd8 ffff88005b7eedd0 ffff8800497d0420 [13549.462350] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.464036] Call Trace: [13549.465334] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.466956] [] schedule+0x29/0x70 [13549.468402] [] taskq_thread+0x3fa/0x440 [spl] [13549.469996] [] ? wake_up_state+0x20/0x20 [13549.471494] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.473121] [] kthread+0xcf/0xe0 [13549.474545] [] ? kthread_create_on_node+0x140/0x140 [13549.476169] [] ret_from_fork+0x58/0x90 [13549.477686] [] ? kthread_create_on_node+0x140/0x140 [13549.479283] z_fr_iss_2 S ffff88006182ee00 0 19970 2 0x00000080 [13549.480918] ffff8800591b7e08 0000000000000046 ffff88005f830000 ffff8800591b7fd8 [13549.482653] ffff8800591b7fd8 ffff8800591b7fd8 ffff88005f830000 ffff8800497d0540 [13549.484396] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.486131] Call Trace: [13549.487391] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.489035] [] schedule+0x29/0x70 [13549.490529] [] taskq_thread+0x3fa/0x440 [spl] [13549.492059] [] ? wake_up_state+0x20/0x20 [13549.493626] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.495233] [] kthread+0xcf/0xe0 [13549.496707] [] ? kthread_create_on_node+0x140/0x140 [13549.498258] [] ret_from_fork+0x58/0x90 [13549.499790] [] ? kthread_create_on_node+0x140/0x140 [13549.501392] z_fr_iss_2 S ffff88006182ee00 0 19971 2 0x00000080 [13549.503025] ffff88006041be08 0000000000000046 ffff88005f830fb0 ffff88006041bfd8 [13549.504774] ffff88006041bfd8 ffff88006041bfd8 ffff88005f830fb0 ffff8800497d0780 [13549.506519] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.508246] Call Trace: [13549.509507] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.511165] [] schedule+0x29/0x70 [13549.512609] [] taskq_thread+0x3fa/0x440 [spl] [13549.514214] [] ? wake_up_state+0x20/0x20 [13549.515717] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.517342] [] kthread+0xcf/0xe0 [13549.518771] [] ? kthread_create_on_node+0x140/0x140 [13549.520386] [] ret_from_fork+0x58/0x90 [13549.521900] [] ? kthread_create_on_node+0x140/0x140 [13549.523505] z_fr_iss_2 S ffff88006182ee00 0 19972 2 0x00000080 [13549.525147] ffff880063afbe08 0000000000000046 ffff88005f831f60 ffff880063afbfd8 [13549.526901] ffff880063afbfd8 ffff880063afbfd8 ffff88005f831f60 ffff8800497d06c0 [13549.528658] ffff88006182ee90 ffffffffa06624b0 ffff88006182eeb0 ffff88006182ee00 [13549.530404] Call Trace: [13549.531678] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.533321] [] schedule+0x29/0x70 [13549.534838] [] taskq_thread+0x3fa/0x440 [spl] [13549.536367] [] ? wake_up_state+0x20/0x20 [13549.537941] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.539556] [] kthread+0xcf/0xe0 [13549.540991] [] ? kthread_create_on_node+0x140/0x140 [13549.542618] [] ret_from_fork+0x58/0x90 [13549.544138] [] ? kthread_create_on_node+0x140/0x140 [13549.545707] z_fr_iss_3 S ffff88006182ed00 0 19973 2 0x00000080 [13549.547446] ffff880036547e08 0000000000000046 ffff88005f832f10 ffff880036547fd8 [13549.549195] ffff880036547fd8 ffff880036547fd8 ffff88005f832f10 ffff8800497d0480 [13549.550886] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.552628] Call Trace: [13549.553894] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.555551] [] schedule+0x29/0x70 [13549.557002] [] taskq_thread+0x3fa/0x440 [spl] [13549.558601] [] ? wake_up_state+0x20/0x20 [13549.560103] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.561730] [] kthread+0xcf/0xe0 [13549.563163] [] ? kthread_create_on_node+0x140/0x140 [13549.564780] [] ret_from_fork+0x58/0x90 [13549.566294] [] ? kthread_create_on_node+0x140/0x140 [13549.567850] z_fr_iss_3 S ffff88006182ed00 0 19974 2 0x00000080 [13549.569541] ffff880063b07e08 0000000000000046 ffff88005f833ec0 ffff880063b07fd8 [13549.571282] ffff880063b07fd8 ffff880063b07fd8 ffff88005f833ec0 ffff8800497d0660 [13549.572981] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.574743] Call Trace: [13549.576013] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.577661] [] schedule+0x29/0x70 [13549.579163] [] taskq_thread+0x3fa/0x440 [spl] [13549.580689] [] ? wake_up_state+0x20/0x20 [13549.582251] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.583821] [] kthread+0xcf/0xe0 [13549.585308] [] ? kthread_create_on_node+0x140/0x140 [13549.586918] [] ret_from_fork+0x58/0x90 [13549.588428] [] ? kthread_create_on_node+0x140/0x140 [13549.590035] z_fr_iss_3 S ffff88006182ed00 0 19975 2 0x00000080 [13549.591673] ffff880065983e08 0000000000000046 ffff88005f834e70 ffff880065983fd8 [13549.593432] ffff880065983fd8 ffff880065983fd8 ffff88005f834e70 ffff8800497d00c0 [13549.595166] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.596854] Call Trace: [13549.598147] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.599721] [] schedule+0x29/0x70 [13549.601232] [] taskq_thread+0x3fa/0x440 [spl] [13549.602760] [] ? wake_up_state+0x20/0x20 [13549.604322] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.605932] [] kthread+0xcf/0xe0 [13549.607404] [] ? kthread_create_on_node+0x140/0x140 [13549.609020] [] ret_from_fork+0x58/0x90 [13549.610482] [] ? kthread_create_on_node+0x140/0x140 [13549.612099] z_fr_iss_3 S ffff88006182ed00 0 19976 2 0x00000080 [13549.613733] ffff8800626c7e08 0000000000000046 ffff88005f835e20 ffff8800626c7fd8 [13549.615481] ffff8800626c7fd8 ffff8800626c7fd8 ffff88005f835e20 ffff8800497d0840 [13549.617217] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.618904] Call Trace: [13549.620199] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.621833] [] schedule+0x29/0x70 [13549.623326] [] taskq_thread+0x3fa/0x440 [spl] [13549.624904] [] ? wake_up_state+0x20/0x20 [13549.626408] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.628032] [] kthread+0xcf/0xe0 [13549.629459] [] ? kthread_create_on_node+0x140/0x140 [13549.631081] [] ret_from_fork+0x58/0x90 [13549.632592] [] ? kthread_create_on_node+0x140/0x140 [13549.634145] z_fr_iss_3 S ffff88006182ed00 0 19977 2 0x00000080 [13549.635829] ffff880046bd7e08 0000000000000046 ffff88005f836dd0 ffff880046bd7fd8 [13549.637580] ffff880046bd7fd8 ffff880046bd7fd8 ffff88005f836dd0 ffff8800497d0060 [13549.639313] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.641015] Call Trace: [13549.642328] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.643905] [] schedule+0x29/0x70 [13549.645412] [] taskq_thread+0x3fa/0x440 [spl] [13549.646994] [] ? wake_up_state+0x20/0x20 [13549.648547] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.650105] [] kthread+0xcf/0xe0 [13549.651594] [] ? kthread_create_on_node+0x140/0x140 [13549.653144] [] ret_from_fork+0x58/0x90 [13549.654679] [] ? kthread_create_on_node+0x140/0x140 [13549.656232] z_fr_iss_3 S ffff88006182ed00 0 19978 2 0x00000080 [13549.657914] ffff880066b5fe08 0000000000000046 ffff880063a98000 ffff880066b5ffd8 [13549.659661] ffff880066b5ffd8 ffff880066b5ffd8 ffff880063a98000 ffff8800497d01e0 [13549.661407] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.663140] Call Trace: [13549.664403] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.666051] [] schedule+0x29/0x70 [13549.667550] [] taskq_thread+0x3fa/0x440 [spl] [13549.669079] [] ? wake_up_state+0x20/0x20 [13549.670648] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.672206] [] kthread+0xcf/0xe0 [13549.673709] [] ? kthread_create_on_node+0x140/0x140 [13549.675315] [] ret_from_fork+0x58/0x90 [13549.676842] [] ? kthread_create_on_node+0x140/0x140 [13549.678450] z_fr_iss_3 S ffff88006182ed00 0 19979 2 0x00000080 [13549.680090] ffff88007a10be08 0000000000000046 ffff880063a98fb0 ffff88007a10bfd8 [13549.681856] ffff88007a10bfd8 ffff88007a10bfd8 ffff880063a98fb0 ffff8800497d05a0 [13549.683609] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.685341] Call Trace: [13549.686604] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.688258] [] schedule+0x29/0x70 [13549.689756] [] taskq_thread+0x3fa/0x440 [spl] [13549.691295] [] ? wake_up_state+0x20/0x20 [13549.692867] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.694448] [] kthread+0xcf/0xe0 [13549.695941] [] ? kthread_create_on_node+0x140/0x140 [13549.697540] [] ret_from_fork+0x58/0x90 [13549.699011] [] ? kthread_create_on_node+0x140/0x140 [13549.700622] z_fr_iss_3 S ffff88006182ed00 0 19980 2 0x00000080 [13549.702323] ffff88006156fe08 0000000000000046 ffff880063a99f60 ffff88006156ffd8 [13549.704019] ffff88006156ffd8 ffff88006156ffd8 ffff880063a99f60 ffff8800497d0de0 [13549.705759] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.707499] Call Trace: [13549.708764] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.710417] [] schedule+0x29/0x70 [13549.711863] [] taskq_thread+0x3fa/0x440 [spl] [13549.713453] [] ? wake_up_state+0x20/0x20 [13549.714953] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.716577] [] kthread+0xcf/0xe0 [13549.718003] [] ? kthread_create_on_node+0x140/0x140 [13549.719619] [] ret_from_fork+0x58/0x90 [13549.721084] [] ? kthread_create_on_node+0x140/0x140 [13549.722694] z_fr_iss_3 S ffff88006182ed00 0 19981 2 0x00000080 [13549.724389] ffff88005de17e08 0000000000000046 ffff880063a9af10 ffff88005de17fd8 [13549.726126] ffff88005de17fd8 ffff88005de17fd8 ffff880063a9af10 ffff8800497d0f00 [13549.727812] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.729566] Call Trace: [13549.730829] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.732475] [] schedule+0x29/0x70 [13549.733968] [] taskq_thread+0x3fa/0x440 [spl] [13549.735491] [] ? wake_up_state+0x20/0x20 [13549.737059] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.738611] [] kthread+0xcf/0xe0 [13549.740098] [] ? kthread_create_on_node+0x140/0x140 [13549.741656] [] ret_from_fork+0x58/0x90 [13549.743179] [] ? kthread_create_on_node+0x140/0x140 [13549.744735] z_fr_iss_3 S ffff88006182ed00 0 19982 2 0x00000080 [13549.746420] ffff88005ab8fe08 0000000000000046 ffff880063a9bec0 ffff88005ab8ffd8 [13549.748161] ffff88005ab8ffd8 ffff88005ab8ffd8 ffff880063a9bec0 ffff880066f50720 [13549.749847] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.751583] Call Trace: [13549.752848] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.754492] [] schedule+0x29/0x70 [13549.755950] [] taskq_thread+0x3fa/0x440 [spl] [13549.757540] [] ? wake_up_state+0x20/0x20 [13549.759115] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.760721] [] kthread+0xcf/0xe0 [13549.762194] [] ? kthread_create_on_node+0x140/0x140 [13549.763796] [] ret_from_fork+0x58/0x90 [13549.765258] [] ? kthread_create_on_node+0x140/0x140 [13549.766876] z_fr_iss_3 S ffff88006182ed00 0 19983 2 0x00000080 [13549.768521] ffff880062a63e08 0000000000000046 ffff880063a9ce70 ffff880062a63fd8 [13549.770258] ffff880062a63fd8 ffff880062a63fd8 ffff880063a9ce70 ffff880066f50000 [13549.771958] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.773707] Call Trace: [13549.774964] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.776618] [] schedule+0x29/0x70 [13549.778109] [] taskq_thread+0x3fa/0x440 [spl] [13549.779686] [] ? wake_up_state+0x20/0x20 [13549.781227] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.782834] [] kthread+0xcf/0xe0 [13549.784302] [] ? kthread_create_on_node+0x140/0x140 [13549.785900] [] ret_from_fork+0x58/0x90 [13549.787364] [] ? kthread_create_on_node+0x140/0x140 [13549.788984] z_fr_iss_3 S ffff88006182ed00 0 19984 2 0x00000080 [13549.790629] ffff880062a6be08 0000000000000046 ffff880063a9de20 ffff880062a6bfd8 [13549.792361] ffff880062a6bfd8 ffff880062a6bfd8 ffff880063a9de20 ffff880066f50960 [13549.794056] ffff88006182ed90 ffffffffa06624b0 ffff88006182edb0 ffff88006182ed00 [13549.795810] Call Trace: [13549.797100] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.798669] [] schedule+0x29/0x70 [13549.800174] [] taskq_thread+0x3fa/0x440 [spl] [13549.801756] [] ? wake_up_state+0x20/0x20 [13549.803299] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.804915] [] kthread+0xcf/0xe0 [13549.806337] [] ? kthread_create_on_node+0x140/0x140 [13549.807971] [] ret_from_fork+0x58/0x90 [13549.809481] [] ? kthread_create_on_node+0x140/0x140 [13549.811033] z_fr_iss_4 S ffff88006182ec00 0 19985 2 0x00000080 [13549.812726] ffff88005ce7be08 0000000000000046 ffff880063a9edd0 ffff88005ce7bfd8 [13549.814471] ffff88005ce7bfd8 ffff88005ce7bfd8 ffff880063a9edd0 ffff880066f50e40 [13549.816207] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.817894] Call Trace: [13549.819223] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.820798] [] schedule+0x29/0x70 [13549.822316] [] taskq_thread+0x3fa/0x440 [spl] [13549.823856] [] ? wake_up_state+0x20/0x20 [13549.825433] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.826996] [] kthread+0xcf/0xe0 [13549.828492] [] ? kthread_create_on_node+0x140/0x140 [13549.830055] [] ret_from_fork+0x58/0x90 [13549.831581] [] ? kthread_create_on_node+0x140/0x140 [13549.833188] z_fr_iss_4 S ffff88006182ec00 0 19986 2 0x00000080 [13549.834819] ffff8800608dfe08 0000000000000046 ffff880047a80000 ffff8800608dffd8 [13549.836575] ffff8800608dffd8 ffff8800608dffd8 ffff880047a80000 ffff880066f508a0 [13549.838318] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.840008] Call Trace: [13549.841299] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.842928] [] schedule+0x29/0x70 [13549.844367] [] taskq_thread+0x3fa/0x440 [spl] [13549.845967] [] ? wake_up_state+0x20/0x20 [13549.847507] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.849114] [] kthread+0xcf/0xe0 [13549.850584] [] ? kthread_create_on_node+0x140/0x140 [13549.852130] [] ret_from_fork+0x58/0x90 [13549.853658] [] ? kthread_create_on_node+0x140/0x140 [13549.855251] z_fr_iss_4 S ffff88006182ec00 0 19987 2 0x00000080 [13549.856886] ffff88007a05be08 0000000000000046 ffff880047a80fb0 ffff88007a05bfd8 [13549.858635] ffff88007a05bfd8 ffff88007a05bfd8 ffff880047a80fb0 ffff880066f50420 [13549.860375] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.862057] Call Trace: [13549.863391] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.864967] [] schedule+0x29/0x70 [13549.866469] [] taskq_thread+0x3fa/0x440 [spl] [13549.868053] [] ? wake_up_state+0x20/0x20 [13549.869590] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.871194] [] kthread+0xcf/0xe0 [13549.872670] [] ? kthread_create_on_node+0x140/0x140 [13549.874263] [] ret_from_fork+0x58/0x90 [13549.875777] [] ? kthread_create_on_node+0x140/0x140 [13549.877369] z_fr_iss_4 S ffff88006182ec00 0 19988 2 0x00000080 [13549.879009] ffff8800689ebe08 0000000000000046 ffff880047a81f60 ffff8800689ebfd8 [13549.880743] ffff8800689ebfd8 ffff8800689ebfd8 ffff880047a81f60 ffff880066f50c00 [13549.882491] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.884219] Call Trace: [13549.885477] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.887113] [] schedule+0x29/0x70 [13549.888605] [] taskq_thread+0x3fa/0x440 [spl] [13549.890189] [] ? wake_up_state+0x20/0x20 [13549.891732] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.893342] [] kthread+0xcf/0xe0 [13549.894762] [] ? kthread_create_on_node+0x140/0x140 [13549.896365] [] ret_from_fork+0x58/0x90 [13549.897839] [] ? kthread_create_on_node+0x140/0x140 [13549.899445] z_fr_iss_4 S ffff88006182ec00 0 19989 2 0x00000080 [13549.901077] ffff88004921be08 0000000000000046 ffff880047a82f10 ffff88004921bfd8 [13549.902821] ffff88004921bfd8 ffff88004921bfd8 ffff880047a82f10 ffff880066f50660 [13549.904552] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.906289] Call Trace: [13549.907543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.909185] [] schedule+0x29/0x70 [13549.910676] [] taskq_thread+0x3fa/0x440 [spl] [13549.912196] [] ? wake_up_state+0x20/0x20 [13549.913763] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.915361] [] kthread+0xcf/0xe0 [13549.916836] [] ? kthread_create_on_node+0x140/0x140 [13549.918387] [] ret_from_fork+0x58/0x90 [13549.919913] [] ? kthread_create_on_node+0x140/0x140 [13549.921516] z_fr_iss_4 S ffff88006182ec00 0 19990 2 0x00000080 [13549.923150] ffff880048337e08 0000000000000046 ffff880047a83ec0 ffff880048337fd8 [13549.924905] ffff880048337fd8 ffff880048337fd8 ffff880047a83ec0 ffff880066f50540 [13549.926635] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.928384] Call Trace: [13549.929646] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.931296] [] schedule+0x29/0x70 [13549.932831] [] taskq_thread+0x3fa/0x440 [spl] [13549.934455] [] ? wake_up_state+0x20/0x20 [13549.936048] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.937680] [] kthread+0xcf/0xe0 [13549.939107] [] ? kthread_create_on_node+0x140/0x140 [13549.940724] [] ret_from_fork+0x58/0x90 [13549.942242] [] ? kthread_create_on_node+0x140/0x140 [13549.943844] z_fr_iss_4 S ffff88006182ec00 0 19991 2 0x00000080 [13549.945491] ffff88005f46fe08 0000000000000046 ffff880047a84e70 ffff88005f46ffd8 [13549.947231] ffff88005f46ffd8 ffff88005f46ffd8 ffff880047a84e70 ffff880066f50180 [13549.948926] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.950651] Call Trace: [13549.951918] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.953563] [] schedule+0x29/0x70 [13549.955061] [] taskq_thread+0x3fa/0x440 [spl] [13549.956593] [] ? wake_up_state+0x20/0x20 [13549.958159] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.959772] [] kthread+0xcf/0xe0 [13549.961241] [] ? kthread_create_on_node+0x140/0x140 [13549.962796] [] ret_from_fork+0x58/0x90 [13549.964328] [] ? kthread_create_on_node+0x140/0x140 [13549.965881] z_fr_iss_4 S ffff88006182ec00 0 19992 2 0x00000080 [13549.967571] ffff880063b27e08 0000000000000046 ffff880047a85e20 ffff880063b27fd8 [13549.969311] ffff880063b27fd8 ffff880063b27fd8 ffff880047a85e20 ffff880066f50c60 [13549.971011] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.972770] Call Trace: [13549.974068] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.975695] [] schedule+0x29/0x70 [13549.977191] [] taskq_thread+0x3fa/0x440 [spl] [13549.978727] [] ? wake_up_state+0x20/0x20 [13549.980285] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.981867] [] kthread+0xcf/0xe0 [13549.983354] [] ? kthread_create_on_node+0x140/0x140 [13549.984908] [] ret_from_fork+0x58/0x90 [13549.986433] [] ? kthread_create_on_node+0x140/0x140 [13549.988041] z_fr_iss_4 S ffff88006182ec00 0 19993 2 0x00000080 [13549.989671] ffff880066d97e08 0000000000000046 ffff880047a86dd0 ffff880066d97fd8 [13549.991433] ffff880066d97fd8 ffff880066d97fd8 ffff880047a86dd0 ffff880066f503c0 [13549.993171] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13549.994860] Call Trace: [13549.996185] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13549.997817] [] schedule+0x29/0x70 [13549.999303] [] taskq_thread+0x3fa/0x440 [spl] [13550.000832] [] ? wake_up_state+0x20/0x20 [13550.002394] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.003953] [] kthread+0xcf/0xe0 [13550.005440] [] ? kthread_create_on_node+0x140/0x140 [13550.006992] [] ret_from_fork+0x58/0x90 [13550.008517] [] ? kthread_create_on_node+0x140/0x140 [13550.010137] z_fr_iss_4 S ffff88006182ec00 0 19994 2 0x00000080 [13550.011769] ffff880060f1be08 0000000000000046 ffff880063a00000 ffff880060f1bfd8 [13550.013518] ffff880060f1bfd8 ffff880060f1bfd8 ffff880063a00000 ffff880066f50a20 [13550.015252] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13550.016935] Call Trace: [13550.018274] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.019850] [] schedule+0x29/0x70 [13550.021355] [] taskq_thread+0x3fa/0x440 [spl] [13550.022936] [] ? wake_up_state+0x20/0x20 [13550.024434] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.026064] [] kthread+0xcf/0xe0 [13550.027544] [] ? kthread_create_on_node+0x140/0x140 [13550.029090] [] ret_from_fork+0x58/0x90 [13550.030619] [] ? kthread_create_on_node+0x140/0x140 [13550.032213] z_fr_iss_4 S ffff88006182ec00 0 19995 2 0x00000080 [13550.033846] ffff880065fa7e08 0000000000000046 ffff880063a00fb0 ffff880065fa7fd8 [13550.035594] ffff880065fa7fd8 ffff880065fa7fd8 ffff880063a00fb0 ffff880066f50600 [13550.037325] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13550.039009] Call Trace: [13550.040299] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.041867] [] schedule+0x29/0x70 [13550.043379] [] taskq_thread+0x3fa/0x440 [spl] [13550.044956] [] ? wake_up_state+0x20/0x20 [13550.046452] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.048085] [] kthread+0xcf/0xe0 [13550.049504] [] ? kthread_create_on_node+0x140/0x140 [13550.051125] [] ret_from_fork+0x58/0x90 [13550.052585] [] ? kthread_create_on_node+0x140/0x140 [13550.054209] z_fr_iss_4 S ffff88006182ec00 0 19996 2 0x00000080 [13550.055841] ffff880053e93e08 0000000000000046 ffff880063a01f60 ffff880053e93fd8 [13550.057590] ffff880053e93fd8 ffff880053e93fd8 ffff880063a01f60 ffff880066f50de0 [13550.059323] ffff88006182ec90 ffffffffa06624b0 ffff88006182ecb0 ffff88006182ec00 [13550.061005] Call Trace: [13550.062298] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.063872] [] schedule+0x29/0x70 [13550.065376] [] taskq_thread+0x3fa/0x440 [spl] [13550.066954] [] ? wake_up_state+0x20/0x20 [13550.068497] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.070052] [] kthread+0xcf/0xe0 [13550.071540] [] ? kthread_create_on_node+0x140/0x140 [13550.073140] [] ret_from_fork+0x58/0x90 [13550.074603] [] ? kthread_create_on_node+0x140/0x140 [13550.076220] z_fr_iss_5 S ffff88006182e900 0 19997 2 0x00000080 [13550.077850] ffff8800497f7e08 0000000000000046 ffff880063a02f10 ffff8800497f7fd8 [13550.079591] ffff8800497f7fd8 ffff8800497f7fd8 ffff880063a02f10 ffff880066f50b40 [13550.081318] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.083002] Call Trace: [13550.084292] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.085879] [] schedule+0x29/0x70 [13550.087384] [] taskq_thread+0x3fa/0x440 [spl] [13550.088913] [] ? wake_up_state+0x20/0x20 [13550.090469] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.092027] [] kthread+0xcf/0xe0 [13550.093509] [] ? kthread_create_on_node+0x140/0x140 [13550.095116] [] ret_from_fork+0x58/0x90 [13550.096631] [] ? kthread_create_on_node+0x140/0x140 [13550.098176] z_fr_iss_5 S ffff88006182e900 0 19998 2 0x00000080 [13550.099862] ffff88005fa27e08 0000000000000046 ffff880063a03ec0 ffff88005fa27fd8 [13550.101611] ffff88005fa27fd8 ffff88005fa27fd8 ffff880063a03ec0 ffff880066f50780 [13550.103352] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.105048] Call Trace: [13550.106343] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.107974] [] schedule+0x29/0x70 [13550.109424] [] taskq_thread+0x3fa/0x440 [spl] [13550.111025] [] ? wake_up_state+0x20/0x20 [13550.112578] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.114142] [] kthread+0xcf/0xe0 [13550.115647] [] ? kthread_create_on_node+0x140/0x140 [13550.117246] [] ret_from_fork+0x58/0x90 [13550.118716] [] ? kthread_create_on_node+0x140/0x140 [13550.120326] z_fr_iss_5 S ffff88006182e900 0 19999 2 0x00000080 [13550.121974] ffff880066f43e08 0000000000000046 ffff880063a04e70 ffff880066f43fd8 [13550.123736] ffff880066f43fd8 ffff880066f43fd8 ffff880063a04e70 ffff880066f50ea0 [13550.125481] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.127211] Call Trace: [13550.128471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.130106] [] schedule+0x29/0x70 [13550.131597] [] taskq_thread+0x3fa/0x440 [spl] [13550.133174] [] ? wake_up_state+0x20/0x20 [13550.134669] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.136294] [] kthread+0xcf/0xe0 [13550.137722] [] ? kthread_create_on_node+0x140/0x140 [13550.139331] [] ret_from_fork+0x58/0x90 [13550.140846] [] ? kthread_create_on_node+0x140/0x140 [13550.142442] z_fr_iss_5 S ffff88006182e900 0 20000 2 0x00000080 [13550.144075] ffff88005f3d7e08 0000000000000046 ffff880063a05e20 ffff88005f3d7fd8 [13550.145817] ffff88005f3d7fd8 ffff88005f3d7fd8 ffff880063a05e20 ffff880066f50f00 [13550.147558] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.149284] Call Trace: [13550.150544] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.152187] [] schedule+0x29/0x70 [13550.153630] [] taskq_thread+0x3fa/0x440 [spl] [13550.155236] [] ? wake_up_state+0x20/0x20 [13550.156732] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.158348] [] kthread+0xcf/0xe0 [13550.159834] [] ? kthread_create_on_node+0x140/0x140 [13550.161378] [] ret_from_fork+0x58/0x90 [13550.162910] [] ? kthread_create_on_node+0x140/0x140 [13550.164507] z_fr_iss_5 S ffff88006182e900 0 20001 2 0x00000080 [13550.166141] ffff88005b6a3e08 0000000000000046 ffff880063a06dd0 ffff88005b6a3fd8 [13550.167881] ffff88005b6a3fd8 ffff88005b6a3fd8 ffff880063a06dd0 ffff880066f504e0 [13550.169630] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.171354] Call Trace: [13550.172614] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.174262] [] schedule+0x29/0x70 [13550.175755] [] taskq_thread+0x3fa/0x440 [spl] [13550.177275] [] ? wake_up_state+0x20/0x20 [13550.178835] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.180387] [] kthread+0xcf/0xe0 [13550.181874] [] ? kthread_create_on_node+0x140/0x140 [13550.183420] [] ret_from_fork+0x58/0x90 [13550.184946] [] ? kthread_create_on_node+0x140/0x140 [13550.186538] z_fr_iss_5 S ffff88006182e900 0 20002 2 0x00000080 [13550.188170] ffff88005f9d7e08 0000000000000046 ffff88005ee30000 ffff88005f9d7fd8 [13550.189905] ffff88005f9d7fd8 ffff88005f9d7fd8 ffff88005ee30000 ffff880066f50ae0 [13550.191652] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.193389] Call Trace: [13550.194649] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.196301] [] schedule+0x29/0x70 [13550.197791] [] taskq_thread+0x3fa/0x440 [spl] [13550.199321] [] ? wake_up_state+0x20/0x20 [13550.200883] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.202488] [] kthread+0xcf/0xe0 [13550.203964] [] ? kthread_create_on_node+0x140/0x140 [13550.205571] [] ret_from_fork+0x58/0x90 [13550.207090] [] ? kthread_create_on_node+0x140/0x140 [13550.208635] z_fr_iss_5 S ffff88006182e900 0 20003 2 0x00000080 [13550.210358] ffff8800661b7e08 0000000000000046 ffff88005ee30fb0 ffff8800661b7fd8 [13550.212051] ffff8800661b7fd8 ffff8800661b7fd8 ffff88005ee30fb0 ffff880066f501e0 [13550.213786] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.215505] Call Trace: [13550.216766] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.218408] [] schedule+0x29/0x70 [13550.219850] [] taskq_thread+0x3fa/0x440 [spl] [13550.221436] [] ? wake_up_state+0x20/0x20 [13550.222936] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.224557] [] kthread+0xcf/0xe0 [13550.226035] [] ? finish_task_switch+0x53/0x180 [13550.227597] [] ? kthread_create_on_node+0x140/0x140 [13550.229143] [] ret_from_fork+0x58/0x90 [13550.230670] [] ? kthread_create_on_node+0x140/0x140 [13550.232264] z_fr_iss_5 S ffff88006182e900 0 20004 2 0x00000080 [13550.233895] ffff8800537efe08 0000000000000046 ffff88005ee31f60 ffff8800537effd8 [13550.235651] ffff8800537effd8 ffff8800537effd8 ffff88005ee31f60 ffff880066f50300 [13550.237397] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.239133] Call Trace: [13550.240379] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.242003] [] schedule+0x29/0x70 [13550.243500] [] taskq_thread+0x3fa/0x440 [spl] [13550.245029] [] ? wake_up_state+0x20/0x20 [13550.246576] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.248176] [] kthread+0xcf/0xe0 [13550.249648] [] ? kthread_create_on_node+0x140/0x140 [13550.251247] [] ret_from_fork+0x58/0x90 [13550.252711] [] ? kthread_create_on_node+0x140/0x140 [13550.254319] z_fr_iss_5 S ffff88006182e900 0 20005 2 0x00000080 [13550.255947] ffff880063ab3e08 0000000000000046 ffff88005ee32f10 ffff880063ab3fd8 [13550.257696] ffff880063ab3fd8 ffff880063ab3fd8 ffff88005ee32f10 ffff880066f50cc0 [13550.259445] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.261175] Call Trace: [13550.262420] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.264042] [] schedule+0x29/0x70 [13550.265484] [] taskq_thread+0x3fa/0x440 [spl] [13550.267078] [] ? wake_up_state+0x20/0x20 [13550.268564] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.270176] [] kthread+0xcf/0xe0 [13550.271649] [] ? kthread_create_on_node+0x140/0x140 [13550.273194] [] ret_from_fork+0x58/0x90 [13550.274724] [] ? kthread_create_on_node+0x140/0x140 [13550.276269] z_fr_iss_5 S ffff88006182e900 0 20006 2 0x00000080 [13550.277954] ffff88005ec5be08 0000000000000046 ffff88005ee33ec0 ffff88005ec5bfd8 [13550.279690] ffff88005ec5bfd8 ffff88005ec5bfd8 ffff88005ee33ec0 ffff880066f50d20 [13550.281438] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.283174] Call Trace: [13550.284425] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.286049] [] schedule+0x29/0x70 [13550.287543] [] taskq_thread+0x3fa/0x440 [spl] [13550.289074] [] ? wake_up_state+0x20/0x20 [13550.290629] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.292226] [] kthread+0xcf/0xe0 [13550.293703] [] ? kthread_create_on_node+0x140/0x140 [13550.295250] [] ret_from_fork+0x58/0x90 [13550.296781] [] ? kthread_create_on_node+0x140/0x140 [13550.298376] z_fr_iss_5 S ffff88006182e900 0 20007 2 0x00000080 [13550.300011] ffff88005f30be08 0000000000000046 ffff88005ee34e70 ffff88005f30bfd8 [13550.301755] ffff88005f30bfd8 ffff88005f30bfd8 ffff88005ee34e70 ffff880066f502a0 [13550.303512] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.305242] Call Trace: [13550.306500] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.308162] [] schedule+0x29/0x70 [13550.309607] [] taskq_thread+0x3fa/0x440 [spl] [13550.311197] [] ? wake_up_state+0x20/0x20 [13550.312744] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.314300] [] kthread+0xcf/0xe0 [13550.315797] [] ? kthread_create_on_node+0x140/0x140 [13550.317390] [] ret_from_fork+0x58/0x90 [13550.318905] [] ? kthread_create_on_node+0x140/0x140 [13550.320453] z_fr_iss_5 S ffff88006182e900 0 20008 2 0x00000080 [13550.322135] ffff88007b2dfe08 0000000000000046 ffff88005ee35e20 ffff88007b2dffd8 [13550.323878] ffff88007b2dffd8 ffff88007b2dffd8 ffff88005ee35e20 ffff880066f50ba0 [13550.325624] ffff88006182e990 ffffffffa06624b0 ffff88006182e9b0 ffff88006182e900 [13550.327368] Call Trace: [13550.328633] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.330282] [] schedule+0x29/0x70 [13550.331731] [] taskq_thread+0x3fa/0x440 [spl] [13550.333324] [] ? wake_up_state+0x20/0x20 [13550.334882] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.336495] [] kthread+0xcf/0xe0 [13550.337926] [] ? finish_task_switch+0x53/0x180 [13550.339511] [] ? kthread_create_on_node+0x140/0x140 [13550.341126] [] ret_from_fork+0x58/0x90 [13550.342640] [] ? kthread_create_on_node+0x140/0x140 [13550.344239] z_fr_iss_6 S ffff88006182e200 0 20009 2 0x00000080 [13550.345870] ffff8800681afe08 0000000000000046 ffff88005ee36dd0 ffff8800681affd8 [13550.347623] ffff8800681affd8 ffff8800681affd8 ffff88005ee36dd0 ffff880066f507e0 [13550.349349] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.351032] Call Trace: [13550.352315] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.353932] [] schedule+0x29/0x70 [13550.355383] [] taskq_thread+0x3fa/0x440 [spl] [13550.356974] [] ? wake_up_state+0x20/0x20 [13550.358460] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.360082] [] kthread+0xcf/0xe0 [13550.361561] [] ? kthread_create_on_node+0x140/0x140 [13550.363115] [] ret_from_fork+0x58/0x90 [13550.364647] [] ? kthread_create_on_node+0x140/0x140 [13550.366249] z_fr_iss_6 S ffff88006182e200 0 20010 2 0x00000080 [13550.367881] ffff88007b1d3e08 0000000000000046 ffff880049b60000 ffff88007b1d3fd8 [13550.369638] ffff88007b1d3fd8 ffff88007b1d3fd8 ffff880049b60000 ffff880066f50240 [13550.371368] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.373055] Call Trace: [13550.374387] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.375957] [] schedule+0x29/0x70 [13550.377474] [] taskq_thread+0x3fa/0x440 [spl] [13550.379056] [] ? wake_up_state+0x20/0x20 [13550.380537] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.382153] [] kthread+0xcf/0xe0 [13550.383578] [] ? kthread_create_on_node+0x140/0x140 [13550.385192] [] ret_from_fork+0x58/0x90 [13550.386710] [] ? kthread_create_on_node+0x140/0x140 [13550.388307] z_fr_iss_6 S ffff88006182e200 0 20011 2 0x00000080 [13550.389943] ffff880065bebe08 0000000000000046 ffff880049b60fb0 ffff880065bebfd8 [13550.391675] ffff880065bebfd8 ffff880065bebfd8 ffff880049b60fb0 ffff880066f50a80 [13550.393400] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.395128] Call Trace: [13550.396381] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.398030] [] schedule+0x29/0x70 [13550.399531] [] taskq_thread+0x3fa/0x440 [spl] [13550.401069] [] ? wake_up_state+0x20/0x20 [13550.402624] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.404173] [] kthread+0xcf/0xe0 [13550.405662] [] ? kthread_create_on_node+0x140/0x140 [13550.407258] [] ret_from_fork+0x58/0x90 [13550.408734] [] ? kthread_create_on_node+0x140/0x140 [13550.410354] z_fr_iss_6 S ffff88006182e200 0 20012 2 0x00000080 [13550.411995] ffff880045ec3e08 0000000000000046 ffff880049b61f60 ffff880045ec3fd8 [13550.413761] ffff880045ec3fd8 ffff880045ec3fd8 ffff880049b61f60 ffff880066f50060 [13550.415506] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.417235] Call Trace: [13550.418494] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.420144] [] schedule+0x29/0x70 [13550.421649] [] taskq_thread+0x3fa/0x440 [spl] [13550.423210] [] ? wake_up_state+0x20/0x20 [13550.424783] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.426384] [] kthread+0xcf/0xe0 [13550.427866] [] ? kthread_create_on_node+0x140/0x140 [13550.429465] [] ret_from_fork+0x58/0x90 [13550.430931] [] ? kthread_create_on_node+0x140/0x140 [13550.432546] z_fr_iss_6 S ffff88006182e200 0 20013 2 0x00000080 [13550.434183] ffff88005f37fe08 0000000000000046 ffff880049b62f10 ffff88005f37ffd8 [13550.435924] ffff88005f37ffd8 ffff88005f37ffd8 ffff880049b62f10 ffff880066f50360 [13550.437686] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.439429] Call Trace: [13550.440698] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.442344] [] schedule+0x29/0x70 [13550.443796] [] taskq_thread+0x3fa/0x440 [spl] [13550.445386] [] ? wake_up_state+0x20/0x20 [13550.446932] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.448537] [] kthread+0xcf/0xe0 [13550.449965] [] ? kthread_create_on_node+0x140/0x140 [13550.451587] [] ret_from_fork+0x58/0x90 [13550.453053] [] ? kthread_create_on_node+0x140/0x140 [13550.454673] z_fr_iss_6 S ffff88006182e200 0 20014 2 0x00000080 [13550.456361] ffff88005b99fe08 0000000000000046 ffff880049b63ec0 ffff88005b99ffd8 [13550.458052] ffff88005b99ffd8 ffff88005b99ffd8 ffff880049b63ec0 ffff880066f50120 [13550.459797] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.461533] Call Trace: [13550.462799] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.464445] [] schedule+0x29/0x70 [13550.465937] [] taskq_thread+0x3fa/0x440 [spl] [13550.467513] [] ? wake_up_state+0x20/0x20 [13550.469064] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.470625] [] kthread+0xcf/0xe0 [13550.472114] [] ? kthread_create_on_node+0x140/0x140 [13550.473667] [] ret_from_fork+0x58/0x90 [13550.475198] [] ? kthread_create_on_node+0x140/0x140 [13550.476753] z_fr_iss_6 S ffff88006182e200 0 20015 2 0x00000080 [13550.478446] ffff8800497fbe08 0000000000000046 ffff880049b64e70 ffff8800497fbfd8 [13550.480184] ffff8800497fbfd8 ffff8800497fbfd8 ffff880049b64e70 ffff880066f50840 [13550.481879] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.483624] Call Trace: [13550.484886] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.486545] [] schedule+0x29/0x70 [13550.488056] [] taskq_thread+0x3fa/0x440 [spl] [13550.489584] [] ? wake_up_state+0x20/0x20 [13550.491152] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.492731] [] kthread+0xcf/0xe0 [13550.494225] [] ? kthread_create_on_node+0x140/0x140 [13550.495784] [] ret_from_fork+0x58/0x90 [13550.497314] [] ? kthread_create_on_node+0x140/0x140 [13550.498874] z_fr_iss_6 S ffff88006182e200 0 20016 2 0x00000080 [13550.500564] ffff880059a1be08 0000000000000046 ffff880049b65e20 ffff880059a1bfd8 [13550.502314] ffff880059a1bfd8 ffff880059a1bfd8 ffff880049b65e20 ffff880066f506c0 [13550.504006] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.505754] Call Trace: [13550.507053] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.508626] [] schedule+0x29/0x70 [13550.510133] [] taskq_thread+0x3fa/0x440 [spl] [13550.511675] [] ? wake_up_state+0x20/0x20 [13550.513235] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.514848] [] kthread+0xcf/0xe0 [13550.516322] [] ? kthread_create_on_node+0x140/0x140 [13550.517873] [] ret_from_fork+0x58/0x90 [13550.519398] [] ? kthread_create_on_node+0x140/0x140 [13550.521010] z_fr_iss_6 S ffff88006182e200 0 20017 2 0x00000080 [13550.522642] ffff880055c23e08 0000000000000046 ffff880049b66dd0 ffff880055c23fd8 [13550.524397] ffff880055c23fd8 ffff880055c23fd8 ffff880049b66dd0 ffff880066f50f60 [13550.526131] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.527821] Call Trace: [13550.529133] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.530771] [] schedule+0x29/0x70 [13550.532217] [] taskq_thread+0x3fa/0x440 [spl] [13550.533811] [] ? wake_up_state+0x20/0x20 [13550.535311] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.536938] [] kthread+0xcf/0xe0 [13550.538364] [] ? kthread_create_on_node+0x140/0x140 [13550.539986] [] ret_from_fork+0x58/0x90 [13550.541451] [] ? kthread_create_on_node+0x140/0x140 [13550.543072] z_fr_iss_6 S ffff88006182e200 0 20018 2 0x00000080 [13550.544704] ffff880046517e08 0000000000000046 ffff88005d2a8000 ffff880046517fd8 [13550.546456] ffff880046517fd8 ffff880046517fd8 ffff88005d2a8000 ffff880066f500c0 [13550.548193] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.549878] Call Trace: [13550.551172] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.552749] [] schedule+0x29/0x70 [13550.554257] [] taskq_thread+0x3fa/0x440 [spl] [13550.555786] [] ? wake_up_state+0x20/0x20 [13550.557345] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.558957] [] kthread+0xcf/0xe0 [13550.560382] [] ? kthread_create_on_node+0x140/0x140 [13550.562013] [] ret_from_fork+0x58/0x90 [13550.563533] [] ? kthread_create_on_node+0x140/0x140 [13550.565146] z_fr_iss_6 S ffff88006182e200 0 20019 2 0x00000080 [13550.566783] ffff8800633dfe08 0000000000000046 ffff88005d2a8fb0 ffff8800633dffd8 [13550.568517] ffff8800633dffd8 ffff8800633dffd8 ffff88005d2a8fb0 ffff880066f509c0 [13550.570266] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.571959] Call Trace: [13550.573252] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.574830] [] schedule+0x29/0x70 [13550.576340] [] taskq_thread+0x3fa/0x440 [spl] [13550.577873] [] ? wake_up_state+0x20/0x20 [13550.579433] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.580990] [] kthread+0xcf/0xe0 [13550.582472] [] ? kthread_create_on_node+0x140/0x140 [13550.584026] [] ret_from_fork+0x58/0x90 [13550.585547] [] ? kthread_create_on_node+0x140/0x140 [13550.587150] z_fr_iss_6 S ffff88006182e200 0 20020 2 0x00000080 [13550.588782] ffff880066f4fe08 0000000000000046 ffff88005d2a9f60 ffff880066f4ffd8 [13550.590523] ffff880066f4ffd8 ffff880066f4ffd8 ffff88005d2a9f60 ffff880066f50d80 [13550.592277] ffff88006182e290 ffffffffa06624b0 ffff88006182e2b0 ffff88006182e200 [13550.593962] Call Trace: [13550.595256] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.596876] [] schedule+0x29/0x70 [13550.598363] [] taskq_thread+0x3fa/0x440 [spl] [13550.599898] [] ? wake_up_state+0x20/0x20 [13550.601458] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.603024] [] kthread+0xcf/0xe0 [13550.604516] [] ? kthread_create_on_node+0x140/0x140 [13550.606068] [] ret_from_fork+0x58/0x90 [13550.607603] [] ? kthread_create_on_node+0x140/0x140 [13550.609150] z_fr_iss_7 S ffff88006182e700 0 20021 2 0x00000080 [13550.610835] ffff88005f6dfe08 0000000000000046 ffff88005d2aaf10 ffff88005f6dffd8 [13550.612578] ffff88005f6dffd8 ffff88005f6dffd8 ffff88005d2aaf10 ffff88005ea0cd20 [13550.614309] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.615991] Call Trace: [13550.617300] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.618873] [] schedule+0x29/0x70 [13550.620383] [] taskq_thread+0x3fa/0x440 [spl] [13550.621971] [] ? wake_up_state+0x20/0x20 [13550.623517] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.625090] [] kthread+0xcf/0xe0 [13550.626573] [] ? kthread_create_on_node+0x140/0x140 [13550.628173] [] ret_from_fork+0x58/0x90 [13550.629725] [] ? kthread_create_on_node+0x140/0x140 [13550.631303] z_fr_iss_7 S ffff88006182e700 0 20022 2 0x00000080 [13550.632985] ffff880047af7e08 0000000000000046 ffff88005d2abec0 ffff880047af7fd8 [13550.634741] ffff880047af7fd8 ffff880047af7fd8 ffff88005d2abec0 ffff88005ea0c4e0 [13550.636481] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.638215] Call Trace: [13550.639471] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.641106] [] schedule+0x29/0x70 [13550.642550] [] taskq_thread+0x3fa/0x440 [spl] [13550.644145] [] ? wake_up_state+0x20/0x20 [13550.645643] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.647261] [] kthread+0xcf/0xe0 [13550.648687] [] ? kthread_create_on_node+0x140/0x140 [13550.650294] [] ret_from_fork+0x58/0x90 [13550.651812] [] ? kthread_create_on_node+0x140/0x140 [13550.653406] z_fr_iss_7 S ffff88006182e700 0 20023 2 0x00000080 [13550.655050] ffff880066eb3e08 0000000000000046 ffff88005d2ace70 ffff880066eb3fd8 [13550.656791] ffff880066eb3fd8 ffff880066eb3fd8 ffff88005d2ace70 ffff88005ea0cd80 [13550.658533] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.660318] Call Trace: [13550.661577] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.663224] [] schedule+0x29/0x70 [13550.664720] [] taskq_thread+0x3fa/0x440 [spl] [13550.666244] [] ? wake_up_state+0x20/0x20 [13550.667811] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.669370] [] kthread+0xcf/0xe0 [13550.670855] [] ? kthread_create_on_node+0x140/0x140 [13550.672454] [] ret_from_fork+0x58/0x90 [13550.673972] [] ? kthread_create_on_node+0x140/0x140 [13550.675581] z_fr_iss_7 S ffff88006182e700 0 20024 2 0x00000080 [13550.677224] ffff88007a1a3e08 0000000000000046 ffff88005d2ade20 ffff88007a1a3fd8 [13550.678953] ffff88007a1a3fd8 ffff88007a1a3fd8 ffff88005d2ade20 ffff88005ea0cf00 [13550.680702] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.682444] Call Trace: [13550.683710] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.685358] [] schedule+0x29/0x70 [13550.686807] [] taskq_thread+0x3fa/0x440 [spl] [13550.688398] [] ? wake_up_state+0x20/0x20 [13550.689901] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.691521] [] kthread+0xcf/0xe0 [13550.692998] [] ? kthread_create_on_node+0x140/0x140 [13550.694550] [] ret_from_fork+0x58/0x90 [13550.696082] [] ? kthread_create_on_node+0x140/0x140 [13550.697688] z_fr_iss_7 S ffff88006182e700 0 20025 2 0x00000080 [13550.699378] ffff88005d2e3e08 0000000000000046 ffff88005d2aedd0 ffff88005d2e3fd8 [13550.701118] ffff88005d2e3fd8 ffff88005d2e3fd8 ffff88005d2aedd0 ffff88005ea0c240 [13550.702805] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.704547] Call Trace: [13550.705813] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.707458] [] schedule+0x29/0x70 [13550.708944] [] taskq_thread+0x3fa/0x440 [spl] [13550.710545] [] ? wake_up_state+0x20/0x20 [13550.712110] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.713719] [] kthread+0xcf/0xe0 [13550.715146] [] ? kthread_create_on_node+0x140/0x140 [13550.716768] [] ret_from_fork+0x58/0x90 [13550.718236] [] ? kthread_create_on_node+0x140/0x140 [13550.719862] z_fr_iss_7 S ffff88006182e700 0 20026 2 0x00000080 [13550.721518] ffff88005fabbe08 0000000000000046 ffff88005b590000 ffff88005fabbfd8 [13550.723269] ffff88005fabbfd8 ffff88005fabbfd8 ffff88005b590000 ffff88005ea0cde0 [13550.724964] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.726659] Call Trace: [13550.727917] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.729494] [] schedule+0x29/0x70 [13550.730943] [] taskq_thread+0x3fa/0x440 [spl] [13550.732482] [] ? wake_up_state+0x20/0x20 [13550.733995] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.735557] [] kthread+0xcf/0xe0 [13550.736982] [] ? kthread_create_on_node+0x140/0x140 [13550.738532] [] ret_from_fork+0x58/0x90 [13550.740002] [] ? kthread_create_on_node+0x140/0x140 [13550.741553] z_fr_iss_7 S ffff88006182e700 0 20027 2 0x00000080 [13550.743185] ffff88005b0d7e08 0000000000000046 ffff88005b590fb0 ffff88005b0d7fd8 [13550.744890] ffff88005b0d7fd8 ffff88005b0d7fd8 ffff88005b590fb0 ffff88005ea0cb40 [13550.746594] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.748284] Call Trace: [13550.749552] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.751120] [] schedule+0x29/0x70 [13550.752562] [] taskq_thread+0x3fa/0x440 [spl] [13550.754087] [] ? wake_up_state+0x20/0x20 [13550.755584] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.757134] [] kthread+0xcf/0xe0 [13550.758551] [] ? kthread_create_on_node+0x140/0x140 [13550.760095] [] ret_from_fork+0x58/0x90 [13550.761552] [] ? kthread_create_on_node+0x140/0x140 [13550.763104] z_fr_iss_7 S ffff88006182e700 0 20028 2 0x00000080 [13550.764738] ffff880048c1fe08 0000000000000046 ffff88005b591f60 ffff880048c1ffd8 [13550.766438] ffff880048c1ffd8 ffff880048c1ffd8 ffff88005b591f60 ffff88005ea0c480 [13550.768139] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.769826] Call Trace: [13550.771081] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.772649] [] schedule+0x29/0x70 [13550.774089] [] taskq_thread+0x3fa/0x440 [spl] [13550.775612] [] ? wake_up_state+0x20/0x20 [13550.777105] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.778661] [] kthread+0xcf/0xe0 [13550.780081] [] ? kthread_create_on_node+0x140/0x140 [13550.781627] [] ret_from_fork+0x58/0x90 [13550.783086] [] ? kthread_create_on_node+0x140/0x140 [13550.784633] z_fr_iss_7 S ffff88006182e700 0 20029 2 0x00000080 [13550.786272] ffff880063a5fe08 0000000000000046 ffff88005b592f10 ffff880063a5ffd8 [13550.787961] ffff880063a5ffd8 ffff880063a5ffd8 ffff88005b592f10 ffff88005ea0cea0 [13550.789661] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.791343] Call Trace: [13550.792602] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.794166] [] schedule+0x29/0x70 [13550.795609] [] taskq_thread+0x3fa/0x440 [spl] [13550.797130] [] ? wake_up_state+0x20/0x20 [13550.798624] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.800177] [] kthread+0xcf/0xe0 [13550.801600] [] ? kthread_create_on_node+0x140/0x140 [13550.803148] [] ret_from_fork+0x58/0x90 [13550.804612] [] ? kthread_create_on_node+0x140/0x140 [13550.806159] z_fr_iss_7 S ffff88006182e700 0 20030 2 0x00000080 [13550.807787] ffff88005ccf7e08 0000000000000046 ffff88005b593ec0 ffff88005ccf7fd8 [13550.809486] ffff88005ccf7fd8 ffff88005ccf7fd8 ffff88005b593ec0 ffff88005ea0c840 [13550.811184] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.812869] Call Trace: [13550.814126] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.815695] [] schedule+0x29/0x70 [13550.817136] [] taskq_thread+0x3fa/0x440 [spl] [13550.818664] [] ? wake_up_state+0x20/0x20 [13550.820158] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.821716] [] kthread+0xcf/0xe0 [13550.823142] [] ? kthread_create_on_node+0x140/0x140 [13550.824692] [] ret_from_fork+0x58/0x90 [13550.826154] [] ? kthread_create_on_node+0x140/0x140 [13550.827706] z_fr_iss_7 S ffff88006182e700 0 20031 2 0x00000080 [13550.829345] ffff880067257e08 0000000000000046 ffff88005b594e70 ffff880067257fd8 [13550.831035] ffff880067257fd8 ffff880067257fd8 ffff88005b594e70 ffff88005ea0c540 [13550.832724] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.834416] Call Trace: [13550.835680] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.837247] [] schedule+0x29/0x70 [13550.838697] [] taskq_thread+0x3fa/0x440 [spl] [13550.840222] [] ? wake_up_state+0x20/0x20 [13550.841732] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.843288] [] kthread+0xcf/0xe0 [13550.844714] [] ? kthread_create_on_node+0x140/0x140 [13550.846272] [] ret_from_fork+0x58/0x90 [13550.847740] [] ? kthread_create_on_node+0x140/0x140 [13550.849290] z_fr_iss_7 S ffff88006182e700 0 20032 2 0x00000080 [13550.850925] ffff880045dbbe08 0000000000000046 ffff88005b595e20 ffff880045dbbfd8 [13550.852626] ffff880045dbbfd8 ffff880045dbbfd8 ffff88005b595e20 ffff88005ea0c8a0 [13550.854318] ffff88006182e790 ffffffffa06624b0 ffff88006182e7b0 ffff88006182e700 [13550.856004] Call Trace: [13550.857277] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.858849] [] schedule+0x29/0x70 [13550.860307] [] taskq_thread+0x3fa/0x440 [spl] [13550.861846] [] ? wake_up_state+0x20/0x20 [13550.863349] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.864918] [] kthread+0xcf/0xe0 [13550.866341] [] ? kthread_create_on_node+0x140/0x140 [13550.867906] [] ret_from_fork+0x58/0x90 [13550.869371] [] ? kthread_create_on_node+0x140/0x140 [13550.870926] z_fr_int S ffff88006182ea00 0 20033 2 0x00000080 [13550.872584] ffff88005acbfe08 0000000000000046 ffff88005b596dd0 ffff88005acbffd8 [13550.874291] ffff88005acbffd8 ffff88005acbffd8 ffff88005b596dd0 ffff88005ea0c0c0 [13550.875992] ffff88006182ea90 ffffffffa06624b0 ffff88006182eab0 ffff88006182ea00 [13550.877701] Call Trace: [13550.878958] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.880526] [] schedule+0x29/0x70 [13550.881977] [] taskq_thread+0x3fa/0x440 [spl] [13550.883505] [] ? wake_up_state+0x20/0x20 [13550.885020] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.886579] [] kthread+0xcf/0xe0 [13550.888002] [] ? kthread_create_on_node+0x140/0x140 [13550.889548] [] ret_from_fork+0x58/0x90 [13550.891018] [] ? kthread_create_on_node+0x140/0x140 [13550.892571] z_cl_iss S ffff88006182e300 0 20034 2 0x00000080 [13550.894211] ffff88006a3cbe08 0000000000000046 ffff880049230000 ffff88006a3cbfd8 [13550.895905] ffff88006a3cbfd8 ffff88006a3cbfd8 ffff880049230000 ffff88005ea0c420 [13550.897599] ffff88006182e390 ffffffffa06624b0 ffff88006182e3b0 ffff88006182e300 [13550.899291] Call Trace: [13550.900548] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.902120] [] schedule+0x29/0x70 [13550.903561] [] taskq_thread+0x3fa/0x440 [spl] [13550.905085] [] ? wake_up_state+0x20/0x20 [13550.906583] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.908137] [] kthread+0xcf/0xe0 [13550.909559] [] ? kthread_create_on_node+0x140/0x140 [13550.911106] [] ret_from_fork+0x58/0x90 [13550.912576] [] ? kthread_create_on_node+0x140/0x140 [13550.914123] z_cl_int S ffff88006182e500 0 20035 2 0x00000080 [13550.915758] ffff88004872be08 0000000000000046 ffff880049230fb0 ffff88004872bfd8 [13550.917459] ffff88004872bfd8 ffff88004872bfd8 ffff880049230fb0 ffff88005ea0c300 [13550.919163] ffff88006182e590 ffffffffa06624b0 ffff88006182e5b0 ffff88006182e500 [13550.921643] Call Trace: [13550.923172] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.924816] [] schedule+0x29/0x70 [13550.926309] [] taskq_thread+0x3fa/0x440 [spl] [13550.927900] [] ? wake_up_state+0x20/0x20 [13550.929444] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.931055] [] kthread+0xcf/0xe0 [13550.932533] [] ? kthread_create_on_node+0x140/0x140 [13550.934150] [] ret_from_fork+0x58/0x90 [13550.935618] [] ? kthread_create_on_node+0x140/0x140 [13550.937234] z_ioctl_iss S ffff88006182ef00 0 20036 2 0x00000080 [13550.938871] ffff88005e70fe08 0000000000000046 ffff880049231f60 ffff88005e70ffd8 [13550.940624] ffff88005e70ffd8 ffff88005e70ffd8 ffff880049231f60 ffff88005ea0c720 [13550.942347] ffff88006182ef90 ffffffffa06624b0 ffff88006182efb0 ffff88006182ef00 [13550.944035] Call Trace: [13550.945327] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.946900] [] schedule+0x29/0x70 [13550.948410] [] taskq_thread+0x3fa/0x440 [spl] [13550.949993] [] ? wake_up_state+0x20/0x20 [13550.951543] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.953104] [] kthread+0xcf/0xe0 [13550.954592] [] ? kthread_create_on_node+0x140/0x140 [13550.956192] [] ret_from_fork+0x58/0x90 [13550.957706] [] ? kthread_create_on_node+0x140/0x140 [13550.959308] z_ioctl_int S ffff880036ac8900 0 20037 2 0x00000080 [13550.960940] ffff88005ec77e08 0000000000000046 ffff880049232f10 ffff88005ec77fd8 [13550.962691] ffff88005ec77fd8 ffff88005ec77fd8 ffff880049232f10 ffff88005ea0cba0 [13550.964435] ffff880036ac8990 ffff8800679d9580 ffff880036ac89b0 ffff880036ac8900 [13550.966160] Call Trace: [13550.967412] [] schedule+0x29/0x70 [13550.968918] [] taskq_thread+0x3fa/0x440 [spl] [13550.970475] [] ? wake_up_state+0x20/0x20 [13550.971963] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.973587] [] kthread+0xcf/0xe0 [13550.975009] [] ? kthread_create_on_node+0x140/0x140 [13550.976630] [] ret_from_fork+0x58/0x90 [13550.978129] [] ? kthread_create_on_node+0x140/0x140 [13550.979676] z_zvol S ffff880036ac8300 0 20038 2 0x00000080 [13550.981389] ffff880055edbe08 0000000000000046 ffff880049233ec0 ffff880055edbfd8 [13550.983118] ffff880055edbfd8 ffff880055edbfd8 ffff880049233ec0 ffff88005ea0c900 [13550.984794] ffff880036ac8390 ffff8800608d1370 ffff880036ac83b0 ffff880036ac8300 [13550.986519] Call Trace: [13550.987769] [] schedule+0x29/0x70 [13550.989283] [] taskq_thread+0x3fa/0x440 [spl] [13550.990798] [] ? wake_up_state+0x20/0x20 [13550.992328] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13550.993932] [] kthread+0xcf/0xe0 [13550.995352] [] ? kthread_create_on_node+0x140/0x140 [13550.996966] [] ret_from_fork+0x58/0x90 [13550.998407] [] ? kthread_create_on_node+0x140/0x140 [13551.000009] metaslab_group_ S ffff880036ac8e00 0 20039 2 0x00000080 [13551.001616] ffff880046bdbe08 0000000000000046 ffff880049234e70 ffff880046bdbfd8 [13551.003334] ffff880046bdbfd8 ffff880046bdbfd8 ffff880049234e70 ffff88005ea0cf60 [13551.005010] ffff880036ac8e90 ffff880049290670 ffff880036ac8eb0 ffff880036ac8e00 [13551.006757] Call Trace: [13551.008051] [] schedule+0x29/0x70 [13551.009547] [] taskq_thread+0x3fa/0x440 [spl] [13551.011063] [] ? wake_up_state+0x20/0x20 [13551.012599] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13551.014174] [] kthread+0xcf/0xe0 [13551.015583] [] ? kthread_create_on_node+0x140/0x140 [13551.017182] [] ret_from_fork+0x58/0x90 [13551.018676] [] ? kthread_create_on_node+0x140/0x140 [13551.020247] z_iput S ffff88007a970b00 0 20041 2 0x00000080 [13551.021864] ffff88006598be08 0000000000000046 ffff880049236dd0 ffff88006598bfd8 [13551.023597] ffff88006598bfd8 ffff88006598bfd8 ffff880049236dd0 ffff88005ea0c7e0 [13551.025301] ffff88007a970b90 ffffffffa06624b0 ffff88007a970bb0 ffff88007a970b00 [13551.026959] Call Trace: [13551.028266] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13551.029876] [] schedule+0x29/0x70 [13551.031355] [] taskq_thread+0x3fa/0x440 [spl] [13551.032871] [] ? wake_up_state+0x20/0x20 [13551.034404] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13551.035991] [] kthread+0xcf/0xe0 [13551.037400] [] ? kthread_create_on_node+0x140/0x140 [13551.039014] [] ret_from_fork+0x58/0x90 [13551.040509] [] ? kthread_create_on_node+0x140/0x140 [13551.042040] z_iput S ffff88007a970b00 0 20042 2 0x00000080 [13551.043686] ffff880068a2be08 0000000000000046 ffff880049235e20 ffff880068a2bfd8 [13551.045416] ffff880068a2bfd8 ffff880068a2bfd8 ffff880049235e20 ffff88005ea0ce40 [13551.047131] ffff88007a970b90 ffffffffa06624b0 ffff88007a970bb0 ffff88007a970b00 [13551.048801] Call Trace: [13551.050074] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13551.051638] [] schedule+0x29/0x70 [13551.053141] [] taskq_thread+0x3fa/0x440 [spl] [13551.054654] [] ? wake_up_state+0x20/0x20 [13551.056186] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13551.057790] [] kthread+0xcf/0xe0 [13551.059213] [] ? kthread_create_on_node+0x140/0x140 [13551.060820] [] ret_from_fork+0x58/0x90 [13551.062319] [] ? kthread_create_on_node+0x140/0x140 [13551.063912] txg_quiesce S 0000000000000000 0 20056 2 0x00000080 [13551.065540] ffff88005b0dfd60 0000000000000046 ffff88005e8f4e70 ffff88005b0dffd8 [13551.067265] ffff88005b0dffd8 ffff88005b0dffd8 ffff88005e8f4e70 ffff8800628a1b08 [13551.068939] ffff8800628a1a10 ffff8800628a1b10 ffff8800628a1a38 0000000000000000 [13551.070686] Call Trace: [13551.071933] [] schedule+0x29/0x70 [13551.073453] [] cv_wait_common+0x10d/0x130 [spl] [13551.075034] [] ? wake_up_atomic_t+0x30/0x30 [13551.076521] [] __cv_wait_sig+0x15/0x20 [spl] [13551.078114] [] txg_quiesce_thread+0x38d/0x3a0 [zfs] [13551.079741] [] ? txg_fini+0x290/0x290 [zfs] [13551.081227] [] thread_generic_wrapper+0x71/0x80 [spl] [13551.082849] [] ? __thread_exit+0x20/0x20 [spl] [13551.084410] [] kthread+0xcf/0xe0 [13551.085828] [] ? kthread_create_on_node+0x140/0x140 [13551.087434] [] ret_from_fork+0x58/0x90 [13551.088891] [] ? kthread_create_on_node+0x140/0x140 [13551.090494] txg_sync D 0000000000000001 0 20057 2 0x00000080 [13551.092116] ffff88006a017ae0 0000000000000046 ffff88005e8f5e20 ffff88006a017fd8 [13551.093832] ffff88006a017fd8 ffff88006a017fd8 ffff88005e8f5e20 ffff88007fc16c40 [13551.095548] 0000000000000000 7fffffffffffffff ffff8800479bfbc8 0000000000000001 [13551.097267] Call Trace: [13551.098504] [] schedule+0x29/0x70 [13551.100004] [] schedule_timeout+0x239/0x2d0 [13551.101536] [] ? kvm_clock_get_cycles+0x1f/0x30 [13551.103042] [] ? ktime_get_ts64+0x4c/0xf0 [13551.104565] [] io_schedule_timeout+0xae/0x130 [13551.106090] [] ? prepare_to_wait_exclusive+0x56/0x90 [13551.107680] [] io_schedule+0x18/0x20 [13551.109107] [] cv_wait_common+0xa7/0x130 [spl] [13551.110672] [] ? wake_up_atomic_t+0x30/0x30 [13551.112221] [] __cv_wait_io+0x18/0x20 [spl] [13551.113767] [] zio_wait+0xc3/0x140 [zfs] [13551.115229] [] dsl_pool_sync+0xbf/0x440 [zfs] [13551.116782] [] spa_sync+0x388/0xb50 [zfs] [13551.118212] [] ? autoremove_wake_function+0x2b/0x40 [13551.119788] [] ? mutex_lock+0x12/0x2f [13551.121266] [] txg_sync_thread+0x3c5/0x620 [zfs] [13551.122816] [] ? txg_init+0x280/0x280 [zfs] [13551.124254] [] thread_generic_wrapper+0x71/0x80 [spl] [13551.125823] [] ? __thread_exit+0x20/0x20 [spl] [13551.127296] [] kthread+0xcf/0xe0 [13551.128720] [] ? kthread_create_on_node+0x140/0x140 [13551.130208] [] ret_from_fork+0x58/0x90 [13551.131679] [] ? kthread_create_on_node+0x140/0x140 [13551.133172] ldlm_cb00_000 S ffff8800604d9400 0 20226 2 0x00000080 [13551.134778] ffff88007bbdbd38 0000000000000046 ffff88005e8f0fb0 ffff88007bbdbfd8 [13551.136461] ffff88007bbdbfd8 ffff88007bbdbfd8 ffff88005e8f0fb0 0000000000000000 [13551.138123] ffff88005e8f0fb0 ffff880062a67000 ffff8800604d9468 ffff8800604d9400 [13551.139733] Call Trace: [13551.140956] [] schedule+0x29/0x70 [13551.142434] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.143999] [] ? wake_up_state+0x20/0x20 [13551.145512] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.146998] [] ? __switch_to+0xd9/0x4c0 [13551.148505] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.150139] [] kthread+0xcf/0xe0 [13551.151528] [] ? kthread_create_on_node+0x140/0x140 [13551.153114] [] ret_from_fork+0x58/0x90 [13551.154591] [] ? kthread_create_on_node+0x140/0x140 [13551.156152] ldlm_cb00_001 S ffff8800604d9400 0 20228 2 0x00000080 [13551.157752] ffff880063b2fd38 0000000000000046 ffff88005e8f1f60 ffff880063b2ffd8 [13551.159489] ffff880063b2ffd8 ffff880063b2ffd8 ffff88005e8f1f60 0000000000000000 [13551.161185] ffff88005e8f1f60 ffff880068966800 ffff8800604d9468 ffff8800604d9400 [13551.162835] Call Trace: [13551.164080] [] schedule+0x29/0x70 [13551.165507] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.167122] [] ? wake_up_state+0x20/0x20 [13551.168612] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.170198] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.171847] [] kthread+0xcf/0xe0 [13551.173297] [] ? kthread_create_on_node+0x140/0x140 [13551.174838] [] ret_from_fork+0x58/0x90 [13551.176351] [] ? kthread_create_on_node+0x140/0x140 [13551.177940] ldlm_cn00_000 S ffff8800604da400 0 20229 2 0x00000080 [13551.179569] ffff88006900fd38 0000000000000046 ffff88005e8f2f10 ffff88006900ffd8 [13551.181301] ffff88006900ffd8 ffff88006900ffd8 ffff88005e8f2f10 0000000000000000 [13551.182975] ffff88005e8f2f10 ffff880068966300 ffff8800604da468 ffff8800604da400 [13551.184707] Call Trace: [13551.185952] [] schedule+0x29/0x70 [13551.187489] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.189129] [] ? wake_up_state+0x20/0x20 [13551.190653] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.192292] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.193992] [] kthread+0xcf/0xe0 [13551.195492] [] ? kthread_create_on_node+0x140/0x140 [13551.197077] [] ret_from_fork+0x58/0x90 [13551.198641] [] ? kthread_create_on_node+0x140/0x140 [13551.200266] ldlm_cn00_001 S ffff8800604da400 0 20230 2 0x00000080 [13551.201928] ffff88005a8f3d38 0000000000000046 ffff88007b00bec0 ffff88005a8f3fd8 [13551.203711] ffff88005a8f3fd8 ffff88005a8f3fd8 ffff88007b00bec0 0000000000000000 [13551.205489] ffff88007b00bec0 ffff880079258780 ffff8800604da468 ffff8800604da400 [13551.207253] Call Trace: [13551.208537] [] schedule+0x29/0x70 [13551.210092] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.211761] [] ? wake_up_state+0x20/0x20 [13551.213356] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.214999] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.216731] [] kthread+0xcf/0xe0 [13551.218195] [] ? kthread_create_on_node+0x140/0x140 [13551.219856] [] ret_from_fork+0x58/0x90 [13551.221403] [] ? kthread_create_on_node+0x140/0x140 [13551.223051] ldlm_cn00_002 S ffff8800604da400 0 20231 2 0x00000080 [13551.224716] ffff880047ce7d38 0000000000000046 ffff88007b008fb0 ffff880047ce7fd8 [13551.226495] ffff880047ce7fd8 ffff880047ce7fd8 ffff88007b008fb0 0000000000000000 [13551.228261] ffff88007b008fb0 ffff880079258700 ffff8800604da468 ffff8800604da400 [13551.229979] Call Trace: [13551.231331] [] schedule+0x29/0x70 [13551.232896] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.234548] [] ? wake_up_state+0x20/0x20 [13551.236175] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.237782] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.239515] [] kthread+0xcf/0xe0 [13551.241041] [] ? kthread_create_on_node+0x140/0x140 [13551.242691] [] ret_from_fork+0x58/0x90 [13551.244244] [] ? kthread_create_on_node+0x140/0x140 [13551.245891] ldlm_bl_01 S ffff880059da7ba8 0 20232 2 0x00000080 [13551.247568] ffff880054bf7e08 0000000000000046 ffff88007b00de20 ffff880054bf7fd8 [13551.249333] ffff880054bf7fd8 ffff880054bf7fd8 ffff88007b00de20 ffff880059da7b80 [13551.251057] ffff88007b00de20 ffff880059da7be4 0000000000000000 ffff880059da7ba8 [13551.252826] Call Trace: [13551.254145] [] schedule+0x29/0x70 [13551.255701] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [13551.257319] [] ? wake_up_state+0x20/0x20 [13551.258941] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [13551.260620] [] kthread+0xcf/0xe0 [13551.262158] [] ? kthread_create_on_node+0x140/0x140 [13551.263762] [] ret_from_fork+0x58/0x90 [13551.265328] [] ? kthread_create_on_node+0x140/0x140 [13551.266980] ldlm_bl_02 S ffff880059da7ba8 0 20233 2 0x00000080 [13551.268665] ffff88007bb77e08 0000000000000046 ffff880059a95e20 ffff88007bb77fd8 [13551.270455] ffff88007bb77fd8 ffff88007bb77fd8 ffff880059a95e20 ffff880059da7b80 [13551.272220] ffff880059a95e20 ffff880059da7be4 0000000000000000 ffff880059da7ba8 [13551.273941] Call Trace: [13551.275337] [] schedule+0x29/0x70 [13551.276851] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [13551.278552] [] ? wake_up_state+0x20/0x20 [13551.280095] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [13551.281833] [] kthread+0xcf/0xe0 [13551.283360] [] ? kthread_create_on_node+0x140/0x140 [13551.285017] [] ret_from_fork+0x58/0x90 [13551.286525] [] ? kthread_create_on_node+0x140/0x140 [13551.288184] ldlm_elt S 0000000000000000 0 20234 2 0x00000080 [13551.289857] ffff880062033e00 0000000000000046 ffff880059a93ec0 ffff880062033fd8 [13551.291645] ffff880062033fd8 ffff880062033fd8 ffff880059a93ec0 ffff880059a93ec0 [13551.293421] 0000000000000000 ffffffffa0d38a00 ffff880059a93ec0 0000000000000000 [13551.295182] Call Trace: [13551.296489] [] ? ldlm_bl_to_thread+0x4a0/0x510 [ptlrpc] [13551.298202] [] schedule+0x29/0x70 [13551.299767] [] expired_lock_main+0x1c2/0x910 [ptlrpc] [13551.301389] [] ? wake_up_state+0x20/0x20 [13551.303009] [] ? ldlm_bl_to_thread+0x510/0x510 [ptlrpc] [13551.304702] [] kthread+0xcf/0xe0 [13551.306181] [] ? kthread_create_on_node+0x140/0x140 [13551.307856] [] ret_from_fork+0x58/0x90 [13551.309365] [] ? kthread_create_on_node+0x140/0x140 [13551.311029] ldlm_poold S 000000000000f888 0 20235 2 0x00000080 [13551.312703] ffff8800636abd70 0000000000000046 ffff880059198000 ffff8800636abfd8 [13551.314487] ffff8800636abfd8 ffff8800636abfd8 ffff880059198000 ffffffff81e380c0 [13551.316259] ffff8800636abda0 0000000100ca3653 ffffffff81e380c0 000000000000f888 [13551.317976] Call Trace: [13551.319304] [] schedule+0x29/0x70 [13551.320795] [] schedule_timeout+0x175/0x2d0 [13551.322406] [] ? internal_add_timer+0x70/0x70 [13551.324028] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [13551.325650] [] ldlm_pools_thread_main+0x23b/0x330 [ptlrpc] [13551.327353] [] ? wake_up_state+0x20/0x20 [13551.328905] [] ? ldlm_pools_recalc+0x3d0/0x3d0 [ptlrpc] [13551.330596] [] kthread+0xcf/0xe0 [13551.332062] [] ? kthread_create_on_node+0x140/0x140 [13551.333719] [] ret_from_fork+0x58/0x90 [13551.335223] [] ? kthread_create_on_node+0x140/0x140 [13551.336881] mgs_params_noti S ffff880059198fb0 0 20236 2 0x00000080 [13551.338543] ffff880062e9bdd8 0000000000000046 ffff880059198fb0 ffff880062e9bfd8 [13551.340318] ffff880062e9bfd8 ffff880062e9bfd8 ffff880059198fb0 ffff880059198fb0 [13551.342032] 0000000200000000 ffff88005dcd4920 ffff88005dcd4800 ffff880059198fb0 [13551.343808] Call Trace: [13551.345110] [] schedule+0x29/0x70 [13551.346576] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [13551.348186] [] ? wake_up_state+0x20/0x20 [13551.349760] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [13551.351409] [] kthread+0xcf/0xe0 [13551.352908] [] ? kthread_create_on_node+0x140/0x140 [13551.354478] [] ret_from_fork+0x58/0x90 [13551.356042] [] ? kthread_create_on_node+0x140/0x140 [13551.357660] ll_mgs_0000 S ffff88006a3a2000 0 20238 2 0x00000080 [13551.359353] ffff880055ae7d38 0000000000000046 ffff88005919af10 ffff880055ae7fd8 [13551.361057] ffff880055ae7fd8 ffff880055ae7fd8 ffff88005919af10 0000000000000000 [13551.362806] ffff88005919af10 ffff880059da7700 ffff88006a3a2068 ffff88006a3a2000 [13551.364555] Call Trace: [13551.365814] [] schedule+0x29/0x70 [13551.367369] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.368966] [] ? wake_up_state+0x20/0x20 [13551.370562] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.372160] [] ? __switch_to+0xd9/0x4c0 [13551.373694] [] ? __schedule+0x3b0/0x990 [13551.375184] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.376879] [] kthread+0xcf/0xe0 [13551.378303] [] ? kthread_create_on_node+0x140/0x140 [13551.379918] [] ret_from_fork+0x58/0x90 [13551.381433] [] ? kthread_create_on_node+0x140/0x140 [13551.383035] ll_mgs_0001 S ffff88006a3a2000 0 20239 2 0x00000080 [13551.384665] ffff88007b43fd38 0000000000000046 ffff88005919bec0 ffff88007b43ffd8 [13551.386411] ffff88007b43ffd8 ffff88007b43ffd8 ffff88005919bec0 0000000000000000 [13551.388132] ffff88005919bec0 ffff880066a9a300 ffff88006a3a2068 ffff88006a3a2000 [13551.389803] Call Trace: [13551.391115] [] schedule+0x29/0x70 [13551.392583] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.394216] [] ? wake_up_state+0x20/0x20 [13551.395774] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.397340] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.399043] [] kthread+0xcf/0xe0 [13551.400530] [] ? finish_task_switch+0x53/0x180 [13551.402062] [] ? kthread_create_on_node+0x140/0x140 [13551.403695] [] ret_from_fork+0x58/0x90 [13551.405211] [] ? kthread_create_on_node+0x140/0x140 [13551.406771] ll_mgs_0002 S ffff88006a3a2000 0 20240 2 0x00000080 [13551.408460] ffff88005e5efd38 0000000000000046 ffff88005919ce70 ffff88005e5effd8 [13551.410202] ffff88005e5effd8 ffff88005e5effd8 ffff88005919ce70 0000000000000000 [13551.411896] ffff88005919ce70 ffff880066a9ae00 ffff88006a3a2068 ffff88006a3a2000 [13551.413654] Call Trace: [13551.414900] [] schedule+0x29/0x70 [13551.416443] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.418039] [] ? wake_up_state+0x20/0x20 [13551.419625] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.421243] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.422959] [] kthread+0xcf/0xe0 [13551.424410] [] ? kthread_create_on_node+0x140/0x140 [13551.426070] [] ret_from_fork+0x58/0x90 [13551.427605] [] ? kthread_create_on_node+0x140/0x140 [13551.429173] ll_evictor S 0000000000000000 0 20241 2 0x00000080 [13551.430878] ffff88007a1dbdd0 0000000000000046 ffff88005919de20 ffff88007a1dbfd8 [13551.432650] ffff88007a1dbfd8 ffff88007a1dbfd8 ffff88005919de20 0000000000000000 [13551.434400] ffff88005919de20 ffffffffa0d76c30 0000000000000000 0000000000000000 [13551.436135] Call Trace: [13551.437422] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [13551.439166] [] schedule+0x29/0x70 [13551.440662] [] ping_evictor_main+0x476/0x6c0 [ptlrpc] [13551.442338] [] ? __switch_to+0xd9/0x4c0 [13551.443854] [] ? wake_up_state+0x20/0x20 [13551.445449] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [13551.447125] [] kthread+0xcf/0xe0 [13551.448663] [] ? kthread_create_on_node+0x140/0x140 [13551.450311] [] ret_from_fork+0x58/0x90 [13551.451825] [] ? kthread_create_on_node+0x140/0x140 [13551.453481] ll_cfg_requeue S ffff88005919edd0 0 20242 2 0x00000080 [13551.455159] ffff8800661bfe18 0000000000000046 ffff88005919edd0 ffff8800661bffd8 [13551.456944] ffff8800661bffd8 ffff8800661bffd8 ffff88005919edd0 ffff88005919edd0 [13551.458728] ffff88005919edd0 ffff88005919edd0 ffff8800661bfe68 ffff88005919edd0 [13551.460494] Call Trace: [13551.461775] [] schedule+0x29/0x70 [13551.463326] [] mgc_requeue_thread+0x605/0x880 [mgc] [13551.464928] [] ? wake_up_state+0x20/0x20 [13551.466505] [] ? mgc_process_config+0x1600/0x1600 [mgc] [13551.468173] [] kthread+0xcf/0xe0 [13551.469686] [] ? kthread_create_on_node+0x140/0x140 [13551.471307] [] ret_from_fork+0x58/0x90 [13551.472912] [] ? kthread_create_on_node+0x140/0x140 [13551.474493] mdt00_000 S ffff88007b077800 0 20243 2 0x00000080 [13551.476200] ffff88005e073d38 0000000000000046 ffff880060abedd0 ffff88005e073fd8 [13551.477971] ffff88005e073fd8 ffff88005e073fd8 ffff880060abedd0 0000000000000000 [13551.479747] ffff880060abedd0 ffff880066a9a180 ffff88007b077868 ffff88007b077800 [13551.481517] Call Trace: [13551.482809] [] schedule+0x29/0x70 [13551.484379] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.486034] [] ? wake_up_state+0x20/0x20 [13551.487628] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.489279] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.490939] [] kthread+0xcf/0xe0 [13551.492467] [] ? kthread_create_on_node+0x140/0x140 [13551.494061] [] ret_from_fork+0x58/0x90 [13551.495630] [] ? kthread_create_on_node+0x140/0x140 [13551.497262] mdt00_001 S ffff88007b077800 0 20244 2 0x00000080 [13551.498931] ffff88005eacfd38 0000000000000046 ffff880060abbec0 ffff88005eacffd8 [13551.500710] ffff88005eacffd8 ffff88005eacffd8 ffff880060abbec0 0000000000000000 [13551.502489] ffff880060abbec0 ffff8800684ad680 ffff88007b077868 ffff88007b077800 [13551.504261] Call Trace: [13551.505558] [] schedule+0x29/0x70 [13551.507123] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.508791] [] ? wake_up_state+0x20/0x20 [13551.510384] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.511996] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.513744] [] kthread+0xcf/0xe0 [13551.515256] [] ? kthread_create_on_node+0x140/0x140 [13551.516903] [] ret_from_fork+0x58/0x90 [13551.518449] [] ? kthread_create_on_node+0x140/0x140 [13551.520030] mdt00_002 S ffff88007b077800 0 20245 2 0x00000080 [13551.521742] ffff88006916fd38 0000000000000046 ffff880060abde20 ffff88006916ffd8 [13551.523511] ffff88006916ffd8 ffff88006916ffd8 ffff880060abde20 0000000000000000 [13551.525274] ffff880060abde20 ffff8800684adf00 ffff88007b077868 ffff88007b077800 [13551.526980] Call Trace: [13551.528293] [] schedule+0x29/0x70 [13551.529796] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.531462] [] ? wake_up_state+0x20/0x20 [13551.533003] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.534671] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.536337] [] kthread+0xcf/0xe0 [13551.537879] [] ? kthread_create_on_node+0x140/0x140 [13551.539519] [] ret_from_fork+0x58/0x90 [13551.541085] [] ? kthread_create_on_node+0x140/0x140 [13551.542682] mdt_rdpg00_000 S ffff8800626c8800 0 20246 2 0x00000080 [13551.544451] ffff8800665a7d38 0000000000000046 ffff880060abce70 ffff8800665a7fd8 [13551.546237] ffff8800665a7fd8 ffff8800665a7fd8 ffff880060abce70 0000000000000000 [13551.547963] ffff880060abce70 ffff8800684ad780 ffff8800626c8868 ffff8800626c8800 [13551.549746] Call Trace: [13551.551051] [] schedule+0x29/0x70 [13551.552629] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.554238] [] ? wake_up_state+0x20/0x20 [13551.555858] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.557470] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.559203] [] kthread+0xcf/0xe0 [13551.560682] [] ? kthread_create_on_node+0x140/0x140 [13551.562343] [] ret_from_fork+0x58/0x90 [13551.563856] [] ? kthread_create_on_node+0x140/0x140 [13551.565513] mdt_rdpg00_001 S ffff8800626c8800 0 20247 2 0x00000080 [13551.567184] ffff880047a93d38 0000000000000046 ffff880060abaf10 ffff880047a93fd8 [13551.568964] ffff880047a93fd8 ffff880047a93fd8 ffff880060abaf10 0000000000000000 [13551.570750] ffff880060abaf10 ffff88007b927380 ffff8800626c8868 ffff8800626c8800 [13551.572528] Call Trace: [13551.573828] [] schedule+0x29/0x70 [13551.575409] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.577027] [] ? wake_up_state+0x20/0x20 [13551.578641] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.580296] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.581961] [] kthread+0xcf/0xe0 [13551.583494] [] ? kthread_create_on_node+0x140/0x140 [13551.585099] [] ret_from_fork+0x58/0x90 [13551.586675] [] ? kthread_create_on_node+0x140/0x140 [13551.588264] mdt_attr00_000 S ffff880000611400 0 20248 2 0x00000080 [13551.589980] ffff880067107d38 0000000000000046 ffff880060ab9f60 ffff880067107fd8 [13551.591765] ffff880067107fd8 ffff880067107fd8 ffff880060ab9f60 0000000000000000 [13551.593546] ffff880060ab9f60 ffff88007b927400 ffff880000611468 ffff880000611400 [13551.595305] Call Trace: [13551.596600] [] schedule+0x29/0x70 [13551.598172] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.599833] [] ? wake_up_state+0x20/0x20 [13551.601422] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.603052] [] ? __switch_to+0xd9/0x4c0 [13551.604560] [] ? __schedule+0x3b0/0x990 [13551.606145] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.607800] [] kthread+0xcf/0xe0 [13551.609322] [] ? kthread_create_on_node+0x140/0x140 [13551.610963] [] ret_from_fork+0x58/0x90 [13551.612456] [] ? kthread_create_on_node+0x140/0x140 [13551.614099] mdt_attr00_001 S ffff880000611400 0 20249 2 0x00000080 [13551.615757] ffff880069077d38 0000000000000046 ffff880060ab8000 ffff880069077fd8 [13551.617528] ffff880069077fd8 ffff880069077fd8 ffff880060ab8000 0000000000000000 [13551.619283] ffff880060ab8000 ffff88005e278900 ffff880000611468 ffff880000611400 [13551.620984] Call Trace: [13551.622281] [] schedule+0x29/0x70 [13551.623772] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.625439] [] ? wake_up_state+0x20/0x20 [13551.626972] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.628595] [] ? __switch_to+0xd9/0x4c0 [13551.630090] [] ? __schedule+0x3b0/0x990 [13551.631663] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.633349] [] kthread+0xcf/0xe0 [13551.634797] [] ? kthread_create_on_node+0x140/0x140 [13551.636436] [] ret_from_fork+0x58/0x90 [13551.637968] [] ? kthread_create_on_node+0x140/0x140 [13551.639583] mdt_out00_000 S ffff88005a5ae000 0 20250 2 0x00000080 [13551.641223] ffff880045f13d38 0000000000000046 ffff880060ab8fb0 ffff880045f13fd8 [13551.642980] ffff880045f13fd8 ffff880045f13fd8 ffff880060ab8fb0 0000000000000000 [13551.644730] ffff880060ab8fb0 ffff88005e278c80 ffff88005a5ae068 ffff88005a5ae000 [13551.646488] Call Trace: [13551.647763] [] schedule+0x29/0x70 [13551.649317] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.650969] [] ? wake_up_state+0x20/0x20 [13551.652546] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.654090] [] ? __switch_to+0xd9/0x4c0 [13551.655647] [] ? __schedule+0x3b0/0x990 [13551.657205] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.658896] [] kthread+0xcf/0xe0 [13551.660335] [] ? kthread_create_on_node+0x140/0x140 [13551.661966] [] ret_from_fork+0x58/0x90 [13551.663446] [] ? kthread_create_on_node+0x140/0x140 [13551.665062] mdt_out00_001 S ffff88005a5ae000 0 20251 2 0x00000080 [13551.666700] ffff880060513d38 0000000000000046 ffff88005e8f6dd0 ffff880060513fd8 [13551.668449] ffff880060513fd8 ffff880060513fd8 ffff88005e8f6dd0 0000000000000000 [13551.670191] ffff88005e8f6dd0 ffff88005e278d00 ffff88005a5ae068 ffff88005a5ae000 [13551.671894] Call Trace: [13551.673188] [] schedule+0x29/0x70 [13551.674655] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.676290] [] ? wake_up_state+0x20/0x20 [13551.677852] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.679397] [] ? __switch_to+0xd9/0x4c0 [13551.680954] [] ? __schedule+0x3b0/0x990 [13551.682453] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.684156] [] kthread+0xcf/0xe0 [13551.685591] [] ? kthread_create_on_node+0x140/0x140 [13551.687217] [] ret_from_fork+0x58/0x90 [13551.688740] [] ? kthread_create_on_node+0x140/0x140 [13551.690346] mdt_seqs_0000 S ffff880065cc2c00 0 20252 2 0x00000080 [13551.691972] ffff8800590afd38 0000000000000046 ffff880067b18fb0 ffff8800590affd8 [13551.693719] ffff8800590affd8 ffff8800590affd8 ffff880067b18fb0 0000000000000000 [13551.695460] ffff880067b18fb0 ffff88005e278d80 ffff880065cc2c68 ffff880065cc2c00 [13551.697180] Call Trace: [13551.698425] [] schedule+0x29/0x70 [13551.699958] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.701536] [] ? wake_up_state+0x20/0x20 [13551.703113] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.704662] [] ? __switch_to+0xd9/0x4c0 [13551.706220] [] ? __schedule+0x3b0/0x990 [13551.707777] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.709453] [] kthread+0xcf/0xe0 [13551.710942] [] ? kthread_create_on_node+0x140/0x140 [13551.712555] [] ret_from_fork+0x58/0x90 [13551.714039] [] ? kthread_create_on_node+0x140/0x140 [13551.715661] mdt_seqs_0001 S ffff880065cc2c00 0 20253 2 0x00000080 [13551.717344] ffff88005e617d38 0000000000000046 ffff880067b18000 ffff88005e617fd8 [13551.719034] ffff88005e617fd8 ffff88005e617fd8 ffff880067b18000 0000000000000000 [13551.720771] ffff880067b18000 ffff88005e278f80 ffff880065cc2c68 ffff880065cc2c00 [13551.722500] Call Trace: [13551.723752] [] schedule+0x29/0x70 [13551.725293] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.726924] [] ? wake_up_state+0x20/0x20 [13551.728484] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.730086] [] ? __switch_to+0xd9/0x4c0 [13551.731625] [] ? __schedule+0x3b0/0x990 [13551.733122] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.734814] [] kthread+0xcf/0xe0 [13551.736295] [] ? kthread_create_on_node+0x140/0x140 [13551.737859] [] ret_from_fork+0x58/0x90 [13551.739391] [] ? kthread_create_on_node+0x140/0x140 [13551.740951] mdt_seqm_0000 S ffff880066833800 0 20254 2 0x00000080 [13551.742636] ffff880066523d38 0000000000000046 ffff88005b8ebec0 ffff880066523fd8 [13551.744385] ffff880066523fd8 ffff880066523fd8 ffff88005b8ebec0 0000000000000000 [13551.746118] ffff88005b8ebec0 ffff88005e278b80 ffff880066833868 ffff880066833800 [13551.747796] Call Trace: [13551.749079] [] schedule+0x29/0x70 [13551.750593] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.752221] [] ? wake_up_state+0x20/0x20 [13551.753794] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.755354] [] ? __switch_to+0xd9/0x4c0 [13551.756916] [] ? __schedule+0x3b0/0x990 [13551.758421] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.760115] [] kthread+0xcf/0xe0 [13551.761606] [] ? kthread_create_on_node+0x140/0x140 [13551.763170] [] ret_from_fork+0x58/0x90 [13551.764732] [] ? kthread_create_on_node+0x140/0x140 [13551.766284] mdt_seqm_0001 S ffff880066833800 0 20255 2 0x00000080 [13551.767957] ffff880060bcfd38 0000000000000046 ffff88005b8e9f60 ffff880060bcffd8 [13551.769715] ffff880060bcffd8 ffff880060bcffd8 ffff88005b8e9f60 0000000000000000 [13551.771467] ffff88005b8e9f60 ffff88005e278580 ffff880066833868 ffff880066833800 [13551.773197] Call Trace: [13551.774446] [] schedule+0x29/0x70 [13551.775981] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.777568] [] ? wake_up_state+0x20/0x20 [13551.779145] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.780706] [] ? __switch_to+0xd9/0x4c0 [13551.782267] [] ? __schedule+0x3b0/0x990 [13551.783831] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.785506] [] kthread+0xcf/0xe0 [13551.786947] [] ? kthread_create_on_node+0x140/0x140 [13551.788584] [] ret_from_fork+0x58/0x90 [13551.790109] [] ? kthread_create_on_node+0x140/0x140 [13551.791722] mdt_fld_0000 S ffff880062210400 0 20256 2 0x00000080 [13551.793409] ffff880049397d38 0000000000000046 ffff88005b8e8fb0 ffff880049397fd8 [13551.795154] ffff880049397fd8 ffff880049397fd8 ffff88005b8e8fb0 0000000000000000 [13551.796859] ffff88005b8e8fb0 ffff88005e278300 ffff880062210468 ffff880062210400 [13551.798600] Call Trace: [13551.799851] [] schedule+0x29/0x70 [13551.801392] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.803036] [] ? wake_up_state+0x20/0x20 [13551.804609] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.806219] [] ? __switch_to+0xd9/0x4c0 [13551.807718] [] ? __schedule+0x3b0/0x990 [13551.809292] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.810931] [] kthread+0xcf/0xe0 [13551.812437] [] ? kthread_create_on_node+0x140/0x140 [13551.814054] [] ret_from_fork+0x58/0x90 [13551.815534] [] ? kthread_create_on_node+0x140/0x140 [13551.817153] mdt_fld_0001 S ffff880062210400 0 20257 2 0x00000080 [13551.818783] ffff88005d8cbd38 0000000000000046 ffff88005b8eedd0 ffff88005d8cbfd8 [13551.820528] ffff88005d8cbfd8 ffff88005d8cbfd8 ffff88005b8eedd0 0000000000000000 [13551.822257] ffff88005b8eedd0 ffff88005e278e00 ffff880062210468 ffff880062210400 [13551.823934] Call Trace: [13551.825245] [] schedule+0x29/0x70 [13551.826764] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13551.828341] [] ? wake_up_state+0x20/0x20 [13551.829921] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13551.831513] [] ? __switch_to+0xd9/0x4c0 [13551.833056] [] ? __schedule+0x3b0/0x990 [13551.834608] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13551.836283] [] kthread+0xcf/0xe0 [13551.837725] [] ? kthread_create_on_node+0x140/0x140 [13551.839351] [] ret_from_fork+0x58/0x90 [13551.840880] [] ? kthread_create_on_node+0x140/0x140 [13551.842433] mgs_lustre_noti S ffff88005b8ede20 0 20258 2 0x00000080 [13551.844108] ffff880045bb7dd8 0000000000000046 ffff88005b8ede20 ffff880045bb7fd8 [13551.845848] ffff880045bb7fd8 ffff880045bb7fd8 ffff88005b8ede20 ffff88005b8ede20 [13551.847593] 0000000200000000 ffff88007b4de720 ffff88007b4de600 ffff88005b8ede20 [13551.849304] Call Trace: [13551.850551] [] schedule+0x29/0x70 [13551.852073] [] mgs_ir_notify+0x1c0/0x2b0 [mgs] [13551.853646] [] ? wake_up_state+0x20/0x20 [13551.855173] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [13551.856831] [] kthread+0xcf/0xe0 [13551.858335] [] ? kthread_create_on_node+0x140/0x140 [13551.859946] [] ret_from_fork+0x58/0x90 [13551.861425] [] ? kthread_create_on_node+0x140/0x140 [13551.863050] qmt_reba_lustre S ffff8800635d2b90 0 20262 2 0x00000080 [13551.864687] ffff88005a597df8 0000000000000046 ffff88005b8eaf10 ffff88005a597fd8 [13551.866437] ffff88005a597fd8 ffff88005a597fd8 ffff88005b8eaf10 0000000000000000 [13551.868163] ffff8800635d2a00 ffff8800635d2ba0 ffff88005b8eaf10 ffff8800635d2b90 [13551.869832] Call Trace: [13551.871108] [] schedule+0x29/0x70 [13551.872609] [] qmt_reba_thread+0x9d0/0xda0 [lquota] [13551.874211] [] ? wake_up_state+0x20/0x20 [13551.875749] [] ? qmt_glimpse_lock.isra.19+0x690/0x690 [lquota] [13551.877427] [] kthread+0xcf/0xe0 [13551.878861] [] ? kthread_create_on_node+0x140/0x140 [13551.880481] [] ret_from_fork+0x58/0x90 [13551.882012] [] ? kthread_create_on_node+0x140/0x140 [13551.883626] osp-pre-0-0 S ffff88005b8ece70 0 20263 2 0x00000080 [13551.885265] ffff880053167d78 0000000000000046 ffff88005b8ece70 ffff880053167fd8 [13551.887015] ffff880053167fd8 ffff880053167fd8 ffff88005b8ece70 00000000ffffffff [13551.888758] ffffffff00000000 ffff88005b8ece70 ffff8800661a8800 ffff88005b8ece70 [13551.890491] Call Trace: [13551.891763] [] schedule+0x29/0x70 [13551.893300] [] osp_precreate_thread+0xb64/0x11f0 [osp] [13551.894949] [] ? kvm_clock_read+0x1f/0x30 [13551.896455] [] ? wake_up_state+0x20/0x20 [13551.898033] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [13551.899593] [] kthread+0xcf/0xe0 [13551.901089] [] ? kthread_create_on_node+0x140/0x140 [13551.902702] [] ret_from_fork+0x58/0x90 [13551.904169] [] ? kthread_create_on_node+0x140/0x140 [13551.905796] osp-syn-0-0 S 0000000000000000 0 20264 2 0x00000080 [13551.907444] ffff88005cc979b8 0000000000000046 ffff88005f2f6dd0 ffff88005cc97fd8 [13551.909190] ffff88005cc97fd8 ffff88005cc97fd8 ffff88005f2f6dd0 ffff8800661a8800 [13551.910879] 0000000000000000 ffff8800661a8b80 ffff8800661a8ae8 0000000000000000 [13551.912610] Call Trace: [13551.913871] [] schedule+0x29/0x70 [13551.915402] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [13551.917088] [] ? wake_up_state+0x20/0x20 [13551.918626] [] llog_process_thread+0x5a5/0x1240 [obdclass] [13551.920312] [] ? osp_sync_thread+0x990/0x990 [osp] [13551.921914] [] llog_process_or_fork+0xbc/0x450 [obdclass] [13551.923627] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [13551.925250] [] llog_process_thread+0x5a5/0x1240 [obdclass] [13551.926944] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [13551.928619] [] llog_process_or_fork+0xbc/0x450 [obdclass] [13551.930316] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [13551.931995] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [13551.933724] [] ? default_wake_function+0x12/0x20 [13551.935339] [] ? __wake_up_common+0x58/0x90 [13551.936878] [] ? osp_sync_thread+0x990/0x990 [osp] [13551.938535] [] llog_cat_process+0x2e/0x30 [obdclass] [13551.940123] [] osp_sync_thread+0x1ba/0x990 [osp] [13551.941736] [] ? __switch_to+0xd9/0x4c0 [13551.943277] [] ? osp_sync_add_rec.isra.16+0xa20/0xa20 [osp] [13551.944974] [] kthread+0xcf/0xe0 [13551.946423] [] ? kthread_create_on_node+0x140/0x140 [13551.948066] [] ret_from_fork+0x58/0x90 [13551.949554] [] ? kthread_create_on_node+0x140/0x140 [13551.951187] osp-pre-1-0 S ffff88005f2f5e20 0 20265 2 0x00000080 [13551.952851] ffff88005fa0fd78 0000000000000046 ffff88005f2f5e20 ffff88005fa0ffd8 [13551.954614] ffff88005fa0ffd8 ffff88005fa0ffd8 ffff88005f2f5e20 00000000ffffffff [13551.956364] ffffffff00000000 ffff88005f2f5e20 ffff88005fb63000 ffff88005f2f5e20 [13551.958057] Call Trace: [13551.959356] [] schedule+0x29/0x70 [13551.960866] [] osp_precreate_thread+0xb64/0x11f0 [osp] [13551.962452] [] ? kvm_clock_read+0x1f/0x30 [13551.964033] [] ? wake_up_state+0x20/0x20 [13551.965585] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [13551.967187] [] kthread+0xcf/0xe0 [13551.968616] [] ? kthread_create_on_node+0x140/0x140 [13551.970231] [] ret_from_fork+0x58/0x90 [13551.971754] [] ? kthread_create_on_node+0x140/0x140 [13551.973316] osp-syn-1-0 S 0000000000000000 0 20266 2 0x00000080 [13551.975008] ffff88007a7979b8 0000000000000046 ffff88005f2f3ec0 ffff88007a797fd8 [13551.976760] ffff88007a797fd8 ffff88007a797fd8 ffff88005f2f3ec0 ffff88005fb63000 [13551.978504] 0000000000000000 0000000000000001 ffff88005fb632e8 0000000000000000 [13551.980236] Call Trace: [13551.981497] [] schedule+0x29/0x70 [13551.983032] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [13551.984709] [] ? libcfs_debug_vmsg2+0x708/0xb70 [libcfs] [13551.986366] [] ? wake_up_state+0x20/0x20 [13551.987935] [] llog_process_thread+0x5a5/0x1240 [obdclass] [13551.989561] [] ? osp_sync_thread+0x990/0x990 [osp] [13551.991200] [] llog_process_or_fork+0xbc/0x450 [obdclass] [13551.992837] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [13551.994524] [] llog_process_thread+0x5a5/0x1240 [obdclass] [13551.996191] [] ? select_task_rq_fair+0x584/0x720 [13551.997829] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [13551.999485] [] llog_process_or_fork+0xbc/0x450 [obdclass] [13552.001177] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [13552.002903] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [13552.004610] [] ? default_wake_function+0x12/0x20 [13552.006165] [] ? __wake_up_common+0x58/0x90 [13552.007756] [] ? osp_sync_thread+0x990/0x990 [osp] [13552.009330] [] llog_cat_process+0x2e/0x30 [obdclass] [13552.010977] [] osp_sync_thread+0x1ba/0x990 [osp] [13552.012525] [] ? __switch_to+0xd9/0x4c0 [13552.014077] [] ? osp_sync_add_rec.isra.16+0xa20/0xa20 [osp] [13552.015703] [] kthread+0xcf/0xe0 [13552.017209] [] ? kthread_create_on_node+0x140/0x140 [13552.018840] [] ret_from_fork+0x58/0x90 [13552.020369] [] ? kthread_create_on_node+0x140/0x140 [13552.021990] tx_commit_cb S ffff88007ab4e100 0 20270 2 0x00000080 [13552.023656] ffff880067597e08 0000000000000046 ffff88005f2f4e70 ffff880067597fd8 [13552.025420] ffff880067597fd8 ffff880067597fd8 ffff88005f2f4e70 ffff8800683f3840 [13552.027170] ffff88007ab4e190 ffff880048d4a070 ffff88007ab4e1b0 ffff88007ab4e100 [13552.028856] Call Trace: [13552.030137] [] schedule+0x29/0x70 [13552.031595] [] taskq_thread+0x3fa/0x440 [spl] [13552.033190] [] ? wake_up_state+0x20/0x20 [13552.034730] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13552.036334] [] kthread+0xcf/0xe0 [13552.037765] [] ? kthread_create_on_node+0x140/0x140 [13552.039387] [] ret_from_fork+0x58/0x90 [13552.040860] [] ? kthread_create_on_node+0x140/0x140 [13552.042485] tx_commit_cb S ffff88007ab4e100 0 20271 2 0x00000080 [13552.044126] ffff88005b0afe08 0000000000000046 ffff88005f2f0000 ffff88005b0affd8 [13552.045877] ffff88005b0affd8 ffff88005b0affd8 ffff88005f2f0000 ffff8800683f3ba0 [13552.047634] ffff88007ab4e190 ffff880048d4a130 ffff88007ab4e1b0 ffff88007ab4e100 [13552.049375] Call Trace: [13552.050637] [] schedule+0x29/0x70 [13552.052152] [] taskq_thread+0x3fa/0x440 [spl] [13552.053684] [] ? wake_up_state+0x20/0x20 [13552.055241] [] ? taskq_thread_spawn+0x60/0x60 [spl] [13552.056797] [] kthread+0xcf/0xe0 [13552.058286] [] ? kthread_create_on_node+0x140/0x140 [13552.059840] [] ret_from_fork+0x58/0x90 [13552.061359] [] ? kthread_create_on_node+0x140/0x140 [13552.062911] lquota_wb_lustr S 000000000000f888 0 20274 2 0x00000080 [13552.064603] ffff8800496ebd30 0000000000000046 0000000000000000 0000000000000286 [13552.066332] ffff88005b0ba000 ffff8800496ebfd8 ffff88005f2f1f60 ffffffff81e380c0 [13552.068007] ffff8800496ebd60 0000000100cb2044 ffffffff81e380c0 000000000000f888 [13552.069757] Call Trace: [13552.071046] [] schedule+0x29/0x70 [13552.072551] [] schedule_timeout+0x175/0x2d0 [13552.074106] [] qsd_update_index+0x2e8/0x570 [lquota] [13552.075684] [] ? qsd_upd_thread+0x9e4/0xd30 [lquota] [13552.077293] [] ? wake_up_state+0x20/0x20 [13552.078815] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [13552.080370] [] ? kthread+0xcf/0xe0 [13552.081847] [] ? kthread_create_on_node+0x140/0x140 [13552.083375] [] ? ret_from_fork+0x58/0x90 [13552.084903] [] ? kthread_create_on_node+0x140/0x140 [13552.086482] tdtd-0 S ffff88005b8e8000 0 20275 2 0x00000080 [13552.088092] ffff88007b5f7d70 0000000000000046 ffff88005b8e8000 ffff88007b5f7fd8 [13552.089810] ffff88007b5f7fd8 ffff88007b5f7fd8 ffff88005b8e8000 ffff88005e4ae580 [13552.091522] 0000000000000000 ffff88007b5f7e30 ffff88005e4ae540 ffff88005b8e8000 [13552.093236] Call Trace: [13552.094462] [] schedule+0x29/0x70 [13552.096012] [] distribute_txn_commit_thread+0xec1/0x1410 [ptlrpc] [13552.097704] [] ? kvm_clock_read+0x1f/0x30 [13552.099176] [] ? set_next_entity+0x3c/0xe0 [13552.100723] [] ? wake_up_state+0x20/0x20 [13552.102264] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [13552.103826] [] kthread+0xcf/0xe0 [13552.105292] [] ? kthread_create_on_node+0x140/0x140 [13552.106883] [] ret_from_fork+0x58/0x90 [13552.108382] [] ? kthread_create_on_node+0x140/0x140 [13552.109923] kworker/0:3 S ffff88007fc16480 0 20907 2 0x00000080 [13552.111603] ffff88005a58be50 0000000000000046 ffff88006726edd0 ffff88005a58bfd8 [13552.113324] ffff88005a58bfd8 ffff88005a58bfd8 ffff88006726edd0 ffff88007fc164a0 [13552.114998] ffff88005e0530b0 ffff88006726edd0 ffff88005e053080 ffff88007fc16480 [13552.116744] Call Trace: [13552.117972] [] schedule+0x29/0x70 [13552.119457] [] worker_thread+0x1d9/0x410 [13552.120970] [] ? rescuer_thread+0x460/0x460 [13552.122505] [] kthread+0xcf/0xe0 [13552.123957] [] ? kthread_create_on_node+0x140/0x140 [13552.125470] [] ret_from_fork+0x58/0x90 [13552.126972] [] ? kthread_create_on_node+0x140/0x140 [13552.128493] mdt00_003 S ffff88007b077800 0 20983 2 0x00000080 [13552.130144] ffff880059a23d38 0000000000000046 ffff880067268fb0 ffff880059a23fd8 [13552.131856] ffff880059a23fd8 ffff880059a23fd8 ffff880067268fb0 0000000000000000 [13552.133580] ffff880067268fb0 ffff88007ab55100 ffff88007b077868 ffff88007b077800 [13552.135278] Call Trace: [13552.136516] [] schedule+0x29/0x70 [13552.138037] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13552.139648] [] ? wake_up_state+0x20/0x20 [13552.141133] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13552.142741] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13552.144338] [] kthread+0xcf/0xe0 [13552.145821] [] ? kthread_create_on_node+0x140/0x140 [13552.147403] [] ret_from_fork+0x58/0x90 [13552.148857] [] ? kthread_create_on_node+0x140/0x140 [13552.150459] ldlm_cn00_003 S ffff8800604da400 0 24141 2 0x00000080 [13552.152086] ffff88005a5d7d38 0000000000000046 ffff88006726bec0 ffff88005a5d7fd8 [13552.153819] ffff88005a5d7fd8 ffff88005a5d7fd8 ffff88006726bec0 0000000000000000 [13552.155548] ffff88006726bec0 ffff8800616b1500 ffff8800604da468 ffff8800604da400 [13552.157302] Call Trace: [13552.158584] [] schedule+0x29/0x70 [13552.160140] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13552.161779] [] ? wake_up_state+0x20/0x20 [13552.163336] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [13552.164885] [] ? __switch_to+0xd9/0x4c0 [13552.166434] [] ? __schedule+0x3b0/0x990 [13552.167935] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13552.169626] [] kthread+0xcf/0xe0 [13552.171097] [] ? kthread_create_on_node+0x140/0x140 [13552.172706] [] ret_from_fork+0x58/0x90 [13552.174215] [] ? kthread_create_on_node+0x140/0x140 [13552.175819] mdt_rdpg00_002 S ffff8800626c8800 0 25538 2 0x00000080 [13552.177455] ffff88004a933d38 000001237c33ced5 ffff88005eaf6540 ffff8800626c8800 [13552.179192] ffff88004a933fd8 0000000000000259 ffff88006227ed00 ffff880045e90300 [13552.180879] ffff88004a933de0 ffff88006227ed00 00058e5174744110 ffff8800626c8800 [13552.182628] Call Trace: [13552.183916] [] ? tgt_request_handle+0x915/0x1320 [ptlrpc] [13552.185605] [] ? schedule+0x29/0x70 [13552.187140] [] ? ptlrpc_wait_event+0x325/0x340 [ptlrpc] [13552.188805] [] ? wake_up_state+0x20/0x20 [13552.190386] [] ? ptlrpc_main+0xaa0/0x1de0 [ptlrpc] [13552.192038] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [13552.193683] [] ? kthread+0xcf/0xe0 [13552.195214] [] ? kthread_create_on_node+0x140/0x140 [13552.196851] [] ? ret_from_fork+0x58/0x90 [13552.198368] [] ? kthread_create_on_node+0x140/0x140 [13552.200027] kworker/1:0 S ffff88007fd16480 0 25691 2 0x00000080 [13552.201696] ffff880066aabe50 0000000000000046 ffff88005f62bec0 ffff880066aabfd8 [13552.203488] ffff880066aabfd8 ffff880066aabfd8 ffff88005f62bec0 ffff88007fd164a0 [13552.205244] ffff88007a1c0330 ffff88005f62bec0 ffff88007a1c0300 ffff88007fd16480 [13552.206945] Call Trace: [13552.208253] [] schedule+0x29/0x70 [13552.209776] [] worker_thread+0x1d9/0x410 [13552.211318] [] ? rescuer_thread+0x460/0x460 [13552.212838] [] kthread+0xcf/0xe0 [13552.214339] [] ? kthread_create_on_node+0x140/0x140 [13552.215905] [] ret_from_fork+0x58/0x90 [13552.217440] [] ? kthread_create_on_node+0x140/0x140 [13552.218999] pickup S ffff88005eaafea8 0 25765 1076 0x00000080 [13552.220695] ffff88005eaafdb8 0000000000000082 ffff88005f62edd0 ffff88005eaaffd8 [13552.222446] ffff88005eaaffd8 ffff88005eaaffd8 ffff88005f62edd0 ffff88005eaafea8 [13552.224179] 0000000005f5e0ff 0000000000000000 ffff88005f62edd0 ffff88005eaafea8 [13552.225862] Call Trace: [13552.227140] [] schedule+0x29/0x70 [13552.228619] [] schedule_hrtimeout_range_clock+0xb2/0x150 [13552.230253] [] ? hrtimer_get_res+0x50/0x50 [13552.231802] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [13552.233449] [] schedule_hrtimeout_range+0x13/0x20 [13552.234990] [] ep_poll+0x23e/0x360 [13552.236497] [] ? wake_up_state+0x20/0x20 [13552.238027] [] SyS_epoll_wait+0xed/0x120 [13552.239490] [] system_call_fastpath+0x16/0x1b [13552.241050] sshd S 0000000000000011 0 25854 921 0x00000080 [13552.242675] ffff88006a60f8e8 0000000000000082 ffff88005f628fb0 ffff88006a60ffd8 [13552.244406] ffff88006a60ffd8 ffff88006a60ffd8 ffff88005f628fb0 0000000000000000 [13552.246118] ffff88006a60fad0 00000000fffffffc 0000000000000011 0000000000000011 [13552.247782] Call Trace: [13552.249052] [] schedule+0x29/0x70 [13552.250531] [] schedule_hrtimeout_range_clock+0x12d/0x150 [13552.252161] [] ? add_wait_queue+0x3c/0x50 [13552.253631] [] ? add_wait_queue+0x3c/0x50 [13552.255151] [] schedule_hrtimeout_range+0x13/0x20 [13552.256718] [] poll_schedule_timeout+0x55/0xb0 [13552.258191] [] do_select+0x6d1/0x7c0 [13552.259670] [] ? ip_finish_output+0x268/0x750 [13552.261120] [] ? ip_output+0x73/0xe0 [13552.262571] [] ? poll_select_copy_remaining+0x150/0x150 [13552.264081] [] ? poll_select_copy_remaining+0x150/0x150 [13552.265650] [] ? poll_select_copy_remaining+0x150/0x150 [13552.267130] [] ? poll_select_copy_remaining+0x150/0x150 [13552.268657] [] ? _raw_spin_unlock_bh+0x1b/0x40 [13552.270104] [] ? release_sock+0x118/0x170 [13552.271471] [] ? tcp_sendmsg+0xd6/0xc40 [13552.272884] [] ? inet_sendmsg+0x64/0xb0 [13552.274266] [] core_sys_select+0x1db/0x300 [13552.275652] [] ? fsnotify+0x316/0x4b0 [13552.277015] [] ? do_sync_write+0x8d/0xd0 [13552.278397] [] SyS_select+0xba/0x110 [13552.279740] [] system_call_fastpath+0x16/0x1b [13552.281141] bash R running task 0 25856 25854 0x00000088 [13552.282618] ffff88005f62de20 00000000714b2521 ffff880062d23e40 ffffffff810c43a8 [13552.284226] ffff88005f62de20 0000000000000000 ffff880062d23e78 ffffffff810c4492 [13552.285763] ffff88005f62e250 00000000714b2521 ffffffff81a7be60 0000000000000074 [13552.287368] Call Trace: [13552.288478] [] sched_show_task+0xa8/0x110 [13552.289916] [] show_state_filter+0x82/0xd0 [13552.291325] [] sysrq_handle_showstate+0x10/0x20 [13552.292747] [] __handle_sysrq+0x107/0x170 [13552.294081] [] write_sysrq_trigger+0x2f/0x40 [13552.295508] [] proc_reg_write+0x3d/0x80 [13552.296865] [] vfs_write+0xbd/0x1e0 [13552.298189] [] SyS_write+0x7f/0xe0 [13552.299457] [] system_call_fastpath+0x16/0x1b [13552.300862] kworker/1:1 S ffff88007fd16480 0 25900 2 0x00000080 [13552.302340] ffff88006681fe50 0000000000000046 ffff88005f62ce70 ffff88006681ffd8 [13552.303860] ffff88006681ffd8 ffff88006681ffd8 ffff88005f62ce70 ffff88007fd164a0 [13552.305432] ffff88005f30c930 ffff88005f62ce70 ffff88005f30c900 ffff88007fd16480 [13552.306966] Call Trace: [13552.308053] [] schedule+0x29/0x70 [13552.309388] [] worker_thread+0x1d9/0x410 [13552.310747] [] ? rescuer_thread+0x460/0x460 [13552.312107] [] kthread+0xcf/0xe0 [13552.313396] [] ? kthread_create_on_node+0x140/0x140 [13552.314825] [] ret_from_fork+0x58/0x90 [13552.316146] [] ? kthread_create_on_node+0x140/0x140 [13552.317566] Sched Debug Version: v0.11, 3.10.0-514.10.2.el7_lustre.x86_64 #1 [13552.318971] ktime : 13552317.561954 [13552.320406] sched_clk : 16550725.421896 [13552.321788] cpu_clk : 13552317.562127 [13552.323218] jiffies : 4308219620 [13552.324549] sched_clock_stable() : 0 [13552.325878] [13552.326880] sysctl_sched [13552.327970] .sysctl_sched_latency : 12.000000 [13552.329288] .sysctl_sched_min_granularity : 10.000000 [13552.330693] .sysctl_sched_wakeup_granularity : 15.000000 [13552.332031] .sysctl_sched_child_runs_first : 0 [13552.333391] .sysctl_sched_features : 77435 [13552.334760] .sysctl_sched_tunable_scaling : 1 (logaritmic) [13552.336176] [13552.337161] cpu#0, 2593.768 MHz [13552.338316] .nr_running : 3 [13552.339576] .load : 1024 [13552.340803] .nr_switches : 34664855 [13552.342120] .nr_load_updates : 3591731 [13552.343408] .nr_uninterruptible : 333 [13552.344669] .next_balance : 4308.219684 [13552.345913] .curr->pid : 25538 [13552.347161] .clock : 13552347.145533 [13552.348419] .cpu_load[0] : 1024 [13552.349663] .cpu_load[1] : 1791 [13552.350866] .cpu_load[2] : 2159 [13552.352050] .cpu_load[3] : 1942 [13552.353179] .cpu_load[4] : 2005 [13552.354358] .avg_idle : 65334 [13552.355526] .max_idle_balance_cost : 500000 [13552.356699] [13552.356699] cfs_rq[0]:/ [13552.357668] .exec_clock : 0.000000 [13552.357668] .MIN_vruntime : 5837845.372694 [13552.357668] .min_vruntime : 5837851.372694 [13552.357668] .max_vruntime : 5837845.372694 [13552.357668] .spread : 0.000000 [13552.357668] .spread0 : 0.000000 [13552.357668] .nr_spread_over : 0 [13552.357668] .nr_running : 1 [13552.357668] .load : 2048 [13552.357668] .runnable_load_avg : 222 [13552.357668] .blocked_load_avg : 1265 [13552.357668] .tg_load_avg : 0 [13552.357668] .tg_load_contrib : 0 [13552.357668] .tg_runnable_contrib : 0 [13552.357668] .tg->runnable_avg : 0 [13552.357668] .tg->cfs_bandwidth.timer_active: 0 [13552.357668] .throttled : 0 [13552.357668] .throttle_count : 0 [13552.357668] .avg->runnable_avg_sum : 46899 [13552.357668] .avg->runnable_avg_period : 46726 [13552.357668] [13552.357668] rt_rq[0]:/ [13552.357668] .rt_nr_running : 0 [13552.357668] .rt_throttled : 0 [13552.357668] .rt_time : 0.013314 [13552.357668] .rt_runtime : 950.000000 [13552.357668] [13552.357668] runnable tasks: [13552.357668] task PID tree-key switches prio wait-time sum-exec sum-sleep [13552.357668] ---------------------------------------------------------------------------------------------------------- [13552.357668] systemd 1 5831612.108256 204060 120 0.000000 33707.066087 0.000000 0 / [13552.357668] ksoftirqd/0 3 5831611.319909 39175 120 0.000000 590.234700 0.000000 0 / [13552.357668] kworker/0:0H 5 2626.095822 7 100 0.000000 0.202784 0.000000 0 / [13552.357668] migration/0 7 0.000000 81953 0 0.000000 852.435716 0.000000 0 / [13552.357668] rcu_bh 8 27.949543 2 120 0.000000 0.001733 0.000000 0 / [13552.357668] rcu_sched 9 5837843.582990 1051876 120 0.000000 6695.436897 0.000000 0 / [13552.357668] watchdog/0 10 0.000000 3391 0 0.000000 50.199682 0.000000 0 / [13552.357668] kdevtmpfs 17 21102.180859 139 120 0.000000 5.957437 0.000000 0 / [13552.357668] khugepaged 33 5837727.837323 1358 139 0.000000 34.998039 0.000000 0 / [13552.357668] fsnotify_mark 34 5831611.307845 48 120 0.000000 0.274196 0.000000 0 / [13552.357668] ipv6_addrconf 48 366.560939 2 100 0.000000 0.028870 0.000000 0 / [13552.357668] kauditd 102 5831231.526870 43820 120 0.000000 501.120541 0.000000 0 / [13552.357668] jbd2/vda1-8 273 5836148.168736 7073 120 0.000000 273.999468 0.000000 0 / [13552.357668] systemd-journal 356 5837844.005036 111413 120 0.000000 31908.799143 0.000000 0 / [13552.357668] lvmetad 377 11616.122228 17 120 0.000000 2.580891 0.000000 0 / [13552.357668] systemd-udevd 383 5833039.800281 727 120 0.000000 67.052594 0.000000 0 / [13552.357668] vballoon 401 1744.390323 2 120 0.000000 0.049792 0.000000 0 / [13552.357668] kworker/u5:0 477 2298.289566 4 100 0.000000 0.014938 0.000000 0 / [13552.357668] ib_addr 480 2015.509627 2 100 0.000000 0.003387 0.000000 0 / [13552.357668] ib_mcast 481 2019.510327 2 100 0.000000 0.001875 0.000000 0 / [13552.357668] ib_nl_sa_wq 482 2023.512441 2 100 0.000000 0.003650 0.000000 0 / [13552.357668] auditd 505 5582890.644672 2930 116 0.000000 66.523552 0.000000 0 / [13552.357668] dbus-daemon 538 5831611.508441 333991 120 0.000000 32727.539553 0.000000 0 / [13552.357668] gssproxy 541 5831581.685617 529 120 0.000000 15.531568 0.000000 0 / [13552.357668] gssproxy 546 2146.802945 1 120 0.000000 0.006654 0.000000 0 / [13552.357668] NetworkManager 551 5831219.828449 91995 120 0.000000 4291.368876 0.000000 0 / [13552.357668] gdbus 579 5831219.724268 30460 120 0.000000 3480.164556 0.000000 0 / [13552.357668] irqbalance 553 5837299.275457 1371 120 0.000000 280.993841 0.000000 0 / [13552.357668] sssd_be 564 5834128.350362 3184 120 0.000000 215.593819 0.000000 0 / [13552.357668] rpc.gssd 570 5831611.317386 47 120 0.000000 2.591922 0.000000 0 / [13552.357668] sssd_nss 599 5834014.987520 87111 120 0.000000 3478.075903 0.000000 0 / [13552.357668] sssd_pam 600 5834151.472553 2723 120 0.000000 66.952256 0.000000 0 / [13552.357668] sssd_ssh 601 5834151.469073 2734 120 0.000000 74.991117 0.000000 0 / [13552.357668] sssd_pac 602 5834203.077103 2742 120 0.000000 80.974863 0.000000 0 / [13552.357668] srp_remove 604 2205.876329 2 100 0.000000 0.006610 0.000000 0 / [13552.357668] target_completi 612 2227.711807 2 100 0.000000 0.005098 0.000000 0 / [13552.357668] tmr-rd_mcp 613 2231.714739 2 100 0.000000 0.003598 0.000000 0 / [13552.357668] xcopy_wq 615 2233.719117 2 100 0.000000 0.005375 0.000000 0 / [13552.357668] dhclient 630 5494798.775266 79 120 0.000000 25.443672 0.000000 0 / [13552.357668] systemd-logind 631 5831611.363223 195240 120 0.000000 17469.750996 0.000000 0 / [13552.357668] iscsi_eh 632 2259.624371 2 100 0.000000 0.012309 0.000000 0 / [13552.357668] kworker/0:1H 704 5835172.934638 770 100 0.000000 9.504595 0.000000 0 / [13552.357668] rsyslogd 899 3467.739697 34 120 0.000000 5.363510 0.000000 0 / [13552.357668] in:imjournal 914 5837844.005971 105087 120 0.000000 9045.332310 0.000000 0 / [13552.357668] rs:main Q:Reg 915 5837844.036493 104821 120 0.000000 11142.657069 0.000000 0 / [13552.357668] tuned 901 5582925.559051 475 120 0.000000 140.080684 0.000000 0 / [13552.357668] tuned 969 3724.904542 2 120 0.000000 1.067931 0.000000 0 / [13552.357668] tuned 971 5837466.724071 13574 120 0.000000 1094.818335 0.000000 0 / [13552.357668] munged 918 5830150.963557 254 120 0.000000 58.284455 0.000000 0 / [13552.357668] munged 920 5494499.386120 4660 120 0.000000 300.341887 0.000000 0 / [13552.357668] nfsiod 925 3485.347715 2 100 0.000000 0.018495 0.000000 0 / [13552.357668] crond 1033 5830150.777039 237 120 0.000000 15.722447 0.000000 0 / [13552.357668] automount 1038 5837799.466941 722 120 0.000000 24.398545 0.000000 0 / [13552.357668] automount 1039 5837809.763690 2178 120 0.000000 49.237029 0.000000 0 / [13552.357668] automount 1077 4186.728067 6 120 0.000000 1.471618 0.000000 0 / [13552.357668] automount 1109 5831496.630674 7 120 0.000000 0.680988 0.000000 0 / [13552.357668] master 1076 5830150.922581 562 120 0.000000 40.525252 0.000000 0 / [13552.357668] agetty 1080 8614.343094 6 120 0.000000 3.828076 0.000000 0 / [13552.357668] agetty 1081 8615.304146 4 120 0.000000 4.783117 0.000000 0 / [13552.357668] qmgr 1104 5830150.830787 90 120 0.000000 10.397705 0.000000 0 / [13552.357668] kdmflush 2532 10390.745203 2 100 0.000000 0.034514 0.000000 0 / [13552.357668] kdmflush 2631 10822.667810 2 100 0.000000 0.017787 0.000000 0 / [13552.357668] bioset 2633 10823.238755 2 100 0.000000 0.004825 0.000000 0 / [13552.357668] spl_kmem_cache 3868 1516782.929451 15 100 0.000000 0.612182 0.000000 0 / [13552.357668] spl_kmem_cache 3869 1571492.803623 15 100 0.000000 0.752715 0.000000 0 / [13552.357668] spl_kmem_cache 3870 1571690.932340 15 100 0.000000 0.644666 0.000000 0 / [13552.357668] spl_kmem_cache 3871 5362973.676189 15 100 0.000000 0.662688 0.000000 0 / [13552.357668] spl_system_task 3872 16443.294387 2 100 0.000000 0.018082 0.000000 0 / [13552.357668] spl_system_task 3873 16449.310470 2 100 0.000000 0.018285 0.000000 0 / [13552.357668] spl_system_task 3874 16455.325956 2 100 0.000000 0.017561 0.000000 0 / [13552.357668] spl_system_task 3875 16461.342229 2 100 0.000000 0.018220 0.000000 0 / [13552.357668] spl_system_task 3876 16467.358276 2 100 0.000000 0.017899 0.000000 0 / [13552.357668] spl_system_task 3877 16473.373728 2 100 0.000000 0.017340 0.000000 0 / [13552.357668] spl_system_task 3878 16479.389787 2 100 0.000000 0.018206 0.000000 0 / [13552.357668] spl_system_task 3879 16485.405606 2 100 0.000000 0.018143 0.000000 0 / [13552.357668] spl_system_task 3880 16491.421503 2 100 0.000000 0.018099 0.000000 0 / [13552.357668] spl_system_task 3881 16497.437442 2 100 0.000000 0.017961 0.000000 0 / [13552.357668] spl_system_task 3882 16503.453442 2 100 0.000000 0.018081 0.000000 0 / [13552.357668] spl_system_task 3883 16509.469248 2 100 0.000000 0.017873 0.000000 0 / [13552.357668] spl_system_task 3884 16515.485143 2 100 0.000000 0.017843 0.000000 0 / [13552.357668] spl_system_task 3885 16521.501087 2 100 0.000000 0.018126 0.000000 0 / [13552.357668] spl_system_task 3886 16527.517818 2 100 0.000000 0.019086 0.000000 0 / [13552.357668] spl_system_task 3887 16533.533810 2 100 0.000000 0.018175 0.000000 0 / [13552.357668] spl_system_task 3888 16539.546882 2 100 0.000000 0.014815 0.000000 0 / [13552.357668] spl_system_task 3889 16545.562890 2 100 0.000000 0.018032 0.000000 0 / [13552.357668] spl_system_task 3890 16551.578633 2 100 0.000000 0.018141 0.000000 0 / [13552.357668] spl_system_task 3891 16557.594535 2 100 0.000000 0.017853 0.000000 0 / [13552.357668] spl_system_task 3892 16563.610365 2 100 0.000000 0.017805 0.000000 0 / [13552.357668] spl_system_task 3893 16569.626065 2 100 0.000000 0.018196 0.000000 0 / [13552.357668] spl_system_task 3894 16575.641668 2 100 0.000000 0.017614 0.000000 0 / [13552.357668] spl_system_task 3895 16581.657263 2 100 0.000000 0.017800 0.000000 0 / [13552.357668] spl_system_task 3896 16587.672946 2 100 0.000000 0.017847 0.000000 0 / [13552.357668] spl_system_task 3897 16593.688884 2 100 0.000000 0.018056 0.000000 0 / [13552.357668] spl_system_task 3898 16599.704907 2 100 0.000000 0.018082 0.000000 0 / [13552.357668] spl_system_task 3899 16605.720948 2 100 0.000000 0.017853 0.000000 0 / [13552.357668] spl_system_task 3900 16611.736391 2 100 0.000000 0.017436 0.000000 0 / [13552.357668] spl_system_task 3901 16617.751978 2 100 0.000000 0.017506 0.000000 0 / [13552.357668] spl_system_task 3902 16623.767583 2 100 0.000000 0.017972 0.000000 0 / [13552.357668] spl_system_task 3903 16629.782766 2 100 0.000000 0.017375 0.000000 0 / [13552.357668] spl_system_task 3904 16635.798505 2 100 0.000000 0.017854 0.000000 0 / [13552.357668] spl_system_task 3905 16641.813803 2 100 0.000000 0.017415 0.000000 0 / [13552.357668] spl_system_task 3906 16647.829933 2 100 0.000000 0.017988 0.000000 0 / [13552.357668] spl_system_task 3907 16653.845199 2 100 0.000000 0.017468 0.000000 0 / [13552.357668] spl_system_task 3908 16659.860983 2 100 0.000000 0.018037 0.000000 0 / [13552.357668] spl_system_task 3909 16665.876725 2 100 0.000000 0.017713 0.000000 0 / [13552.357668] spl_system_task 3910 16671.892602 2 100 0.000000 0.017986 0.000000 0 / [13552.357668] spl_system_task 3911 16677.909423 2 100 0.000000 0.018785 0.000000 0 / [13552.357668] spl_system_task 3912 16683.925185 2 100 0.000000 0.017720 0.000000 0 / [13552.357668] spl_system_task 3913 16689.940902 2 100 0.000000 0.017894 0.000000 0 / [13552.357668] spl_system_task 3914 16695.956394 2 100 0.000000 0.017583 0.000000 0 / [13552.357668] spl_system_task 3915 16701.971996 2 100 0.000000 0.017683 0.000000 0 / [13552.357668] spl_system_task 3916 16707.987465 2 100 0.000000 0.017529 0.000000 0 / [13552.357668] spl_system_task 3917 16714.003220 2 100 0.000000 0.017520 0.000000 0 / [13552.357668] spl_system_task 3918 16720.018902 2 100 0.000000 0.017953 0.000000 0 / [13552.357668] spl_system_task 3919 16726.034304 2 100 0.000000 0.017326 0.000000 0 / [13552.357668] spl_system_task 3920 16732.049832 2 100 0.000000 0.017448 0.000000 0 / [13552.357668] spl_system_task 3921 16738.065592 2 100 0.000000 0.017838 0.000000 0 / [13552.357668] spl_system_task 3922 16744.081111 2 100 0.000000 0.017472 0.000000 0 / [13552.357668] spl_system_task 3923 16750.096902 2 100 0.000000 0.017997 0.000000 0 / [13552.357668] spl_system_task 3924 16756.112509 2 100 0.000000 0.017954 0.000000 0 / [13552.357668] spl_system_task 3925 16762.128111 2 100 0.000000 0.017672 0.000000 0 / [13552.357668] spl_system_task 3926 16768.143910 2 100 0.000000 0.017700 0.000000 0 / [13552.357668] spl_system_task 3927 16774.159852 2 100 0.000000 0.017896 0.000000 0 / [13552.357668] spl_system_task 3928 16780.175680 2 100 0.000000 0.017811 0.000000 0 / [13552.357668] spl_system_task 3929 16786.191129 2 100 0.000000 0.017365 0.000000 0 / [13552.357668] spl_system_task 3930 16792.206744 2 100 0.000000 0.017590 0.000000 0 / [13552.357668] spl_system_task 3931 16798.222391 2 100 0.000000 0.017791 0.000000 0 / [13552.357668] spl_system_task 3932 16804.237983 2 100 0.000000 0.017557 0.000000 0 / [13552.357668] spl_system_task 3933 16810.253818 2 100 0.000000 0.017910 0.000000 0 / [13552.357668] spl_system_task 3935 16810.607962 2 100 0.000000 0.017369 0.000000 0 / [13552.357668] spl_dynamic_tas 3936 16816.623616 2 100 0.000000 0.016969 0.000000 0 / [13552.357668] dbu_evict 3948 5837960.520456 5209662 120 0.000000 38853.482261 0.000000 0 / [13552.357668] arc_prune 3949 16855.153463 2 120 0.000000 0.024186 0.000000 0 / [13552.357668] arc_prune 3950 16861.171957 2 120 0.000000 0.018497 0.000000 0 / [13552.357668] arc_reclaim 3951 5837490.829981 13035 120 0.000000 203.443003 0.000000 0 / [13552.357668] arc_user_evicts 3952 5837503.513393 13035 120 0.000000 106.655003 0.000000 0 / [13552.357668] l2arc_feed 3953 5837520.448342 13035 120 0.000000 101.976284 0.000000 0 / [13552.357668] z_unmount 3954 16893.267530 2 120 0.000000 0.021274 0.000000 0 / [13552.357668] ptlrpc_hr00_001 5911 5482421.376637 168639 120 0.000000 7815.010449 0.000000 0 / [13552.357668] socknal_reaper 5918 5837564.616857 13027 120 0.000000 121.377577 0.000000 0 / [13552.357668] socknal_sd00_00 5919 5837316.295044 67790 120 0.000000 8344.381852 0.000000 0 / [13552.357668] socknal_sd00_01 5920 5836699.599826 1898749 120 0.000000 111639.564870 0.000000 0 / [13552.357668] ptlrpcd_rcv 5923 5837825.689511 13186 120 0.000000 202.525328 0.000000 0 / [13552.357668] ptlrpcd_00_00 5924 5837968.689599 7829420 120 0.000000 97384.873950 0.000000 0 / [13552.357668] ptlrpcd_00_01 5925 5837969.377358 7825967 120 0.000000 97558.535363 0.000000 0 / [13552.357668] ll_ping 5926 5836402.449939 2675 120 0.000000 95.474076 0.000000 0 / [13552.357668] sptlrpc_gc 5927 5790471.762359 9 120 0.000000 0.055863 0.000000 0 / [13552.357668] kworker/u4:2 17953 5836949.657342 13320 120 0.000000 176.901209 0.000000 0 / [13552.357668] kworker/0:2 9350 5837539.325804 4137 120 0.000000 40.487516 0.000000 0 / [13552.357668] z_rd_iss 19726 5460533.683010 2 100 0.000000 0.018935 0.000000 0 / [13552.357668] z_rd_iss 19727 5460539.698710 2 100 0.000000 0.019646 0.000000 0 / [13552.357668] z_rd_iss 19728 5460545.714000 2 100 0.000000 0.019142 0.000000 0 / [13552.357668] z_rd_iss 19729 5460551.729469 2 100 0.000000 0.019409 0.000000 0 / [13552.357668] z_rd_iss 19730 5460557.744460 2 100 0.000000 0.018895 0.000000 0 / [13552.357668] z_rd_iss 19731 5460563.759519 2 100 0.000000 0.018970 0.000000 0 / [13552.357668] z_rd_iss 19732 5460569.773056 2 100 0.000000 0.017481 0.000000 0 / [13552.357668] z_rd_iss 19733 5460575.789528 2 100 0.000000 0.024420 0.000000 0 / [13552.357668] z_rd_int_0 19734 5473969.914691 11 100 0.000000 0.601726 0.000000 0 / [13552.357668] z_rd_int_0 19735 5473969.914688 11 100 0.000000 0.203516 0.000000 0 / [13552.357668] z_rd_int_0 19736 5473969.914688 11 100 0.000000 0.109043 0.000000 0 / [13552.357668] z_rd_int_0 19737 5473969.914689 11 100 0.000000 0.139570 0.000000 0 / [13552.357668] z_rd_int_0 19738 5473969.914702 11 100 0.000000 0.114742 0.000000 0 / [13552.357668] z_rd_int_0 19739 5480324.668092 12 100 0.000000 0.183419 0.000000 0 / [13552.357668] z_rd_int_0 19740 5480324.668055 12 100 0.000000 0.186436 0.000000 0 / [13552.357668] z_rd_int_0 19741 5473556.757924 11 100 0.000000 0.097724 0.000000 0 / [13552.357668] z_rd_int_0 19742 5473575.104369 11 100 0.000000 0.150265 0.000000 0 / [13552.357668] z_rd_int_0 19743 5473969.914776 10 100 0.000000 0.193024 0.000000 0 / [13552.357668] z_rd_int_0 19744 5473969.914695 10 100 0.000000 0.125798 0.000000 0 / [13552.357668] z_rd_int_0 19745 5473969.914696 10 100 0.000000 0.154424 0.000000 0 / [13552.357668] z_rd_int_1 19746 5473969.914684 11 100 0.000000 0.252531 0.000000 0 / [13552.357668] z_rd_int_1 19747 5467476.137454 10 100 0.000000 0.107174 0.000000 0 / [13552.357668] z_rd_int_1 19749 5473969.914695 10 100 0.000000 0.099340 0.000000 0 / [13552.357668] z_rd_int_1 19751 5473969.914686 11 100 0.000000 0.122410 0.000000 0 / [13552.357668] z_rd_int_1 19752 5473969.914688 10 100 0.000000 0.138718 0.000000 0 / [13552.357668] z_rd_int_1 19753 5473969.914689 10 100 0.000000 0.126913 0.000000 0 / [13552.357668] z_rd_int_1 19754 5473969.914689 10 100 0.000000 0.136959 0.000000 0 / [13552.357668] z_rd_int_1 19755 5473969.914686 10 100 0.000000 0.158519 0.000000 0 / [13552.357668] z_rd_int_1 19756 5473969.914684 10 100 0.000000 0.137756 0.000000 0 / [13552.357668] z_rd_int_2 19758 5467476.137367 10 100 0.000000 0.217398 0.000000 0 / [13552.357668] z_rd_int_2 19759 5467476.137397 10 100 0.000000 0.510634 0.000000 0 / [13552.357668] z_rd_int_2 19762 5473969.914695 11 100 0.000000 0.180984 0.000000 0 / [13552.357668] z_rd_int_2 19764 5473969.914685 11 100 0.000000 0.157132 0.000000 0 / [13552.357668] z_rd_int_2 19765 5473969.914685 10 100 0.000000 0.126009 0.000000 0 / [13552.357668] z_rd_int_2 19766 5473969.914687 10 100 0.000000 0.108077 0.000000 0 / [13552.357668] z_rd_int_2 19768 5473969.914689 11 100 0.000000 0.174219 0.000000 0 / [13552.357668] z_rd_int_3 19771 5473969.914689 11 100 0.000000 0.173262 0.000000 0 / [13552.357668] z_rd_int_3 19772 5473969.914693 10 100 0.000000 0.144176 0.000000 0 / [13552.357668] z_rd_int_3 19773 5473969.914690 10 100 0.000000 0.152811 0.000000 0 / [13552.357668] z_rd_int_3 19774 5473969.914684 10 100 0.000000 0.145987 0.000000 0 / [13552.357668] z_rd_int_3 19776 5473969.914686 11 100 0.000000 0.132319 0.000000 0 / [13552.357668] z_rd_int_3 19777 5473969.914572 10 100 0.000000 0.119132 0.000000 0 / [13552.357668] z_rd_int_3 19778 5480324.668056 11 100 0.000000 0.149916 0.000000 0 / [13552.357668] z_rd_int_3 19779 5467815.602270 9 100 0.000000 0.125406 0.000000 0 / [13552.357668] z_rd_int_3 19780 5467815.602141 9 100 0.000000 0.109362 0.000000 0 / [13552.357668] z_rd_int_3 19781 5467815.602193 9 100 0.000000 2.787698 0.000000 0 / [13552.357668] z_rd_int_4 19782 5467476.137551 11 100 0.000000 0.134017 0.000000 0 / [13552.357668] z_rd_int_4 19783 5473969.914700 11 100 0.000000 0.131696 0.000000 0 / [13552.357668] z_rd_int_4 19784 5473969.914697 11 100 0.000000 0.158198 0.000000 0 / [13552.357668] z_rd_int_4 19785 5473969.914688 11 100 0.000000 0.136317 0.000000 0 / [13552.357668] z_rd_int_4 19786 5473969.914798 11 100 0.000000 0.143275 0.000000 0 / [13552.357668] z_rd_int_4 19787 5473969.914685 11 100 0.000000 0.175008 0.000000 0 / [13552.357668] z_rd_int_4 19789 5467476.137640 10 100 0.000000 0.168668 0.000000 0 / [13552.357668] z_rd_int_4 19791 5467476.137402 10 100 0.000000 0.124892 0.000000 0 / [13552.357668] z_rd_int_4 19793 5467476.137393 10 100 0.000000 0.105107 0.000000 0 / [13552.357668] z_rd_int_5 19795 5473969.914686 10 100 0.000000 0.474158 0.000000 0 / [13552.357668] z_rd_int_5 19796 5473969.914688 10 100 0.000000 0.463374 0.000000 0 / [13552.357668] z_rd_int_5 19797 5473969.914683 10 100 0.000000 0.133979 0.000000 0 / [13552.357668] z_rd_int_5 19798 5473969.914683 10 100 0.000000 0.164911 0.000000 0 / [13552.357668] z_rd_int_5 19799 5473969.914688 10 100 0.000000 0.159289 0.000000 0 / [13552.357668] z_rd_int_5 19800 5473969.914690 10 100 0.000000 0.143236 0.000000 0 / [13552.357668] z_rd_int_5 19801 5473969.914686 11 100 0.000000 0.163662 0.000000 0 / [13552.357668] z_rd_int_5 19802 5473969.914684 10 100 0.000000 0.171210 0.000000 0 / [13552.357668] z_rd_int_5 19803 5473969.914866 10 100 0.000000 0.138151 0.000000 0 / [13552.357668] z_rd_int_5 19804 5473969.914698 9 100 0.000000 0.116036 0.000000 0 / [13552.357668] z_rd_int_6 19806 5473556.812206 11 100 0.000000 0.383082 0.000000 0 / [13552.357668] z_rd_int_6 19807 5473969.914687 11 100 0.000000 0.153026 0.000000 0 / [13552.357668] z_rd_int_6 19808 5473969.914697 11 100 0.000000 0.174830 0.000000 0 / [13552.357668] z_rd_int_6 19809 5473969.914684 11 100 0.000000 0.213759 0.000000 0 / [13552.357668] z_rd_int_6 19810 5473969.914669 12 100 0.000000 0.154510 0.000000 0 / [13552.357668] z_rd_int_6 19811 5473969.914572 11 100 0.000000 0.197587 0.000000 0 / [13552.357668] z_rd_int_6 19815 5467476.137436 10 100 0.000000 0.137013 0.000000 0 / [13552.357668] z_rd_int_6 19816 5467476.137428 11 100 0.000000 0.160690 0.000000 0 / [13552.357668] z_rd_int_6 19817 5467476.137403 10 100 0.000000 0.172539 0.000000 0 / [13552.357668] z_rd_int_7 19818 5473969.914691 11 100 0.000000 0.289097 0.000000 0 / [13552.357668] z_rd_int_7 19819 5473969.914685 11 100 0.000000 0.312444 0.000000 0 / [13552.357668] z_rd_int_7 19820 5473969.914686 11 100 0.000000 0.671409 0.000000 0 / [13552.357668] z_rd_int_7 19821 5473969.914692 11 100 0.000000 0.135872 0.000000 0 / [13552.357668] z_rd_int_7 19822 5473969.914688 11 100 0.000000 0.151418 0.000000 0 / [13552.357668] z_rd_int_7 19823 5473969.914668 12 100 0.000000 0.146750 0.000000 0 / [13552.357668] z_rd_int_7 19824 5467476.137421 10 100 0.000000 0.109307 0.000000 0 / [13552.357668] z_rd_int_7 19825 5480324.668207 11 100 0.000000 0.159468 0.000000 0 / [13552.357668] z_rd_int_7 19827 5467815.602234 10 100 0.000000 0.152450 0.000000 0 / [13552.357668] z_rd_int_7 19828 5473556.812244 10 100 0.000000 0.184498 0.000000 0 / [13552.357668] z_rd_int_7 19829 5473969.914695 10 100 0.000000 0.158380 0.000000 0 / [13552.357668] z_wr_iss 19830 5832657.964772 6650 101 0.000000 416.059200 0.000000 0 / [13552.357668] z_wr_iss_h 19831 5461141.914597 2 100 0.000000 0.018884 0.000000 0 / [13552.357668] z_wr_iss_h 19832 5461147.929524 2 100 0.000000 0.019086 0.000000 0 / [13552.357668] z_wr_iss_h 19834 5461148.284040 2 100 0.000000 0.020160 0.000000 0 / [13552.357668] z_wr_iss_h 19835 5461154.298901 2 100 0.000000 0.020495 0.000000 0 / [13552.357668] z_wr_int_0 19846 5832960.233239 359 100 0.000000 9.399549 0.000000 0 / [13552.357668] z_wr_int_2 19866 5833387.201535 360 100 0.000000 9.518656 0.000000 0 / [13552.357668] z_wr_int_2 19867 5834717.094271 360 100 0.000000 9.369039 0.000000 0 / [13552.357668] z_wr_int_3 19880 5833835.513915 368 100 0.000000 9.558978 0.000000 0 / [13552.357668] z_wr_int_3 19881 5833956.188349 368 100 0.000000 9.663975 0.000000 0 / [13552.357668] z_wr_int_3 19882 5834112.341991 368 100 0.000000 9.889963 0.000000 0 / [13552.357668] z_wr_int_3 19883 5837070.885633 368 100 0.000000 9.756338 0.000000 0 / [13552.357668] z_wr_int_4 19892 5835088.681710 357 100 0.000000 9.218928 0.000000 0 / [13552.357668] z_wr_int_4 19893 5835149.278209 357 100 0.000000 9.434904 0.000000 0 / [13552.357668] z_wr_int_5 19904 5836125.209398 360 100 0.000000 9.410361 0.000000 0 / [13552.357668] z_wr_int_5 19905 5836158.054544 360 100 0.000000 9.611719 0.000000 0 / [13552.357668] z_wr_int_5 19906 5837196.156437 360 100 0.000000 9.459602 0.000000 0 / [13552.357668] z_wr_int_6 19908 5834872.738012 353 100 0.000000 9.262789 0.000000 0 / [13552.357668] z_wr_int_6 19909 5837040.652331 353 100 0.000000 9.698373 0.000000 0 / [13552.357668] z_wr_int_6 19919 5833419.817311 352 100 0.000000 10.583116 0.000000 0 / [13552.357668] z_wr_int_7 19924 5836617.268758 355 100 0.000000 9.557659 0.000000 0 / [13552.357668] z_wr_int_h 19932 5461486.218673 2 100 0.000000 0.016765 0.000000 0 / [13552.357668] z_wr_int_h 19933 5461492.233453 2 100 0.000000 0.015906 0.000000 0 / [13552.357668] z_wr_int_h 19934 5461498.248381 2 100 0.000000 0.016079 0.000000 0 / [13552.357668] z_wr_int_h 19935 5461504.263192 2 100 0.000000 0.015971 0.000000 0 / [13552.357668] z_wr_int_h 19936 5461510.277850 2 100 0.000000 0.015843 0.000000 0 / [13552.357668] z_fr_iss_0 19937 5461516.292638 2 100 0.000000 0.015843 0.000000 0 / [13552.357668] z_fr_iss_0 19938 5461522.307630 2 100 0.000000 0.016056 0.000000 0 / [13552.357668] z_fr_iss_0 19939 5461528.322723 2 100 0.000000 0.016241 0.000000 0 / [13552.357668] z_fr_iss_0 19940 5461534.337613 2 100 0.000000 0.015942 0.000000 0 / [13552.357668] z_fr_iss_0 19941 5461540.352375 2 100 0.000000 0.015861 0.000000 0 / [13552.357668] z_fr_iss_0 19942 5461546.367180 2 100 0.000000 0.015828 0.000000 0 / [13552.357668] z_fr_iss_0 19943 5461552.381804 2 100 0.000000 0.015685 0.000000 0 / [13552.357668] z_fr_iss_0 19944 5461558.396638 2 100 0.000000 0.016031 0.000000 0 / [13552.357668] z_fr_iss_0 19945 5461564.411450 2 100 0.000000 0.015926 0.000000 0 / [13552.357668] z_fr_iss_0 19946 5461570.426431 2 100 0.000000 0.016040 0.000000 0 / [13552.357668] z_fr_iss_0 19947 5461576.440980 2 100 0.000000 0.015559 0.000000 0 / [13552.357668] z_fr_iss_0 19948 5461582.455648 2 100 0.000000 0.015700 0.000000 0 / [13552.357668] z_fr_iss_1 19949 5461588.470323 2 100 0.000000 0.015771 0.000000 0 / [13552.357668] z_fr_iss_1 19950 5461594.484952 2 100 0.000000 0.015703 0.000000 0 / [13552.357668] z_fr_iss_1 19951 5461600.500064 2 100 0.000000 0.016248 0.000000 0 / [13552.357668] z_fr_iss_1 19952 5461606.514829 2 100 0.000000 0.015928 0.000000 0 / [13552.357668] z_fr_iss_1 19953 5461612.529660 2 100 0.000000 0.015940 0.000000 0 / [13552.357668] z_fr_iss_1 19954 5461618.544280 2 100 0.000000 0.019591 0.000000 0 / [13552.357668] z_fr_iss_1 19955 5461624.560036 2 100 0.000000 0.016810 0.000000 0 / [13552.357668] z_fr_iss_1 19956 5461630.574974 2 100 0.000000 0.016058 0.000000 0 / [13552.357668] z_fr_iss_1 19957 5461636.587401 2 100 0.000000 0.013536 0.000000 0 / [13552.357668] z_fr_iss_1 19958 5461642.599600 2 100 0.000000 0.013326 0.000000 0 / [13552.357668] z_fr_iss_1 19959 5461648.611561 2 100 0.000000 0.013054 0.000000 0 / [13552.357668] z_fr_iss_1 19960 5461654.623491 2 100 0.000000 0.012985 0.000000 0 / [13552.357668] z_fr_iss_2 19961 5461660.635456 2 100 0.000000 0.013117 0.000000 0 / [13552.357668] z_fr_iss_2 19962 5461666.647562 2 100 0.000000 0.013173 0.000000 0 / [13552.357668] z_fr_iss_2 19963 5461672.659810 2 100 0.000000 0.013283 0.000000 0 / [13552.357668] z_fr_iss_2 19964 5461678.671776 2 100 0.000000 0.013016 0.000000 0 / [13552.357668] z_fr_iss_2 19965 5461684.683624 2 100 0.000000 0.012964 0.000000 0 / [13552.357668] z_fr_iss_2 19966 5461690.695648 2 100 0.000000 0.013048 0.000000 0 / [13552.357668] z_fr_iss_2 19967 5461696.707577 2 100 0.000000 0.013056 0.000000 0 / [13552.357668] z_fr_iss_2 19968 5461702.719481 2 100 0.000000 0.013035 0.000000 0 / [13552.357668] z_fr_iss_2 19969 5461708.731317 2 100 0.000000 0.012906 0.000000 0 / [13552.357668] z_fr_iss_2 19970 5461714.743251 2 100 0.000000 0.012956 0.000000 0 / [13552.357668] z_fr_iss_2 19971 5461720.755103 2 100 0.000000 0.012918 0.000000 0 / [13552.357668] z_fr_iss_2 19972 5461726.767188 2 100 0.000000 0.013084 0.000000 0 / [13552.357668] z_fr_iss_3 19973 5461732.779278 2 100 0.000000 0.013138 0.000000 0 / [13552.357668] z_fr_iss_3 19974 5461738.791151 2 100 0.000000 0.013005 0.000000 0 / [13552.357668] z_fr_iss_3 19975 5461744.803158 2 100 0.000000 0.013090 0.000000 0 / [13552.357668] z_fr_iss_3 19976 5461750.815258 2 100 0.000000 0.013248 0.000000 0 / [13552.357668] z_fr_iss_3 19977 5461756.827141 2 100 0.000000 0.012972 0.000000 0 / [13552.357668] z_fr_iss_3 19978 5461762.839219 2 100 0.000000 0.013110 0.000000 0 / [13552.357668] z_fr_iss_3 19979 5461768.851322 2 100 0.000000 0.013187 0.000000 0 / [13552.357668] z_fr_iss_3 19980 5461774.863464 2 100 0.000000 0.013202 0.000000 0 / [13552.357668] z_fr_iss_3 19981 5461780.875461 2 100 0.000000 0.013290 0.000000 0 / [13552.357668] z_fr_iss_3 19982 5461786.893802 2 100 0.000000 0.019503 0.000000 0 / [13552.357668] z_fr_iss_3 19983 5461792.905687 2 100 0.000000 0.013024 0.000000 0 / [13552.357668] z_fr_iss_3 19984 5461798.918000 2 100 0.000000 0.013399 0.000000 0 / [13552.357668] z_fr_iss_4 19985 5461804.930190 2 100 0.000000 0.013271 0.000000 0 / [13552.357668] z_fr_iss_4 19986 5461810.942623 2 100 0.000000 0.013444 0.000000 0 / [13552.357668] z_fr_iss_4 19987 5461816.954802 2 100 0.000000 0.013250 0.000000 0 / [13552.357668] z_fr_iss_4 19988 5461822.966805 2 100 0.000000 0.013087 0.000000 0 / [13552.357668] z_fr_iss_4 19989 5461828.978721 2 100 0.000000 0.013049 0.000000 0 / [13552.357668] z_fr_iss_4 19990 5461834.990712 2 100 0.000000 0.013056 0.000000 0 / [13552.357668] z_fr_iss_4 19991 5461841.002778 2 100 0.000000 0.013125 0.000000 0 / [13552.357668] z_fr_iss_4 19992 5461847.014999 2 100 0.000000 0.013313 0.000000 0 / [13552.357668] z_fr_iss_4 19993 5461853.027118 2 100 0.000000 0.013149 0.000000 0 / [13552.357668] z_fr_iss_4 19994 5461859.039288 2 100 0.000000 0.013199 0.000000 0 / [13552.357668] z_fr_iss_4 19995 5461865.051283 2 100 0.000000 0.013048 0.000000 0 / [13552.357668] z_fr_iss_4 19996 5461871.063166 2 100 0.000000 0.012936 0.000000 0 / [13552.357668] z_fr_iss_5 19997 5461877.075194 2 100 0.000000 0.013041 0.000000 0 / [13552.357668] z_fr_iss_5 19998 5461883.087105 2 100 0.000000 0.013005 0.000000 0 / [13552.357668] z_fr_iss_5 19999 5461889.099247 2 100 0.000000 0.013180 0.000000 0 / [13552.357668] z_fr_iss_5 20000 5461895.111140 2 100 0.000000 0.012950 0.000000 0 / [13552.357668] z_fr_iss_5 20001 5461901.122971 2 100 0.000000 0.012910 0.000000 0 / [13552.357668] z_fr_iss_5 20002 5461907.135022 2 100 0.000000 0.013021 0.000000 0 / [13552.357668] z_fr_iss_5 20003 5461913.147117 2 100 0.000000 0.013107 0.000000 0 / [13552.357668] z_fr_iss_5 20004 5461919.159023 2 100 0.000000 0.012957 0.000000 0 / [13552.357668] z_fr_iss_5 20005 5461925.170878 2 100 0.000000 0.012983 0.000000 0 / [13552.357668] z_fr_iss_5 20006 5461931.182841 2 100 0.000000 0.013113 0.000000 0 / [13552.357668] z_fr_iss_5 20007 5461937.194771 2 100 0.000000 0.013070 0.000000 0 / [13552.357668] z_fr_iss_5 20008 5461943.211243 2 100 0.000000 0.017524 0.000000 0 / [13552.357668] z_fr_iss_6 20009 5461949.223390 2 100 0.000000 0.013142 0.000000 0 / [13552.357668] z_fr_iss_6 20010 5461955.235443 2 100 0.000000 0.013094 0.000000 0 / [13552.357668] z_fr_iss_6 20011 5461961.246211 2 100 0.000000 0.011792 0.000000 0 / [13552.357668] z_fr_iss_6 20012 5461967.261267 2 100 0.000000 0.016144 0.000000 0 / [13552.357668] z_fr_iss_6 20013 5461973.276068 2 100 0.000000 0.015903 0.000000 0 / [13552.357668] z_fr_iss_6 20014 5461979.290877 2 100 0.000000 0.015930 0.000000 0 / [13552.357668] z_fr_iss_6 20015 5461985.305524 2 100 0.000000 0.015747 0.000000 0 / [13552.357668] z_fr_iss_6 20016 5461991.320387 2 100 0.000000 0.016002 0.000000 0 / [13552.357668] z_fr_iss_6 20017 5461997.335431 2 100 0.000000 0.016141 0.000000 0 / [13552.357668] z_fr_iss_6 20018 5462003.350244 2 100 0.000000 0.015868 0.000000 0 / [13552.357668] z_fr_iss_6 20019 5462009.364711 2 100 0.000000 0.015542 0.000000 0 / [13552.357668] z_fr_iss_6 20020 5462015.379363 2 100 0.000000 0.015666 0.000000 0 / [13552.357668] z_fr_iss_7 20021 5462021.394150 2 100 0.000000 0.015891 0.000000 0 / [13552.357668] z_fr_iss_7 20022 5462027.408829 2 100 0.000000 0.015917 0.000000 0 / [13552.357668] z_fr_iss_7 20023 5462033.423602 2 100 0.000000 0.015862 0.000000 0 / [13552.357668] z_fr_iss_7 20024 5462039.438276 2 100 0.000000 0.015817 0.000000 0 / [13552.357668] z_fr_iss_7 20025 5462045.452854 2 100 0.000000 0.015601 0.000000 0 / [13552.357668] z_fr_iss_7 20026 5462051.467751 2 100 0.000000 0.015936 0.000000 0 / [13552.357668] z_fr_iss_7 20027 5462057.482739 2 100 0.000000 0.016070 0.000000 0 / [13552.357668] z_fr_iss_7 20028 5462063.497365 2 100 0.000000 0.015700 0.000000 0 / [13552.357668] z_fr_iss_7 20029 5462069.512172 2 100 0.000000 0.015848 0.000000 0 / [13552.357668] z_fr_iss_7 20030 5462075.527135 2 100 0.000000 0.016089 0.000000 0 / [13552.357668] z_fr_iss_7 20031 5462081.541985 2 100 0.000000 0.015911 0.000000 0 / [13552.357668] z_fr_iss_7 20032 5462087.556867 2 100 0.000000 0.016013 0.000000 0 / [13552.357668] z_fr_int 20033 5462093.571921 2 100 0.000000 0.016065 0.000000 0 / [13552.357668] z_cl_iss 20034 5462099.588406 2 100 0.000000 0.017509 0.000000 0 / [13552.357668] z_cl_int 20035 5462105.603491 2 100 0.000000 0.016142 0.000000 0 / [13552.357668] z_ioctl_iss 20036 5462111.618705 2 100 0.000000 0.016322 0.000000 0 / [13552.357668] metaslab_group_ 20039 5830150.727613 910 100 0.000000 5.132366 0.000000 0 / [13552.357668] z_iput 20041 5462145.597234 2 120 0.000000 0.018271 0.000000 0 / [13552.357668] z_iput 20042 5462151.614375 2 120 0.000000 0.017144 0.000000 0 / [13552.357668] txg_quiesce 20056 5832446.140766 1776 120 0.000000 9.645524 0.000000 0 / [13552.357668] txg_sync 20057 5832446.206552 12058 120 0.000000 248.083973 0.000000 0 / [13552.357668] ldlm_cb00_000 20226 5490447.503595 45 120 0.000000 0.473300 0.000000 0 / [13552.357668] ldlm_cn00_001 20230 5483133.786301 19 120 0.000000 0.687989 0.000000 0 / [13552.357668] ldlm_cn00_002 20231 5483133.746761 14 120 0.000000 0.458343 0.000000 0 / [13552.357668] ldlm_bl_01 20232 5837761.434567 3827 120 0.000000 8.984502 0.000000 0 / [13552.357668] ldlm_bl_02 20233 5837761.433516 3823 120 0.000000 13.569306 0.000000 0 / [13552.357668] ldlm_elt 20234 5463133.869703 2 120 0.000000 0.011677 0.000000 0 / [13552.357668] ldlm_poold 20235 5837761.444255 3821 120 0.000000 72.118035 0.000000 0 / [13552.357668] mgs_params_noti 20236 5463145.894978 2 118 0.000000 0.016213 0.000000 0 / [13552.357668] ll_mgs_0000 20238 5463228.891172 4 120 0.000000 0.047931 0.000000 0 / [13552.357668] ll_mgs_0001 20239 5836699.521366 3421 120 0.000000 65.403529 0.000000 0 / [13552.357668] ll_mgs_0002 20240 5837175.829046 3822 120 0.000000 153.771375 0.000000 0 / [13552.357668] ll_evictor 20241 5463176.221325 2 120 0.000000 0.015703 0.000000 0 / [13552.357668] ll_cfg_requeue 20242 5486931.118887 38 120 0.000000 1.229420 0.000000 0 / [13552.357668] mdt00_000 20243 5837175.827259 2440 120 0.000000 84.041612 0.000000 0 / [13552.357668] mdt00_001 20244 5836271.958479 1798 120 0.000000 53.539254 0.000000 0 / [13552.357668] mdt00_002 20245 5836271.958160 2481 120 0.000000 76.146719 0.000000 0 / [13552.357668] mdt_rdpg00_000 20246 5833664.846544 2759678 120 0.000000 139026.856596 0.000000 0 / [13552.357668] mdt_rdpg00_001 20247 5838128.003247 3787043 120 0.000000 195149.201136 0.000000 0 / [13552.357668] mdt_attr00_000 20248 5463218.445161 2 120 0.000000 0.043869 0.000000 0 / [13552.357668] mdt_attr00_001 20249 5463224.461926 2 120 0.000000 0.018251 0.000000 0 / [13552.357668] osp-pre-0-0 20263 5837315.057349 769 120 0.000000 18.395105 0.000000 0 / [13552.357668] osp-pre-1-0 20265 5837315.060249 769 120 0.000000 21.524617 0.000000 0 / [13552.357668] tx_commit_cb 20270 5830146.582641 324 120 0.000000 2531.855097 0.000000 0 / [13552.357668] Rlquota_wb_lustr 20274 5838132.511498 5123469 120 0.000000 279760.182453 0.000000 0 / [13552.357668] kworker/0:3 20907 5494476.289163 68 120 0.000000 0.246661 0.000000 0 / [13552.357668] mdt00_003 20983 5836699.520144 2222 120 0.000000 72.455620 0.000000 0 / [13552.357668] ldlm_cn00_003 24141 5483139.681262 3 120 0.000000 0.014395 0.000000 0 / [13552.357668] R mdt_rdpg00_002 25538 5838141.638500 3698373 120 0.000000 163960.162102 0.000000 0 / [13552.357668] pickup 25765 5830150.775906 29 120 0.000000 7.451990 0.000000 0 / [13552.930749] [13552.931652] cpu#1, 2593.768 MHz [13552.932567] .nr_running : 6 [13552.933600] .load : 91833 [13552.934662] .nr_switches : 36901241 [13552.935752] .nr_load_updates : 3334859 [13552.936842] .nr_uninterruptible : -332 [13552.937913] .next_balance : 4308.220355 [13552.939042] .curr->pid : 25856 [13552.940134] .clock : 13539245.178936 [13552.941299] .cpu_load[0] : 91833 [13552.942398] .cpu_load[1] : 91833 [13552.943497] .cpu_load[2] : 91833 [13552.944640] .cpu_load[3] : 91833 [13552.946046] .cpu_load[4] : 91833 [13552.948730] .avg_idle : 77099 [13552.952340] .max_idle_balance_cost : 500000 [13552.953436] [13552.953436] cfs_rq[1]:/ [13552.954415] .exec_clock : 0.000000 [13552.954415] .MIN_vruntime : 5348758.960237 [13552.954415] .min_vruntime : 5348764.960237 [13552.954415] .max_vruntime : 5348758.960237 [13552.954415] .spread : 0.000000 [13552.954415] .spread0 : -489382.089177 [13552.954415] .nr_spread_over : 0 [13552.954415] .nr_running : 4 [13552.954415] .load : 91833 [13552.954415] .runnable_load_avg : 238 [13552.954415] .blocked_load_avg : 5405 [13552.954415] .tg_load_avg : 0 [13552.954415] .tg_load_contrib : 0 [13552.954415] .tg_runnable_contrib : 0 [13552.954415] .tg->runnable_avg : 0 [13552.954415] .tg->cfs_bandwidth.timer_active: 0 [13552.954415] .throttled : 0 [13552.954415] .throttle_count : 0 [13552.954415] .avg->runnable_avg_sum : 36553 [13552.954415] .avg->runnable_avg_period : 46358 [13552.954415] [13552.954415] rt_rq[1]:/ [13552.954415] .rt_nr_running : 1 [13552.954415] .rt_throttled : 0 [13552.954415] .rt_time : 0.000000 [13552.954415] .rt_runtime : 950.000000 [13552.954415] [13552.954415] runnable tasks: [13552.954415] task PID tree-key switches prio wait-time sum-exec sum-sleep [13552.954415] ---------------------------------------------------------------------------------------------------------- [13552.954415] kthreadd 2 5348748.800282 19421 120 0.000000 343.432935 0.000000 0 / [13552.954415] kworker/u4:0 6 4849592.253357 13165 120 0.000000 178.695900 0.000000 0 / [13552.954415] watchdog/1 11 -3.000000 3387 0 0.000000 55.407122 0.000000 0 / [13552.954415] migration/1 12 0.000000 75042 0 0.000000 789.087539 0.000000 0 / [13552.954415] ksoftirqd/1 13 5348758.960237 231829 120 0.000000 4234.163929 0.000000 0 / [13552.954415] kworker/1:0H 15 3298.054011 7 100 0.000000 0.094032 0.000000 0 / [13552.954415] netns 18 10.983951 2 100 0.000000 0.009614 0.000000 0 / [13552.954415] khungtaskd 19 5339848.830664 114 120 0.000000 7.858317 0.000000 0 / [13552.954415] writeback 20 18.992804 2 100 0.000000 0.000000 0.000000 0 / [13552.954415] kintegrityd 21 22.992803 2 100 0.000000 0.000000 0.000000 0 / [13552.954415] bioset 22 26.992802 2 100 0.000000 0.000000 0.000000 0 / [13552.954415] kblockd 23 30.992801 2 100 0.000000 0.000000 0.000000 0 / [13552.954415] md 24 35.000258 2 100 0.000000 0.007458 0.000000 0 / [13552.954415] kswapd0 31 341.909319 3 120 0.000000 0.055087 0.000000 0 / [13552.954415] ksmd 32 294.931436 2 125 0.000000 0.037502 0.000000 0 / [13552.954415] crypto 35 307.364154 2 100 0.000000 0.011603 0.000000 0 / [13552.954415] kthrotld 43 339.831610 2 100 0.000000 0.027970 0.000000 0 / [13552.954415] kmpath_rdacd 45 347.893259 2 100 0.000000 0.037845 0.000000 0 / [13552.954415] kpsmoused 46 350.948567 2 100 0.000000 0.020893 0.000000 0 / [13552.954415] deferwq 67 466.825360 2 100 0.000000 0.011122 0.000000 0 / [13552.954415] ata_sff 238 779.680457 2 100 0.000000 0.005835 0.000000 0 / [13552.954415] scsi_eh_0 240 880.355988 4 120 0.000000 0.326281 0.000000 0 / [13552.954415] scsi_tmf_0 241 790.735617 2 100 0.000000 0.002765 0.000000 0 / [13552.954415] scsi_eh_1 242 880.300297 4 120 0.000000 0.258967 0.000000 0 / [13552.954415] scsi_tmf_1 243 798.737874 2 100 0.000000 0.003051 0.000000 0 / [13552.954415] ttm_swap 252 826.418362 2 100 0.000000 0.004034 0.000000 0 / [13552.954415] ext4-rsv-conver 274 908.431948 2 100 0.000000 0.020178 0.000000 0 / [13552.954415] rpciod 367 1120.787419 2 100 0.000000 0.005597 0.000000 0 / [13552.954415] ib-comp-wq 478 1298.877941 2 100 0.000000 0.007953 0.000000 0 / [13552.954415] iw_cm_wq 496 1320.470193 2 100 0.000000 0.006885 0.000000 0 / [13552.954415] auditd 499 5348532.055719 43293 116 0.000000 1737.892883 0.000000 0 / [13552.954415] ib_cm 515 1340.837671 2 100 0.000000 0.022196 0.000000 0 / [13552.954415] rdma_cm 523 1366.304588 2 100 0.000000 0.007845 0.000000 0 / [13552.954415] gssproxy 544 1406.107921 1 120 0.000000 0.001612 0.000000 0 / [13552.954415] gssproxy 545 1405.108746 1 120 0.000000 0.002439 0.000000 0 / [13552.954415] gssproxy 547 1404.522188 1 120 0.000000 0.015881 0.000000 0 / [13552.954415] gssproxy 548 5348676.253010 32 120 0.000000 5.494765 0.000000 0 / [13552.954415] gmain 573 1442.272081 3 120 0.000000 0.047874 0.000000 0 / [13552.954415] polkitd 552 5348531.259398 99883 120 0.000000 4787.571209 0.000000 0 / [13552.954415] gmain 568 1436.554446 4 120 0.000000 0.045310 0.000000 0 / [13552.954415] gdbus 572 5348531.332524 97877 120 0.000000 6725.292792 0.000000 0 / [13552.954415] JS GC Helper 574 1442.850552 1 120 0.000000 0.026345 0.000000 0 / [13552.954415] JS Sour~ Thread 576 1442.535027 8 120 0.000000 0.070224 0.000000 0 / [13552.954415] runaway-killer- 578 1447.479724 5 120 0.000000 0.065719 0.000000 0 / [13552.954415] sssd 559 5347748.740739 708 120 0.000000 992.397212 0.000000 0 / [13552.954415] chronyd 567 5306809.324015 146 120 0.000000 22.657244 0.000000 0 / [13552.954415] ipoib_flush 580 1450.571768 2 100 0.000000 0.006759 0.000000 0 / [13552.954415] frwr_recovery 649 1610.225136 2 100 0.000000 0.009292 0.000000 0 / [13552.954415] xprtrdma_receiv 650 1614.229341 2 100 0.000000 0.005300 0.000000 0 / [13552.954415] gmain 968 3427.224256 1 120 0.000000 0.030295 0.000000 0 / [13552.954415] tuned 972 15221.004981 27 120 0.000000 7.319844 0.000000 0 / [13552.954415] xinetd 908 5055306.600212 18638 120 0.000000 4019.312093 0.000000 0 / [13552.954415] munged 911 5055168.825828 9324 120 0.000000 227.095162 0.000000 0 / [13552.954415] munged 919 5055038.957224 4659 120 0.000000 290.190038 0.000000 0 / [13552.954415] sshd 921 5348529.193606 225 120 0.000000 31.948161 0.000000 0 / [13552.954415] kworker/1:1H 930 5348758.966015 1382 100 0.000000 61.417520 0.000000 0 / [13552.954415] nfsv4.0-svc 931 5348758.981934 46 120 0.000000 3.290966 0.000000 0 / [13552.954415] atd 1034 5141386.793743 8 120 0.000000 1.875844 0.000000 0 / [13552.954415] automount 1036 3923.282921 36 120 0.000000 10.692584 0.000000 0 / [13552.954415] automount 1089 3845.820978 6 120 0.000000 0.632188 0.000000 0 / [13552.954415] automount 1093 3913.469454 9 120 0.000000 5.739777 0.000000 0 / [13552.954415] bioset 2533 9144.095208 2 100 0.000000 0.012631 0.000000 0 / [13552.954415] kdmflush 2582 9395.346109 2 100 0.000000 0.020209 0.000000 0 / [13552.954415] bioset 2584 9399.045381 2 100 0.000000 0.010393 0.000000 0 / [13552.954415] spl_system_task 3934 15071.061806 2 100 0.000000 0.011718 0.000000 0 / [13552.954415] cfs_rh_00 5899 4952902.913184 59 120 0.000000 8.493047 0.000000 0 / [13552.954415] cfs_rh_01 5900 4949097.711753 58 120 0.000000 5.744517 0.000000 0 / [13552.954415] obd_zombid 5905 5023146.089115 519 120 0.000000 15.440274 0.000000 0 / [13552.954415] ptlrpc_hr00_000 5910 5043262.166132 168725 120 0.000000 7673.116812 0.000000 0 / [13552.954415] socknal_cd00 5914 4546005.329443 10 120 0.000000 0.549721 0.000000 0 / [13552.954415] socknal_cd01 5915 4546005.237259 6 120 0.000000 0.536048 0.000000 0 / [13552.954415] socknal_cd02 5916 4546005.328543 8 120 0.000000 0.659238 0.000000 0 / [13552.954415] socknal_cd03 5917 926893.130454 8 120 0.000000 0.362802 0.000000 0 / [13552.954415] acceptor_000 5921 4546005.350891 22 120 0.000000 1.222410 0.000000 0 / [13552.954415] router_checker 5922 24788.950858 2 120 0.000000 0.022348 0.000000 0 / [13552.954415] kworker/1:3 5031 5348758.960237 24457 120 0.000000 276.288446 0.000000 0 / [13552.954415] z_null_iss 19724 5347744.076725 1111 100 0.000000 39.008839 0.000000 0 / [13552.954415] z_null_int 19725 5347744.076012 1347 100 0.000000 24.811134 0.000000 0 / [13552.954415] z_rd_int_1 19748 5032904.147220 12 100 0.000000 0.146326 0.000000 0 / [13552.954415] z_rd_int_1 19750 5035421.023711 10 100 0.000000 0.173810 0.000000 0 / [13552.954415] z_rd_int_1 19757 5029951.597603 10 100 0.000000 0.113273 0.000000 0 / [13552.954415] z_rd_int_2 19760 5032904.147269 10 100 0.000000 0.575686 0.000000 0 / [13552.954415] z_rd_int_2 19761 5032904.147176 11 100 0.000000 0.145749 0.000000 0 / [13552.954415] z_rd_int_2 19763 5034754.532824 10 100 0.000000 0.149677 0.000000 0 / [13552.954415] z_rd_int_2 19767 5035747.850499 10 100 0.000000 0.125807 0.000000 0 / [13552.954415] z_rd_int_2 19769 5041646.290904 10 100 0.000000 0.204778 0.000000 0 / [13552.954415] z_rd_int_3 19770 5034754.532870 10 100 0.000000 0.467105 0.000000 0 / [13552.954415] z_rd_int_3 19775 5035747.850510 10 100 0.000000 0.137783 0.000000 0 / [13552.954415] z_rd_int_4 19788 5029951.597764 11 100 0.000000 0.169071 0.000000 0 / [13552.954415] z_rd_int_4 19790 5029951.597896 10 100 0.000000 0.170635 0.000000 0 / [13552.954415] z_rd_int_4 19792 5029951.597588 11 100 0.000000 0.138051 0.000000 0 / [13552.954415] z_rd_int_5 19794 5035747.850506 10 100 0.000000 0.620934 0.000000 0 / [13552.954415] z_rd_int_5 19805 5035747.850564 9 100 0.000000 0.144275 0.000000 0 / [13552.954415] z_rd_int_6 19812 5029951.597784 10 100 0.000000 0.169154 0.000000 0 / [13552.954415] z_rd_int_6 19813 5029951.597648 10 100 0.000000 0.147443 0.000000 0 / [13552.954415] z_rd_int_6 19814 5029951.597595 10 100 0.000000 0.113942 0.000000 0 / [13552.954415] z_rd_int_7 19826 5029951.597644 10 100 0.000000 0.139269 0.000000 0 / [13552.954415] z_wr_iss_h 19833 5025012.034755 2 100 0.000000 0.002842 0.000000 0 / [13552.954415] z_wr_int_0 19836 5347744.075569 359 100 0.000000 9.254469 0.000000 0 / [13552.954415] z_wr_int_0 19837 5347744.075712 359 100 0.000000 9.326187 0.000000 0 / [13552.954415] z_wr_int_0 19838 5347744.075711 359 100 0.000000 9.587955 0.000000 0 / [13552.954415] z_wr_int_0 19839 5347744.075647 359 100 0.000000 9.411443 0.000000 0 / [13552.954415] z_wr_int_0 19840 5347744.075574 359 100 0.000000 9.530034 0.000000 0 / [13552.954415] z_wr_int_0 19841 5347744.075560 359 100 0.000000 9.457390 0.000000 0 / [13552.954415] z_wr_int_0 19842 5347744.075609 359 100 0.000000 9.354358 0.000000 0 / [13552.954415] z_wr_int_0 19843 5347744.075576 359 100 0.000000 9.776860 0.000000 0 / [13552.954415] z_wr_int_0 19844 5347744.075558 359 100 0.000000 9.354933 0.000000 0 / [13552.954415] z_wr_int_0 19845 5347744.075650 359 100 0.000000 9.334431 0.000000 0 / [13552.954415] z_wr_int_0 19847 5347744.075615 358 100 0.000000 9.131524 0.000000 0 / [13552.954415] z_wr_int_1 19848 5347744.075879 366 100 0.000000 9.621063 0.000000 0 / [13552.954415] z_wr_int_1 19849 5347744.075808 366 100 0.000000 9.776997 0.000000 0 / [13552.954415] z_wr_int_1 19850 5347744.075944 366 100 0.000000 9.567391 0.000000 0 / [13552.954415] z_wr_int_1 19851 5347744.075992 366 100 0.000000 9.460557 0.000000 0 / [13552.954415] z_wr_int_1 19852 5347744.075619 365 100 0.000000 9.765418 0.000000 0 / [13552.954415] z_wr_int_1 19853 5347744.075549 365 100 0.000000 9.650822 0.000000 0 / [13552.954415] z_wr_int_1 19854 5347744.075595 365 100 0.000000 9.615734 0.000000 0 / [13552.954415] z_wr_int_1 19855 5347744.075572 365 100 0.000000 9.550241 0.000000 0 / [13552.954415] z_wr_int_1 19856 5347744.075571 365 100 0.000000 9.700033 0.000000 0 / [13552.954415] z_wr_int_1 19857 5347744.075574 365 100 0.000000 9.718066 0.000000 0 / [13552.954415] z_wr_int_1 19858 5347744.075603 365 100 0.000000 9.979234 0.000000 0 / [13552.954415] z_wr_int_1 19859 5347744.075575 365 100 0.000000 9.776558 0.000000 0 / [13552.954415] z_wr_int_2 19860 5347744.075554 360 100 0.000000 9.461570 0.000000 0 / [13552.954415] z_wr_int_2 19861 5347744.075556 360 100 0.000000 9.418778 0.000000 0 / [13552.954415] z_wr_int_2 19862 5347744.075681 360 100 0.000000 9.338690 0.000000 0 / [13552.954415] z_wr_int_2 19863 5347744.075542 360 100 0.000000 10.715293 0.000000 0 / [13552.954415] z_wr_int_2 19864 5347744.075557 360 100 0.000000 9.630159 0.000000 0 / [13552.954415] z_wr_int_2 19865 5347744.075689 360 100 0.000000 9.531945 0.000000 0 / [13552.954415] z_wr_int_2 19868 5347714.305296 359 100 0.000000 9.468770 0.000000 0 / [13552.954415] z_wr_int_2 19869 5347744.075591 359 100 0.000000 9.485509 0.000000 0 / [13552.954415] z_wr_int_2 19870 5347744.075704 359 100 0.000000 9.339068 0.000000 0 / [13552.954415] z_wr_int_2 19871 5347744.075610 359 100 0.000000 9.710757 0.000000 0 / [13552.954415] z_wr_int_3 19872 5347744.075601 368 100 0.000000 9.819348 0.000000 0 / [13552.954415] z_wr_int_3 19873 5347744.075563 368 100 0.000000 9.456247 0.000000 0 / [13552.954415] z_wr_int_3 19874 5347744.075571 368 100 0.000000 9.796549 0.000000 0 / [13552.954415] z_wr_int_3 19875 5347744.075565 368 100 0.000000 9.666176 0.000000 0 / [13552.954415] z_wr_int_3 19876 5347744.075640 368 100 0.000000 9.296697 0.000000 0 / [13552.954415] z_wr_int_3 19877 5347744.075590 368 100 0.000000 9.512062 0.000000 0 / [13552.954415] z_wr_int_3 19878 5347744.075676 368 100 0.000000 9.622265 0.000000 0 / [13552.954415] z_wr_int_3 19879 5347744.075874 368 100 0.000000 10.516408 0.000000 0 / [13552.954415] z_wr_int_4 19884 5347744.075588 357 100 0.000000 9.542233 0.000000 0 / [13552.954415] z_wr_int_4 19885 5347744.075563 357 100 0.000000 9.516269 0.000000 0 / [13552.954415] z_wr_int_4 19886 5347744.075598 357 100 0.000000 9.523671 0.000000 0 / [13552.954415] z_wr_int_4 19887 5347744.075577 357 100 0.000000 9.791491 0.000000 0 / [13552.954415] z_wr_int_4 19888 5347744.075840 357 100 0.000000 9.379751 0.000000 0 / [13552.954415] z_wr_int_4 19889 5347744.075983 357 100 0.000000 9.738751 0.000000 0 / [13552.954415] z_wr_int_4 19890 5347744.076020 357 100 0.000000 9.297979 0.000000 0 / [13552.954415] z_wr_int_4 19891 5347744.075889 357 100 0.000000 9.387181 0.000000 0 / [13552.954415] z_wr_int_4 19894 5347744.075570 356 100 0.000000 9.917230 0.000000 0 / [13552.954415] z_wr_int_4 19895 5347744.075566 356 100 0.000000 9.530309 0.000000 0 / [13552.954415] z_wr_int_5 19896 5347744.075580 360 100 0.000000 9.578101 0.000000 0 / [13552.954415] z_wr_int_5 19897 5347744.075560 360 100 0.000000 9.172256 0.000000 0 / [13552.954415] z_wr_int_5 19898 5347744.075613 360 100 0.000000 9.590014 0.000000 0 / [13552.954415] z_wr_int_5 19899 5347744.075562 360 100 0.000000 9.918360 0.000000 0 / [13552.954415] z_wr_int_5 19900 5347744.075881 360 100 0.000000 9.826756 0.000000 0 / [13552.954415] z_wr_int_5 19901 5347744.075904 360 100 0.000000 9.609622 0.000000 0 / [13552.954415] z_wr_int_5 19902 5347744.075932 360 100 0.000000 9.476154 0.000000 0 / [13552.954415] z_wr_int_5 19903 5347744.075919 360 100 0.000000 9.526866 0.000000 0 / [13552.954415] z_wr_int_5 19907 5347744.075550 359 100 0.000000 9.544137 0.000000 0 / [13552.954415] z_wr_int_6 19910 5348758.960237 352 100 0.000000 9.338374 0.000000 0 / [13552.954415] z_wr_int_6 19911 5347744.075574 352 100 0.000000 9.314116 0.000000 0 / [13552.954415] z_wr_int_6 19912 5347744.075649 352 100 0.000000 9.254229 0.000000 0 / [13552.954415] z_wr_int_6 19913 5347744.075609 352 100 0.000000 10.581237 0.000000 0 / [13552.954415] z_wr_int_6 19914 5347744.075630 352 100 0.000000 9.521651 0.000000 0 / [13552.954415] z_wr_int_6 19915 5347744.075687 352 100 0.000000 9.253855 0.000000 0 / [13552.954415] z_wr_int_6 19916 5347744.075983 352 100 0.000000 9.258623 0.000000 0 / [13552.954415] z_wr_int_6 19917 5347744.076028 352 100 0.000000 9.455677 0.000000 0 / [13552.954415] z_wr_int_6 19918 5347744.076202 352 100 0.000000 9.529398 0.000000 0 / [13552.954415] z_wr_int_7 19920 5347744.075585 355 100 0.000000 16.417914 0.000000 0 / [13552.954415] z_wr_int_7 19921 5347744.075551 355 100 0.000000 9.674237 0.000000 0 / [13552.954415] z_wr_int_7 19922 5347744.075572 355 100 0.000000 10.478997 0.000000 0 / [13552.954415] z_wr_int_7 19923 5347744.075671 355 100 0.000000 9.494746 0.000000 0 / [13552.954415] z_wr_int_7 19925 5347713.452049 354 100 0.000000 9.276263 0.000000 0 / [13552.954415] z_wr_int_7 19926 5347714.088316 354 100 0.000000 9.511661 0.000000 0 / [13552.954415] z_wr_int_7 19927 5347744.075614 354 100 0.000000 9.317761 0.000000 0 / [13552.954415] z_wr_int_7 19928 5347744.075575 354 100 0.000000 9.463458 0.000000 0 / [13552.954415] z_wr_int_7 19929 5347744.075729 354 100 0.000000 9.284397 0.000000 0 / [13552.954415] z_wr_int_7 19930 5347744.075691 354 100 0.000000 10.570341 0.000000 0 / [13552.954415] z_wr_int_7 19931 5347744.075578 354 100 0.000000 10.810737 0.000000 0 / [13552.954415] z_ioctl_int 20037 5347744.075388 688 100 0.000000 5.340673 0.000000 0 / [13552.954415] z_zvol 20038 5025147.013774 12 120 0.000000 1.629265 0.000000 0 / [13552.954415] ldlm_cb00_001 20228 5050481.834848 64 120 0.000000 3.553962 0.000000 0 / [13552.954415] ldlm_cn00_000 20229 5044029.845118 12 120 0.000000 0.495570 0.000000 0 / [13552.954415] mdt_out00_000 20250 5025940.160463 2 120 0.000000 0.100632 0.000000 0 / [13552.954415] mdt_out00_001 20251 5025946.198981 2 120 0.000000 0.038521 0.000000 0 / [13552.954415] mdt_seqs_0000 20252 5025952.257237 2 120 0.000000 0.058259 0.000000 0 / [13552.954415] mdt_seqs_0001 20253 5025958.291117 2 120 0.000000 0.033883 0.000000 0 / [13552.954415] mdt_seqm_0000 20254 5025964.347894 2 120 0.000000 0.056780 0.000000 0 / [13552.954415] mdt_seqm_0001 20255 5025970.381776 2 120 0.000000 0.033885 0.000000 0 / [13552.954415] mdt_fld_0000 20256 5025976.442534 2 120 0.000000 0.060761 0.000000 0 / [13552.954415] mdt_fld_0001 20257 5025982.476843 2 120 0.000000 0.034312 0.000000 0 / [13552.954415] mgs_lustre_noti 20258 5026067.883854 5 118 0.000000 0.111936 0.000000 0 / [13552.954415] qmt_reba_lustre 20262 5044455.021689 11 120 0.000000 0.266466 0.000000 0 / [13552.954415] osp-syn-0-0 20264 5039553.144879 18 120 0.000000 1.564752 0.000000 0 / [13552.954415] osp-syn-1-0 20266 5026047.031044 13 120 0.000000 0.697134 0.000000 0 / [13552.954415] tx_commit_cb 20271 5347744.899054 318 120 0.000000 2784.250613 0.000000 0 / [13552.954415] tdtd-0 20275 5026097.914829 2 120 0.000000 0.031082 0.000000 0 / [13552.954415] kworker/1:0 25691 5348707.371096 6 120 0.000000 0.019352 0.000000 0 / [13552.954415] sshd 25854 5348538.385813 28 120 0.000000 14.562289 0.000000 0 / [13552.954415] R bash 25856 5348770.809867 45 120 0.000000 20.708053 0.000000 0 / [13552.954415] kworker/1:1 25900 5348758.963968 4 120 0.000000 0.010094 0.000000 0 / [13553.269241] Console [trevis-41vm7] disconnected from at 04-05 19:50. Console [trevis-41vm7] connected to at 04-05 19:50. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x35e0c000-0x36efdfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 528MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=420143da-7f83-49b4-8a24-6cffb1d410f7 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1701832k/2097140k available (6765k kernel code, 400k absent, 394908k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008196] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.012355] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014909] Mount-cache hash table entries: 4096 [ 0.016296] Initializing cgroup subsys memory [ 0.017010] Initializing cgroup subsys devices [ 0.019005] Initializing cgroup subsys freezer [ 0.020005] Initializing cgroup subsys net_cls [ 0.021005] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.024008] Initializing cgroup subsys hugetlb [ 0.025005] Initializing cgroup subsys pids [ 0.026005] Initializing cgroup subsys net_prio [ 0.028120] Disabled fast string operations [ 0.029019] mce: CPU supports 10 MCE banks [ 0.030057] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] tlb_flushall_shift: 6 [ 0.034181] Freeing SMP alternatives: 28k freed [ 0.040519] ACPI: Core revision 20130517 [ 0.042629] ACPI: All ACPI Tables successfully acquired [ 0.044026] ftrace: allocating 25813 entries in 101 pages [ 0.052282] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.055004] smpboot: Max logical packages: 2 [ 0.057243] Enabling x2apic [ 0.058000] Enabled x2apic [ 0.058004] Switched APIC routing to physical x2apic. [ 0.061270] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.062003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.065025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.067582] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.070041] Brought up 2 CPUs [ 0.070031] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.072005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.073601] devtmpfs: initialized [ 0.075879] EVM: security.selinux [ 0.076005] EVM: security.ima [ 0.077004] EVM: security.capability [ 0.078753] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.079029] pinctrl core: initialized pinctrl subsystem [ 0.080167] NET: Registered protocol family 16 [ 0.081273] ACPI: bus type PCI registered [ 0.082005] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.083087] PCI: Using configuration type 1 for base access [ 0.087088] ACPI: Added _OSI(Module Device) [ 0.088013] ACPI: Added _OSI(Processor Device) [ 0.089006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.090009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.092685] ACPI: Interpreter enabled [ 0.093013] ACPI: (supports S0 S5) [ 0.094005] ACPI: Using IOAPIC for interrupt routing [ 0.095017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.098621] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.099010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.100010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.101018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.102244] acpiphp: Slot [1] registered [ 0.103031] acpiphp: Slot [2] registered [ 0.104022] acpiphp: Slot [3] registered [ 0.105009] acpiphp: Slot [4] registered [ 0.106022] acpiphp: Slot [5] registered [ 0.107022] acpiphp: Slot [6] registered [ 0.108022] acpiphp: Slot [7] registered [ 0.109022] acpiphp: Slot [8] registered [ 0.110030] acpiphp: Slot [9] registered [ 0.111022] acpiphp: Slot [10] registered [ 0.112022] acpiphp: Slot [11] registered [ 0.113025] acpiphp: Slot [12] registered [ 0.114022] acpiphp: Slot [13] registered [ 0.115021] acpiphp: Slot [14] registered [ 0.116021] acpiphp: Slot [15] registered [ 0.117031] acpiphp: Slot [16] registered [ 0.118030] acpiphp: Slot [17] registered [ 0.119028] acpiphp: Slot [18] registered [ 0.120021] acpiphp: Slot [19] registered [ 0.121021] acpiphp: Slot [20] registered [ 0.122022] acpiphp: Slot [21] registered [ 0.123022] acpiphp: Slot [22] registered [ 0.124029] acpiphp: Slot [23] registered [ 0.125022] acpiphp: Slot [24] registered [ 0.126021] acpiphp: Slot [25] registered [ 0.127022] acpiphp: Slot [26] registered [ 0.128021] acpiphp: Slot [27] registered [ 0.129029] acpiphp: Slot [28] registered [ 0.130022] acpiphp: Slot [29] registered [ 0.131022] acpiphp: Slot [30] registered [ 0.132029] acpiphp: Slot [31] registered [ 0.133265] PCI host bridge to bus 0000:00 [ 0.134006] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.135006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.136005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.137006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.138005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.139981] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.140006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.141005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.142005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.143812] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.144013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.161671] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.163076] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.164800] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.166069] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.167775] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.169276] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.170295] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.171012] vgaarb: loaded [ 0.172005] vgaarb: bridge control possible 0000:00:02.0 [ 0.173212] SCSI subsystem initialized [ 0.174052] ACPI: bus type USB registered [ 0.175037] usbcore: registered new interface driver usbfs [ 0.176017] usbcore: registered new interface driver hub [ 0.177050] usbcore: registered new device driver usb [ 0.178194] PCI: Using ACPI for IRQ routing [ 0.179350] NetLabel: Initializing [ 0.180007] NetLabel: domain hash size = 128 [ 0.181004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.182057] NetLabel: unlabeled traffic allowed by default [ 0.183150] Switched to clocksource kvm-clock [ 0.193563] pnp: PnP ACPI init [ 0.194688] ACPI: bus type PNP registered [ 0.196396] pnp: PnP ACPI: found 5 devices [ 0.197626] ACPI: bus type PNP unregistered [ 0.207448] NET: Registered protocol family 2 [ 0.209095] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.211504] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.213241] TCP: Hash tables configured (established 16384 bind 16384) [ 0.214861] TCP: reno registered [ 0.215991] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.217504] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.219177] NET: Registered protocol family 1 [ 0.220416] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.221787] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.223149] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.224846] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.226514] Unpacking initramfs... [ 0.514477] Freeing initrd memory: 17352k freed [ 0.520373] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.521966] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.523888] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.525409] Initialise system trusted keyring [ 0.526702] audit: initializing netlink socket (disabled) [ 0.528107] type=2000 audit(1491421876.831:1): initialized [ 0.549038] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.550550] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.553134] zpool: loaded [ 0.554202] zbud: loaded [ 0.555472] VFS: Disk quotas dquot_6.5.2 [ 0.556718] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.558488] msgmni has been set to 3357 [ 0.559759] Key type big_key registered [ 0.562487] NET: Registered protocol family 38 [ 0.563776] Key type asymmetric registered [ 0.565010] Asymmetric key parser 'x509' registered [ 0.566341] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.568685] io scheduler noop registered [ 0.569905] io scheduler deadline registered (default) [ 0.571285] io scheduler cfq registered [ 0.572592] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.573999] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.575695] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.577971] ACPI: Power Button [PWRF] [ 0.579354] GHES: HEST is not enabled! [ 0.580642] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.603469] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.605536] Non-volatile memory driver v1.3 [ 0.606779] Linux agpgart interface v0.103 [ 0.608159] crash memory driver: version 1.1 [ 0.609482] rdac: device handler registered [ 0.610785] hp_sw: device handler registered [ 0.627715] emc: device handler registered [ 0.628954] alua: device handler registered [ 0.630229] libphy: Fixed MDIO Bus: probed [ 0.631549] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.633078] ehci-pci: EHCI PCI platform driver [ 0.634351] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.635826] ohci-pci: OHCI PCI platform driver [ 0.637105] uhci_hcd: USB Universal Host Controller Interface driver [ 0.638834] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.640291] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.642584] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.643996] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.645515] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.647115] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.649428] usb usb1: Product: UHCI Host Controller [ 0.650811] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.653181] usb usb1: SerialNumber: 0000:00:01.2 [ 0.654650] hub 1-0:1.0: USB hub found [ 0.655892] hub 1-0:1.0: 2 ports detected [ 0.657284] usbcore: registered new interface driver usbserial [ 0.658795] usbcore: registered new interface driver usbserial_generic [ 0.660354] usbserial: USB Serial support registered for generic [ 0.661913] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.664934] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.666298] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.667805] mousedev: PS/2 mouse device common for all mice [ 0.669562] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.672206] rtc_cmos 00:00: RTC can wake from S4 [ 0.672735] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.672985] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.678787] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.680295] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.681778] cpuidle: using governor menu [ 0.683057] hidraw: raw HID events driver (C) Jiri Kosina [ 0.684493] usbcore: registered new interface driver usbhid [ 0.685869] usbhid: USB HID core driver [ 0.687074] drop_monitor: Initializing network drop monitor service [ 0.688606] TCP: cubic registered [ 0.689701] Initializing XFRM netlink socket [ 0.691034] NET: Registered protocol family 10 [ 0.692604] NET: Registered protocol family 17 [ 0.694072] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.695562] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.698027] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.701149] Loading compiled-in X.509 certificates [ 0.702467] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.704993] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.708056] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.710618] registered taskstats version 1 [ 0.714691] Key type trusted registered [ 0.718081] Key type encrypted registered [ 0.721341] IMA: No TPM chip found, activating TPM-bypass! [ 0.723049] rtc_cmos 00:00: setting system clock to 2017-04-05 19:51:15 UTC (1491421875) [ 0.726234] Freeing unused kernel memory: 1680k freed [ 0.732432] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.736884] systemd[1]: Detected virtualization kvm. [ 0.738269] systemd[1]: Detected architecture x86-64. [ 0.739654] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.741867] systemd[1]: Set hostname to . [ OK ] Reached target Timers. [ 0.774782] systemd[1]: Reached target Timers. [ 0.776088] systemd[1]: Starting Timers. [ OK ] Created slice Root Slice. [ 0.778002] systemd[1]: Created slice Root Slice. [ 0.779333] systemd[1]: Starting Root Slice. [ OK ] Reached target Local File Systems. [ 0.781231] systemd[1]: Reached target Local File Systems. [ 0.782638] systemd[1]: Starting Local File Systems. [ OK ] Listening on udev Control Socket. [ 0.784640] systemd[1]: Listening on udev Control Socket. [ 0.786035] systemd[1]: Starting udev Control Socket. [ OK ] Listening on Journal Socket. [ 0.788032] systemd[1]: Listening on Journal Socket. [ 0.789381] systemd[1]: Starting Journal Socket. [ OK ] Reached target Swap. [ 0.791250] systemd[1]: Reached target Swap. [ 0.792507] systemd[1]: Starting Swap. [ OK ] Listening on udev Kernel Socket. [ 0.794334] systemd[1]: Listening on udev Kernel Socket. [ 0.795717] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 0.797624] systemd[1]: Reached target Sockets. [ 0.798926] systemd[1]: Starting Sockets. [ OK ] Created slice System Slice. [ 0.800841] systemd[1]: Created slice System Slice. [ 0.802191] systemd[1]: Starting System Slice. [ 0.803893] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ 0.806440] systemd[1]: Reached target Slices. [ 0.807726] systemd[1]: Starting Slices. [ 0.809640] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ 0.887682] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.890087] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 0.892597] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Started Load Kernel Modules. [ 0.902370] systemd[1]: Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 0.905411] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 0.908619] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ 0.912437] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ 0.917938] systemd[1]: Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. Starting dracut initqueue hook... [ 1.067086] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.068855] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.071723] [drm] Initialized drm 1.1.0 20060810 [ 1.072653] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.078402] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.080315] FDC 0 is a S82078B [ 1.081776] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.083150] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.087748] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.102412] scsi host0: ata_piix [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.113088] scsi host1: ata_piix [ 1.114289] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.115835] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.119415] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.120952] [TTM] Initializing pool allocator [ 1.122441] [TTM] Initializing DMA pool allocator [ 1.124349] [drm] fb mappable at 0xF0000000 [ 1.137209] [drm] vram aper at 0xF0000000 [ 1.138413] [drm] size 33554432 [ 1.139515] [drm] fb depth is 24 %G[ 1.140681] [drm] pitch is 3072 %G[ 1.143386] fbcon: cirrusdrmfb (fb0) is primary device [ 1.169869] Console: switching to colour frame buffer device 128x48 [ 1.181612] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.186024] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 1.203041] vda: vda1 vda2 vda4 [ OK ] Found device /dev/disk/by-uuid/420143da-7f83-49b4-8a24-6cffb1d410f7. Starting File System Check on /dev/...a-7f83-49b4-8a24-6cffb1d410f7... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 1.521050] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Started File System Check on /dev/d...3da-7f83-49b4-8a24-6cffb1d410f7. Mounting /sysroot... [ 4.130101] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 4.131599] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Load Kernel Modules. Stopping Load Kernel Modules... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 4.480120] systemd-journald[98]: Received SIGTERM from PID 1 (systemd). [ 5.799882] SELinux: Disabled at runtime. [ 5.838029] type=1404 audit(1491421880.614:2): selinux=0 auid=4294967295 ses=4294967295 [ 6.133040] ip_tables: (C) 2000-2006 Netfilter Core Team [ 6.134532] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Load Kernel Modules... Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Mounting Debug File System... Mounting Huge Pages File System... [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on udev Control Socket. Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Reached target Encrypted Volumes. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Stopped Journal Service. Starting Journal Service... Starting Remount Root and Kernel File Systems... [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on LVM2 poll daemon socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Read and set NIS domainname from /etc/sysconfig/network... Mounting NFSD configuration filesystem... Mounting POSIX Message Queue File System... [ OK ] Started Load Kernel Modules. Starting Apply Kernel Variables... [ OK ] Started Apply Kernel Variables. [ 7.853819] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... Starting Load/Save Random Seed... Starting udev Coldplug all Devices... [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Load/Save Random Seed. Starting Create Static Device Nodes in /dev... [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Configure read-only root support. [ OK ] Started udev Coldplug all Devices. [ 8.476898] RPC: Registered named UNIX socket transport module. [ 8.478448] RPC: Registered udp transport module. [ 8.479908] RPC: Registered tcp transport module. [ 8.481333] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ 9.129267] systemd-journald[349]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ 9.188424] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ OK ] Mounted NFSD configuration filesystem. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started udev Kernel Device Manager. [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. Activating swap /dev/disk/by-uuid/d...6-1e76-4988-acad-74e281f3721c... [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 253:4... [ 9.803253] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ OK ] Activated swap /dev/disk/by-uuid/d3560f36-1e76-4988-acad-74e281f3721c. [ OK ] Reached target Swap. %G%G[ 9.933875] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 10.099185] device-mapper: uevent: version 1.0.3 [ 10.101137] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Preprocess NFS configuration... [ 10.176080] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 10.401575] ppdev: user-space parallel port driver [ OK ] Started Preprocess NFS configuration. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... Mounting RPC Pipe File System... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Mounted RPC Pipe File System. [ 11.057229] AES CTR mode by8 optimization enabled [ 11.061616] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 11.079659] alg: No test for crc32 (crc32-pclmul) [ 11.203517] intel_rapl: no valid rapl domains found in package 0 [ 11.226685] intel_rapl: no valid rapl domains found in package 0 [ 11.254978] intel_powerclamp: No package C-state available [ 11.260538] intel_powerclamp: No package C-state available[ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting GSSAPI Proxy Daemon... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting Authorization Manager... Starting Dump dmesg to /var/log/dmesg... Starting System Security Services Daemon... Starting NTP client/server... [ OK ] Started D-Bus System Message Bus. [ 11.619847] type=1305 audit(1491421886.395:3): audit_pid=436 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Dump dmesg to /var/log/dmesg. Starting RPC security service for NFS client and server... [ OK ] Started NTP client/server. [ OK ] Found device /dev/lvm-Role_MDS/forcemnt. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ 13.437095] spl: loading out-of-tree module taints kernel. [ 13.438680] spl: module verification failed: signature and/or required key missing - tainting kernel [ 13.444746] SPL: Loaded module v0.6.5.9-1 [ 13.446578] znvpair: module license 'CDDL' taints kernel. [ 13.448024] Disabling lock debugging due to kernel taint [ 13.468234] ZFS: Loaded module v0.6.5.9-1, ZFS pool version 5000, ZFS filesystem version 5 [ OK ] Started LVM2 PV scan on device 253:4. [ 13.673366] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ OK ] Started Authorization Manager. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ 13.977446] Loading iSCSI transport class v2.0-870. Starting Hostname Service... [ OK ] Started Hostname Service. [ 14.092633] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 14.094546] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 14.145127] iscsi: registered transport (iser) [ 14.338043] RPC: Registered rdma transport module. [ 14.339602] RPC: Registered rdma backchannel transport module. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting Xinetd A Powerful Replacement For Inetd... Starting OpenSSH server daemon... Starting Postfix Mail Transport Agent... Starting Dynamic System Tuning Daemon... Starting System Logging Service... Starting Notify NFS peers of a restart... Starting MUNGE authentication service... [ OK ] Reached target Network is Online. Mounting /scratch... Mounting /mnt/forcemnt... [ OK ] Started System Logging Service. [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started Notify NFS peers of a restart. [ 19.954031] EXT4-fs (dm-2): mounting ext3 file system using the ext4 subsystem [ OK ] Started MUNGE authentication service. [ 19.967111] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /mnt/forcemnt. [ 20.085364] FS-Cache: Loaded [ 20.291719] FS-Cache: Netfs 'nfs' registered for caching [ 20.385384] Key type dns_resolver registered [ 20.608485] NFS: Registering the id_resolver key type [ 20.609970] Key type id_resolver registered [ 20.611464] Key type id_legacy registered [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Permit User Sessions... Starting Crash recovery kernel arming... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ OK ] Started Job spooling tools. Starting Job spooling tools... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: Console [trevis-41vm7] disconnected from at 04-05 20:46. Console [trevis-41vm7] connected to at 04-05 20:47. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005035] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008196] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011877] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014339] Mount-cache hash table entries: 4096 [ 0.015301] Initializing cgroup subsys memory [ 0.017010] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.019006] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.023006] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.025005] Initializing cgroup subsys net_prio [ 0.027087] Disabled fast string operations [ 0.028020] mce: CPU supports 10 MCE banks [ 0.029058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] tlb_flushall_shift: 6 [ 0.033180] Freeing SMP alternatives: 28k freed [ 0.039049] ACPI: Core revision 20130517 [ 0.041115] ACPI: All ACPI Tables successfully acquired [ 0.042026] ftrace: allocating 25811 entries in 101 pages [ 0.051273] smpboot: Max logical packages: 2 [ 0.052005] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.055258] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056005] Switched APIC routing to physical x2apic. [ 0.058524] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.059003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.062025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.064598] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.317040] Brought up 2 CPUs [ 0.317016] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.319005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.320585] devtmpfs: initialized [ 0.323954] EVM: security.selinux [ 0.324005] EVM: security.ima [ 0.325004] EVM: security.capability [ 0.326737] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.327010] pinctrl core: initialized pinctrl subsystem [ 0.328157] NET: Registered protocol family 16 [ 0.329319] ACPI: bus type PCI registered [ 0.330006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.331090] PCI: Using configuration type 1 for base access [ 0.335087] ACPI: Added _OSI(Module Device) [ 0.336013] ACPI: Added _OSI(Processor Device) [ 0.337006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.338005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.340674] ACPI: Interpreter enabled [ 0.341013] ACPI: (supports S0 S5) [ 0.342005] ACPI: Using IOAPIC for interrupt routing [ 0.343028] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.346642] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.347010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.348010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.349018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.350256] acpiphp: Slot [1] registered [ 0.351031] acpiphp: Slot [2] registered [ 0.352022] acpiphp: Slot [3] registered [ 0.353022] acpiphp: Slot [4] registered [ 0.354022] acpiphp: Slot [5] registered [ 0.355022] acpiphp: Slot [6] registered [ 0.356026] acpiphp: Slot [7] registered [ 0.357022] acpiphp: Slot [8] registered [ 0.358029] acpiphp: Slot [9] registered [ 0.359022] acpiphp: Slot [10] registered [ 0.360022] acpiphp: Slot [11] registered [ 0.361021] acpiphp: Slot [12] registered [ 0.362021] acpiphp: Slot [13] registered [ 0.363021] acpiphp: Slot [14] registered [ 0.364022] acpiphp: Slot [15] registered [ 0.365029] acpiphp: Slot [16] registered [ 0.366030] acpiphp: Slot [17] registered [ 0.367022] acpiphp: Slot [18] registered [ 0.368021] acpiphp: Slot [19] registered [ 0.369021] acpiphp: Slot [20] registered [ 0.370022] acpiphp: Slot [21] registered [ 0.371021] acpiphp: Slot [22] registered [ 0.372029] acpiphp: Slot [23] registered [ 0.373022] acpiphp: Slot [24] registered [ 0.374022] acpiphp: Slot [25] registered [ 0.375021] acpiphp: Slot [26] registered [ 0.376033] acpiphp: Slot [27] registered [ 0.377022] acpiphp: Slot [28] registered [ 0.378022] acpiphp: Slot [29] registered [ 0.379029] acpiphp: Slot [30] registered [ 0.380022] acpiphp: Slot [31] registered [ 0.381261] PCI host bridge to bus 0000:00 [ 0.382007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.383006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.384006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.385005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.386006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.387967] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.391807] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.392013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.409648] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.411075] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.412760] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.414067] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.415729] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.417201] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.418263] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.419014] vgaarb: loaded [ 0.420006] vgaarb: bridge control possible 0000:00:02.0 [ 0.421214] SCSI subsystem initialized [ 0.422052] ACPI: bus type USB registered [ 0.423039] usbcore: registered new interface driver usbfs [ 0.424017] usbcore: registered new interface driver hub [ 0.425052] usbcore: registered new device driver usb [ 0.426204] PCI: Using ACPI for IRQ routing [ 0.427340] NetLabel: Initializing [ 0.428008] NetLabel: domain hash size = 128 [ 0.429004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.430054] NetLabel: unlabeled traffic allowed by default [ 0.431155] Switched to clocksource kvm-clock [ 0.441350] pnp: PnP ACPI init [ 0.442426] ACPI: bus type PNP registered [ 0.444071] pnp: PnP ACPI: found 5 devices [ 0.445238] ACPI: bus type PNP unregistered [ 0.454987] NET: Registered protocol family 2 [ 0.456556] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.458845] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.460527] TCP: Hash tables configured (established 16384 bind 16384) [ 0.462093] TCP: reno registered [ 0.463159] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.464601] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.466199] NET: Registered protocol family 1 [ 0.467435] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.468810] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.470156] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.471830] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.473479] Unpacking initramfs... [ 4.604996] Freeing initrd memory: 42360k freed [ 4.624299] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.625898] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.627639] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.629066] Initialise system trusted keyring [ 4.630284] audit: initializing netlink socket (disabled) [ 4.631613] type=2000 audit(1491425259.531:1): initialized [ 4.652494] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.653992] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.656570] zpool: loaded [ 4.657768] zbud: loaded [ 4.659295] VFS: Disk quotas dquot_6.5.2 [ 4.660486] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.662127] msgmni has been set to 3357 [ 4.663322] Key type big_key registered [ 4.665664] NET: Registered protocol family 38 [ 4.666884] Key type asymmetric registered [ 4.668038] Asymmetric key parser 'x509' registered [ 4.669380] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.671883] io scheduler noop registered [ 4.673032] io scheduler deadline registered (default) [ 4.674344] io scheduler cfq registered [ 4.675646] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.677011] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.678773] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.680990] ACPI: Power Button [PWRF] [ 4.682272] GHES: HEST is not enabled! [ 4.683462] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.706694] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.708404] Non-volatile memory driver v1.3 [ 4.709594] Linux agpgart interface v0.103 [ 4.710869] crash memory driver: version 1.1 [ 4.712122] rdac: device handler registered [ 4.713355] hp_sw: device handler registered [ 4.714963] emc: device handler registered [ 4.716249] alua: device handler registered [ 4.717486] libphy: Fixed MDIO Bus: probed [ 4.718716] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.720205] ehci-pci: EHCI PCI platform driver [ 4.721428] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.722846] ohci-pci: OHCI PCI platform driver [ 4.724057] uhci_hcd: USB Universal Host Controller Interface driver [ 4.725777] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.727145] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.729326] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.730658] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.732105] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.733625] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.735799] usb usb1: Product: UHCI Host Controller [ 4.737094] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.738736] usb usb1: SerialNumber: 0000:00:01.2 [ 4.740607] hub 1-0:1.0: USB hub found [ 4.742128] hub 1-0:1.0: 2 ports detected [ 4.743568] usbcore: registered new interface driver usbserial [ 4.744972] usbcore: registered new interface driver usbserial_generic [ 4.746516] usbserial: USB Serial support registered for generic [ 4.747970] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.750879] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.752288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.753822] mousedev: PS/2 mouse device common for all mice [ 4.755522] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.758107] rtc_cmos 00:00: RTC can wake from S4 [ 4.758636] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.758874] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.766099] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.768245] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.769798] cpuidle: using governor menu [ 4.771081] hidraw: raw HID events driver (C) Jiri Kosina [ 4.772604] usbcore: registered new interface driver usbhid [ 4.773969] usbhid: USB HID core driver [ 4.775178] drop_monitor: Initializing network drop monitor service [ 4.776726] TCP: cubic registered [ 4.777843] Initializing XFRM netlink socket [ 4.779136] NET: Registered protocol family 10 [ 4.780710] NET: Registered protocol family 17 [ 4.782185] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.783603] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.785042] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.787680] Loading compiled-in X.509 certificates [ 4.788994] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.791688] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.794807] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.797391] registered taskstats version 1 [ 4.801135] Key type trusted registered [ 4.804651] Key type encrypted registered [ 4.807885] IMA: No TPM chip found, activating TPM-bypass! [ 4.809600] rtc_cmos 00:00: setting system clock to 2017-04-05 20:47:38 UTC (1491425258) [ 4.812729] Freeing unused kernel memory: 1680k freed [ 4.819016] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.839301] systemd[1]: Detected virtualization kvm. [ 4.840657] systemd[1]: Detected architecture x86-64. [ 4.841970] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.844069] systemd[1]: No hostname configured. [ 4.845315] systemd[1]: Set hostname to . [ 4.846649] systemd[1]: Initializing machine ID from KVM UUID. [ 4.872049] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Reached target Local File Systems. [ 4.875445] systemd[1]: Reached target Local File Systems. [ 4.876920] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 4.879416] systemd[1]: Reached target Timers. [ 4.880912] systemd[1]: Starting Timers. [ OK ] Reached target Swap. [ 4.882867] systemd[1]: Reached target Swap. [ 4.884358] systemd[1]: Starting Swap. [ OK ] Reached target Encrypted Volumes. [ 4.886194] systemd[1]: Reached target Encrypted Volumes. [ 4.887565] systemd[1]: Starting Encrypted Volumes. [ OK ] Created slice Root Slice. [ 4.889545] systemd[1]: Created slice Root Slice. [ 4.890796] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 4.892620] systemd[1]: Listening on udev Kernel Socket. [ 4.893952] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on Journal Socket. [ 4.895861] systemd[1]: Listening on Journal Socket. [ 4.897175] systemd[1]: Starting Journal Socket. [ OK ] Created slice System Slice. [ 4.899072] systemd[1]: Created slice System Slice. [ 4.900333] systemd[1]: Starting System Slice. [ 4.901988] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.904570] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 4.908656] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.912861] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Listening on udev Control Socket. [ 4.915621] systemd[1]: Listening on udev Control Socket. [ 4.915641] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Slices. [ 4.918925] systemd[1]: Reached target Slices. [ 4.920151] systemd[1]: Starting Slices. [ 4.921832] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ 4.926766] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 4.995881] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.071959] dracut-cmdline[89]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.206506] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.211429] Loading iSCSI transport class v2.0-870. [ 5.215972] iscsi: registered transport (tcp) [ 5.251285] FDC 0 is a S82078B [ 5.257801] iBFT detected. [ 5.258900] iBFT has incorrect checksum (0x54)! [ 5.269241] md: raid0 personality registered for level 0 [ 5.272918] md: raid1 personality registered for level 1 [ 5.276261] async_tx: api initialized (async) [ 5.278272] xor: automatically using best checksumming function: [ 5.289006] avx : 22412.000 MB/sec [ 5.309009] raid6: sse2x1 gen() 7921 MB/s [ 5.327008] raid6: sse2x2 gen() 9945 MB/s [ 5.345008] raid6: sse2x4 gen() 11539 MB/s [ 5.346201] raid6: using algorithm sse2x4 gen() (11539 MB/s) [ 5.347553] raid6: using ssse3x2 recovery algorithm [ 5.353322] md: raid6 personality registered for level 6 [ 5.354648] md: raid5 personality registered for level 5 [ 5.355961] md: raid4 personality registered for level 4 [ 5.362705] md: raid10 personality registered for level 10 [ 5.366047] md: linear personality registered for level -1 [ 5.371639] device-mapper: uevent: version 1.0.3 [ 5.372938] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.387716] device-mapper: multipath: version 1.11.0 loaded [ 5.391077] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.422136] RPC: Registered named UNIX socket transport module. [ 5.423535] RPC: Registered udp transport module. [ 5.424767] RPC: Registered tcp transport module. [ 5.426006] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... Starting Show Plymouth Boot Screen... [ OK ] Mounted Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ 5.625129] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Reached target Sockets. [ 5.642517] scsi host0: ata_piix [ 5.644684] scsi host1: ata_piix [ OK ] Started Device-Mapper Multipath Device Controller. [ 5.649617] [drm] Initialized drm 1.1.0 20060810 [ 5.650266] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.650267] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Starting Open-iSCSI... [ OK ] Started Show Plymouth Boot Screen. [ OK ] Started Open-iSCSI. [ 5.656972] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 5.657791] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 5.660233] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 Starting dracut initqueue hook... [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 5.669166] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.671192] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 5.674063] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 5.676118] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.689328] AES CTR mode by8 optimization enabled [ 5.698193] 8021q: 802.1Q VLAN Support v1.8 [ 5.707746] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.709193] [TTM] Initializing pool allocator [ 5.710662] [TTM] Initializing DMA pool allocator [ 5.712929] [drm] fb mappable at 0xF0000000 [ 5.714120] [drm] vram aper at 0xF0000000 [ 5.715310] [drm] size 33554432 [ 5.716361] [drm] fb depth is 24 [ 5.717438] [drm] pitch is 3072 %G[ 5.718666] fbcon: cirrusdrmfb (fb0) is primary device [ 5.734484] vda: vda1 vda2 vda4 [ 5.751907] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 5.768134] alg: No test for crc32 (crc32-pclmul) [ 5.778037] Console: switching to colour frame buffer device 128x48 [ 5.790933] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device %G[ 5.794040] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 5.844526] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.562524] dracut-initqueue[564]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.562841] dracut-initqueue[564]: Dload Upload Total Spent Left Speed [ 8.657809] dracut-initqueue[564]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14606 100 14606 0 0 148k 0 --:--:-- --:--:-- --:--:-- 150k [ 9.173222] dracut-initqueue[564]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.174080] dracut-initqueue[564]: Dload Upload Total Spent Left Speed [ 12.534976] dracut-initqueue[564]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 26 317M 26 83.1M 0 0 100M 0 0:00:03 --:--:-- 0:00:03 100M 54 317M 54 172M 0 0 94.5M 0 0:00:03 0:00:01 0:00:02 94.5M 82 317M 82 260M 0 0 92.1M 0 0:00:03 0:00:02 0:00:01 92.1M 100 317M 100 317M 0 0 94.3M 0 0:00:03 0:00:03 --:--:-- 94.3M [ 12.639129] loop: module loaded [ OK ] Started dracut initqueue hook. Starting dracut pre-mount hook... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. Starting dracut mount hook... [ OK ] Reached target Initrd File Systems. [ 12.818700] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 14.842804] systemd-journald[85]: Received SIGTERM from PID 1 (systemd). [ 15.383573] type=1403 audit(1491425269.073:2): policy loaded auid=4294967295 ses=4294967295 [ 15.388884] systemd[1]: Successfully loaded SELinux policy in 262.192ms. [ 15.487865] systemd[1]: Relabelled /dev and /run in 18.961ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Reached target Timers. Mounting Temporary Directory... [ OK ] Reached target Swap. [ OK ] Created slice User and Session Slice. Mounting Debug File System... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Created slice system-anaconda\x2dtmux.slice. [ OK ] Reached target Slices. Starting Create list of required st... nodes for the current kernel... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Mounted Temporary Directory. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Remount Root and Kernel File Systems. Starting Rebuild Hardware Database... Starting Configure read-only root support... Starting Load/Save Random Seed... [ OK ] Started Apply Kernel Variables. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Configure read-only root support. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... Starting udev Wait for Complete Device Initialization... %G%G[ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... [ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Rebuild Journal Catalog... Starting Import network configuration from initramfs... [ OK ] Reached target Encrypted Volumes. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... Starting Login Service... [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting firewalld - dynamic firewall daemon... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... Starting pre-anaconda logging service... Starting Service enabling compressing RAM with zRam... Starting Dump dmesg to /var/log/dmesg... %G (BStarting installer, one moment...                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (B [anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (B anaconda 21.48.22.93-1 for CentOS Linux 7 started.  * installation log files are stored in /tmp during the installation  * shell is available on TTY2  * when reporting a bug add logs from /tmp as separate text/plain attachments 20:47:55 Running pre-installation scripts 20:47:56 Not asking for VNC because of an automated install 20:47:56 Not asking for VNC because text mode was explicitly asked for in kickstart Starting automated install.......................Checking software selectionGenerating updated storage configuration Checking storage configuration... ================================================================================ ================================================================================ Installation  1) [x] Language settings 2) [x] Time settings (English (United States)) (Etc/UTC timezone) 3) [x] Installation source 4) [x] Software selection (http://10.9.0.10/cblr/links/Ce (Custom software selected)  ntOS-7.3-x86_64) 6) [x] Kdump  5) [x] Installation Destination (Kdump is enabled)  (Custom partitioning selected) 8) [ ] User creation  7) [x] Network configuration (No user will be created)  (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Paths. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... Stopping Load/Save Random Seed... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... Unmounting /mnt/sysimage/proc... Unmounting /mnt/sysimage/sys/fs/selinux... Unmounting Temporary Directory... Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/run... Unmounting /mnt/sysimage/dev/shm... Unmounting Configuration File System... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Load/Save Random Seed. [ OK ] Unmounted /mnt/sysimage/proc. [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. Unmounting /mnt/sysimage/sys... [ OK ] Unmounted Temporary Directory. [ OK ] Unmounted /mnt/sysimage/dev/pts. [ OK ] Unmounted /mnt/sysimage/run. [ OK ] Unmounted /mnt/sysimage/dev/shm. [ OK ] Unmounted Configuration File System. Unmounting /mnt/sysimage/dev... [ OK ] Unmounted /mnt/sysimage/sys. [ OK ] Unmounted /mnt/sysimage/dev. Unmounting /mnt/sysimage... [ OK ] Deactivated swap /dev/zram1. [ OK ] Deactivated swap /dev/zram0. [ OK ] Started Restore /run/initramfs. [ OK ] Unmounted /mnt/sysimage. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 964.796661] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=0d9f39ef-f730-4d66-ba62-54980bd39277 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fd7000-0x367e3fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=0d9f39ef-f730-4d66-ba62-54980bd39277 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694556k/2097140k available (6765k kernel code, 400k absent, 402184k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.005030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008199] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011119] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013184] Mount-cache hash table entries: 4096 [ 0.014243] Initializing cgroup subsys memory [ 0.015012] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.018005] Initializing cgroup subsys net_cls [ 0.019005] Initializing cgroup subsys blkio [ 0.020005] Initializing cgroup subsys perf_event [ 0.022005] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.024005] Initializing cgroup subsys net_prio [ 0.025106] Disabled fast string operations [ 0.027019] mce: CPU supports 10 MCE banks [ 0.028059] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] tlb_flushall_shift: 6 [ 0.032171] Freeing SMP alternatives: 28k freed [ 0.037724] ACPI: Core revision 20130517 [ 0.039763] ACPI: All ACPI Tables successfully acquired [ 0.041015] ftrace: allocating 25813 entries in 101 pages [ 0.049190] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.050005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.052004] smpboot: Max logical packages: 2 [ 0.053254] Enabling x2apic [ 0.054000] Enabled x2apic [ 0.054004] Switched APIC routing to physical x2apic. [ 0.057404] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.058003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.061025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.063431] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.066036] Brought up 2 CPUs [ 0.066031] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.068005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.069503] devtmpfs: initialized [ 0.071860] EVM: security.selinux [ 0.072005] EVM: security.ima [ 0.073004] EVM: security.capability [ 0.074739] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.075027] pinctrl core: initialized pinctrl subsystem [ 0.076141] NET: Registered protocol family 16 [ 0.077272] ACPI: bus type PCI registered [ 0.078006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.079084] PCI: Using configuration type 1 for base access [ 0.083079] ACPI: Added _OSI(Module Device) [ 0.084013] ACPI: Added _OSI(Processor Device) [ 0.085006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.086005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.088663] ACPI: Interpreter enabled [ 0.089014] ACPI: (supports S0 S5) [ 0.090005] ACPI: Using IOAPIC for interrupt routing [ 0.091017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.094543] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.095010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.096009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.097018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.098239] acpiphp: Slot [1] registered [ 0.099030] acpiphp: Slot [2] registered [ 0.100022] acpiphp: Slot [3] registered [ 0.101022] acpiphp: Slot [4] registered [ 0.102022] acpiphp: Slot [5] registered [ 0.103035] acpiphp: Slot [6] registered [ 0.104021] acpiphp: Slot [7] registered [ 0.105021] acpiphp: Slot [8] registered [ 0.106032] acpiphp: Slot [9] registered [ 0.107022] acpiphp: Slot [10] registered [ 0.108021] acpiphp: Slot [11] registered [ 0.109021] acpiphp: Slot [12] registered [ 0.110021] acpiphp: Slot [13] registered [ 0.111021] acpiphp: Slot [14] registered [ 0.112026] acpiphp: Slot [15] registered [ 0.113028] acpiphp: Slot [16] registered [ 0.114029] acpiphp: Slot [17] registered [ 0.115022] acpiphp: Slot [18] registered [ 0.116034] acpiphp: Slot [19] registered [ 0.117022] acpiphp: Slot [20] registered [ 0.118021] acpiphp: Slot [21] registered [ 0.119021] acpiphp: Slot [22] registered [ 0.120028] acpiphp: Slot [23] registered [ 0.121022] acpiphp: Slot [24] registered [ 0.122021] acpiphp: Slot [25] registered [ 0.123021] acpiphp: Slot [26] registered [ 0.124021] acpiphp: Slot [27] registered [ 0.125028] acpiphp: Slot [28] registered [ 0.126022] acpiphp: Slot [29] registered [ 0.127021] acpiphp: Slot [30] registered [ 0.128028] acpiphp: Slot [31] registered [ 0.129255] PCI host bridge to bus 0000:00 [ 0.130006] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.131006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.132005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.133006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.134006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.135964] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.136006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.137005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.138005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.139801] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.140013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.157594] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.159076] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.160754] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.162070] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.163728] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.165198] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.166269] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.167012] vgaarb: loaded [ 0.167998] vgaarb: bridge control possible 0000:00:02.0 [ 0.168203] SCSI subsystem initialized [ 0.169050] ACPI: bus type USB registered [ 0.170036] usbcore: registered new interface driver usbfs [ 0.171016] usbcore: registered new interface driver hub [ 0.172048] usbcore: registered new device driver usb [ 0.173176] PCI: Using ACPI for IRQ routing [ 0.174339] NetLabel: Initializing [ 0.175007] NetLabel: domain hash size = 128 [ 0.176004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.177040] NetLabel: unlabeled traffic allowed by default [ 0.178139] Switched to clocksource kvm-clock [ 0.187027] pnp: PnP ACPI init [ 0.188094] ACPI: bus type PNP registered [ 0.189708] pnp: PnP ACPI: found 5 devices [ 0.190884] ACPI: bus type PNP unregistered [ 0.201117] NET: Registered protocol family 2 [ 0.202660] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.204912] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.206492] TCP: Hash tables configured (established 16384 bind 16384) [ 0.208033] TCP: reno registered [ 0.209110] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.210531] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.212110] NET: Registered protocol family 1 [ 0.213352] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.214743] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.216110] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.217762] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.219421] Unpacking initramfs... [ 0.613396] Freeing initrd memory: 24628k freed [ 0.628783] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.630415] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.632250] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.633691] Initialise system trusted keyring [ 0.634915] audit: initializing netlink socket (disabled) [ 0.636232] type=2000 audit(1491426247.718:1): initialized [ 0.657077] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.658538] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.661043] zpool: loaded [ 0.662041] zbud: loaded [ 0.663239] VFS: Disk quotas dquot_6.5.2 [ 0.664471] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666116] msgmni has been set to 3357 [ 0.667315] Key type big_key registered [ 0.669902] NET: Registered protocol family 38 [ 0.671123] Key type asymmetric registered [ 0.672288] Asymmetric key parser 'x509' registered [ 0.673575] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.675799] io scheduler noop registered [ 0.676952] io scheduler deadline registered (default) [ 0.678257] io scheduler cfq registered [ 0.679500] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.680857] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.682477] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.684616] ACPI: Power Button [PWRF] [ 0.685869] GHES: HEST is not enabled! [ 0.687097] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.709794] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.711540] Non-volatile memory driver v1.3 [ 0.712736] Linux agpgart interface v0.103 [ 0.714043] crash memory driver: version 1.1 [ 0.715298] rdac: device handler registered [ 0.716534] hp_sw: device handler registered [ 0.732222] emc: device handler registered [ 0.733400] alua: device handler registered [ 0.734595] libphy: Fixed MDIO Bus: probed [ 0.735819] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.737296] ehci-pci: EHCI PCI platform driver [ 0.738497] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.739927] ohci-pci: OHCI PCI platform driver [ 0.741154] uhci_hcd: USB Universal Host Controller Interface driver [ 0.742809] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.744157] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.746349] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.747668] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.749137] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.750645] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.752825] usb usb1: Product: UHCI Host Controller [ 0.754124] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.756380] usb usb1: SerialNumber: 0000:00:01.2 [ 0.757755] hub 1-0:1.0: USB hub found [ 0.758921] hub 1-0:1.0: 2 ports detected [ 0.760274] usbcore: registered new interface driver usbserial [ 0.761694] usbcore: registered new interface driver usbserial_generic [ 0.763207] usbserial: USB Serial support registered for generic [ 0.764669] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.767578] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.768941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.770464] mousedev: PS/2 mouse device common for all mice [ 0.772216] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.774818] rtc_cmos 00:00: RTC can wake from S4 [ 0.775460] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.779051] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.779301] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.783069] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.784591] cpuidle: using governor menu [ 0.785886] hidraw: raw HID events driver (C) Jiri Kosina [ 0.787374] usbcore: registered new interface driver usbhid [ 0.788744] usbhid: USB HID core driver [ 0.789943] drop_monitor: Initializing network drop monitor service [ 0.791486] TCP: cubic registered [ 0.792578] Initializing XFRM netlink socket [ 0.793856] NET: Registered protocol family 10 [ 0.795413] NET: Registered protocol family 17 [ 0.796798] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.798251] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.799692] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.802201] Loading compiled-in X.509 certificates [ 0.803531] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.806038] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.809113] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.811642] registered taskstats version 1 [ 0.815356] Key type trusted registered [ 0.818583] Key type encrypted registered [ 0.821756] IMA: No TPM chip found, activating TPM-bypass! [ 0.823445] rtc_cmos 00:00: setting system clock to 2017-04-05 21:04:07 UTC (1491426247) [ 0.826570] Freeing unused kernel memory: 1680k freed [ 0.832857] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.837196] systemd[1]: Detected virtualization kvm. [ 0.838594] systemd[1]: Detected architecture x86-64. [ 0.839957] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.842193] systemd[1]: Set hostname to . [ OK ] Reached target Local File Systems. [ 0.874868] systemd[1]: Reached target Local File Systems. [ 0.876241] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 0.878102] systemd[1]: Reached target Timers. [ 0.879339] systemd[1]: Starting Timers. [ OK ] Reached target Swap. [ 0.881053] systemd[1]: Reached target Swap. [ 0.882242] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 0.884050] systemd[1]: Created slice Root Slice. [ 0.885296] systemd[1]: Starting Root Slice. [ OK ] Created slice System Slice. [ 0.887150] systemd[1]: Created slice System Slice. [ 0.888409] systemd[1]: Starting System Slice. [ OK ] Listening on udev Control Socket. [ 0.890291] systemd[1]: Listening on udev Control Socket. [ 0.891630] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Slices. [ 0.893473] systemd[1]: Reached target Slices. [ 0.894688] systemd[1]: Starting Slices. [ OK ] Listening on Journal Socket. [ 0.896482] systemd[1]: Listening on Journal Socket. [ 0.897796] systemd[1]: Starting Journal Socket. [ 0.899561] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.901905] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ 0.984252] systemd[1]: Listening on udev Kernel Socket. [ 0.985594] systemd[1]: Starting udev Kernel Socket. [ 0.987274] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.989628] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 0.991988] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Sockets. [ 0.999321] systemd[1]: Reached target Sockets. [ 1.000584] systemd[1]: Starting Sockets. [ OK ] Started Apply Kernel Variables. [ 1.002929] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 1.007185] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 1.010078] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 1.016827] systemd[1]: Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ 1.023118] systemd[1]: Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 1.213507] scsi host0: ata_piix [ 1.214869] scsi host1: ata_piix [ 1.215958] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.217425] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [[ 1.219639] FDC 0 is a S82078B [ 1.220947] [drm] Initialized drm 1.1.0 20060810  OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK [ 1.225370] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) ] Reached target Basic System. [ 1.227273] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.229916] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.233445] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.236656] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.238767] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.241439] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.256043] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.257494] [TTM] Initializing pool allocator [ 1.259785] [TTM] Initializing DMA pool allocator [ 1.261530] [drm] fb mappable at 0xF0000000 [ 1.262709] [drm] vram aper at 0xF0000000 [ 1.263864] [drm] size 33554432 [ 1.264905] [drm] fb depth is 24 [ 1.265953] [drm] pitch is 3072 %G[ 1.267550] fbcon: cirrusdrmfb (fb0) is primary device [ 1.302602] vda: vda1 vda2 [ 1.310435] Console: switching to colour frame buffer device 128x48 [ 1.323088] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.328028] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 %G[ OK ] Found device /dev/disk/by-uuid/0d9f39ef-f730-4d66-ba62-54980bd39277. Starting File System Check on /dev/...f-f730-4d66-ba62-54980bd39277... [ OK ] Started File System Check on /dev/d...9ef-f730-4d66-ba62-54980bd39277. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 1.418363] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.419699] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ 1.631115] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.813732] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 2.073662] SELinux: Disabled at runtime. [ 2.112029] type=1404 audit(1491426248.788:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.118905] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.120393] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Reached target Encrypted Volumes. Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Remote File Systems. Mounting Huge Pages File System... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on LVM2 poll daemon socket. Starting Remount Root and Kernel File Systems... Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Debug File System... Mounting POSIX Message Queue File System... [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ 2.595233] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Started Remount Root and Kernel File Systems. Starting Load/Save Random Seed... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ 2.607132] systemd-journald[343]: Received request to flush runtime journal from PID 1 Starting Rebuild Hardware Database... [ OK ] Started Load/Save Random Seed. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Configure read-only root support. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ 3.660313] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ OK ] Found device /dev/ttyS0. [ 3.670349] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 3.687429] AES CTR mode by8 optimization enabled [ 3.691753] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ OK ] Found device /dev/disk/by-uuid/b159939a-b01a-44bc-9698-eb97bb18b8b4. Activating swap /dev/disk/by-uuid/b...a-b01a-44bc-9698-eb97bb18b8b4... [ 3.705256] alg: No test for crc32 (crc32-pclmul) [ 3.725400] intel_rapl: no valid rapl domains found in package 0 [ 3.727977] ppdev: user-space parallel port driver %G%G[ 3.730938] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ OK ] Activated swap /dev/disk/by-uuid/b159939a-b01a-44bc-9698-eb97bb18b8b4. [ OK ] Reached target Swap. [ 3.752163] intel_powerclamp: No package C-state available[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Rebuild Journal Catalog... Starting Import network configuration from initramfs... Starting Mark the need to relabel after reboot... [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Update is Completed. [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Reached target Paths. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Dump dmesg to /var/log/dmesg... Starting NTP client/server... [ 3.919336] type=1305 audit(1491426250.595:3): audit_pid=451 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting OpenSSH Server Key Generation... Starting Permit User Sessions... Starting Authorization Manager... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting Network Manager... Starting Login Service... [ OK ] Reached target Timers. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started Permit User Sessions. Starting Terminate Plymouth Boot Screen... Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Job spooling tools. Starting Job spooling tools... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ 4.667852] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.670761] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-05T21:04:22+00:00] WARN: ***************************************** [2017-04-05T21:04:22+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-05T21:04:22+00:00] WARN: ***************************************** [2017-04-05T21:04:31+00:00] INFO: Forking chef instance to converge... [2017-04-05T21:04:32+00:00] INFO: *** Chef 12.4.3 *** [2017-04-05T21:04:32+00:00] INFO: Chef-client pid: 8126 [2017-04-05T21:04:33+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-05T21:04:34+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-05T21:04:34+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-05T21:04:34+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-05T21:04:34+00:00] INFO: Running start handlers [2017-04-05T21:04:34+00:00] INFO: Start handlers complete. [2017-04-05T21:04:34+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-05T21:04:34+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-05T21:04:34+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-05T21:04:34+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-05T21:04:34+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170405210434.534990 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170405210434.537739 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170405210434.541362 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170405210434.544894 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170405210434.548655 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170405210434.552303 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-05T21:04:34+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170405210434.555990 [2017-04-05T21:04:34+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-05T21:04:34+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-05T21:04:34+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170405210434.564571 [2017-04-05T21:04:34+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-05T21:04:34+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-05T21:04:40+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-05T21:04:40+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170405210440.718289 [2017-04-05T21:04:40+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-05T21:04:40+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-05T21:04:40+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-05T21:05:24+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-05T21:05:24+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-05T21:05:24+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-05T21:05:38+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-05T21:05:38+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-05T21:05:38+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-05T21:05:38+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-05T21:05:38+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-05T21:05:38+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-05T21:05:38+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-05T21:05:39+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-05T21:05:39+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-05T21:05:39+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170405210539.328347 [2017-04-05T21:05:39+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-05T21:05:39+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-05T21:05:39+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-05T21:05:39+00:00] INFO: service[sssd] restarted [2017-04-05T21:05:39+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-05T21:05:39+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:39+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:05:39+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-05T21:05:40+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-05T21:05:40+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-05T21:05:40+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-05T21:05:40+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-05T21:05:40+00:00] INFO: service[ntpd] enabled [2017-04-05T21:05:40+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-05T21:05:40+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170405210540.829739 [2017-04-05T21:05:40+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-05T21:05:40+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-05T21:05:40+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-05T21:05:41+00:00] INFO: service[ntpd] restarted [2017-04-05T21:05:41+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-05T21:05:41+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170405210541.127132 [2017-04-05T21:05:41+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-05T21:05:41+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-05T21:05:41+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-05T21:05:41+00:00] INFO: service[ntpd] restarted [2017-04-05T21:05:41+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-05T21:05:41+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-05T21:05:41+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-05T21:05:41+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-05T21:05:41+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-05T21:05:41+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170405210541.926139 [2017-04-05T21:05:41+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-05T21:05:41+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-05T21:05:41+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-05T21:05:41+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170405210541.933007 [2017-04-05T21:05:41+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-05T21:05:41+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-05T21:05:41+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T21:05:42+00:00] INFO: service[autofs] restarted [2017-04-05T21:05:42+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-05T21:05:42+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170405210542.226423 [2017-04-05T21:05:42+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-05T21:05:42+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-05T21:05:42+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T21:05:42+00:00] INFO: service[autofs] restarted [2017-04-05T21:05:42+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-05T21:05:42+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-05T21:05:42+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-05T21:05:42+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-05T21:05:43+00:00] INFO: service[autofs] restarted [2017-04-05T21:05:43+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-05T21:05:44+00:00] INFO: service[autofs] enabled [2017-04-05T21:05:44+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-05T21:05:44+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-05T21:05:44+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-05T21:05:44+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-05T21:05:44+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-05T21:05:44+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170405210544.562691 [2017-04-05T21:05:44+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-05T21:05:44+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-05T21:05:44+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170405210544.570014 [2017-04-05T21:05:44+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-05T21:05:44+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-05T21:05:44+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-05T21:05:44+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-05T21:05:44+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-05T21:05:44+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170405210544.580841 [2017-04-05T21:05:44+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-05T21:05:44+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-05T21:05:44+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-05T21:05:46+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-05T21:05:46+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-05T21:05:46+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-05T21:05:46+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-05T21:05:47+00:00] INFO: service[munge] restarted [2017-04-05T21:05:47+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-05T21:05:47+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-05T21:05:47+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-05T21:05:47+00:00] INFO: service[munge] restarted [2017-04-05T21:05:47+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-05T21:05:47+00:00] INFO: service[munge] enabled [2017-04-05T21:05:47+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-05T21:05:48+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-05T21:05:48+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-05T21:05:51+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-05T21:05:51+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-05T21:05:51+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-05T21:05:53+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-05T21:05:53+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170405210553.913511 [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-05T21:05:53+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170405210553.917959 [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-05T21:05:53+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-05T21:05:53+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-05T21:05:53+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:53+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-05T21:05:53+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-05T21:05:53+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:53+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-05T21:05:53+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-05T21:05:54+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-05T21:05:54+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-05T21:05:54+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:54+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-05T21:05:54+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-05T21:05:54+00:00] INFO: service[xinetd] started [2017-04-05T21:05:54+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-05T21:05:55+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-05T21:05:55+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-05T21:05:55+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-05T21:05:55+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-05T21:05:55+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-05T21:05:55+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170405210555.820464 [2017-04-05T21:05:55+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-05T21:05:55+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-05T21:05:55+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-05T21:05:55+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-05T21:05:55+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-05T21:05:55+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-05T21:05:55+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-05T21:05:55+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-05T21:05:55+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-05T21:05:55+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-05T21:05:55+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-05T21:05:55+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-05T21:05:55+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-05T21:05:55+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-05T21:05:55+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-05T21:05:55+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-05T21:05:55+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:05:55+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-05T21:05:55+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-05T21:05:55+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-05T21:05:57+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-05T21:05:57+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-05T21:05:58+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-05T21:06:00+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-05T21:06:00+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-05T21:06:00+00:00] INFO: service[rsh.socket] enabled [2017-04-05T21:06:00+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-05T21:06:00+00:00] INFO: service[rsh.socket] started [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-05T21:06:00+00:00] INFO: service[rlogin.socket] enabled [2017-04-05T21:06:00+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-05T21:06:00+00:00] INFO: service[rlogin.socket] started [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-05T21:06:00+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-05T21:06:00+00:00] INFO: execute[Guard resource] ran successfully [2017-04-05T21:06:00+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-05T21:06:00+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-05T21:06:00+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-05T21:06:00+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-05T21:06:00+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-05T21:06:02+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-05T21:06:02+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-05T21:06:03+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-05T21:06:03+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-05T21:06:03+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-05T21:06:03+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-05T21:06:05+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-05T21:06:05+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-05T21:06:05+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.el7 from addon-epel7-x86_64 repository [2017-04-05T21:06:08+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.el7 [2017-04-05T21:06:08+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-05T21:06:09+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-05T21:06:09+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-05T21:06:09+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-05T21:06:09+00:00] INFO: service[rdma] enabled [2017-04-05T21:06:09+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 122.872530] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 122.881643] Loading iSCSI transport class v2.0-870. [ 122.885146] iscsi: registered transport (iser) [ 122.907719] RPC: Registered named UNIX socket transport module. [ 122.908384] RPC: Registered udp transport module. [ 122.908802] RPC: Registered tcp transport module. [ 122.909245] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 122.969149] RPC: Registered rdma transport module. [ 122.969623] RPC: Registered rdma backchannel transport module. [2017-04-05T21:06:10+00:00] INFO: service[rdma] started [2017-04-05T21:06:10+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-05T21:06:10+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-05T21:06:10+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 123.056406] FS-Cache: Loaded [ 123.065601] FS-Cache: Netfs 'nfs' registered for caching [ 123.068995] Key type dns_resolver registered [ 123.077543] NFS: Registering the id_resolver key type [ 123.078110] Key type id_resolver registered [ 123.078473] Key type id_legacy registered [2017-04-05T21:06:10+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-05T21:06:10+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-05T21:06:10+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170405210610.558412 [2017-04-05T21:06:10+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-05T21:06:10+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-05T21:06:10+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-05T21:06:50+00:00] INFO: service[kdump] restarted [2017-04-05T21:06:50+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-05T21:06:50+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-05T21:06:50+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-05T21:06:50+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-05T21:06:50+00:00] INFO: group[quota_usr] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-05T21:06:50+00:00] INFO: user[quota_usr] created [2017-04-05T21:06:50+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-05T21:06:50+00:00] INFO: group[quota_2usr] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-05T21:06:50+00:00] INFO: user[quota_2usr] created [2017-04-05T21:06:50+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-05T21:06:50+00:00] INFO: group[sanityusr] created [2017-04-05T21:06:50+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-05T21:06:50+00:00] INFO: user[sanityusr] created [2017-04-05T21:06:50+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-05T21:06:50+00:00] INFO: group[sanityusr1] created [2017-04-05T21:06:50+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-05T21:06:50+00:00] INFO: user[sanityusr1] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_1] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_2] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_3] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_4] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_5] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_6] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_7] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_8] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-05T21:06:50+00:00] INFO: user[quota15_9] created [2017-04-05T21:06:50+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_10] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_11] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_12] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_13] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_14] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_15] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_16] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_17] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_18] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_19] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_20] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_21] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_22] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_23] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_24] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_25] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_26] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_27] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_28] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_29] created [2017-04-05T21:06:51+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-05T21:06:51+00:00] INFO: user[quota15_30] created [2017-04-05T21:06:51+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-05T21:06:51+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-05T21:06:51+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-05T21:06:51+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-05T21:06:51+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-05T21:06:51+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-05T21:06:51+00:00] INFO: bash[set_hostid] ran successfully [2017-04-05T21:06:51+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-05T21:06:51+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-05T21:06:51+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-05T21:06:51+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-05T21:06:51+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-05T21:06:51+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-05T21:06:51+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-05T21:06:51+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-05T21:06:51+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-05T21:06:51+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-05T21:06:52+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-05T21:06:52+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-05T21:06:52+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-05T21:06:52+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-05T21:06:52+00:00] INFO: user[mpiuser] created [2017-04-05T21:06:52+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-05T21:06:52+00:00] INFO: group[mpiuser] created [2017-04-05T21:06:52+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-05T21:06:52+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-05T21:06:52+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-05T21:06:52+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-05T21:06:52+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-05T21:06:52+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-05T21:06:52+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-05T21:06:52+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-05T21:06:52+00:00] INFO: service[postfix] restarted [2017-04-05T21:06:52+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-05T21:06:52+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-05T21:06:52+00:00] INFO: execute[newaliases] ran successfully [2017-04-05T21:06:52+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-05T21:06:52+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-05T21:06:53+00:00] INFO: service[sshd] restarted [2017-04-05T21:06:53+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-05T21:06:53+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-05T21:06:53+00:00] INFO: service[rsyslog] restarted [2017-04-05T21:06:53+00:00] INFO: Chef Run complete in 139.421959718 seconds [2017-04-05T21:06:53+00:00] INFO: Skipping removal of unused files from the cache [2017-04-05T21:06:53+00:00] INFO: Running report handlers [2017-04-05T21:06:53+00:00] INFO: Creating JSON run report [2017-04-05T21:06:53+00:00] INFO: Report handlers complete [ OK ] Stopped Login Service. [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped target User and Group Name Lookups. [ OK ] Stopped Postfix Mail Transport Agent. [ OK ] Unmounted /scratch. [ OK ] Stopped MUNGE authentication service. [ OK ] Stopped System Logging Service. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Stopped Automounts filesystems on demand. Stopping System Security Services Daemon... [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped target Swap. [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... Stopping Load/Save Random Seed... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ 5525.211913] type=1305 audit(1491431772.643:1137): audit_pid=0 old=451 auid=4294967295 ses=4294967295 res=1 [ 5525.214391] type=1130 audit(1491431772.646:1138): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.217685] type=1131 audit(1491431772.649:1139): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.221543] type=1130 audit(1491431772.653:1140): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.225253] type=1131 audit(1491431772.657:1141): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Security Auditing Service. [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. [ 5525.233995] type=1130 audit(1491431772.665:1142): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.234049] type=1131 audit(1491431772.666:1143): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.245352] type=1130 audit(1491431772.677:1144): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 5525.249491] type=1131 audit(1491431772.681:1145): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting /run/user/0... [ OK ] Stopped Configure read-only root support. [ 5525.265920] type=1130 audit(1491431772.697:1146): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Configure read-only root support... [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Reached target Shutdown. [ 5525.296666] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 5525.300824] systemd-journald[343]: Received SIGTERM from PID 1 (systemd-shutdow). [ 5525.306328] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 5525.309312] systemd-shutdown[1]: Unmounting file systems. [ 5525.467510] EXT4-fs (vda1): re-mounted. Opts: (null) [ 5525.487087] EXT4-fs (vda1): re-mounted. Opts: (null) [ 5525.489258] systemd-shutdown[1]: All filesystems unmounted. [ 5525.491491] systemd-shutdown[1]: Deactivating swaps. [ 5525.493744] systemd-shutdown[1]: All swaps deactivated. [ 5525.495925] systemd-shutdown[1]: Detaching loop devices. [ 5525.498554] systemd-shutdown[1]: All loop devices detached. [ 5525.500703] systemd-shutdown[1]: Detaching DM devices. [ 5525.502877] systemd-shutdown[1]: All DM devices detached. [ 5525.533804] systemd-shutdown[1]: Rebooting. [ 5525.536576] Restarting system. [ 5525.538387] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=0d9f39ef-f730-4d66-ba62-54980bd39277 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x35e0a000-0x36efcfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 528MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=0d9f39ef-f730-4d66-ba62-54980bd39277 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1701828k/2097140k available (6765k kernel code, 400k absent, 394912k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006029] Security Framework initialized [ 0.007010] SELinux: Initializing. [ 0.008194] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011610] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013528] Mount-cache hash table entries: 4096 [ 0.015244] Initializing cgroup subsys memory [ 0.016012] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.019004] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.023005] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.025005] Initializing cgroup subsys net_prio [ 0.027074] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.029060] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] tlb_flushall_shift: 6 [ 0.033184] Freeing SMP alternatives: 28k freed [ 0.039411] ACPI: Core revision 20130517 [ 0.041512] ACPI: All ACPI Tables successfully acquired [ 0.043015] ftrace: allocating 25813 entries in 101 pages [ 0.051232] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.053004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.054005] smpboot: Max logical packages: 2 [ 0.056188] Enabling x2apic [ 0.057000] Enabled x2apic [ 0.057005] Switched APIC routing to physical x2apic. [ 0.060411] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.061003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.064029] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.066498] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.070044] Brought up 2 CPUs [ 0.070034] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.072006] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.073568] devtmpfs: initialized [ 0.075907] EVM: security.selinux [ 0.076005] EVM: security.ima [ 0.077004] EVM: security.capability [ 0.078788] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.079016] pinctrl core: initialized pinctrl subsystem [ 0.080166] NET: Registered protocol family 16 [ 0.081302] ACPI: bus type PCI registered [ 0.082007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.083092] PCI: Using configuration type 1 for base access [ 0.087103] ACPI: Added _OSI(Module Device) [ 0.088013] ACPI: Added _OSI(Processor Device) [ 0.089006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.090012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.092696] ACPI: Interpreter enabled [ 0.093014] ACPI: (supports S0 S5) [ 0.094005] ACPI: Using IOAPIC for interrupt routing [ 0.095021] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.098565] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.099011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.100010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.101023] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.102246] acpiphp: Slot [1] registered [ 0.103032] acpiphp: Slot [2] registered [ 0.104023] acpiphp: Slot [3] registered [ 0.105023] acpiphp: Slot [4] registered [ 0.106023] acpiphp: Slot [5] registered [ 0.107023] acpiphp: Slot [6] registered [ 0.108024] acpiphp: Slot [7] registered [ 0.109024] acpiphp: Slot [8] registered [ 0.110033] acpiphp: Slot [9] registered [ 0.111023] acpiphp: Slot [10] registered [ 0.112026] acpiphp: Slot [11] registered [ 0.113023] acpiphp: Slot [12] registered [ 0.114024] acpiphp: Slot [13] registered [ 0.115023] acpiphp: Slot [14] registered [ 0.116024] acpiphp: Slot [15] registered [ 0.117032] acpiphp: Slot [16] registered [ 0.118030] acpiphp: Slot [17] registered [ 0.119023] acpiphp: Slot [18] registered [ 0.120023] acpiphp: Slot [19] registered [ 0.121023] acpiphp: Slot [20] registered [ 0.122023] acpiphp: Slot [21] registered [ 0.123024] acpiphp: Slot [22] registered [ 0.124031] acpiphp: Slot [23] registered [ 0.125023] acpiphp: Slot [24] registered [ 0.126023] acpiphp: Slot [25] registered [ 0.127023] acpiphp: Slot [26] registered [ 0.128023] acpiphp: Slot [27] registered [ 0.129030] acpiphp: Slot [28] registered [ 0.130023] acpiphp: Slot [29] registered [ 0.131023] acpiphp: Slot [30] registered [ 0.132030] acpiphp: Slot [31] registered [ 0.133263] PCI host bridge to bus 0000:00 [ 0.134009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.135007] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.136007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.137007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.138007] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.140005] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.141006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.142006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.143006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.144812] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.145013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.169697] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.171078] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.172803] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.174072] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.175785] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.177303] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.178318] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.179007] vgaarb: loaded [ 0.180005] vgaarb: bridge control possible 0000:00:02.0 [ 0.181230] SCSI subsystem initialized [ 0.182054] ACPI: bus type USB registered [ 0.183041] usbcore: registered new interface driver usbfs [ 0.184018] usbcore: registered new interface driver hub [ 0.185057] usbcore: registered new device driver usb [ 0.186200] PCI: Using ACPI for IRQ routing [ 0.187390] NetLabel: Initializing [ 0.188008] NetLabel: domain hash size = 128 [ 0.189004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.190044] NetLabel: unlabeled traffic allowed by default [ 0.191169] Switched to clocksource kvm-clock [ 0.200588] pnp: PnP ACPI init [ 0.201719] ACPI: bus type PNP registered [ 0.203467] pnp: PnP ACPI: found 5 devices [ 0.204687] ACPI: bus type PNP unregistered [ 0.215644] NET: Registered protocol family 2 [ 0.217319] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.219696] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.221368] TCP: Hash tables configured (established 16384 bind 16384) [ 0.222947] TCP: reno registered [ 0.224088] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.225575] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.227235] NET: Registered protocol family 1 [ 0.228464] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.229845] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.231197] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.232871] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.234544] Unpacking initramfs... [ 0.507574] Freeing initrd memory: 17356k freed [ 0.513574] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.515299] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.517301] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.518819] Initialise system trusted keyring [ 0.520154] audit: initializing netlink socket (disabled) [ 0.521556] type=2000 audit(1491431801.362:1): initialized [ 0.542603] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.544126] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.546667] zpool: loaded [ 0.547736] zbud: loaded [ 0.549036] VFS: Disk quotas dquot_6.5.2 [ 0.550272] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.552044] msgmni has been set to 3357 [ 0.553316] Key type big_key registered [ 0.555952] NET: Registered protocol family 38 [ 0.557244] Key type asymmetric registered [ 0.558465] Asymmetric key parser 'x509' registered [ 0.559799] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.562147] io scheduler noop registered [ 0.563364] io scheduler deadline registered (default) [ 0.564720] io scheduler cfq registered [ 0.566041] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.567451] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.569153] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.571440] ACPI: Power Button [PWRF] [ 0.572813] GHES: HEST is not enabled! [ 0.574107] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.596852] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.598847] Non-volatile memory driver v1.3 [ 0.600113] Linux agpgart interface v0.103 [ 0.601463] crash memory driver: version 1.1 [ 0.602782] rdac: device handler registered [ 0.604100] hp_sw: device handler registered [ 0.621009] emc: device handler registered [ 0.622251] alua: device handler registered [ 0.623522] libphy: Fixed MDIO Bus: probed [ 0.624821] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.626367] ehci-pci: EHCI PCI platform driver [ 0.627626] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.629098] ohci-pci: OHCI PCI platform driver [ 0.630384] uhci_hcd: USB Universal Host Controller Interface driver [ 0.632115] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.633571] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.635867] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.637267] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.638778] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.640401] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.642704] usb usb1: Product: UHCI Host Controller [ 0.644277] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.646676] usb usb1: SerialNumber: 0000:00:01.2 [ 0.648201] hub 1-0:1.0: USB hub found [ 0.649419] hub 1-0:1.0: 2 ports detected [ 0.650790] usbcore: registered new interface driver usbserial [ 0.652300] usbcore: registered new interface driver usbserial_generic [ 0.653869] usbserial: USB Serial support registered for generic [ 0.655398] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.658428] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.659814] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.661319] mousedev: PS/2 mouse device common for all mice [ 0.663108] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.666285] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.666286] rtc_cmos 00:00: RTC can wake from S4 [ 0.666727] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.666858] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.666936] cpuidle: using governor menu [ 0.667061] hidraw: raw HID events driver (C) Jiri Kosina [ 0.667169] usbcore: registered new interface driver usbhid [ 0.667171] usbhid: USB HID core driver [ 0.667213] drop_monitor: Initializing network drop monitor service [ 0.667285] TCP: cubic registered [ 0.667288] Initializing XFRM netlink socket [ 0.667409] NET: Registered protocol family 10 [ 0.668976] NET: Registered protocol family 17 [ 0.669832] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.670185] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.670234] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.670431] Loading compiled-in X.509 certificates [ 0.670480] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.670496] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.670965] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.670993] registered taskstats version 1 [ 0.676521] Key type trusted registered [ 0.678497] Key type encrypted registered [ 0.680465] IMA: No TPM chip found, activating TPM-bypass! [ 0.681114] rtc_cmos 00:00: setting system clock to 2017-04-05 22:36:41 UTC (1491431801) [ 0.706440] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.709851] Freeing unused kernel memory: 1680k freed [ 0.715716] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.719904] systemd[1]: Detected virtualization kvm. [ 0.721248] systemd[1]: Detected architecture x86-64. [ 0.722547] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.724606] systemd[1]: Set hostname to . [ OK ] Reached target Local File Systems. [ 0.757034] systemd[1]: Reached target Local File Systems. [ 0.758448] systemd[1]: Starting Local File Systems. [ OK ] Created slice Root Slice. [ 0.760473] systemd[1]: Created slice Root Slice. [ 0.761781] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 0.763669] systemd[1]: Listening on udev Kernel Socket. [ 0.765052] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on Journal Socket. [ 0.767014] systemd[1]: Listening on Journal Socket. [ 0.768364] systemd[1]: Starting Journal Socket. [ OK ] Reached target Timers. [ 0.770222] systemd[1]: Reached target Timers. [ 0.771499] systemd[1]: Starting Timers. [ OK ] Listening on udev Control Socket. [ 0.773357] systemd[1]: Listening on udev Control Socket. [ 0.774732] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Sockets. [ 0.776625] systemd[1]: Reached target Sockets. [ 0.777911] systemd[1]: Starting Sockets. [ OK ] Reached target Swap. [ 0.779665] systemd[1]: Reached target Swap. [ 0.780923] systemd[1]: Starting Swap. [ OK ] Created slice System Slice. [ 0.782745] systemd[1]: Created slice System Slice. [ 0.784066] systemd[1]: Starting System Slice. [ 0.785834] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.788201] systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... [ OK ] Reached target Slices. [ 0.790659] systemd[1]: Reached target Slices. [ 0.791956] systemd[1]: Starting Slices. [ 0.793699] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.799059] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.801555] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ 0.881500] systemd[1]: Started Journal Service. [ OK ] Started Load Kernel Modules. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... Starting Apply Kernel Variables... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 0.981490] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 0.983314] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 0.986758] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 0.991324] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 0.991341] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 0.994782] [drm] Initialized drm 1.1.0 20060810 [ 0.994943] FDC 0 is a S82078B [ 1.004964] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.007641] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.024066] scsi host0: ata_piix [ 1.033358] scsi host1: ata_piix [ 1.046122] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 %G[ 1.047623] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.048665] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.048666] [TTM] Initializing pool allocator [ 1.048668] [TTM] Initializing DMA pool allocator [ 1.048888] [drm] fb mappable at 0xF0000000 [ 1.048888] [drm] vram aper at 0xF0000000 [ 1.048889] [drm] size 33554432 [ 1.048889] [drm] fb depth is 24 [ 1.048889] [drm] pitch is 3072 [ 1.052483] vda: vda1 vda2 vda4 [ 1.053929] fbcon: cirrusdrmfb (fb0) is primary device [ 1.080124] Console: switching to colour frame buffer device 128x48 [ 1.091726] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device %G[ 1.116029] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 1.514083] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Found device /dev/disk/by-uuid/0d9f39ef-f730-4d66-ba62-54980bd39277. Starting File System Check on /dev/...f-f730-4d66-ba62-54980bd39277... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d...9ef-f730-4d66-ba62-54980bd39277. Mounting /sysroot... [ 2.655844] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 2.657301] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Load Kernel Modules. Stopping Load Kernel Modules... [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 4.573527] systemd-journald[92]: Received SIGTERM from PID 1 (systemd). [ 6.250325] SELinux: Disabled at runtime. [ 6.288031] type=1404 audit(1491431807.107:2): selinux=0 auid=4294967295 ses=4294967295 [ 6.383148] ip_tables: (C) 2000-2006 Netfilter Core Team [ 6.385150] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. Starting Read and set NIS domainname from /etc/sysconfig/network... [ 8.126783] RPC: Registered named UNIX socket transport module. [ 8.128171] RPC: Registered udp transport module. [ 8.129410] RPC: Registered tcp transport module. [ OK ] Listening on LVM2 metadata daemon socket. Starting Create list of required st... nodes for the current kernel... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Kernel Module supporting RPCSEC_GSS... [ 8.133065] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Encrypted Volumes. Mounting NFSD configuration filesystem... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... Mounting Debug File System... Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Starting Load Kernel Modules... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Load Kernel Modules. Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ 8.361029] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... Starting udev Coldplug all Devices... Starting Load/Save Random Seed... Starting Flush Journal to Persistent Storage... [ 8.381324] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ OK ] Mounted NFSD configuration filesystem. [ OK ] Started Load/Save Random Seed. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Configure read-only root support. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ 8.577243] systemd-journald[345]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Kernel Device Manager. [ 8.774729] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 8.813089] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/b159939a-b01a-44bc-9698-eb97bb18b8b4. Activating swap /dev/disk/by-uuid/b...a-b01a-44bc-9698-eb97bb18b8b4... %G[ 8.918014] AES CTR mode by8 optimization enabled %G[ 8.922639] ppdev: user-space parallel port driver [ 8.927127] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 8.932079] alg: No test for crc32 (crc32-pclmul) [ 8.939196] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 8.962131] intel_rapl: no valid rapl domains found in package 0 [ OK ] Activated swap /dev/disk/by-uuid/b159939a-b01a-44bc-9698-eb97bb18b8b4. [ 8.990503] intel_rapl: no valid rapl domains found in package 0 [ 9.000301] intel_powerclamp: No package C-state available[ OK ] Reached target Swap. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Preprocess NFS configuration... [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Create Volatile Files and Directories. Mounting RPC Pipe File System... Starting Security Auditing Service... [ OK ] Mounted RPC Pipe File System. [ OK ] Started Preprocess NFS configuration. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. Starting System Security Services Daemon... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting NTP client/server... Starting GSSAPI Proxy Daemon... Starting Authorization Manager... Starting Dump dmesg to /var/log/dmesg... [ OK ] Started D-Bus System Message Bus. [ 9.267859] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 9.278430] Loading iSCSI transport class v2.0-870. [ 9.361887] iscsi: registered transport (iser) [ 9.571246] type=1305 audit(1491431810.390:3): audit_pid=501 old=0 auid=4294967295 ses=4294967295 res=1 [ 9.671965] RPC: Registered rdma transport module. [ 9.673428] RPC: Registered rdma backchannel transport module. Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Reached target Timers. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started NTP client/server. [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ OK ] Started Authorization Manager. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ OK ] Started Hostname Service. [ 12.302137] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 12.303866] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Notify NFS peers of a restart... Starting MUNGE authentication service... Starting Dynamic System Tuning Daemon... Starting Postfix Mail Transport Agent... Starting System Logging Service... Starting Xinetd A Powerful Replacement For Inetd... [ OK ] Reached target Network is Online. Mounting /scratch... [ OK ] Started Notify NFS peers of a restart. [ 19.094913] FS-Cache: Loaded [ 19.102187] FS-Cache: Netfs 'nfs' registered for caching [ OK ] Started OpenSSH server daemon. [ 19.107408] Key type dns_resolver registered [ 19.117788] NFS: Registering the id_resolver key type [ 19.119220] Key type id_resolver registered [ 19.120531] Key type id_legacy registered [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started MUNGE authentication service. [ OK ] Started System Logging Service. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Permit User Sessions... Starting Crash recovery kernel arming... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. Starting Command Scheduler... Starting Terminate Plymouth Boot Screen... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: Console [trevis-41vm7] log opened at 2017-04-05 22:37:42 UTC. Console [trevis-41vm7] connected to at 04-05 22:37. [ 127.494243] device-mapper: uevent: version 1.0.3 [ 127.494775] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 213.089090] spl: loading out-of-tree module taints kernel. [ 213.089692] spl: module verification failed: signature and/or required key missing - tainting kernel [ 213.095328] SPL: Loaded module v0.6.5.9-1 [ 213.096121] znvpair: module license 'CDDL' taints kernel. [ 213.096770] Disabling lock debugging due to kernel taint [ 213.117283] ZFS: Loaded module v0.6.5.9-1, ZFS pool version 5000, ZFS filesystem version 5 [ 213.196662] SPL: using hostid 0x61303930 [ 222.998197] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 223.001218] alg: No test for adler32 (adler32-zlib) [ 223.001737] alg: No test for crc32 (crc32-table) [ 223.810753] Lustre: Lustre: Build Version: 2.9.54_26_gaaeee33 [ 223.847357] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 223.847932] LNet: Accept all, port 7988 [ 223.873608] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 223.948051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 224.123445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 224.148338] Lustre: lustre-MDT0000: new disk, initializing [ 224.490081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.560933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.933600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 225.220184] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 225.701502] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 225.986846] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 226.271980] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 226.558392] Lustre: DEBUG MARKER: sync; sync; sync [ 229.448247] Lustre: lustre-MDT0000: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 231.926089] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 232.215235] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 235.118171] Lustre: MGS: Connection restored to ae0cff62-4df9-6df9-aa7a-87848c801b97 (at 10.9.5.244@tcp) [ 244.283190] Lustre: lustre-MDT0000: Connection restored to ae0cff62-4df9-6df9-aa7a-87848c801b97 (at 10.9.5.244@tcp) [ 244.284158] Lustre: Skipped 1 previous similar message [ 251.989822] Lustre: MGS: Connection restored to 6a04c9a8-6892-d84a-f4e7-7a2f1fa7b691 (at 10.9.5.237@tcp) [ 255.703926] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 256.035381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 256.221655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.391293] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 256.705809] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 266.128398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.54_26_gaaeee33 [ 266.311036] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.54_26_gaaeee33 [ 266.482262] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 266.812827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.54_26_gaaeee33 [ 267.003115] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.54_26_gaaeee33 [ 267.528562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.54_26_gaaeee33 [ 267.706923] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.54_26_gaaeee33 [ 267.932887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: ost-pools ============----- Wed Apr 5 22:41:09 UTC 2017 [ 268.116374] Lustre: DEBUG MARKER: -----============= acceptance-small: ost-pools ============----- Wed Apr 5 22:41:09 UTC 2017 [ 268.403570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration -o sec ========================================================== 22:41:09 \(1491432069\) [ 268.591079] Lustre: DEBUG MARKER: == ost-pools test complete, duration -o sec ========================================================== 22:41:09 (1491432069) [ 270.078181] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 270.078181] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 270.078181] if [ $running -ne $mpts ]; then [ 270.078181] echo $(hostname) env are INSANE!; [ 270.078181] exit 1; [ 270.078181] fi [ 270.416862] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 270.416862] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 270.416862] if [ $running -ne $mpts ]; then [ 270.416862] echo $(hostname) env are INSANE!; [ 270.416862] exit 1; [ 270.416862] fi [ 271.306669] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 271.599177] Lustre: DEBUG MARKER: cat /proc/mounts [ 273.283276] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 273.604564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 273.791338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.964443] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 274.313568] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 274.858983] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 275.191224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 275.382249] Lustre: DEBUG MARKER: excepting tests: [ 275.589579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 18 23b [ 275.770679] Lustre: DEBUG MARKER: skipping tests SLOW=no: 18 23b [ 276.023113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 22:41:17 \(1491432077\) [ 276.214274] Lustre: DEBUG MARKER: == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 22:41:17 (1491432077) [ 276.383324] Lustre: DEBUG MARKER: lctl pool_new lustre.p [ 286.680535] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [ 286.970207] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [ 287.281657] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 287.568692] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.p [ 287.858111] Lustre: DEBUG MARKER: lctl pool_destroy lustre.p [ 292.156281] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [ 292.444042] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [ 292.790558] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 293.528963] Lustre: DEBUG MARKER: rc=0; [ 293.528963] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 293.528963] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 293.528963] echo $(hostname -s): $val; [ 293.528963] rc=$val; [ 293.528963] fi; [ 293.528963] exit $rc [ 294.202208] Lustre: DEBUG MARKER: dmesg [ 294.627221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 22:41:35 \(1491432095\) [ 294.812748] Lustre: DEBUG MARKER: == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 22:41:35 (1491432095) [ 294.972666] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12 [ 297.266337] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [ 297.552264] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [ 299.897099] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 300.188338] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool12 [ 300.484195] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool12 [ 306.780222] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [ 307.066230] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [ 309.438514] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 310.092107] Lustre: DEBUG MARKER: rc=0; [ 310.092107] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 310.092107] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 310.092107] echo $(hostname -s): $val; [ 310.092107] rc=$val; [ 310.092107] fi; [ 310.092107] exit $rc [ 310.769159] Lustre: DEBUG MARKER: dmesg [ 311.182199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 22:41:52 \(1491432112\) [ 311.380914] Lustre: DEBUG MARKER: == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 22:41:52 (1491432112) [ 311.540292] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [ 321.832599] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 322.121395] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 322.440099] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 322.728851] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [ 323.018951] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [ 329.312436] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 329.600771] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 329.950131] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 330.596070] Lustre: DEBUG MARKER: rc=0; [ 330.596070] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 330.596070] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 330.596070] echo $(hostname -s): $val; [ 330.596070] rc=$val; [ 330.596070] fi; [ 330.596070] exit $rc [ 331.271166] Lustre: DEBUG MARKER: dmesg [ 331.695499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1d: Create a pool with a 16 char pool name\; should fail ============================ 22:42:13 \(1491432133\) [ 331.883506] Lustre: DEBUG MARKER: == ost-pools test 1d: Create a pool with a 16 char pool name; should fail ============================ 22:42:13 (1491432133) [ 332.071352] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12345678 [ 332.403428] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 333.085601] Lustre: DEBUG MARKER: rc=0; [ 333.085601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 333.085601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 333.085601] echo $(hostname -s): $val; [ 333.085601] rc=$val; [ 333.085601] fi; [ 333.085601] exit $rc [ 333.778964] Lustre: DEBUG MARKER: dmesg [ 334.187257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1e: Create a pool with a 1000 char pool name\; should fail ========================== 22:42:15 \(1491432135\) [ 334.379392] Lustre: DEBUG MARKER: == ost-pools test 1e: Create a pool with a 1000 char pool name; should fail ========================== 22:42:15 (1491432135) [ 334.559244] Lustre: DEBUG MARKER: lctl pool_new lustre.testpooloooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo [ 334.881118] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 335.541347] Lustre: DEBUG MARKER: rc=0; [ 335.541347] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 335.541347] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 335.541347] echo $(hostname -s): $val; [ 335.541347] rc=$val; [ 335.541347] fi; [ 335.541347] exit $rc [ 336.215973] Lustre: DEBUG MARKER: dmesg [ 336.634296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 22:42:17 \(1491432137\) [ 336.824168] Lustre: DEBUG MARKER: == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 22:42:17 (1491432137) [ 336.993885] Lustre: DEBUG MARKER: lctl pool_new .testpool [ 337.313121] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 337.968946] Lustre: DEBUG MARKER: rc=0; [ 337.968946] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 337.968946] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 337.968946] echo $(hostname -s): $val; [ 337.968946] rc=$val; [ 337.968946] fi; [ 337.968946] exit $rc [ 338.639474] Lustre: DEBUG MARKER: dmesg [ 339.060111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 22:42:20 \(1491432140\) [ 339.261782] Lustre: DEBUG MARKER: == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 22:42:20 (1491432140) [ 339.422612] Lustre: DEBUG MARKER: lctl pool_new testpool [ 339.741872] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 340.397763] Lustre: DEBUG MARKER: rc=0; [ 340.397763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 340.397763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 340.397763] echo $(hostname -s): $val; [ 340.397763] rc=$val; [ 340.397763] fi; [ 340.397763] exit $rc [ 341.072666] Lustre: DEBUG MARKER: dmesg [ 341.494927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 22:42:22 \(1491432142\) [ 341.679555] Lustre: DEBUG MARKER: == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 22:42:22 (1491432142) [ 341.846803] Lustre: DEBUG MARKER: lctl pool_new lustre. [ 342.166888] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 342.846176] Lustre: DEBUG MARKER: rc=0; [ 342.846176] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 342.846176] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 342.846176] echo $(hostname -s): $val; [ 342.846176] rc=$val; [ 342.846176] fi; [ 342.846176] exit $rc [ 343.526661] Lustre: DEBUG MARKER: dmesg [ 343.935233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 22:42:25 \(1491432145\) [ 344.119846] Lustre: DEBUG MARKER: == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 22:42:25 (1491432145) [ 344.284682] Lustre: DEBUG MARKER: lctl pool_new . [ 344.604171] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 345.254411] Lustre: DEBUG MARKER: rc=0; [ 345.254411] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 345.254411] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 345.254411] echo $(hostname -s): $val; [ 345.254411] rc=$val; [ 345.254411] fi; [ 345.254411] exit $rc [ 345.925179] Lustre: DEBUG MARKER: dmesg [ 346.338899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 22:42:27 \(1491432147\) [ 346.531384] Lustre: DEBUG MARKER: == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 22:42:27 (1491432147) [ 346.697098] Lustre: DEBUG MARKER: lctl pool_new lustre,testpool [ 347.027216] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 347.685157] Lustre: DEBUG MARKER: rc=0; [ 347.685157] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 347.685157] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 347.685157] echo $(hostname -s): $val; [ 347.685157] rc=$val; [ 347.685157] fi; [ 347.685157] exit $rc [ 348.361897] Lustre: DEBUG MARKER: dmesg [ 348.782692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 22:42:30 \(1491432150\) [ 348.972860] Lustre: DEBUG MARKER: == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 22:42:30 (1491432150) [ 349.141762] Lustre: DEBUG MARKER: lctl pool_new lustre/testpool [ 349.461985] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 350.126823] Lustre: DEBUG MARKER: rc=0; [ 350.126823] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 350.126823] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 350.126823] echo $(hostname -s): $val; [ 350.126823] rc=$val; [ 350.126823] fi; [ 350.126823] exit $rc [ 350.805983] Lustre: DEBUG MARKER: dmesg [ 351.228406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 22:42:32 \(1491432152\) [ 351.412679] Lustre: DEBUG MARKER: == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 22:42:32 (1491432152) [ 351.587659] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 357.894688] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 358.194695] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 358.511558] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 358.801942] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 359.091168] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 359.376522] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 365.670768] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 365.956951] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 368.329804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 368.982586] Lustre: DEBUG MARKER: rc=0; [ 368.982586] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 368.982586] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 368.982586] echo $(hostname -s): $val; [ 368.982586] rc=$val; [ 368.982586] fi; [ 368.982586] exit $rc [ 369.657370] Lustre: DEBUG MARKER: dmesg [ 370.066139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 22:42:51 \(1491432171\) [ 370.250608] Lustre: DEBUG MARKER: == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 22:42:51 (1491432171) [ 370.419054] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [ 378.714183] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 379.004325] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 379.321402] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567 OST0000 [ 387.614954] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [ 387.614954] sort -u | tr '\n' ' ' [ 387.906918] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [ 387.906918] sort -u | tr '\n' ' ' [ 391.218977] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1234567 OST0000 [ 398.599338] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 398.891079] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [ 399.178485] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [ 405.478655] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 405.771235] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [ 408.147969] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 408.807711] Lustre: DEBUG MARKER: rc=0; [ 408.807711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 408.807711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 408.807711] echo $(hostname -s): $val; [ 408.807711] rc=$val; [ 408.807711] fi; [ 408.807711] exit $rc [ 409.511218] Lustre: DEBUG MARKER: dmesg [ 409.928691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 22:43:31 \(1491432211\) [ 410.127388] Lustre: DEBUG MARKER: == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 22:43:31 (1491432211) [ 410.299270] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 410.585613] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [ 410.907722] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 411.575529] Lustre: DEBUG MARKER: rc=0; [ 411.575529] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 411.575529] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 411.575529] echo $(hostname -s): $val; [ 411.575529] rc=$val; [ 411.575529] fi; [ 411.575529] exit $rc [ 412.250668] Lustre: DEBUG MARKER: dmesg [ 412.664979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 22:43:33 \(1491432213\) [ 412.849524] Lustre: DEBUG MARKER: == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 22:43:33 (1491432213) [ 413.023781] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567890 lustre-OST0000 [ 413.347446] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 414.003379] Lustre: DEBUG MARKER: rc=0; [ 414.003379] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 414.003379] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 414.003379] echo $(hostname -s): $val; [ 414.003379] rc=$val; [ 414.003379] fi; [ 414.003379] exit $rc [ 414.680685] Lustre: DEBUG MARKER: dmesg [ 415.092566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2c: pool_add: OST index combinations =============================================== 22:43:36 \(1491432216\) [ 415.274658] Lustre: DEBUG MARKER: == ost-pools test 2c: pool_add: OST index combinations =============================================== 22:43:36 (1491432216) [ 415.442231] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 423.733355] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 424.022112] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 424.332925] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [ 434.623672] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [ 442.958395] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [ 451.251384] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [ 459.563613] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [ 467.857096] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 476.171170] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000,0001,] [ 486.469437] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000,0001,] [ 494.782224] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 504.131850] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0001/1] [ 514.446266] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 514.736340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 515.020620] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 523.313210] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 523.600813] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 525.970798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 526.615928] Lustre: DEBUG MARKER: rc=0; [ 526.615928] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 526.615928] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 526.615928] echo $(hostname -s): $val; [ 526.615928] rc=$val; [ 526.615928] fi; [ 526.615928] exit $rc [ 527.290318] Lustre: DEBUG MARKER: dmesg [ 527.703353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2d: pool_add: OSTs that don\'t exist should be rejected ============================= 22:45:29 \(1491432329\) [ 527.889176] Lustre: DEBUG MARKER: == ost-pools test 2d: pool_add: OSTs that don't exist should be rejected ============================= 22:45:29 (1491432329) [ 528.063859] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 536.355238] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 536.642351] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 537.963668] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0002_UUID [ 538.252593] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 538.545854] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 538.839189] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 549.138074] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 549.427682] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 549.772228] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 550.431384] Lustre: DEBUG MARKER: rc=0; [ 550.431384] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 550.431384] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 550.431384] echo $(hostname -s): $val; [ 550.431384] rc=$val; [ 550.431384] fi; [ 550.431384] exit $rc [ 551.107924] Lustre: DEBUG MARKER: dmesg [ 551.529477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 22:45:52 \(1491432352\) [ 551.714458] Lustre: DEBUG MARKER: == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 22:45:52 (1491432352) [ 551.878904] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 562.173945] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 562.459634] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 562.770048] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [ 569.078538] Lustre: DEBUG MARKER: LOCALE=C /usr/sbin/lctl pool_add lustre.testpool lustre-OST0000_UUID 2>&1 [ 569.377869] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 569.666340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 569.952939] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 580.244346] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 588.539648] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 588.828300] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 589.172969] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 589.834291] Lustre: DEBUG MARKER: rc=0; [ 589.834291] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 589.834291] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 589.834291] echo $(hostname -s): $val; [ 589.834291] rc=$val; [ 589.834291] fi; [ 589.834291] exit $rc [ 590.508562] Lustre: DEBUG MARKER: dmesg [ 590.923563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3a: pool_remove: non-existant pool ================================================= 22:46:32 \(1491432392\) [ 591.118379] Lustre: DEBUG MARKER: == ost-pools test 3a: pool_remove: non-existant pool ================================================= 22:46:32 (1491432392) [ 591.275055] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [ 591.600655] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 592.260582] Lustre: DEBUG MARKER: rc=0; [ 592.260582] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 592.260582] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 592.260582] echo $(hostname -s): $val; [ 592.260582] rc=$val; [ 592.260582] fi; [ 592.260582] exit $rc [ 592.932474] Lustre: DEBUG MARKER: dmesg [ 593.346736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 22:46:34 \(1491432394\) [ 593.545287] Lustre: DEBUG MARKER: == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 22:46:34 (1491432394) [ 593.702508] Lustre: DEBUG MARKER: lctl pool_remove nonexistantfs.testpool OST0000 [ 594.023114] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 594.680553] Lustre: DEBUG MARKER: rc=0; [ 594.680553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 594.680553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 594.680553] echo $(hostname -s): $val; [ 594.680553] rc=$val; [ 594.680553] fi; [ 594.680553] exit $rc [ 595.359525] Lustre: DEBUG MARKER: dmesg [ 595.797032] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 22:46:37 \(1491432397\) [ 595.984761] Lustre: DEBUG MARKER: == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 22:46:37 (1491432397) [ 596.141438] Lustre: DEBUG MARKER: lctl pool_remove lustre.p1234567891234567890 lustre-OST0000 [ 596.456144] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 597.126863] Lustre: DEBUG MARKER: rc=0; [ 597.126863] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 597.126863] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 597.126863] echo $(hostname -s): $val; [ 597.126863] rc=$val; [ 597.126863] fi; [ 597.126863] exit $rc [ 597.797693] Lustre: DEBUG MARKER: dmesg [ 598.230733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3d: pool_remove: OST index combinations ============================================ 22:46:39 \(1491432399\) [ 598.413556] Lustre: DEBUG MARKER: == ost-pools test 3d: pool_remove: OST index combinations ============================================ 22:46:39 (1491432399) [ 598.582801] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 604.880591] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 605.178821] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 609.539956] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [ 619.830381] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [ 629.175668] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [ 633.466716] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [ 639.778622] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [ 648.070736] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 656.416686] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 660.710884] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 660.710884] sort -u | tr '\n' ' ' [ 660.999741] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 660.999741] sort -u | tr '\n' ' ' [ 663.342116] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0001/1] [ 673.679289] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 673.967334] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 674.252332] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 676.548502] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 676.835623] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 682.244956] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 682.899151] Lustre: DEBUG MARKER: rc=0; [ 682.899151] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 682.899151] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 682.899151] echo $(hostname -s): $val; [ 682.899151] rc=$val; [ 682.899151] fi; [ 682.899151] exit $rc [ 683.573666] Lustre: DEBUG MARKER: dmesg [ 683.992773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 22:48:05 \(1491432485\) [ 684.172660] Lustre: DEBUG MARKER: == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 22:48:05 (1491432485) [ 684.343413] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 684.668420] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 685.332781] Lustre: DEBUG MARKER: rc=0; [ 685.332781] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 685.332781] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 685.332781] echo $(hostname -s): $val; [ 685.332781] rc=$val; [ 685.332781] fi; [ 685.332781] exit $rc [ 686.014413] Lustre: DEBUG MARKER: dmesg [ 686.435032] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 22:48:07 \(1491432487\) [ 686.622684] Lustre: DEBUG MARKER: == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 22:48:07 (1491432487) [ 686.791689] Lustre: DEBUG MARKER: lctl pool_destroy nonexistantfs.testpool [ 687.108564] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 687.789876] Lustre: DEBUG MARKER: rc=0; [ 687.789876] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 687.789876] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 687.789876] echo $(hostname -s): $val; [ 687.789876] rc=$val; [ 687.789876] fi; [ 687.789876] exit $rc [ 688.465369] Lustre: DEBUG MARKER: dmesg [ 688.878602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 22:48:10 \(1491432490\) [ 689.072794] Lustre: DEBUG MARKER: == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 22:48:10 (1491432490) [ 689.238154] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 699.537946] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 699.836627] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 700.149502] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [ 708.440907] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 708.440907] sort -u | tr '\n' ' ' [ 708.732187] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 708.732187] sort -u | tr '\n' ' ' [ 710.056765] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 710.346123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 710.633600] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 710.929777] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 719.220628] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 727.513903] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 727.801802] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 728.157610] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 728.805038] Lustre: DEBUG MARKER: rc=0; [ 728.805038] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 728.805038] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 728.805038] echo $(hostname -s): $val; [ 728.805038] rc=$val; [ 728.805038] fi; [ 728.805038] exit $rc [ 729.485727] Lustre: DEBUG MARKER: dmesg [ 729.899457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5a: lfs pool_list from client ====================================================== 22:48:51 \(1491432531\) [ 730.083592] Lustre: DEBUG MARKER: == ost-pools test 5a: lfs pool_list from client ====================================================== 22:48:51 (1491432531) [ 730.255150] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 730.544810] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 730.829160] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 737.120854] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 737.406477] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 740.756254] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 743.050221] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 743.338064] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 747.702268] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 754.018143] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 754.307443] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 754.591737] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 760.879148] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 771.171334] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 781.465172] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 781.753821] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 782.072688] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 782.363198] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 782.648566] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 784.940717] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 785.231251] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 792.651875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 793.303937] Lustre: DEBUG MARKER: rc=0; [ 793.303937] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 793.303937] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 793.303937] echo $(hostname -s): $val; [ 793.303937] rc=$val; [ 793.303937] fi; [ 793.303937] exit $rc [ 793.981630] Lustre: DEBUG MARKER: dmesg [ 794.403328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5b: lctl pool_list from MDS ======================================================== 22:49:55 \(1491432595\) [ 794.587523] Lustre: DEBUG MARKER: == ost-pools test 5b: lctl pool_list from MDS ======================================================== 22:49:55 (1491432595) [ 794.756057] Lustre: DEBUG MARKER: lctl pool_list [ 795.045913] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 795.333438] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 795.619775] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 803.913047] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 804.199795] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 805.523085] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 809.816693] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 810.102900] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 812.432835] Lustre: DEBUG MARKER: lctl pool_list lustre [ 812.719851] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 819.010227] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [ 819.299027] Lustre: DEBUG MARKER: lctl pool_list nonexistantfs [ 819.584291] Lustre: DEBUG MARKER: lctl pool_list lustre.nonexistantpool [ 819.870858] Lustre: DEBUG MARKER: lctl pool_list /mnt/lustrenonexistant [ 820.156192] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 820.441331] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 820.727322] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 825.020692] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 833.311515] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 837.604056] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 837.890022] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 843.259462] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 843.555575] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 843.844056] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 854.136971] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 854.423390] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 854.766797] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 855.429028] Lustre: DEBUG MARKER: rc=0; [ 855.429028] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 855.429028] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 855.429028] echo $(hostname -s): $val; [ 855.429028] rc=$val; [ 855.429028] fi; [ 855.429028] exit $rc [ 856.104997] Lustre: DEBUG MARKER: dmesg [ 856.519532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 6: getstripe\/setstripe ============================================================= 22:50:57 \(1491432657\) [ 856.704581] Lustre: DEBUG MARKER: == ost-pools test 6: getstripe/setstripe ============================================================= 22:50:57 (1491432657) [ 856.867758] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 865.163847] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 865.451986] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 866.775796] Lustre: DEBUG MARKER: lctl pool_list lustre [ 867.065891] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 871.358834] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 871.358834] sort -u | tr '\n' ' ' [ 871.653259] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 871.653259] sort -u | tr '\n' ' ' [ 873.003636] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 877.295988] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 877.580503] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 879.914466] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [ 882.212642] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 882.212642] sort -u | tr '\n' ' ' [ 882.501277] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 882.501277] sort -u | tr '\n' ' ' [ 886.895692] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 887.556174] Lustre: DEBUG MARKER: rc=0; [ 887.556174] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 887.556174] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 887.556174] echo $(hostname -s): $val; [ 887.556174] rc=$val; [ 887.556174] fi; [ 887.556174] exit $rc [ 888.230960] Lustre: DEBUG MARKER: dmesg [ 888.596958] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 888.886205] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 889.172548] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 899.462497] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 907.769125] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 918.062910] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 918.351179] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 918.665264] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 918.954767] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 919.242476] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [ 927.533937] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 935.827946] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 936.115743] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 937.506522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7a: create various pool name ======================================================= 22:52:18 \(1491432738\) [ 937.714277] Lustre: DEBUG MARKER: == ost-pools test 7a: create various pool name ======================================================= 22:52:18 (1491432738) [ 937.924819] Lustre: DEBUG MARKER: lctl pool_new lustre.6 [ 946.219879] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.6 2>/dev/null || echo foo [ 946.507408] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.6 2>/dev/null || echo foo [ 946.821602] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.6 lustre-OST[0-1/1] [ 951.111510] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.6 | [ 951.111510] sort -u | tr '\n' ' ' [ 951.398547] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.6 | [ 951.398547] sort -u | tr '\n' ' ' [ 957.850027] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.6 [ 958.138681] Lustre: DEBUG MARKER: lctl pool_remove lustre.6 lustre-OST0000_UUID [ 960.428811] Lustre: DEBUG MARKER: lctl pool_remove lustre.6 lustre-OST0001_UUID [ 966.722809] Lustre: DEBUG MARKER: lctl pool_destroy lustre.6 [ 975.017416] Lustre: DEBUG MARKER: lctl pool_new lustre.666214970 [ 983.310587] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666214970 2>/dev/null || echo foo [ 983.600938] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666214970 2>/dev/null || echo foo [ 983.916945] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.666214970 lustre-OST[0-1/1] [ 992.218376] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666214970 | [ 992.218376] sort -u | tr '\n' ' ' [ 992.528397] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666214970 | [ 992.528397] sort -u | tr '\n' ' ' [ 992.884705] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.666214970 [ 993.173522] Lustre: DEBUG MARKER: lctl pool_remove lustre.666214970 lustre-OST0000_UUID [ 1001.466838] Lustre: DEBUG MARKER: lctl pool_remove lustre.666214970 lustre-OST0001_UUID [ 1009.760554] Lustre: DEBUG MARKER: lctl pool_destroy lustre.666214970 [ 1020.055628] Lustre: DEBUG MARKER: lctl pool_new lustre.666228531473 [ 1030.347718] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666228531473 2>/dev/null || echo foo [ 1030.634870] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666228531473 2>/dev/null || echo foo [ 1030.954712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.666228531473 lustre-OST[0-1/1] [ 1037.247853] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666228531473 | [ 1037.247853] sort -u | tr '\n' ' ' [ 1037.540883] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.666228531473 | [ 1037.540883] sort -u | tr '\n' ' ' [ 1041.929445] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.666228531473 [ 1042.219408] Lustre: DEBUG MARKER: lctl pool_remove lustre.666228531473 lustre-OST0000_UUID [ 1048.509767] Lustre: DEBUG MARKER: lctl pool_remove lustre.666228531473 lustre-OST0001_UUID [ 1056.801639] Lustre: DEBUG MARKER: lctl pool_destroy lustre.666228531473 [ 1063.135532] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1063.798642] Lustre: DEBUG MARKER: rc=0; [ 1063.798642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1063.798642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1063.798642] echo $(hostname -s): $val; [ 1063.798642] rc=$val; [ 1063.798642] fi; [ 1063.798642] exit $rc [ 1064.487651] Lustre: DEBUG MARKER: dmesg [ 1064.870544] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1065.158327] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1065.446568] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1065.881546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 22:54:27 \(1491432867\) [ 1066.070613] Lustre: DEBUG MARKER: == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 22:54:27 (1491432867) [ 1066.238960] Lustre: DEBUG MARKER: lctl pool_new qwerty [ 1066.525256] Lustre: DEBUG MARKER: lctl pool_new lustre. [ 1066.811743] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789^bdef [ 1067.097576] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abdefg [ 1067.418161] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1068.078756] Lustre: DEBUG MARKER: rc=0; [ 1068.078756] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1068.078756] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1068.078756] echo $(hostname -s): $val; [ 1068.078756] rc=$val; [ 1068.078756] fi; [ 1068.078756] exit $rc [ 1068.751956] Lustre: DEBUG MARKER: dmesg [ 1069.171323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 22:54:30 \(1491432870\) [ 1069.360122] Lustre: DEBUG MARKER: == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 22:54:30 (1491432870) [ 1069.528961] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abcde [ 1079.827135] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [ 1080.116967] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [ 1080.435308] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.0123456789abcde lustre-OST[0-1/1] [ 1090.734346] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [ 1090.734346] sort -u | tr '\n' ' ' [ 1091.023922] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [ 1091.023922] sort -u | tr '\n' ' ' [ 1091.340447] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.0123456789abcde [ 1091.628801] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0000_UUID [ 1099.922927] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0001_UUID [ 1110.212647] Lustre: DEBUG MARKER: lctl pool_destroy lustre.0123456789abcde [ 1114.546170] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1115.224858] Lustre: DEBUG MARKER: rc=0; [ 1115.224858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1115.224858] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1115.224858] echo $(hostname -s): $val; [ 1115.224858] rc=$val; [ 1115.224858] fi; [ 1115.224858] exit $rc [ 1115.929164] Lustre: DEBUG MARKER: dmesg [ 1116.290367] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1116.642991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 11: OSTs in overlapping\/multiple pools ============================================= 22:55:17 \(1491432917\) [ 1116.830956] Lustre: DEBUG MARKER: == ost-pools test 11: OSTs in overlapping/multiple pools ============================================= 22:55:17 (1491432917) [ 1117.004606] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1123.297691] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1123.582283] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1123.895378] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 1132.192906] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1132.479147] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1132.795798] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0001/2] [ 1141.090569] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/2] [ 1147.382287] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1147.382287] sort -u | tr '\n' ' ' [ 1147.669810] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1147.669810] sort -u | tr '\n' ' ' [ 1151.160098] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1151.455834] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1151.759337] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1152.056945] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1152.359306] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1152.653441] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1152.955412] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1153.248261] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1153.552250] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1153.847340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1154.149312] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1154.446186] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1154.748314] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1155.045035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1155.349154] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1155.645573] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1155.948877] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1156.245095] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1156.549431] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1156.842860] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1157.146857] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1157.442034] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1157.745701] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1158.044584] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1158.348722] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1158.644674] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1158.952532] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1159.247592] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1159.549956] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1159.844806] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1160.148932] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1160.444882] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1160.746752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1161.041446] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1161.342651] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1161.638560] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1161.938465] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1162.232578] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1162.532324] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1162.827560] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1163.129108] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1163.421959] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1163.722735] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1164.017600] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1164.319153] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1164.613840] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1164.911900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1165.206228] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1165.504858] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1165.795351] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1166.095808] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1166.390076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1166.687850] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1166.979808] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1167.288585] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1167.585054] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1167.884626] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1168.180114] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1168.478105] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1168.775366] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1169.077600] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1169.371159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1169.669656] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1169.961278] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1170.261470] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1170.555117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1170.856612] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1171.158075] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1171.458764] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1171.749076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1172.048299] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1172.342792] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1172.642976] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1172.934348] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1173.232932] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1173.525277] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1173.825135] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1174.120723] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1174.421586] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1174.713310] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1175.010749] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1175.303245] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1175.602608] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1175.897162] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1176.196048] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1176.488306] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1176.785590] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1177.084685] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1177.386246] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1177.677832] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1177.979393] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1178.273753] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1178.575620] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1178.871102] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1179.172332] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1179.467700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1179.767697] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1180.061962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1180.363551] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1180.661632] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1180.965556] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1181.260558] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1181.562510] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1181.857356] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1182.163116] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1182.458427] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1182.757841] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1183.050822] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1183.352873] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1183.643205] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1183.943976] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1184.237478] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1184.541642] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1184.835083] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1185.134125] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1185.430731] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1185.732487] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1186.026873] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1186.331689] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1186.622318] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1186.921540] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1187.220919] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1187.524548] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1187.827994] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1188.127491] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1188.421876] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1188.717979] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1189.010694] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1189.310027] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1189.602583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1189.908458] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1190.213507] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1190.512949] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1190.814109] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1191.125445] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1191.420462] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1191.725878] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1192.023212] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1192.330161] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1192.623403] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1192.921184] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1193.215738] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1193.519650] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1193.816320] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1194.116695] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1194.408549] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1194.710677] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1195.004746] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1195.313193] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1195.609300] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1195.910783] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1196.207046] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1196.507902] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1196.799720] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1197.104792] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1197.401894] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1197.701078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1197.993894] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1198.291107] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1198.585741] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1198.883812] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1199.176189] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1199.482259] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1199.790265] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1200.098773] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1200.395647] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1200.703495] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1200.997678] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1201.299052] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1201.597117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1201.897754] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1202.193067] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1202.490163] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1202.783330] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1203.085077] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1203.378607] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1203.679036] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1203.977887] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1204.276870] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1204.569620] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1204.871866] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1205.165443] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1205.465942] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1205.756709] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1206.055123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1206.348991] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1206.645575] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1206.940510] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1207.245665] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1207.537633] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1207.835594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1208.129825] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1208.430110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1208.721645] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1209.022496] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1209.317784] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1209.616946] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1209.910433] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1210.208118] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1210.498337] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1210.902558] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1211.196510] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1211.494493] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1211.785376] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1212.088138] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1212.385635] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1212.688130] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1212.981876] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1213.282856] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1213.578117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1213.878698] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1214.173532] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1214.476475] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1214.767679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1215.068530] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1215.364005] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1215.664376] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1215.958316] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1216.260695] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1216.556632] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1216.856434] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1217.154426] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1217.457665] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1217.749596] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1218.048897] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1218.345770] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1218.643776] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1218.937192] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1219.245870] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1219.541730] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1219.844953] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1220.138401] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1220.439545] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1220.734354] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1221.035727] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1221.335340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1221.639191] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1221.932341] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1222.239252] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1222.533244] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1222.833359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1223.128678] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1223.430390] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1223.723101] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1224.021473] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1224.313484] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1224.613455] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1224.905330] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1225.202259] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1225.495749] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1225.793471] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1226.085149] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1226.385752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1226.677288] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1226.973070] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1227.269191] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1227.567048] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1227.860314] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1228.163263] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1228.456630] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1228.757537] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1229.050752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1229.352735] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1229.647339] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1229.945347] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1230.237137] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1230.538370] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1230.832588] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1231.141650] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1231.437885] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1231.738676] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1232.036367] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1232.345582] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1232.638857] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1232.943484] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1233.237583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1233.536670] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1233.830654] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1234.130340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1234.426524] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1234.724752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1235.017656] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1235.332269] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1235.626647] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1235.929532] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1236.227891] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1236.530355] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1236.825402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1237.129486] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1237.423621] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1237.720823] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1238.013467] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1238.312586] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1238.604992] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1238.904049] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1239.198761] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1239.497824] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1239.792717] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1240.096333] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1240.392340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1240.695911] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1240.989665] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1241.290847] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1241.589723] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1241.891942] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1242.187776] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1242.487636] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1242.780794] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1243.079721] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1243.372377] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1243.671551] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1243.965296] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1244.269094] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1244.564331] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1244.869825] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1245.167786] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1245.470780] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1245.764968] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1246.065752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1246.358355] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1246.657648] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1246.952623] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1247.259892] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1247.554994] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1247.855301] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1248.150101] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1248.454597] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1248.749220] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1249.050175] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1249.345799] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1249.647183] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1249.941616] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1250.241852] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1250.534160] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1250.833571] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1251.125679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1251.425679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1251.717747] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1252.016726] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1252.313080] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1252.613111] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1252.910748] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1253.216132] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1253.511334] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1253.810782] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1254.103117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1254.404561] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1254.697754] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1254.996171] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1255.286566] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1255.588854] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1255.879711] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1256.184395] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1256.476797] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1256.774090] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1257.066035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1257.368470] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1257.660268] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1257.957491] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1258.258540] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1258.561683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1258.856137] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1259.155700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1259.450058] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1259.747763] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1260.041452] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1260.341576] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1260.634640] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1260.933392] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1261.226061] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1261.524272] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1261.820420] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1262.122661] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1262.416952] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1262.719606] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1263.012564] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1263.312468] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1263.607725] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1263.908111] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1264.202454] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1264.505402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1264.798547] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1265.098312] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1265.391076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1265.688881] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1265.980402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1266.277751] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1266.569137] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1266.868601] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1267.165185] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1267.463141] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1267.756892] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1268.057866] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1268.350636] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1268.649825] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1268.942600] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1269.241512] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1269.535095] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1269.836191] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1270.129843] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1270.590387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1271.255553] Lustre: DEBUG MARKER: rc=0; [ 1271.255553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1271.255553] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1271.255553] echo $(hostname -s): $val; [ 1271.255553] rc=$val; [ 1271.255553] fi; [ 1271.255553] exit $rc [ 1271.930110] Lustre: DEBUG MARKER: dmesg [ 1272.314875] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1272.604212] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1272.890511] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1281.190235] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1289.493700] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1289.780531] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1290.095891] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1290.385654] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1290.674679] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [ 1296.965357] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 1305.259575] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1305.549961] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1305.926841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 12: OST Pool Membership ============================================================ 22:58:27 \(1491433107\) [ 1306.115551] Lustre: DEBUG MARKER: == ost-pools test 12: OST Pool Membership ============================================================ 22:58:27 (1491433107) [ 1306.324422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_12 Need at least 3 OSTs [ 1306.501372] Lustre: DEBUG MARKER: SKIP: ost-pools test_12 Need at least 3 OSTs [ 1306.703421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1307.370619] Lustre: DEBUG MARKER: rc=0; [ 1307.370619] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1307.370619] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1307.370619] echo $(hostname -s): $val; [ 1307.370619] rc=$val; [ 1307.370619] fi; [ 1307.370619] exit $rc [ 1308.050996] Lustre: DEBUG MARKER: dmesg [ 1308.456404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 13: Striping characteristics in a pool ============================================= 22:58:29 \(1491433109\) [ 1308.640856] Lustre: DEBUG MARKER: == ost-pools test 13: Striping characteristics in a pool ============================================= 22:58:29 (1491433109) [ 1308.845161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_13 Need at least 3 OSTs [ 1309.020356] Lustre: DEBUG MARKER: SKIP: ost-pools test_13 Need at least 3 OSTs [ 1309.222582] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1309.884897] Lustre: DEBUG MARKER: rc=0; [ 1309.884897] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1309.884897] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1309.884897] echo $(hostname -s): $val; [ 1309.884897] rc=$val; [ 1309.884897] fi; [ 1309.884897] exit $rc [ 1310.574172] Lustre: DEBUG MARKER: dmesg [ 1311.000166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 22:58:32 \(1491433112\) [ 1311.192923] Lustre: DEBUG MARKER: == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 22:58:32 (1491433112) [ 1311.407566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_14 Need at least 3 OSTs [ 1311.584766] Lustre: DEBUG MARKER: SKIP: ost-pools test_14 Need at least 3 OSTs [ 1311.791331] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1312.458988] Lustre: DEBUG MARKER: rc=0; [ 1312.458988] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1312.458988] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1312.458988] echo $(hostname -s): $val; [ 1312.458988] rc=$val; [ 1312.458988] fi; [ 1312.458988] exit $rc [ 1313.153090] Lustre: DEBUG MARKER: dmesg [ 1313.584325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 15: One directory per OST\/pool ===================================================== 22:58:34 \(1491433114\) [ 1313.775630] Lustre: DEBUG MARKER: == ost-pools test 15: One directory per OST/pool ===================================================== 22:58:34 (1491433114) [ 1313.957288] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool0 [ 1324.251878] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [ 1324.539632] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [ 1324.862340] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool0 lustre-OST[0000] [ 1335.160671] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [ 1335.160671] sort -u | tr '\n' ' ' [ 1335.448329] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [ 1335.448329] sort -u | tr '\n' ' ' [ 1336.905209] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [ 1343.200795] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 1343.487790] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 1344.817231] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 lustre-OST[0001] [ 1357.112141] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [ 1357.112141] sort -u | tr '\n' ' ' [ 1357.401405] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [ 1357.401405] sort -u | tr '\n' ' ' [ 1358.892182] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1359.561086] Lustre: DEBUG MARKER: rc=0; [ 1359.561086] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1359.561086] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1359.561086] echo $(hostname -s): $val; [ 1359.561086] rc=$val; [ 1359.561086] fi; [ 1359.561086] exit $rc [ 1360.240419] Lustre: DEBUG MARKER: dmesg [ 1360.614124] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1360.906815] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool0 [ 1361.193539] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool0 lustre-OST0000_UUID [ 1371.487519] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool0 [ 1379.779872] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [ 1380.067537] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [ 1380.387615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1380.686146] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [ 1380.981059] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0001_UUID [ 1387.271142] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [ 1397.564126] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 1397.852683] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 1398.222286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 16: Inheritance of pool properties ================================================= 22:59:59 \(1491433199\) [ 1398.405521] Lustre: DEBUG MARKER: == ost-pools test 16: Inheritance of pool properties ================================================= 22:59:59 (1491433199) [ 1398.576120] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1406.868984] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1407.155809] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1409.490521] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/2] [ 1415.782214] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1415.782214] sort -u | tr '\n' ' ' [ 1416.074803] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1416.074803] sort -u | tr '\n' ' ' [ 1416.431733] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1416.726724] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1417.025680] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1417.324415] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1417.626123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1417.919586] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1418.217773] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1418.513159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1418.813707] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1419.107593] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1419.413645] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1419.708559] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1420.010224] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1420.303497] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1420.608473] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1420.906058] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1421.204447] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1421.495741] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1421.792868] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1422.085886] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1422.470926] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1423.135702] Lustre: DEBUG MARKER: rc=0; [ 1423.135702] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1423.135702] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1423.135702] echo $(hostname -s): $val; [ 1423.135702] rc=$val; [ 1423.135702] fi; [ 1423.135702] exit $rc [ 1423.812752] Lustre: DEBUG MARKER: dmesg [ 1424.168540] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1424.456916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1424.744503] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1427.034082] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1431.330942] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1431.617642] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1431.986522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 17: Referencing an empty pool ====================================================== 23:00:33 \(1491433233\) [ 1432.186508] Lustre: DEBUG MARKER: == ost-pools test 17: Referencing an empty pool ====================================================== 23:00:33 (1491433233) [ 1432.338238] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1436.631181] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1436.918609] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1439.261115] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 1445.556193] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1445.556193] sort -u | tr '\n' ' ' [ 1445.844520] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1445.844520] sort -u | tr '\n' ' ' [ 1446.189950] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1446.503149] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1446.818853] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1447.117624] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1447.425726] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1447.721410] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1448.023360] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1448.317246] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1448.620951] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1448.916937] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1449.224683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1449.522364] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1449.827197] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1450.125738] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1450.430585] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1450.725399] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1451.026736] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1451.319808] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1451.622469] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1451.917108] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1452.224811] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1452.511842] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1452.796755] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1461.089929] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 1471.382148] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1477.675244] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1477.961886] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1478.349680] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1479.011406] Lustre: DEBUG MARKER: rc=0; [ 1479.011406] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1479.011406] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1479.011406] echo $(hostname -s): $val; [ 1479.011406] rc=$val; [ 1479.011406] fi; [ 1479.011406] exit $rc [ 1479.706935] Lustre: DEBUG MARKER: dmesg [ 1480.121680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_18 skipping SLOW test 18 [ 1480.312064] Lustre: DEBUG MARKER: SKIP: ost-pools test_18 skipping SLOW test 18 [ 1480.538146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 19: Pools should not come into play when not specified ============================= 23:01:21 \(1491433281\) [ 1480.720598] Lustre: DEBUG MARKER: == ost-pools test 19: Pools should not come into play when not specified ============================= 23:01:21 (1491433281) [ 1480.891402] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1485.194641] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1485.478631] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1487.816700] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/2] [ 1490.111441] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1490.111441] sort -u | tr '\n' ' ' [ 1490.398313] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1490.398313] sort -u | tr '\n' ' ' [ 1496.814186] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1497.114803] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1497.420380] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1497.718515] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1498.018879] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1498.312054] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1498.610950] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1498.903345] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1499.204313] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1499.495105] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1499.796936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1500.090923] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1500.392383] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1500.686308] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1500.986877] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1501.280100] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1501.580308] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1501.877287] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1502.182433] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1502.482537] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1502.787868] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1503.086706] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1503.389052] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1503.685315] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1504.102975] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1504.769049] Lustre: DEBUG MARKER: rc=0; [ 1504.769049] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1504.769049] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1504.769049] echo $(hostname -s): $val; [ 1504.769049] rc=$val; [ 1504.769049] fi; [ 1504.769049] exit $rc [ 1505.449410] Lustre: DEBUG MARKER: dmesg [ 1505.834456] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1506.122773] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1506.411230] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1516.705155] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1524.999565] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1525.288993] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1525.662059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 23:02:06 \(1491433326\) [ 1525.848539] Lustre: DEBUG MARKER: == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 23:02:06 (1491433326) [ 1526.014705] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1532.308478] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1532.595046] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1534.932300] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 1541.222807] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1541.508584] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1544.856828] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/2] [ 1555.158387] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1555.158387] sort -u | tr '\n' ' ' [ 1555.459045] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1555.459045] sort -u | tr '\n' ' ' [ 1555.777788] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0001/2] [ 1566.071803] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 1566.071803] sort -u | tr '\n' ' ' [ 1566.363080] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 1566.363080] sort -u | tr '\n' ' ' [ 1566.724380] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1567.020814] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1567.325615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1567.618191] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1568.002018] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1568.667409] Lustre: DEBUG MARKER: rc=0; [ 1568.667409] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1568.667409] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1568.667409] echo $(hostname -s): $val; [ 1568.667409] rc=$val; [ 1568.667409] fi; [ 1568.667409] exit $rc [ 1569.355644] Lustre: DEBUG MARKER: dmesg [ 1569.720889] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1570.009425] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1570.296831] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1578.588318] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1588.883837] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1589.171088] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1589.486593] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1589.772931] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1590.059377] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [ 1600.354877] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 1608.649482] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1608.939352] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1609.318997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 23:03:30 \(1491433410\) [ 1609.510701] Lustre: DEBUG MARKER: == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 23:03:30 (1491433410) [ 1609.673376] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1617.967471] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1618.256507] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1618.575344] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/2] [ 1624.869097] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1624.869097] sort -u | tr '\n' ' ' [ 1625.160541] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1625.160541] sort -u | tr '\n' ' ' [ 1627.526366] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1627.822630] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1628.168218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1628.824178] Lustre: DEBUG MARKER: rc=0; [ 1628.824178] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1628.824178] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1628.824178] echo $(hostname -s): $val; [ 1628.824178] rc=$val; [ 1628.824178] fi; [ 1628.824178] exit $rc [ 1629.507378] Lustre: DEBUG MARKER: dmesg [ 1629.881684] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1630.168719] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1630.457340] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1640.749455] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1647.045883] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1647.332912] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1648.711353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 23:04:10 \(1491433450\) [ 1648.904103] Lustre: DEBUG MARKER: == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 23:04:10 (1491433450) [ 1649.072546] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1655.373091] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1655.661876] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1655.984765] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [ 1664.277493] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1664.277493] sort -u | tr '\n' ' ' [ 1664.568839] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1664.568839] sort -u | tr '\n' ' ' [ 1664.878253] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [ 1673.172682] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1673.461344] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1674.794981] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [ 1679.099822] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 1679.099822] sort -u | tr '\n' ' ' [ 1679.398682] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [ 1679.398682] sort -u | tr '\n' ' ' [ 1682.761566] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1682.765476] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1683.061039] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1683.341171] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1683.619662] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1683.898623] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1684.178291] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1684.458859] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1684.739040] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1685.019382] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1685.301965] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0001/2] [ 1687.074308] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1691.366309] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1699.656917] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1709.952516] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1716.246426] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1722.540053] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1730.834931] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1741.126052] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1747.419805] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1753.712239] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1764.005969] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1772.302554] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1780.598947] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1788.891526] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1795.182923] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1803.477943] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1813.768573] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1824.067230] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0001/1] [ 1832.361395] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0001/1] [ 1838.695401] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1839.360789] Lustre: DEBUG MARKER: rc=0; [ 1839.360789] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1839.360789] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1839.360789] echo $(hostname -s): $val; [ 1839.360789] rc=$val; [ 1839.360789] fi; [ 1839.360789] exit $rc [ 1840.047396] Lustre: DEBUG MARKER: dmesg [ 1840.415508] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1840.703579] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1840.989984] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1845.282692] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1853.577884] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1853.863171] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1854.183202] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1854.472491] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [ 1854.760852] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [ 1861.054710] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [ 1867.357417] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1867.642204] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [ 1868.015816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 23a: OST pools and quota =========================================================== 23:07:49 \(1491433669\) [ 1868.207067] Lustre: DEBUG MARKER: == ost-pools test 23a: OST pools and quota =========================================================== 23:07:49 (1491433669) [ 1868.610806] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1872.901692] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1873.187426] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1877.553920] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/3] [ 1881.847217] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1881.847217] sort -u | tr '\n' ' ' [ 1882.139138] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1882.139138] sort -u | tr '\n' ' ' [ 1887.519989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1887.813393] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 1946.442744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first dd failed with EDQUOT. [ 1946.624544] Lustre: DEBUG MARKER: first dd failed with EDQUOT. [ 1946.843439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1947.490844] Lustre: DEBUG MARKER: rc=0; [ 1947.490844] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1947.490844] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1947.490844] echo $(hostname -s): $val; [ 1947.490844] rc=$val; [ 1947.490844] fi; [ 1947.490844] exit $rc [ 1948.171362] Lustre: DEBUG MARKER: dmesg [ 1948.543481] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 1948.835279] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1949.133175] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 1959.424204] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 1969.718293] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1970.005111] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1970.448364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_23b skipping SLOW test 23b [ 1970.640040] Lustre: DEBUG MARKER: SKIP: ost-pools test_23b skipping SLOW test 23b [ 1970.879647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 23:09:32 \(1491433772\) [ 1971.063595] Lustre: DEBUG MARKER: == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 23:09:32 (1491433772) [ 1971.225834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 1971.515233] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [ 1979.806857] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1980.092983] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 1980.411833] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0001/1] [ 1988.708596] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1988.708596] sort -u | tr '\n' ' ' [ 1989.001184] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [ 1989.001184] sort -u | tr '\n' ' ' [ 1989.375235] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1989.674590] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1989.993679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1990.290725] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1990.609958] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1990.906968] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1991.223037] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1991.522316] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1991.836756] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1992.131994] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1992.454959] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1992.767276] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1993.093805] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1993.392376] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1993.711747] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1994.005547] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1994.322689] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1994.624828] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1994.945068] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1995.240750] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1995.581620] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1995.873700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1996.186985] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1996.483551] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1996.798195] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1997.093807] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1997.404456] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1997.699772] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1998.015075] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1998.312676] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1998.623586] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1998.917286] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1999.229898] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1999.522469] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 1999.833781] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2000.128042] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2000.442346] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2000.736533] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2001.047217] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2001.342507] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2002.029721] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2002.700478] Lustre: DEBUG MARKER: rc=0; [ 2002.700478] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2002.700478] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2002.700478] echo $(hostname -s): $val; [ 2002.700478] rc=$val; [ 2002.700478] fi; [ 2002.700478] exit $rc [ 2003.382768] Lustre: DEBUG MARKER: dmesg [ 2003.756499] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 2004.047579] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [ 2004.332452] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [ 2012.622741] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [ 2020.916150] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [ 2029.211148] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 2029.502604] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [ 2029.873307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 25: Create new pool and restart MDS ================================================ 23:10:31 \(1491433831\) [ 2030.058921] Lustre: DEBUG MARKER: == ost-pools test 25: Create new pool and restart MDS ================================================ 23:10:31 (1491433831) [ 2030.232218] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [ 2034.531133] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 2034.819030] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 2039.173622] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 OST0000; sync [ 2047.577444] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2047.865611] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2048.008561] Lustre: Failing over lustre-MDT0000 [ 2048.176099] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.293168] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.296378] LustreError: Skipped 1 previous similar message [ 2052.058904] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.042033] Lustre: 29669:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491433848/real 1491433848] req@ffff88005e081800 x1563879826088240/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491433854 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2054.060587] Lustre: server umount lustre-MDT0000 complete [ 2054.203642] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2054.499921] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2054.499921] grep -q ^lustre-mdt1/ /proc/mounts || [ 2054.499921] zpool export lustre-mdt1 [ 2065.064073] Lustre: DEBUG MARKER: hostname [ 2065.379772] Lustre: DEBUG MARKER: modprobe zfs; [ 2065.379772] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2065.379772] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2065.989723] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2066.275355] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2066.459845] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2066.462130] Lustre: Skipped 3 previous similar messages [ 2066.622622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.827899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2067.116373] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 2067.603621] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2067.891128] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2068.179221] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2068.473553] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2068.767413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2069.061139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2069.372068] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2069.662464] Lustre: DEBUG MARKER: lctl get_param -n at_min [ 2069.952301] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 2070.176947] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2070.177278] Lustre: MGS: Connection restored to e2ce2e05-a1b3-0c7a-22f6-b0410d9ccdfc (at 10.9.5.238@tcp) [ 2070.591746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2070.592901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2070.777910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2070.777918] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2074.061180] Lustre: lustre-MDT0000: Connection restored to 6a04c9a8-6892-d84a-f4e7-7a2f1fa7b691 (at 10.9.5.237@tcp) [ 2074.061184] Lustre: MGS: Connection restored to 6a04c9a8-6892-d84a-f4e7-7a2f1fa7b691 (at 10.9.5.237@tcp) [ 2074.061186] Lustre: Skipped 5 previous similar messages [ 2074.149797] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2074.440097] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 2074.728075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 2082.087563] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [ 2082.384581] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [ 2082.727077] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2083.396889] Lustre: DEBUG MARKER: rc=0; [ 2083.396889] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2083.396889] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2083.396889] echo $(hostname -s): $val; [ 2083.396889] rc=$val; [ 2083.396889] fi; [ 2083.396889] exit $rc [ 2084.084363] Lustre: DEBUG MARKER: dmesg [ 2084.469148] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 2084.772840] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [ 2085.076688] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0000_UUID [ 2095.385818] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [ 2101.681227] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 2101.969746] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [ 2105.382846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 23:11:46 \(1491433906\) [ 2105.578576] Lustre: DEBUG MARKER: == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 23:11:46 (1491433906) [ 2105.785101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_26 Need at least 3 OSTs [ 2105.973246] Lustre: DEBUG MARKER: SKIP: ost-pools test_26 Need at least 3 OSTs [ 2106.184995] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2106.840480] Lustre: DEBUG MARKER: rc=0; [ 2106.840480] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2106.840480] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2106.840480] echo $(hostname -s): $val; [ 2106.840480] rc=$val; [ 2106.840480] fi; [ 2106.840480] exit $rc [ 2107.522974] Lustre: DEBUG MARKER: dmesg [ 2107.938318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration 1840 sec ======================================================== 23:11:49 \(1491433909\) [ 2108.139882] Lustre: DEBUG MARKER: == ost-pools test complete, duration 1840 sec ======================================================== 23:11:49 (1491433909) [ 2108.729266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-lfsck ============----- Wed Apr 5 23:11:50 UTC 2017 [ 2108.921821] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Wed Apr 5 23:11:50 UTC 2017 [ 2109.234103] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test complete, duration -o sec ======================================================= 23:11:50 \(1491433910\) [ 2109.452816] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration -o sec ======================================================= 23:11:50 (1491433910) [ 2109.844401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2110.499832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2111.537961] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2111.825448] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2114.224193] LustreError: 5891:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079bfc300 x1563879826097168/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2116.308667] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2121.303987] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.004046] Lustre: 1881:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491433918/real 1491433918] req@ffff88005e2eb300 x1563879826097280/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491433924 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2124.024139] Lustre: server umount lustre-MDT0000 complete [ 2124.169121] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2124.468399] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2124.468399] grep -q ^lustre-mdt1/ /proc/mounts || [ 2124.468399] zpool export lustre-mdt1 [ 2133.733855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2134.023580] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2134.620647] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2134.869429] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2135.134050] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2135.134050] grep -q ^lustre-mdt1/ /proc/mounts || [ 2135.134050] zpool export lustre-mdt1 [ 2135.407242] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=zfs --device-size=100000 --reformat lustre-mdt1/mdt1 /dev/lv [ 2136.061762] Lustre: DEBUG MARKER: zpool set cachefile=none lustre-mdt1 [ 2136.351412] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2136.351412] grep -q ^lustre-mdt1/ /proc/mounts || [ 2136.351412] zpool export lustre-mdt1 [ 2141.607505] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2141.607505] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2141.607505] if [ $running -ne $mpts ]; then [ 2141.607505] echo $(hostname) env are INSANE!; [ 2141.607505] exit 1; [ 2141.607505] fi [ 2141.904111] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2141.904111] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2141.904111] if [ $running -ne $mpts ]; then [ 2141.904111] echo $(hostname) env are INSANE!; [ 2141.904111] exit 1; [ 2141.904111] fi [ 2142.818644] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2143.098386] Lustre: DEBUG MARKER: modprobe zfs; [ 2143.098386] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2143.098386] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2143.565120] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2143.847070] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2144.208470] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2144.220977] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripecount in log lustre-MDT0000 [ 2144.223377] Lustre: Skipped 4 previous similar messages [ 2144.269054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 2144.271166] Lustre: Skipped 1 previous similar message [ 2144.286290] Lustre: lustre-MDT0000: new disk, initializing [ 2144.459208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.520349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2144.710837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2144.999805] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2145.482845] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2145.772601] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2146.060897] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2146.350682] Lustre: DEBUG MARKER: sync; sync; sync [ 2151.700397] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2151.990057] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2162.760484] Lustre: lustre-MDT0000: Connection restored to ae0cff62-4df9-6df9-aa7a-87848c801b97 (at 10.9.5.244@tcp) [ 2162.763072] Lustre: Skipped 3 previous similar messages [ 2172.600446] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2172.924861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2173.116137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2173.279813] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2178.594286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2179.224777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2179.567149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 31 [ 2179.748427] Lustre: DEBUG MARKER: excepting tests: 31 [ 2179.971451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 0: Control LFSCK manually ======================================================= 23:13:01 \(1491433981\) [ 2180.159852] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually ======================================================= 23:13:01 (1491433981) [ 2180.877902] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x1600 [ 2181.169291] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2181.492729] LustreError: 6048:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [ 2181.635098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2181.923829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2182.209854] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [ 2184.495035] LustreError: 6048:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 2184.722049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2185.012523] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2185.193464] LustreError: 6416:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [ 2185.339983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2185.626866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2185.914330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2185.914330] awk '/^status/ { print $2 }' [ 2186.205980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2186.205980] awk '/^status/ { print $2 }' [ 2187.500298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2187.500298] awk '/^status/ { print $2 }' [ 2188.196022] LustreError: 6416:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 2188.799363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2188.799363] awk '/^status/ { print $2 }' [ 2189.089697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2189.382696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2189.670882] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2190.069808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2190.069808] awk '/^status/ { print $2 }' [ 2190.357870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2190.357870] awk '/^status/ { print $2 }' [ 2190.647683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2191.700791] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2191.994041] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2192.768861] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2192.771147] Lustre: Skipped 1 previous similar message [ 2195.088063] LustreError: 5891:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006399c300 x1563879826099056/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2195.092837] LustreError: 5891:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2196.176050] LustreError: 5891:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079bfc000 x1563879826099088/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2202.760736] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.765641] LustreError: Skipped 1 previous similar message [ 2204.178068] Lustre: 7452:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491433998/real 1491433998] req@ffff880079bfc000 x1563879826099168/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434004 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2204.198486] Lustre: server umount lustre-MDT0000 complete [ 2204.345153] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2204.643883] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2204.643883] grep -q ^lustre-mdt1/ /proc/mounts || [ 2204.643883] zpool export lustre-mdt1 [ 2213.877192] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2214.439594] Lustre: DEBUG MARKER: rc=0; [ 2214.439594] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2214.439594] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2214.439594] echo $(hostname -s): $val; [ 2214.439594] rc=$val; [ 2214.439594] fi; [ 2214.439594] exit $rc [ 2215.014197] Lustre: DEBUG MARKER: dmesg [ 2215.377093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================= 23:13:36 \(1491434016\) [ 2215.534788] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================= 23:13:36 (1491434016) [ 2215.704226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_1a OI Scrub not implemented for ZFS [ 2215.857519] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_1a OI Scrub not implemented for ZFS [ 2216.028383] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2216.601927] Lustre: DEBUG MARKER: rc=0; [ 2216.601927] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2216.601927] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2216.601927] echo $(hostname -s): $val; [ 2216.601927] rc=$val; [ 2216.601927] fi; [ 2216.601927] exit $rc [ 2217.167264] Lustre: DEBUG MARKER: dmesg [ 2217.526182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ======================== 23:13:38 \(1491434018\) [ 2217.676563] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ======================== 23:13:38 (1491434018) [ 2217.845997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_1b OI Scrub not implemented for ZFS [ 2217.990600] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_1b OI Scrub not implemented for ZFS [ 2218.158118] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2218.742667] Lustre: DEBUG MARKER: rc=0; [ 2218.742667] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2218.742667] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2218.742667] echo $(hostname -s): $val; [ 2218.742667] rc=$val; [ 2218.742667] fi; [ 2218.742667] exit $rc [ 2219.302598] Lustre: DEBUG MARKER: dmesg [ 2219.706885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================== 23:13:41 \(1491434021\) [ 2219.853925] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================== 23:13:41 (1491434021) [ 2220.575634] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2220.575634] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2220.575634] if [ $running -ne $mpts ]; then [ 2220.575634] echo $(hostname) env are INSANE!; [ 2220.575634] exit 1; [ 2220.575634] fi [ 2220.867601] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2220.867601] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2220.867601] if [ $running -ne $mpts ]; then [ 2220.867601] echo $(hostname) env are INSANE!; [ 2220.867601] exit 1; [ 2220.867601] fi [ 2221.778345] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2222.062036] Lustre: DEBUG MARKER: modprobe zfs; [ 2222.062036] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2222.062036] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2222.516980] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2222.801669] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2222.983977] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2222.986857] Lustre: Skipped 4 previous similar messages [ 2223.118637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.311297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2223.599097] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2224.084714] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2224.372989] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2224.662677] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2224.956665] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2225.246168] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2228.087071] Lustre: 5890:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434023/real 1491434023] req@ffff880079bfd200 x1563879826099344/t0(0) o8->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491434028 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2234.916074] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2235.242303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2235.441387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2235.607414] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2241.432986] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [ 2241.582346] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2241.727964] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2242.228644] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2242.611450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2242.611450] awk '/^status/ { print $2 }' [ 2242.898991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2242.898991] awk '/^status/ { print $2 }' [ 2243.187464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2243.579412] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2244.260641] Lustre: DEBUG MARKER: rc=0; [ 2244.260641] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2244.260641] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2244.260641] echo $(hostname -s): $val; [ 2244.260641] rc=$val; [ 2244.260641] fi; [ 2244.260641] exit $rc [ 2244.972328] Lustre: DEBUG MARKER: dmesg [ 2245.420412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================== 23:14:06 \(1491434046\) [ 2245.625124] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================== 23:14:06 (1491434046) [ 2246.347782] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [ 2246.496456] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2246.642169] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2247.135346] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2247.519481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2247.519481] awk '/^status/ { print $2 }' [ 2247.808309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2247.808309] awk '/^status/ { print $2 }' [ 2248.102921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2248.491327] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2249.166718] Lustre: DEBUG MARKER: rc=0; [ 2249.166718] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2249.166718] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2249.166718] echo $(hostname -s): $val; [ 2249.166718] rc=$val; [ 2249.166718] fi; [ 2249.166718] exit $rc [ 2249.851327] Lustre: DEBUG MARKER: dmesg [ 2250.271051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================= 23:14:11 \(1491434051\) [ 2250.456347] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================= 23:14:11 (1491434051) [ 2251.430449] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1605 [ 2251.579441] Lustre: *** cfs_fail_loc=1605, val=0*** [ 2251.725538] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2252.195819] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2252.587282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2252.587282] awk '/^status/ { print $2 }' [ 2252.875114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2252.875114] awk '/^status/ { print $2 }' [ 2253.165970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2253.546396] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2254.214961] Lustre: DEBUG MARKER: rc=0; [ 2254.214961] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2254.214961] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2254.214961] echo $(hostname -s): $val; [ 2254.214961] rc=$val; [ 2254.214961] fi; [ 2254.214961] exit $rc [ 2254.903837] Lustre: DEBUG MARKER: dmesg [ 2255.346056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ================================== 23:14:16 \(1491434056\) [ 2255.542641] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ================================== 23:14:16 (1491434056) [ 2256.424250] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161d [ 2256.571997] Lustre: *** cfs_fail_loc=161d, val=0*** [ 2256.717144] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2257.184492] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2257.579740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2257.579740] awk '/^status/ { print $2 }' [ 2257.871599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2257.871599] awk '/^status/ { print $2 }' [ 2258.162121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2258.540406] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2259.189891] Lustre: DEBUG MARKER: rc=0; [ 2259.189891] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2259.189891] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2259.189891] echo $(hostname -s): $val; [ 2259.189891] rc=$val; [ 2259.189891] fi; [ 2259.189891] exit $rc [ 2259.875218] Lustre: DEBUG MARKER: dmesg [ 2260.296019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ============================= 23:14:21 \(1491434061\) [ 2260.494298] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ============================= 23:14:21 (1491434061) [ 2260.704309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_2e We need at least 2 MDSes for this test [ 2260.888135] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_2e We need at least 2 MDSes for this test [ 2261.087628] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2261.745599] Lustre: DEBUG MARKER: rc=0; [ 2261.745599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2261.745599] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2261.745599] echo $(hostname -s): $val; [ 2261.745599] rc=$val; [ 2261.745599] fi; [ 2261.745599] exit $rc [ 2262.426259] Lustre: DEBUG MARKER: dmesg [ 2262.843127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ===================================== 23:14:24 \(1491434064\) [ 2263.028657] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ===================================== 23:14:24 (1491434064) [ 2263.731069] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [ 2263.885064] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2264.033079] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [ 2264.189679] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2264.331170] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2264.618252] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2264.960651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2264.960651] awk '/^status/ { print $2 }' [ 2265.251431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2265.251431] awk '/^status/ { print $2 }' [ 2265.540690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2265.826530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2266.153722] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2266.828746] Lustre: DEBUG MARKER: rc=0; [ 2266.828746] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2266.828746] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2266.828746] echo $(hostname -s): $val; [ 2266.828746] rc=$val; [ 2266.828746] fi; [ 2266.828746] exit $rc [ 2267.520095] Lustre: DEBUG MARKER: dmesg [ 2267.935083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup\/restore ============= 23:14:29 \(1491434069\) [ 2268.125035] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ============= 23:14:29 (1491434069) [ 2268.335785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_4 OI Scrub not implemented for ZFS [ 2268.530435] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_4 OI Scrub not implemented for ZFS [ 2268.729324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2269.381647] Lustre: DEBUG MARKER: rc=0; [ 2269.381647] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2269.381647] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2269.381647] echo $(hostname -s): $val; [ 2269.381647] rc=$val; [ 2269.381647] fi; [ 2269.381647] exit $rc [ 2270.074121] Lustre: DEBUG MARKER: dmesg [ 2270.503104] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ======================================= 23:14:31 \(1491434071\) [ 2270.690526] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ======================================= 23:14:31 (1491434071) [ 2270.910851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_5 OI Scrub not implemented for ZFS [ 2271.099651] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_5 OI Scrub not implemented for ZFS [ 2271.306558] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2271.964096] Lustre: DEBUG MARKER: rc=0; [ 2271.964096] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2271.964096] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2271.964096] echo $(hostname -s): $val; [ 2271.964096] rc=$val; [ 2271.964096] fi; [ 2271.964096] exit $rc [ 2272.641159] Lustre: DEBUG MARKER: dmesg [ 2273.075037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 6a: LFSCK resumes from last checkpoint \(1\) ====================================== 23:14:34 \(1491434074\) [ 2273.258623] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ====================================== 23:14:34 (1491434074) [ 2273.914263] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1600 [ 2274.201200] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2274.551462] LustreError: 16081:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [ 2274.711517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2275.553032] LustreError: 16081:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 2276.555093] LustreError: 16081:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [ 2276.557341] LustreError: 16081:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2277.559033] LustreError: 16081:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [ 2277.561186] LustreError: 16081:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2278.006462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001608 [ 2278.299255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2278.299255] awk '/^status/ { print $2 }' [ 2278.563031] Lustre: *** cfs_fail_loc=1608, val=0*** [ 2278.592879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2278.592879] awk '/^status/ { print $2 }' [ 2278.889197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2279.176790] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1600 [ 2279.466512] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2279.829805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2280.118699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2280.407971] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2280.696621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2280.696621] awk '/^status/ { print $2 }' [ 2280.991490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2280.991490] awk '/^status/ { print $2 }' [ 2281.322463] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2281.988658] Lustre: DEBUG MARKER: rc=0; [ 2281.988658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2281.988658] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2281.988658] echo $(hostname -s): $val; [ 2281.988658] rc=$val; [ 2281.988658] fi; [ 2281.988658] exit $rc [ 2282.668709] Lustre: DEBUG MARKER: dmesg [ 2283.100585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 6b: LFSCK resumes from last checkpoint \(2\) ====================================== 23:14:44 \(1491434084\) [ 2283.288818] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ====================================== 23:14:44 (1491434084) [ 2283.956927] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [ 2284.241680] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2284.487997] LustreError: 17412:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [ 2284.490600] LustreError: 17412:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2284.651357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2285.493066] LustreError: 17412:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 2285.495332] LustreError: 17412:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2289.958100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001609 [ 2290.249683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2290.249683] awk '/^status/ { print $2 }' [ 2290.497057] Lustre: *** cfs_fail_loc=1609, val=0*** [ 2290.538685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2290.538685] awk '/^status/ { print $2 }' [ 2290.832161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2291.122660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2291.420655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [ 2291.723513] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2292.067396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2292.356115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2292.641043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2292.923038] LustreError: 18003:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [ 2292.925268] LustreError: 18003:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2292.931079] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2293.216647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2293.216647] awk '/^status/ { print $2 }' [ 2293.506910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2293.506910] awk '/^status/ { print $2 }' [ 2293.927032] LustreError: 18003:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 2293.929222] LustreError: 18003:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2294.795826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2294.795826] awk '/^status/ { print $2 }' [ 2295.115119] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2295.774869] Lustre: DEBUG MARKER: rc=0; [ 2295.774869] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2295.774869] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2295.774869] echo $(hostname -s): $val; [ 2295.774869] rc=$val; [ 2295.774869] fi; [ 2295.774869] exit $rc [ 2296.458895] Lustre: DEBUG MARKER: dmesg [ 2296.884482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount \(1\) ================ 23:14:58 \(1491434098\) [ 2297.079751] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ================ 23:14:58 (1491434098) [ 2298.180324] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [ 2298.467859] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2298.846895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2302.138818] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2302.430332] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2302.576446] Lustre: Failing over lustre-MDT0000 [ 2302.799744] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.732046] Lustre: 19183:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434103/real 1491434103] req@ffff88006399e400 x1563879826110864/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434109 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2308.738369] Lustre: 19183:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2308.754445] Lustre: server umount lustre-MDT0000 complete [ 2308.898683] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2309.201565] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2309.201565] grep -q ^lustre-mdt1/ /proc/mounts || [ 2309.201565] zpool export lustre-mdt1 [ 2309.666986] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2309.917938] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2310.199821] Lustre: DEBUG MARKER: modprobe zfs; [ 2310.199821] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2310.199821] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2310.744447] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2311.045432] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2311.244398] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2311.247917] Lustre: Skipped 16 previous similar messages [ 2311.492272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.654055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2311.942937] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2312.429616] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2312.718097] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2313.006974] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2313.300067] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2313.588431] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2313.876351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2313.876351] awk '/^status/ { print $2 }' [ 2314.165087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2314.165087] awk '/^status/ { print $2 }' [ 2315.454762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2315.454762] awk '/^status/ { print $2 }' [ 2316.747717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2316.747717] awk '/^status/ { print $2 }' [ 2318.043686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2318.043686] awk '/^status/ { print $2 }' [ 2319.345751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2319.345751] awk '/^status/ { print $2 }' [ 2320.637173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2320.637173] awk '/^status/ { print $2 }' [ 2321.931364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2321.931364] awk '/^status/ { print $2 }' [ 2323.224085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2323.224085] awk '/^status/ { print $2 }' [ 2324.512616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2324.512616] awk '/^status/ { print $2 }' [ 2324.810883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2324.949407] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2325.808867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2325.808867] awk '/^status/ { print $2 }' [ 2326.149421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2326.823549] Lustre: DEBUG MARKER: rc=0; [ 2326.823549] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2326.823549] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2326.823549] echo $(hostname -s): $val; [ 2326.823549] rc=$val; [ 2326.823549] fi; [ 2326.823549] exit $rc [ 2327.512733] Lustre: DEBUG MARKER: dmesg [ 2327.953560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount \(2\) ================ 23:15:29 \(1491434129\) [ 2328.154086] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ================ 23:15:29 (1491434129) [ 2329.071237] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2329.071237] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2329.071237] if [ $running -ne $mpts ]; then [ 2329.071237] echo $(hostname) env are INSANE!; [ 2329.071237] exit 1; [ 2329.071237] fi [ 2329.404052] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2329.404052] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2329.404052] if [ $running -ne $mpts ]; then [ 2329.404052] echo $(hostname) env are INSANE!; [ 2329.404052] exit 1; [ 2329.404052] fi [ 2330.386419] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2330.705481] Lustre: DEBUG MARKER: modprobe zfs; [ 2330.705481] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2330.705481] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2330.999326] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2331.287980] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2331.579587] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2336.906030] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2337.238005] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2337.430439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2337.602866] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2343.457758] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [ 2343.605731] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2343.803757] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1602 [ 2344.091081] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [ 2344.386525] LustreError: 23476:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 1000ms [ 2344.388708] LustreError: 23476:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 2344.471988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2344.471988] awk '/^status/ { print $2 }' [ 2344.759621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2344.759621] awk '/^status/ { print $2 }' [ 2345.252891] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2345.390025] LustreError: 23476:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [ 2345.392106] LustreError: 23476:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 2345.542498] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2345.687193] Lustre: Failing over lustre-MDT0000 [ 2347.361395] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.365780] LustreError: Skipped 3 previous similar messages [ 2352.426035] Lustre: 23772:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434147/real 1491434147] req@ffff88006390f300 x1563879826114336/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434153 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2352.445041] Lustre: server umount lustre-MDT0000 complete [ 2352.590134] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2352.889863] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2352.889863] grep -q ^lustre-mdt1/ /proc/mounts || [ 2352.889863] zpool export lustre-mdt1 [ 2353.376571] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2353.626611] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2353.905119] Lustre: DEBUG MARKER: modprobe zfs; [ 2353.905119] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2353.905119] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2354.435286] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2354.736199] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2355.160114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.328934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2355.617297] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2356.100495] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2356.386416] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2356.675892] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2356.823727] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2356.865387] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2356.879131] LustreError: 25835:0:(osd_index.c:1241:osd_index_it_get()) NOT IMPLEMETED YET (move to 0xa013000002000000) [ 2356.983649] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2357.275681] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2357.562614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2357.562614] awk '/^status/ { print $2 }' [ 2357.850623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2357.850623] awk '/^status/ { print $2 }' [ 2358.176984] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2358.872607] Lustre: DEBUG MARKER: rc=0; [ 2358.872607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2358.872607] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2358.872607] echo $(hostname -s): $val; [ 2358.872607] rc=$val; [ 2358.872607] fi; [ 2358.872607] exit $rc [ 2359.565096] Lustre: DEBUG MARKER: dmesg [ 2359.998323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 8: LFSCK state machine ========================================================== 23:16:01 \(1491434161\) [ 2360.183222] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ========================================================== 23:16:01 (1491434161) [ 2360.993570] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2361.282367] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2369.458045] Lustre: 26562:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434164/real 1491434164] req@ffff88006390f300 x1563879826116176/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434170 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2369.477846] Lustre: server umount lustre-MDT0000 complete [ 2369.626066] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2369.926690] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2369.926690] grep -q ^lustre-mdt1/ /proc/mounts || [ 2369.926690] zpool export lustre-mdt1 [ 2373.051527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2373.353778] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2373.950221] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2374.199311] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2374.461271] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2374.461271] grep -q ^lustre-mdt1/ /proc/mounts || [ 2374.461271] zpool export lustre-mdt1 [ 2374.731099] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=zfs --device-size=100000 --reformat lustre-mdt1/mdt1 /dev/lv [ 2375.449311] Lustre: DEBUG MARKER: zpool set cachefile=none lustre-mdt1 [ 2375.740782] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2375.740782] grep -q ^lustre-mdt1/ /proc/mounts || [ 2375.740782] zpool export lustre-mdt1 [ 2380.875948] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2380.875948] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2380.875948] if [ $running -ne $mpts ]; then [ 2380.875948] echo $(hostname) env are INSANE!; [ 2380.875948] exit 1; [ 2380.875948] fi [ 2381.188168] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2381.188168] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2381.188168] if [ $running -ne $mpts ]; then [ 2381.188168] echo $(hostname) env are INSANE!; [ 2381.188168] exit 1; [ 2381.188168] fi [ 2382.113450] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2382.399980] Lustre: DEBUG MARKER: modprobe zfs; [ 2382.399980] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2382.399980] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2382.846160] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2383.131167] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2383.552929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2383.555192] Lustre: Skipped 4 previous similar messages [ 2383.605748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 2383.607850] Lustre: Skipped 1 previous similar message [ 2383.622826] Lustre: lustre-MDT0000: new disk, initializing [ 2383.824593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.896933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2384.161437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2384.466524] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2384.968891] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2385.257449] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2385.550154] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2385.852229] Lustre: DEBUG MARKER: sync; sync; sync [ 2391.155053] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2391.446048] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2412.085714] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2412.416719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2412.620918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2412.792073] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2418.774186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2419.062831] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [ 2419.210273] Lustre: *** cfs_fail_loc=1603, val=0*** [ 2419.353796] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [ 2419.501167] Lustre: *** cfs_fail_loc=1604, val=0*** [ 2419.503108] Lustre: Skipped 19 previous similar messages [ 2419.833851] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x1601 [ 2420.122111] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2420.416138] LustreError: 30543:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 2000ms [ 2420.418643] LustreError: 30543:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2420.563432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2420.851701] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [ 2422.420047] LustreError: 30543:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [ 2422.422334] LustreError: 30543:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2422.739925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2423.046370] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2423.475510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2423.765887] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001609 [ 2424.053992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2424.053992] awk '/^status/ { print $2 }' [ 2424.343215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2424.343215] awk '/^status/ { print $2 }' [ 2425.329042] Lustre: *** cfs_fail_loc=1609, val=0*** [ 2425.633395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2425.633395] awk '/^status/ { print $2 }' [ 2425.921925] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1600 [ 2426.208547] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2426.572081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2426.859823] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160a [ 2428.428049] Lustre: *** cfs_fail_loc=160a, val=0*** [ 2432.156951] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2432.448575] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2432.594717] Lustre: Failing over lustre-MDT0000 [ 2434.986602] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.990887] LustreError: Skipped 3 previous similar messages [ 2438.628076] Lustre: 31650:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434233/real 1491434233] req@ffff88005b0d0300 x1563879826117920/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434239 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2438.649241] Lustre: server umount lustre-MDT0000 complete [ 2438.793735] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2439.090480] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2439.090480] grep -q ^lustre-mdt1/ /proc/mounts || [ 2439.090480] zpool export lustre-mdt1 [ 2439.506586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160b [ 2439.756278] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2440.038280] Lustre: DEBUG MARKER: modprobe zfs; [ 2440.038280] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2440.038280] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2440.498948] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2440.789459] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2440.990125] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2440.993487] Lustre: Skipped 21 previous similar messages [ 2441.125307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.286386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2441.580187] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2441.989526] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2442.067575] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2442.071629] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2442.078901] Lustre: *** cfs_fail_loc=160b, val=0*** [ 2442.367827] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2442.660069] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2442.954680] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2443.244215] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2443.542578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2443.542578] awk '/^status/ { print $2 }' [ 2443.833762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2444.120912] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1601 [ 2444.409295] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2444.703782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2445.006395] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2445.293663] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2445.440310] Lustre: Failing over lustre-MDT0000 [ 2452.741944] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2453.042075] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2453.042075] grep -q ^lustre-mdt1/ /proc/mounts || [ 2453.042075] zpool export lustre-mdt1 [ 2453.524047] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160b [ 2453.771465] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2454.058901] Lustre: DEBUG MARKER: modprobe zfs; [ 2454.058901] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2454.058901] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2454.529060] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2454.813725] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2455.344844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2455.635472] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2456.121331] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2456.409572] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.701040] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.994774] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2457.282661] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2457.571568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2457.571568] awk '/^status/ { print $2 }' [ 2458.862505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2458.862505] awk '/^status/ { print $2 }' [ 2459.004773] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2459.124073] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2459.137175] Lustre: *** cfs_fail_loc=160b, val=0*** [ 2460.161813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2460.161813] awk '/^status/ { print $2 }' [ 2460.453985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2460.743504] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2461.028790] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2461.173834] Lustre: Failing over lustre-MDT0000 [ 2467.360976] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2467.661401] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2467.661401] grep -q ^lustre-mdt1/ /proc/mounts || [ 2467.661401] zpool export lustre-mdt1 [ 2468.134445] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2468.412860] Lustre: DEBUG MARKER: modprobe zfs; [ 2468.412860] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2468.412860] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2468.889608] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2469.174960] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,skip_lfsck lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2469.653895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2469.941549] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2470.427114] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2470.714908] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2471.009098] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2471.304788] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2471.594741] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2471.885656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2471.885656] awk '/^status/ { print $2 }' [ 2473.178075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2473.178075] awk '/^status/ { print $2 }' [ 2473.509739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2473.582352] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2474.474488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [ 2474.474488] awk '/^status/ { print $2 }' [ 2474.766824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2475.052499] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x1602 [ 2475.340434] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [ 2475.638190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2475.638190] awk '/^status/ { print $2 }' [ 2475.928607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2475.928607] awk '/^status/ { print $2 }' [ 2476.219780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2476.509747] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2476.798635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2476.798635] awk '/^status/ { print $2 }' [ 2477.087340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2477.087340] awk '/^status/ { print $2 }' [ 2478.385514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2478.385514] awk '/^status/ { print $2 }' [ 2478.685576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2479.017026] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2479.686711] Lustre: DEBUG MARKER: rc=0; [ 2479.686711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2479.686711] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2479.686711] echo $(hostname -s): $val; [ 2479.686711] rc=$val; [ 2479.686711] fi; [ 2479.686711] exit $rc [ 2480.392397] Lustre: DEBUG MARKER: dmesg [ 2480.845077] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 9a: LFSCK speed control \(1\) ===================================================== 23:18:02 \(1491434282\) [ 2481.041694] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ===================================================== 23:18:02 (1491434282) [ 2481.250750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_9a Need MDS version \>= 2.7.50 [ 2481.436663] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_9a Need MDS version >= 2.7.50 [ 2481.643850] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2482.316640] Lustre: DEBUG MARKER: rc=0; [ 2482.316640] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2482.316640] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2482.316640] echo $(hostname -s): $val; [ 2482.316640] rc=$val; [ 2482.316640] fi; [ 2482.316640] exit $rc [ 2483.002249] Lustre: DEBUG MARKER: dmesg [ 2483.425518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 9b: LFSCK speed control \(2\) ===================================================== 23:18:04 \(1491434284\) [ 2483.612121] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ===================================================== 23:18:04 (1491434284) [ 2483.823863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_9b Need MDS version \>= 2.7.50 [ 2484.007024] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_9b Need MDS version >= 2.7.50 [ 2484.216579] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2484.893910] Lustre: DEBUG MARKER: rc=0; [ 2484.893910] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2484.893910] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2484.893910] echo $(hostname -s): $val; [ 2484.893910] rc=$val; [ 2484.893910] fi; [ 2484.893910] exit $rc [ 2485.588908] Lustre: DEBUG MARKER: dmesg [ 2486.044675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 10: System is available during LFSCK scanning =================================== 23:18:07 \(1491434287\) [ 2486.225399] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning =================================== 23:18:07 (1491434287) [ 2486.442190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_10 lookup\(..\)\/linkea on ZFS issue [ 2486.625131] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_10 lookup(..)/linkea on ZFS issue [ 2486.829145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2487.489514] Lustre: DEBUG MARKER: rc=0; [ 2487.489514] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2487.489514] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2487.489514] echo $(hostname -s): $val; [ 2487.489514] rc=$val; [ 2487.489514] fi; [ 2487.489514] exit $rc [ 2488.165676] Lustre: DEBUG MARKER: dmesg [ 2488.601606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ============================================= 23:18:09 \(1491434289\) [ 2488.789513] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ============================================= 23:18:09 (1491434289) [ 2489.688633] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2489.688633] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2489.688633] if [ $running -ne $mpts ]; then [ 2489.688633] echo $(hostname) env are INSANE!; [ 2489.688633] exit 1; [ 2489.688633] fi [ 2490.024512] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2490.024512] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2490.024512] if [ $running -ne $mpts ]; then [ 2490.024512] echo $(hostname) env are INSANE!; [ 2490.024512] exit 1; [ 2490.024512] fi [ 2490.985876] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2491.305187] Lustre: DEBUG MARKER: modprobe zfs; [ 2491.305187] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2491.305187] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2491.600844] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2491.898177] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2492.193113] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2497.559659] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2497.894376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2498.085286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2498.251038] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2504.901595] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2505.205524] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2508.061744] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2508.063886] Lustre: Skipped 1 previous similar message [ 2509.520097] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880060ffb000 x1563879826121616/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2509.524525] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2517.383038] Lustre: 10181:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434312/real 1491434312] req@ffff880060ffb000 x1563879826121712/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434318 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2517.389189] Lustre: 10181:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2517.404202] Lustre: server umount lustre-MDT0000 complete [ 2517.406147] Lustre: Skipped 2 previous similar messages [ 2517.549618] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2517.848750] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2517.848750] grep -q ^lustre-mdt1/ /proc/mounts || [ 2517.848750] zpool export lustre-mdt1 [ 2590.113685] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2590.716185] Lustre: DEBUG MARKER: rc=0; [ 2590.716185] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2590.716185] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2590.716185] echo $(hostname -s): $val; [ 2590.716185] rc=$val; [ 2590.716185] fi; [ 2590.716185] exit $rc [ 2591.330834] Lustre: DEBUG MARKER: dmesg [ 2591.736720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================== 23:19:53 \(1491434393\) [ 2591.910393] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================== 23:19:53 (1491434393) [ 2592.714676] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2592.714676] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2592.714676] if [ $running -ne $mpts ]; then [ 2592.714676] echo $(hostname) env are INSANE!; [ 2592.714676] exit 1; [ 2592.714676] fi [ 2593.006825] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2593.006825] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2593.006825] if [ $running -ne $mpts ]; then [ 2593.006825] echo $(hostname) env are INSANE!; [ 2593.006825] exit 1; [ 2593.006825] fi [ 2593.911330] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2594.188467] Lustre: DEBUG MARKER: modprobe zfs; [ 2594.188467] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2594.188467] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2594.732059] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2595.017905] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2595.393721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.395851] Lustre: Skipped 2 previous similar messages [ 2595.586521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2595.876667] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2596.400225] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2596.686413] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2596.982452] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2597.279174] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2597.569476] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2622.101863] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2622.435924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2622.627698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2622.792868] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2628.658600] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2628.944682] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 2629.318576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_seq [ 2631.557294] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2631.845481] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2631.989699] Lustre: Failing over lustre-MDT0000 [ 2632.131453] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_disconnect to node 10.9.5.244@tcp failed: rc = -107 [ 2632.279708] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.284109] LustreError: Skipped 10 previous similar messages [ 2638.176665] Lustre: server umount lustre-MDT0000 complete [ 2638.320555] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2638.621395] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2638.621395] grep -q ^lustre-mdt1/ /proc/mounts || [ 2638.621395] zpool export lustre-mdt1 [ 2639.110400] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2639.359906] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2639.644635] Lustre: DEBUG MARKER: modprobe zfs; [ 2639.644635] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2639.644635] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2640.197718] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2640.480719] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2641.052345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2641.341943] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2641.826244] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2641.999400] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2642.091073] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2642.128802] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2642.421340] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2642.714635] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2643.005548] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2652.096495] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 2652.099106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2652.213105] LustreError: 14907:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 [ 2653.227163] LustreError: 14907:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 [ 2654.236824] LustreError: 14907:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 [ 2655.529152] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2655.817576] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2656.769599] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2656.772141] Lustre: Skipped 1 previous similar message [ 2663.996040] Lustre: 15694:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434458/real 1491434458] req@ffff880065b1f600 x1563879826127168/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434464 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2664.003778] Lustre: 15694:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2664.161063] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2664.459575] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2664.459575] grep -q ^lustre-mdt1/ /proc/mounts || [ 2664.459575] zpool export lustre-mdt1 [ 2673.806507] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2674.388763] Lustre: DEBUG MARKER: rc=0; [ 2674.388763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2674.388763] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2674.388763] echo $(hostname -s): $val; [ 2674.388763] rc=$val; [ 2674.388763] fi; [ 2674.388763] exit $rc [ 2674.962041] Lustre: DEBUG MARKER: dmesg [ 2675.341909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ============================= 23:21:16 \(1491434476\) [ 2675.496803] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ============================= 23:21:16 (1491434476) [ 2675.674361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_12a We need at least 2 MDSes for test_12a [ 2675.821992] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_12a We need at least 2 MDSes for test_12a [ 2675.986664] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2676.566749] Lustre: DEBUG MARKER: rc=0; [ 2676.566749] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2676.566749] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2676.566749] echo $(hostname -s): $val; [ 2676.566749] rc=$val; [ 2676.566749] fi; [ 2676.566749] exit $rc [ 2677.132059] Lustre: DEBUG MARKER: dmesg [ 2677.514620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 12b: auto detect Lustre device ================================================== 23:21:18 \(1491434478\) [ 2677.665228] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ================================================== 23:21:18 (1491434478) [ 2678.386149] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2678.386149] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2678.386149] if [ $running -ne $mpts ]; then [ 2678.386149] echo $(hostname) env are INSANE!; [ 2678.386149] exit 1; [ 2678.386149] fi [ 2678.681790] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2678.681790] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2678.681790] if [ $running -ne $mpts ]; then [ 2678.681790] echo $(hostname) env are INSANE!; [ 2678.681790] exit 1; [ 2678.681790] fi [ 2679.602143] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2679.885765] Lustre: DEBUG MARKER: modprobe zfs; [ 2679.885765] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2679.885765] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2680.419366] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2680.700453] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2681.046446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.048508] Lustre: Skipped 1 previous similar message [ 2681.237981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2681.525503] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2682.011257] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2682.300421] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2682.591570] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2682.884685] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2683.174505] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2692.652551] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2693.001983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2693.187305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2693.361571] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2698.845523] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -A -r [ 2699.435733] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [ 2699.435733] awk '/^namespace_mdts_completed/ { print $2 }' [ 2699.734168] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [ 2699.734168] awk '/^layout_mdts_completed/ { print $2 }' [ 2700.027453] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 2700.367505] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2701.033253] Lustre: DEBUG MARKER: rc=0; [ 2701.033253] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2701.033253] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2701.033253] echo $(hostname -s): $val; [ 2701.033253] rc=$val; [ 2701.033253] fi; [ 2701.033253] exit $rc [ 2701.723610] Lustre: DEBUG MARKER: dmesg [ 2702.154491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ============================================= 23:21:43 \(1491434503\) [ 2702.337811] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ============================================= 23:21:43 (1491434503) [ 2702.523398] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160f [ 2702.673868] Lustre: *** cfs_fail_loc=160f, val=0*** [ 2702.851494] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2703.139099] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2703.599413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2703.599413] awk '/^status/ { print $2 }' [ 2703.888376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2703.888376] awk '/^status/ { print $2 }' [ 2704.179203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2704.499118] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2705.164590] Lustre: DEBUG MARKER: rc=0; [ 2705.164590] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2705.164590] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2705.164590] echo $(hostname -s): $val; [ 2705.164590] rc=$val; [ 2705.164590] fi; [ 2705.164590] exit $rc [ 2705.863669] Lustre: DEBUG MARKER: dmesg [ 2706.283236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference \(1\) ============= 23:21:47 \(1491434507\) [ 2706.468727] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ============= 23:21:47 (1491434507) [ 2707.012701] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2707.302599] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 2707.972380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2708.293415] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 2708.685804] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 2709.039542] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2709.326665] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 2709.797430] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2710.244868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2710.244868] awk '/^status/ { print $2 }' [ 2710.541557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2710.541557] awk '/^status/ { print $2 }' [ 2710.831676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2711.119648] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -c [ 2711.606856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2711.606856] awk '/^status/ { print $2 }' [ 2711.899858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2711.899858] awk '/^status/ { print $2 }' [ 2712.217090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2712.543189] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 2713.177746] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2713.557215] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2714.209454] Lustre: DEBUG MARKER: rc=0; [ 2714.209454] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2714.209454] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2714.209454] echo $(hostname -s): $val; [ 2714.209454] rc=$val; [ 2714.209454] fi; [ 2714.209454] exit $rc [ 2714.886891] Lustre: DEBUG MARKER: dmesg [ 2715.318185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference \(2\) ============= 23:21:56 \(1491434516\) [ 2715.503968] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ============= 23:21:56 (1491434516) [ 2716.081594] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2716.368375] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 2717.058791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2717.379607] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 2717.770535] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 2718.134359] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 2718.421494] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 2718.915565] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -d [ 2719.407501] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [ 2719.407501] awk '/^layout_mdts_completed/ { print $2 }' [ 2719.699703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2719.989649] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c -d [ 2720.506618] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [ 2720.506618] awk '/^layout_mdts_completed/ { print $2 }' [ 2720.836855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2721.159636] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 2721.802780] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2722.188274] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2722.890851] Lustre: DEBUG MARKER: rc=0; [ 2722.890851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2722.890851] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2722.890851] echo $(hostname -s): $val; [ 2722.890851] rc=$val; [ 2722.890851] fi; [ 2722.890851] exit $rc [ 2723.569678] Lustre: DEBUG MARKER: dmesg [ 2724.009247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object\/OST-object pairs \(1\) ================= 23:22:05 \(1491434525\) [ 2724.194420] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ================= 23:22:05 (1491434525) [ 2725.173869] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2725.580338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2725.580338] awk '/^status/ { print $2 }' [ 2725.868889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2725.868889] awk '/^status/ { print $2 }' [ 2726.159632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2726.487439] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2727.153201] Lustre: DEBUG MARKER: rc=0; [ 2727.153201] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2727.153201] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2727.153201] echo $(hostname -s): $val; [ 2727.153201] rc=$val; [ 2727.153201] fi; [ 2727.153201] exit $rc [ 2727.830398] Lustre: DEBUG MARKER: dmesg [ 2728.271772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object\/OST-object pairs \(2\) ================= 23:22:09 \(1491434529\) [ 2728.457865] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ================= 23:22:09 (1491434529) [ 2729.384489] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2729.859793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2729.859793] awk '/^status/ { print $2 }' [ 2730.151833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2730.151833] awk '/^status/ { print $2 }' [ 2730.441449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2730.763834] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2731.426796] Lustre: DEBUG MARKER: rc=0; [ 2731.426796] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2731.426796] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2731.426796] echo $(hostname -s): $val; [ 2731.426796] rc=$val; [ 2731.426796] fi; [ 2731.426796] exit $rc [ 2732.102589] Lustre: DEBUG MARKER: dmesg [ 2732.544272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object\/OST-object pairs \(3\) ================= 23:22:13 \(1491434533\) [ 2732.728283] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ================= 23:22:13 (1491434533) [ 2732.935962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_15c We need at least 2 MDSes for this test [ 2733.125830] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c We need at least 2 MDSes for this test [ 2733.330710] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2733.986180] Lustre: DEBUG MARKER: rc=0; [ 2733.986180] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2733.986180] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2733.986180] echo $(hostname -s): $val; [ 2733.986180] rc=$val; [ 2733.986180] fi; [ 2733.986180] exit $rc [ 2734.667072] Lustre: DEBUG MARKER: dmesg [ 2735.096259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object\/OST-object owner =================== 23:22:16 \(1491434536\) [ 2735.288172] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner =================== 23:22:16 (1491434536) [ 2735.512144] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1613 [ 2735.667742] Lustre: *** cfs_fail_loc=1613, val=0*** [ 2735.812072] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2736.099286] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2736.542472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2736.542472] awk '/^status/ { print $2 }' [ 2736.832910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2736.832910] awk '/^status/ { print $2 }' [ 2737.122431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2737.450929] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2738.106163] Lustre: DEBUG MARKER: rc=0; [ 2738.106163] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2738.106163] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2738.106163] echo $(hostname -s): $val; [ 2738.106163] rc=$val; [ 2738.106163] fi; [ 2738.106163] exit $rc [ 2738.783555] Lustre: DEBUG MARKER: dmesg [ 2739.222180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 17: LFSCK can repair multiple references ======================================== 23:22:20 \(1491434540\) [ 2739.408277] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ======================================== 23:22:20 (1491434540) [ 2739.592930] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0x1614 [ 2739.741557] Lustre: *** cfs_fail_loc=1614, val=0*** [ 2739.743426] Lustre: Skipped 31 previous similar messages [ 2740.123559] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2740.422637] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2740.882971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2740.882971] awk '/^status/ { print $2 }' [ 2741.171574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2741.171574] awk '/^status/ { print $2 }' [ 2741.462163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2741.821802] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2742.475617] Lustre: DEBUG MARKER: rc=0; [ 2742.475617] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2742.475617] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2742.475617] echo $(hostname -s): $val; [ 2742.475617] rc=$val; [ 2742.475617] fi; [ 2742.475617] exit $rc [ 2743.152121] Lustre: DEBUG MARKER: dmesg [ 2743.598047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18a: Find out orphan OST-object and repair it \(1\) =============================== 23:22:24 \(1491434544\) [ 2743.785359] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) =============================== 23:22:24 (1491434544) [ 2744.037984] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1615 [ 2744.189403] Lustre: *** cfs_fail_loc=1615, val=0*** [ 2744.191254] Lustre: Skipped 1 previous similar message [ 2746.428053] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2746.731570] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2747.152208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2747.152208] awk '/^status/ { print $2 }' [ 2747.442903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2747.442903] awk '/^status/ { print $2 }' [ 2748.410109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2748.747023] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2749.407493] Lustre: DEBUG MARKER: rc=0; [ 2749.407493] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2749.407493] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2749.407493] echo $(hostname -s): $val; [ 2749.407493] rc=$val; [ 2749.407493] fi; [ 2749.407493] exit $rc [ 2750.086646] Lustre: DEBUG MARKER: dmesg [ 2750.521123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18b: Find out orphan OST-object and repair it \(2\) =============================== 23:22:31 \(1491434551\) [ 2750.717928] Lustre: DEBUG MARKER: == sanity-lfsck test 18b: Find out orphan OST-object and repair it (2) =============================== 23:22:31 (1491434551) [ 2750.970694] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [ 2751.119576] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2751.121455] Lustre: Skipped 1 previous similar message [ 2753.317286] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2753.610116] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2754.098393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2754.098393] awk '/^status/ { print $2 }' [ 2754.388420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2754.388420] awk '/^status/ { print $2 }' [ 2755.347035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2755.698691] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2756.354890] Lustre: DEBUG MARKER: rc=0; [ 2756.354890] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2756.354890] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2756.354890] echo $(hostname -s): $val; [ 2756.354890] rc=$val; [ 2756.354890] fi; [ 2756.354890] exit $rc [ 2757.039326] Lustre: DEBUG MARKER: dmesg [ 2757.477222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18c: Find out orphan OST-object and repair it \(3\) =============================== 23:22:38 \(1491434558\) [ 2757.666581] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) =============================== 23:22:38 (1491434558) [ 2758.243794] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [ 2758.391716] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2758.393541] Lustre: Skipped 1 previous similar message [ 2760.558874] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2760.854513] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2761.322095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2761.322095] awk '/^status/ { print $2 }' [ 2761.612725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2761.612725] awk '/^status/ { print $2 }' [ 2762.568108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2762.909434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2763.570141] Lustre: DEBUG MARKER: rc=0; [ 2763.570141] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2763.570141] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2763.570141] echo $(hostname -s): $val; [ 2763.570141] rc=$val; [ 2763.570141] fi; [ 2763.570141] exit $rc [ 2764.244863] Lustre: DEBUG MARKER: dmesg [ 2764.676710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18d: Find out orphan OST-object and repair it \(4\) =============================== 23:22:46 \(1491434566\) [ 2764.867906] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) =============================== 23:22:46 (1491434566) [ 2765.091907] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1618 [ 2765.247019] Lustre: *** cfs_fail_loc=1618, val=0*** [ 2765.248851] Lustre: Skipped 1 previous similar message [ 2767.417122] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2768.379479] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2768.667827] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2769.808084] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005a28a700 x1563879826148144/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2773.023944] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2773.026044] Lustre: Skipped 1 previous similar message [ 2780.855781] Lustre: server umount lustre-MDT0000 complete [ 2780.857707] Lustre: Skipped 1 previous similar message [ 2781.001808] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2781.298736] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2781.298736] grep -q ^lustre-mdt1/ /proc/mounts || [ 2781.298736] zpool export lustre-mdt1 [ 2791.109828] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2791.109828] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2791.109828] if [ $running -ne $mpts ]; then [ 2791.109828] echo $(hostname) env are INSANE!; [ 2791.109828] exit 1; [ 2791.109828] fi [ 2791.406768] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2791.406768] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2791.406768] if [ $running -ne $mpts ]; then [ 2791.406768] echo $(hostname) env are INSANE!; [ 2791.406768] exit 1; [ 2791.406768] fi [ 2792.336925] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2792.624842] Lustre: DEBUG MARKER: modprobe zfs; [ 2792.624842] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2792.624842] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2793.237137] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2793.520522] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2793.703090] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 2793.705184] Lustre: Skipped 47 previous similar messages [ 2794.054672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2794.342756] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2794.829890] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2795.117586] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2795.405988] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2795.699960] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2795.989530] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2805.489798] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2805.825129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2806.016456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2806.180070] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2811.492575] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c -d [ 2812.211484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2812.211484] awk '/^status/ { print $2 }' [ 2812.508921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2812.508921] awk '/^status/ { print $2 }' [ 2813.461839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2813.751136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2814.089194] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2814.761150] Lustre: DEBUG MARKER: rc=0; [ 2814.761150] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2814.761150] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2814.761150] echo $(hostname -s): $val; [ 2814.761150] rc=$val; [ 2814.761150] fi; [ 2814.761150] exit $rc [ 2815.448643] Lustre: DEBUG MARKER: dmesg [ 2815.890213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18e: Find out orphan OST-object and repair it \(5\) =============================== 23:23:37 \(1491434617\) [ 2816.087702] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) =============================== 23:23:37 (1491434617) [ 2816.391662] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1618 [ 2816.545348] Lustre: *** cfs_fail_loc=1618, val=0*** [ 2816.547321] Lustre: Skipped 1 previous similar message [ 2818.714960] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2819.641706] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2819.932035] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2820.432066] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005af6f900 x1563879826149360/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 2820.436899] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2821.000702] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2821.000703] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 2826.000327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.004877] LustreError: Skipped 3 previous similar messages [ 2828.263834] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2828.560878] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2828.560878] grep -q ^lustre-mdt1/ /proc/mounts || [ 2828.560878] zpool export lustre-mdt1 [ 2838.362645] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2838.362645] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2838.362645] if [ $running -ne $mpts ]; then [ 2838.362645] echo $(hostname) env are INSANE!; [ 2838.362645] exit 1; [ 2838.362645] fi [ 2838.661804] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 2838.661804] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 2838.661804] if [ $running -ne $mpts ]; then [ 2838.661804] echo $(hostname) env are INSANE!; [ 2838.661804] exit 1; [ 2838.661804] fi [ 2839.599507] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 2839.883379] Lustre: DEBUG MARKER: modprobe zfs; [ 2839.883379] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 2839.883379] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 2840.441380] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 2840.724772] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 2841.073041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.075088] Lustre: Skipped 1 previous similar message [ 2841.294534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2841.599494] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2842.092432] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2842.380484] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2842.668998] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2842.968075] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 2843.258879] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2857.842032] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 2858.187259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 2858.378321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2858.547409] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 2863.872959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x1602 [ 2864.163906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2864.501117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 2864.894573] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 2865.245313] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c [ 2865.659622] LustreError: 2644:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 10000ms [ 2865.661907] LustreError: 2644:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 2865.724195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2865.724195] awk '/^status/ { print $2 }' [ 2866.012711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2866.012711] awk '/^status/ { print $2 }' [ 2868.383676] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 2868.672633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2868.672633] awk '/^status/ { print $2 }' [ 2868.960670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2868.960670] awk '/^status/ { print $2 }' [ 2870.251999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2870.251999] awk '/^status/ { print $2 }' [ 2871.546509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2871.546509] awk '/^status/ { print $2 }' [ 2872.843068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2872.843068] awk '/^status/ { print $2 }' [ 2874.142120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2874.142120] awk '/^status/ { print $2 }' [ 2875.435809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2875.435809] awk '/^status/ { print $2 }' [ 2875.664046] LustreError: 2644:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [ 2875.666511] LustreError: 2644:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 2876.727462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2876.727462] awk '/^status/ { print $2 }' [ 2877.714704] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 2878.351141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2878.698780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2879.070113] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2879.735200] Lustre: DEBUG MARKER: rc=0; [ 2879.735200] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2879.735200] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2879.735200] echo $(hostname -s): $val; [ 2879.735200] rc=$val; [ 2879.735200] fi; [ 2879.735200] exit $rc [ 2880.422730] Lustre: DEBUG MARKER: dmesg [ 2880.955857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18f: Skip the failed OST\(s\) when handle orphan OST-objects ====================== 23:24:42 \(1491434682\) [ 2881.168076] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ====================== 23:24:42 (1491434682) [ 2881.555145] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [ 2881.702949] Lustre: *** cfs_fail_loc=1616, val=0*** [ 2881.704944] Lustre: Skipped 1 previous similar message [ 2883.873868] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2884.175918] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161c fail_val=0 [ 2884.464915] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2885.351418] Lustre: *** cfs_fail_loc=161c, val=0*** [ 2885.516895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2885.516895] awk '/^status/ { print $2 }' [ 2885.811969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2885.811969] awk '/^status/ { print $2 }' [ 2887.412123] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2887.701095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2887.989692] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2888.539613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2888.539613] awk '/^status/ { print $2 }' [ 2888.829551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2888.829551] awk '/^status/ { print $2 }' [ 2889.775808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2890.102138] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2890.770904] Lustre: DEBUG MARKER: rc=0; [ 2890.770904] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2890.770904] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2890.770904] echo $(hostname -s): $val; [ 2890.770904] rc=$val; [ 2890.770904] fi; [ 2890.770904] exit $rc [ 2891.468433] Lustre: DEBUG MARKER: dmesg [ 2891.899394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18g: Find out orphan OST-object and repair it \(7\) =============================== 23:24:53 \(1491434693\) [ 2892.087129] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) =============================== 23:24:53 (1491434693) [ 2892.356025] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162e [ 2892.643866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2892.935938] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2893.706303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2893.706303] awk '/^status/ { print $2 }' [ 2894.010118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2894.010118] awk '/^status/ { print $2 }' [ 2894.964533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2895.312698] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2895.967892] Lustre: DEBUG MARKER: rc=0; [ 2895.967892] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2895.967892] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2895.967892] echo $(hostname -s): $val; [ 2895.967892] rc=$val; [ 2895.967892] fi; [ 2895.967892] exit $rc [ 2896.648502] Lustre: DEBUG MARKER: dmesg [ 2897.088249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 19a: OST-object inconsistency self detect ======================================= 23:24:58 \(1491434698\) [ 2897.277973] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ======================================= 23:24:58 (1491434698) [ 2898.849143] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2899.506676] Lustre: DEBUG MARKER: rc=0; [ 2899.506676] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2899.506676] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2899.506676] echo $(hostname -s): $val; [ 2899.506676] rc=$val; [ 2899.506676] fi; [ 2899.506676] exit $rc [ 2900.186770] Lustre: DEBUG MARKER: dmesg [ 2900.624334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 19b: OST-object inconsistency self repair ======================================= 23:25:01 \(1491434701\) [ 2900.814380] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ======================================= 23:25:01 (1491434701) [ 2903.706772] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2904.367906] Lustre: DEBUG MARKER: rc=0; [ 2904.367906] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2904.367906] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2904.367906] echo $(hostname -s): $val; [ 2904.367906] rc=$val; [ 2904.367906] fi; [ 2904.367906] exit $rc [ 2905.059723] Lustre: DEBUG MARKER: dmesg [ 2905.499605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 20: Handle the orphan with dummy LOV EA slot properly =========================== 23:25:06 \(1491434706\) [ 2905.689644] Lustre: DEBUG MARKER: == sanity-lfsck test 20: Handle the orphan with dummy LOV EA slot properly =========================== 23:25:06 (1491434706) [ 2906.055985] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [ 2906.344557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161a [ 2906.634455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 [ 2909.073702] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2909.717466] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [ 2913.633255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2913.633255] awk '/^status/ { print $2 }' [ 2913.921050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2913.921050] awk '/^status/ { print $2 }' [ 2915.210184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [ 2915.210184] awk '/^status/ { print $2 }' [ 2916.186783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2916.935511] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2917.599545] Lustre: DEBUG MARKER: rc=0; [ 2917.599545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2917.599545] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2917.599545] echo $(hostname -s): $val; [ 2917.599545] rc=$val; [ 2917.599545] fi; [ 2917.599545] exit $rc [ 2918.285754] Lustre: DEBUG MARKER: dmesg [ 2918.718702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 21: run all LFSCK components by default ========================================= 23:25:20 \(1491434720\) [ 2918.903428] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================= 23:25:20 (1491434720) [ 2919.076525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 2919.504737] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -s 1 -r [ 2920.151060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2920.452236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2920.743358] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [ 2921.234357] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2921.904646] Lustre: DEBUG MARKER: rc=0; [ 2921.904646] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2921.904646] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2921.904646] echo $(hostname -s): $val; [ 2921.904646] rc=$val; [ 2921.904646] fi; [ 2921.904646] exit $rc [ 2922.614237] Lustre: DEBUG MARKER: dmesg [ 2923.073415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 22a: LFSCK can repair unmatched pairs \(1\) ======================================= 23:25:24 \(1491434724\) [ 2923.254891] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ======================================= 23:25:24 (1491434724) [ 2923.454441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_22a We need at least 2 MDSes for this test [ 2923.641280] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_22a We need at least 2 MDSes for this test [ 2923.842937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2924.499617] Lustre: DEBUG MARKER: rc=0; [ 2924.499617] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2924.499617] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2924.499617] echo $(hostname -s): $val; [ 2924.499617] rc=$val; [ 2924.499617] fi; [ 2924.499617] exit $rc [ 2925.172358] Lustre: DEBUG MARKER: dmesg [ 2925.616157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 22b: LFSCK can repair unmatched pairs \(2\) ======================================= 23:25:26 \(1491434726\) [ 2925.803217] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ======================================= 23:25:26 (1491434726) [ 2926.006185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_22b We need at least 2 MDSes for this test [ 2926.195718] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_22b We need at least 2 MDSes for this test [ 2926.395443] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2927.061427] Lustre: DEBUG MARKER: rc=0; [ 2927.061427] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2927.061427] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2927.061427] echo $(hostname -s): $val; [ 2927.061427] rc=$val; [ 2927.061427] fi; [ 2927.061427] exit $rc [ 2927.740506] Lustre: DEBUG MARKER: dmesg [ 2928.189908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23a: LFSCK can repair dangling name entry \(1\) =================================== 23:25:29 \(1491434729\) [ 2928.379059] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) =================================== 23:25:29 (1491434729) [ 2928.590127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_23a We need at least 2 MDSes for this test [ 2928.779098] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_23a We need at least 2 MDSes for this test [ 2928.990496] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2929.646074] Lustre: DEBUG MARKER: rc=0; [ 2929.646074] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2929.646074] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2929.646074] echo $(hostname -s): $val; [ 2929.646074] rc=$val; [ 2929.646074] fi; [ 2929.646074] exit $rc [ 2930.327275] Lustre: DEBUG MARKER: dmesg [ 2930.773660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23b: LFSCK can repair dangling name entry \(2\) =================================== 23:25:32 \(1491434732\) [ 2930.957063] Lustre: DEBUG MARKER: == sanity-lfsck test 23b: LFSCK can repair dangling name entry (2) =================================== 23:25:32 (1491434732) [ 2931.266459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=119 fail_loc=0x1621 [ 2931.415961] Lustre: *** cfs_fail_loc=1621, val=0*** [ 2931.417852] Lustre: Skipped 4 previous similar messages [ 2931.560290] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 2931.862073] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -C [ 2932.309553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2932.309553] awk '/^status/ { print $2 }' [ 2932.597064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2932.597064] awk '/^status/ { print $2 }' [ 2932.886733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2933.174761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2933.502793] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2934.182158] Lustre: DEBUG MARKER: rc=0; [ 2934.182158] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2934.182158] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2934.182158] echo $(hostname -s): $val; [ 2934.182158] rc=$val; [ 2934.182158] fi; [ 2934.182158] exit $rc [ 2934.870722] Lustre: DEBUG MARKER: dmesg [ 2935.312940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23c: LFSCK can repair dangling name entry \(3\) =================================== 23:25:36 \(1491434736\) [ 2935.506919] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) =================================== 23:25:36 (1491434736) [ 2935.677312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2936.007181] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 2936.389830] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 2936.824635] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=133 fail_loc=0x1621 [ 2937.114657] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 2937.436272] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x1602 [ 2937.722587] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -C [ 2938.189705] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [ 2938.475784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2938.475784] awk '/^status/ { print $2 }' [ 2938.764069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2938.764069] awk '/^status/ { print $2 }' [ 2940.051031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2940.051031] awk '/^status/ { print $2 }' [ 2941.340866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2941.340866] awk '/^status/ { print $2 }' [ 2942.630501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2942.630501] awk '/^status/ { print $2 }' [ 2943.927178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2943.927178] awk '/^status/ { print $2 }' [ 2945.220819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2945.220819] awk '/^status/ { print $2 }' [ 2946.509117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2946.509117] awk '/^status/ { print $2 }' [ 2947.803066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2947.803066] awk '/^status/ { print $2 }' [ 2949.098592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [ 2949.098592] awk '/^status/ { print $2 }' [ 2949.422151] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 2950.068772] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2950.423390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2950.758329] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2951.428290] Lustre: DEBUG MARKER: rc=0; [ 2951.428290] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2951.428290] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2951.428290] echo $(hostname -s): $val; [ 2951.428290] rc=$val; [ 2951.428290] fi; [ 2951.428290] exit $rc [ 2952.120542] Lustre: DEBUG MARKER: dmesg [ 2952.561499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ============================= 23:25:53 \(1491434753\) [ 2952.744037] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ============================= 23:25:53 (1491434753) [ 2952.945060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_24 We need at least 2 MDSes for this test [ 2953.136803] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_24 We need at least 2 MDSes for this test [ 2953.346470] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2953.999753] Lustre: DEBUG MARKER: rc=0; [ 2953.999753] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2953.999753] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2953.999753] echo $(hostname -s): $val; [ 2953.999753] rc=$val; [ 2953.999753] fi; [ 2953.999753] exit $rc [ 2954.682714] Lustre: DEBUG MARKER: dmesg [ 2955.132338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ============================ 23:25:56 \(1491434756\) [ 2955.320122] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ============================ 23:25:56 (1491434756) [ 2955.526724] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_25 Only support to inject failure on ldiskfs [ 2955.712114] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_25 Only support to inject failure on ldiskfs [ 2955.919321] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2956.579216] Lustre: DEBUG MARKER: rc=0; [ 2956.579216] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2956.579216] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2956.579216] echo $(hostname -s): $val; [ 2956.579216] rc=$val; [ 2956.579216] fi; [ 2956.579216] exit $rc [ 2957.264425] Lustre: DEBUG MARKER: dmesg [ 2957.710480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace =========== 23:25:59 \(1491434759\) [ 2957.899870] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace =========== 23:25:59 (1491434759) [ 2958.106254] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [ 2958.399798] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2958.688638] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [ 2959.048985] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [ 2959.048985] awk '/^namespace_mdts_completed/ { print $2 }' [ 2959.337384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2959.672089] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2960.349297] Lustre: DEBUG MARKER: rc=0; [ 2960.349297] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2960.349297] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2960.349297] echo $(hostname -s): $val; [ 2960.349297] rc=$val; [ 2960.349297] fi; [ 2960.349297] exit $rc [ 2961.037762] Lustre: DEBUG MARKER: dmesg [ 2961.479657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========== 23:26:02 \(1491434762\) [ 2961.671449] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========== 23:26:02 (1491434762) [ 2961.876776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_26b We need at least 2 MDSes for this test [ 2962.063667] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_26b We need at least 2 MDSes for this test [ 2962.274438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2962.930915] Lustre: DEBUG MARKER: rc=0; [ 2962.930915] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2962.930915] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2962.930915] echo $(hostname -s): $val; [ 2962.930915] rc=$val; [ 2962.930915] fi; [ 2962.930915] exit $rc [ 2963.611188] Lustre: DEBUG MARKER: dmesg [ 2964.059298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan =============== 23:26:05 \(1491434765\) [ 2964.259306] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan =============== 23:26:05 (1491434765) [ 2964.443670] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [ 2964.592877] Lustre: *** cfs_fail_loc=1624, val=0*** [ 2964.594689] Lustre: Skipped 4 previous similar messages [ 2964.739706] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2965.032863] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [ 2965.493222] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [ 2965.493222] awk '/^namespace_mdts_completed/ { print $2 }' [ 2965.787892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2966.127060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2966.791896] Lustre: DEBUG MARKER: rc=0; [ 2966.791896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2966.791896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2966.791896] echo $(hostname -s): $val; [ 2966.791896] rc=$val; [ 2966.791896] fi; [ 2966.791896] exit $rc [ 2967.473389] Lustre: DEBUG MARKER: dmesg [ 2967.910466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ============== 23:26:09 \(1491434769\) [ 2968.100136] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ============== 23:26:09 (1491434769) [ 2968.310917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_27b We need at least 2 MDSes for this test [ 2968.500586] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_27b We need at least 2 MDSes for this test [ 2968.711160] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2969.371654] Lustre: DEBUG MARKER: rc=0; [ 2969.371654] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2969.371654] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2969.371654] echo $(hostname -s): $val; [ 2969.371654] rc=$val; [ 2969.371654] fi; [ 2969.371654] exit $rc [ 2970.060565] Lustre: DEBUG MARKER: dmesg [ 2970.501811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 28: Skip the failed MDT\(s\) when handle orphan MDT-objects ======================= 23:26:11 \(1491434771\) [ 2970.693755] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ======================= 23:26:11 (1491434771) [ 2970.895568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_28 The test needs at least 2 MDTs [ 2971.076433] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_28 The test needs at least 2 MDTs [ 2971.290209] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2971.950218] Lustre: DEBUG MARKER: rc=0; [ 2971.950218] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2971.950218] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2971.950218] echo $(hostname -s): $val; [ 2971.950218] rc=$val; [ 2971.950218] fi; [ 2971.950218] exit $rc [ 2972.645665] Lustre: DEBUG MARKER: dmesg [ 2973.077706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 29b: LFSCK can repair bad nlink count \(2\) ======================================= 23:26:14 \(1491434774\) [ 2973.262649] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ======================================= 23:26:14 (1491434774) [ 2973.453821] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1626 [ 2973.745450] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2974.039856] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [ 2974.461476] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [ 2974.461476] awk '/^namespace_mdts_completed/ { print $2 }' [ 2974.755220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2975.091804] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2975.746533] Lustre: DEBUG MARKER: rc=0; [ 2975.746533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2975.746533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2975.746533] echo $(hostname -s): $val; [ 2975.746533] rc=$val; [ 2975.746533] fi; [ 2975.746533] exit $rc [ 2976.447885] Lustre: DEBUG MARKER: dmesg [ 2976.889696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 29c: verify linkEA size limitation ============================================== 23:26:18 \(1491434778\) [ 2977.070737] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation ============================================== 23:26:18 (1491434778) [ 2980.474302] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [ 2980.856971] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [ 2980.856971] awk '/^namespace_mdts_completed/ { print $2 }' [ 2981.146694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2981.434371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [ 2981.838512] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2982.497041] Lustre: DEBUG MARKER: rc=0; [ 2982.497041] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2982.497041] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2982.497041] echo $(hostname -s): $val; [ 2982.497041] rc=$val; [ 2982.497041] fi; [ 2982.497041] exit $rc [ 2983.189473] Lustre: DEBUG MARKER: dmesg [ 2983.634957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 30: LFSCK can recover the orphans from backend \/lost+found ====================== 23:26:24 \(1491434784\) [ 2983.834144] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ====================== 23:26:24 (1491434784) [ 2984.034944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_30 Only support backend \/lost+found for ldiskfs [ 2984.213911] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_30 Only support backend /lost+found for ldiskfs [ 2984.425988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2985.088729] Lustre: DEBUG MARKER: rc=0; [ 2985.088729] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2985.088729] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2985.088729] echo $(hostname -s): $val; [ 2985.088729] rc=$val; [ 2985.088729] fi; [ 2985.088729] exit $rc [ 2985.775750] Lustre: DEBUG MARKER: dmesg [ 2986.205266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31a skipping ALWAYS excluded test 31a \(base 31\) [ 2986.389396] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31a skipping ALWAYS excluded test 31a (base 31) [ 2986.609064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31b skipping ALWAYS excluded test 31b \(base 31\) [ 2986.800758] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31b skipping ALWAYS excluded test 31b (base 31) [ 2987.016786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31c skipping ALWAYS excluded test 31c \(base 31\) [ 2987.211746] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31c skipping ALWAYS excluded test 31c (base 31) [ 2987.430738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31d skipping ALWAYS excluded test 31d \(base 31\) [ 2987.619716] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31d skipping ALWAYS excluded test 31d (base 31) [ 2987.828922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31e skipping ALWAYS excluded test 31e \(base 31\) [ 2988.027791] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31e skipping ALWAYS excluded test 31e (base 31) [ 2988.250881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31f skipping ALWAYS excluded test 31f \(base 31\) [ 2988.440427] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31f skipping ALWAYS excluded test 31f (base 31) [ 2988.651242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31g skipping ALWAYS excluded test 31g \(base 31\) [ 2988.842780] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31g skipping ALWAYS excluded test 31g (base 31) [ 2989.052695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_31h skipping ALWAYS excluded test 31h \(base 31\) [ 2989.250587] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_31h skipping ALWAYS excluded test 31h (base 31) [ 2989.474679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 32: stop LFSCK when some OST failed ============================================= 23:26:30 \(1491434790\) [ 2989.666846] Lustre: DEBUG MARKER: == sanity-lfsck test 32: stop LFSCK when some OST failed ============================================= 23:26:30 (1491434790) [ 2990.442176] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x162d [ 2990.732384] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [ 2991.319300] Lustre: *** cfs_fail_loc=162d, val=0*** [ 2991.465922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [ 2993.507458] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [ 2994.326980] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation out_update to node 10.9.5.244@tcp failed: rc = -107 [ 2994.329786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2994.808345] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [ 2994.956485] LustreError: 17516:0:(osp_object.c:527:osp_attr_get()) lustre-OST0000-osc-MDT0000:osp_attr_get update error [0x100000000:0x28d:0x0]: rc = -4 [ 2995.201750] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2995.839631] Lustre: DEBUG MARKER: rc=0; [ 2995.839631] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2995.839631] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2995.839631] echo $(hostname -s): $val; [ 2995.839631] rc=$val; [ 2995.839631] fi; [ 2995.839631] exit $rc [ 2996.511164] Lustre: DEBUG MARKER: dmesg [ 2997.567715] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2997.856428] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 2998.262600] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 3000.880083] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006464b300 x1563879826155792/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 3010.033033] Lustre: 18103:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491434804/real 1491434804] req@ffff880068af8000 x1563879826155888/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491434810 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3010.040705] Lustre: 18103:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3010.199081] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3010.497517] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3010.497517] grep -q ^lustre-mdt1/ /proc/mounts || [ 3010.497517] zpool export lustre-mdt1 [ 3019.153069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3019.462823] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3020.074902] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3020.324577] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3020.588918] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3020.588918] grep -q ^lustre-mdt1/ /proc/mounts || [ 3020.588918] zpool export lustre-mdt1 [ 3020.866133] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=zfs --device-size=3145728 --reformat lustre-mdt1/mdt1 /dev/l [ 3021.586447] Lustre: DEBUG MARKER: zpool set cachefile=none lustre-mdt1 [ 3021.885429] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3021.885429] grep -q ^lustre-mdt1/ /proc/mounts || [ 3021.885429] zpool export lustre-mdt1 [ 3026.466856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test complete, duration 917 sec ====================================================== 23:27:07 \(1491434827\) [ 3026.623371] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 917 sec ====================================================== 23:27:07 (1491434827) [ 3026.815122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-hsm ============----- Wed Apr 5 23:27:08 UTC 2017 [ 3026.965204] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-hsm ============----- Wed Apr 5 23:27:08 UTC 2017 [ 3027.203916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration -o sec ========================================================= 23:27:08 \(1491434828\) [ 3027.362589] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration -o sec ========================================================= 23:27:08 (1491434828) [ 3028.180805] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 3028.180805] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 3028.180805] if [ $running -ne $mpts ]; then [ 3028.180805] echo $(hostname) env are INSANE!; [ 3028.180805] exit 1; [ 3028.180805] fi [ 3028.475451] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 3028.475451] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 3028.475451] if [ $running -ne $mpts ]; then [ 3028.475451] echo $(hostname) env are INSANE!; [ 3028.475451] exit 1; [ 3028.475451] fi [ 3029.380746] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 3029.656410] Lustre: DEBUG MARKER: modprobe zfs; [ 3029.656410] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 3029.656410] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 3030.112913] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 3030.396467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 3030.855935] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 3030.858466] Lustre: Skipped 4 previous similar messages [ 3030.918467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 3030.920766] Lustre: Skipped 1 previous similar message [ 3030.936267] Lustre: lustre-MDT0000: new disk, initializing [ 3031.256220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3031.519863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3031.809447] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 3032.294780] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3032.583481] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3032.873474] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3033.165271] Lustre: DEBUG MARKER: sync; sync; sync [ 3038.519223] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 3038.810964] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3060.438262] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 3060.765145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 3060.955903] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3061.123404] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 3066.445305] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3066.733510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3067.316973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 3067.497281] Lustre: DEBUG MARKER: excepting tests: [ 3068.865691] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [ 3069.013276] Lustre: Modifying parameter general.mdt.lustre-MDT0000.hsm_control in log params [ 3069.015489] Lustre: Skipped 4 previous similar messages [ 3089.167345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3089.461083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3089.759535] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [ 3090.102192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1: lfs hsm flags root\/non-root access ============================================= 23:28:11 \(1491434891\) [ 3090.288551] Lustre: DEBUG MARKER: == sanity-hsm test 1: lfs hsm flags root/non-root access ============================================= 23:28:11 (1491434891) [ 3090.549054] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3091.212983] Lustre: DEBUG MARKER: rc=0; [ 3091.212983] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3091.212983] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3091.212983] echo $(hostname -s): $val; [ 3091.212983] rc=$val; [ 3091.212983] fi; [ 3091.212983] exit $rc [ 3091.891459] Lustre: DEBUG MARKER: dmesg [ 3092.326702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1a: mmap & cat a HSM released file ================================================ 23:28:13 \(1491434893\) [ 3092.520561] Lustre: DEBUG MARKER: == sanity-hsm test 1a: mmap [ 3094.037465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3094.329860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3095.619527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3096.913089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3101.944004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3102.232933] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3102.520913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3102.809175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3104.099994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3105.391876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3106.681266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3107.969959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3109.262308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3109.550355] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3109.842085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3110.128343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3110.952545] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3111.614419] Lustre: DEBUG MARKER: rc=0; [ 3111.614419] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3111.614419] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3111.614419] echo $(hostname -s): $val; [ 3111.614419] rc=$val; [ 3111.614419] fi; [ 3111.614419] exit $rc [ 3112.306951] Lustre: DEBUG MARKER: dmesg [ 3112.735978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 23:28:34 \(1491434914\) [ 3112.926557] Lustre: DEBUG MARKER: == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 23:28:34 (1491434914) [ 3113.184983] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3113.852613] Lustre: DEBUG MARKER: rc=0; [ 3113.852613] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3113.852613] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3113.852613] echo $(hostname -s): $val; [ 3113.852613] rc=$val; [ 3113.852613] fi; [ 3113.852613] exit $rc [ 3114.537903] Lustre: DEBUG MARKER: dmesg [ 3114.966721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 23:28:36 \(1491434916\) [ 3115.150557] Lustre: DEBUG MARKER: == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 23:28:36 (1491434916) [ 3115.555234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3116.199161] Lustre: DEBUG MARKER: rc=0; [ 3116.199161] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3116.199161] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3116.199161] echo $(hostname -s): $val; [ 3116.199161] rc=$val; [ 3116.199161] fi; [ 3116.199161] exit $rc [ 3116.872280] Lustre: DEBUG MARKER: dmesg [ 3117.330560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 4: Useless cancel must not be registered ========================================== 23:28:38 \(1491434918\) [ 3117.527150] Lustre: DEBUG MARKER: == sanity-hsm test 4: Useless cancel must not be registered ========================================== 23:28:38 (1491434918) [ 3118.082713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 3118.404911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3119.056495] Lustre: DEBUG MARKER: rc=0; [ 3119.056495] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3119.056495] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3119.056495] echo $(hostname -s): $val; [ 3119.056495] rc=$val; [ 3119.056495] fi; [ 3119.056495] exit $rc [ 3119.733747] Lustre: DEBUG MARKER: dmesg [ 3120.169227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 8: Test default archive number ==================================================== 23:28:41 \(1491434921\) [ 3120.361517] Lustre: DEBUG MARKER: == sanity-hsm test 8: Test default archive number ==================================================== 23:28:41 (1491434921) [ 3121.286884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3121.578596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3124.392458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3124.680941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3124.968918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3125.254843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3126.540515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3127.825414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3129.115038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3130.404935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3131.691661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3131.980223] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3132.274843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3132.560497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3133.364253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3134.020404] Lustre: DEBUG MARKER: rc=0; [ 3134.020404] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3134.020404] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3134.020404] echo $(hostname -s): $val; [ 3134.020404] rc=$val; [ 3134.020404] fi; [ 3134.020404] exit $rc [ 3134.705148] Lustre: DEBUG MARKER: dmesg [ 3135.140492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 23:28:56 \(1491434936\) [ 3135.322813] Lustre: DEBUG MARKER: == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 23:28:56 (1491434936) [ 3138.005181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3138.291477] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3138.578668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3138.862532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3140.148162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3141.438093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3142.724943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3143.013659] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3143.300900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3143.585882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3147.940810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 06f64bfb-e331-eb0b-d3fb-dd9b8f1621ba [ 3148.252372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3148.539884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3151.359213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3151.647894] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3151.934289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3152.220859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3153.509233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3154.802153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3156.091601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3157.381155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3158.670286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3158.958505] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3159.245240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3159.529631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3160.336386] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3160.981125] Lustre: DEBUG MARKER: rc=0; [ 3160.981125] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3160.981125] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3160.981125] echo $(hostname -s): $val; [ 3160.981125] rc=$val; [ 3160.981125] fi; [ 3160.981125] exit $rc [ 3161.668640] Lustre: DEBUG MARKER: dmesg [ 3162.147861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9a: Multiple remote agents ======================================================== 23:29:23 \(1491434963\) [ 3162.320909] Lustre: DEBUG MARKER: == sanity-hsm test 9a: Multiple remote agents ======================================================== 23:29:23 (1491434963) [ 3162.518510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [ 3162.703233] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [ 3162.911543] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3163.566687] Lustre: DEBUG MARKER: rc=0; [ 3163.566687] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3163.566687] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3163.566687] echo $(hostname -s): $val; [ 3163.566687] rc=$val; [ 3163.566687] fi; [ 3163.566687] exit $rc [ 3164.252992] Lustre: DEBUG MARKER: dmesg [ 3164.700182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10a: Archive a file =============================================================== 23:29:26 \(1491434966\) [ 3164.892423] Lustre: DEBUG MARKER: == sanity-hsm test 10a: Archive a file =============================================================== 23:29:26 (1491434966) [ 3165.801200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3166.094748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3169.495197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3169.784540] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3170.072948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3170.358653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3171.646462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3172.935833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3174.223421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3175.511583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3176.798434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3177.087064] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3177.374521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3177.660246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3178.477507] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3179.130114] Lustre: DEBUG MARKER: rc=0; [ 3179.130114] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3179.130114] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3179.130114] echo $(hostname -s): $val; [ 3179.130114] rc=$val; [ 3179.130114] fi; [ 3179.130114] exit $rc [ 3179.803361] Lustre: DEBUG MARKER: dmesg [ 3180.242236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 23:29:41 \(1491434981\) [ 3180.428858] Lustre: DEBUG MARKER: == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 23:29:41 (1491434981) [ 3181.348410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3181.652090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3181.953433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000402:0x6:0x0'.*action='ARCHIVE'/ {n++}; END {print n}' [ 3184.771353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3185.059748] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3185.348379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3185.633264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3186.918841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3188.214451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3189.510822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3190.797563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3192.082301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3192.369106] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3192.658319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3192.947246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3193.748989] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3194.414340] Lustre: DEBUG MARKER: rc=0; [ 3194.414340] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3194.414340] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3194.414340] echo $(hostname -s): $val; [ 3194.414340] rc=$val; [ 3194.414340] fi; [ 3194.414340] exit $rc [ 3195.098224] Lustre: DEBUG MARKER: dmesg [ 3195.536059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10c: Check forbidden archive ====================================================== 23:29:56 \(1491434996\) [ 3195.725875] Lustre: DEBUG MARKER: == sanity-hsm test 10c: Check forbidden archive ====================================================== 23:29:56 (1491434996) [ 3199.174644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3199.462933] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3199.752284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3200.038343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3201.325436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3202.613080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3202.901750] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3203.192327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3203.480395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3204.272378] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3204.935903] Lustre: DEBUG MARKER: rc=0; [ 3204.935903] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3204.935903] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3204.935903] echo $(hostname -s): $val; [ 3204.935903] rc=$val; [ 3204.935903] fi; [ 3204.935903] exit $rc [ 3205.613915] Lustre: DEBUG MARKER: dmesg [ 3206.087773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 23:30:07 \(1491435007\) [ 3206.274990] Lustre: DEBUG MARKER: == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 23:30:07 (1491435007) [ 3207.187721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3207.480426] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3207.777843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 3210.595554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3210.885831] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3211.175293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3211.463165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3212.754438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3214.048069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3215.347715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3216.643295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3217.931077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3218.215939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3218.503235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3218.790969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3219.593450] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3220.259250] Lustre: DEBUG MARKER: rc=0; [ 3220.259250] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3220.259250] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3220.259250] echo $(hostname -s): $val; [ 3220.259250] rc=$val; [ 3220.259250] fi; [ 3220.259250] exit $rc [ 3220.937481] Lustre: DEBUG MARKER: dmesg [ 3221.370815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11a: Import a file ================================================================ 23:30:22 \(1491435022\) [ 3221.561136] Lustre: DEBUG MARKER: == sanity-hsm test 11a: Import a file ================================================================ 23:30:22 (1491435022) [ 3222.969910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3223.620557] Lustre: DEBUG MARKER: rc=0; [ 3223.620557] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3223.620557] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3223.620557] echo $(hostname -s): $val; [ 3223.620557] rc=$val; [ 3223.620557] fi; [ 3223.620557] exit $rc [ 3224.298785] Lustre: DEBUG MARKER: dmesg [ 3224.750731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 23:30:26 \(1491435026\) [ 3224.943667] Lustre: DEBUG MARKER: == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 23:30:26 (1491435026) [ 3225.864911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3226.158870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3229.399554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3229.687447] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3229.973327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3230.259554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3231.545517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3232.833373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3234.119892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3235.406316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3236.693938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3236.985735] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3237.272977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3237.557087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3238.362299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3239.014964] Lustre: DEBUG MARKER: rc=0; [ 3239.014964] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3239.014964] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3239.014964] echo $(hostname -s): $val; [ 3239.014964] rc=$val; [ 3239.014964] fi; [ 3239.014964] exit $rc [ 3239.691727] Lustre: DEBUG MARKER: dmesg [ 3240.142378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 23:30:41 \(1491435041\) [ 3240.333988] Lustre: DEBUG MARKER: == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 23:30:41 (1491435041) [ 3241.978036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3242.269909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3245.333520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3245.622342] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3245.910802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3246.202247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3247.495515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3248.781971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3250.067853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3251.355705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3252.641657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3252.930458] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3253.218698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3253.505170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3254.317796] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3254.976878] Lustre: DEBUG MARKER: rc=0; [ 3254.976878] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3254.976878] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3254.976878] echo $(hostname -s): $val; [ 3254.976878] rc=$val; [ 3254.976878] fi; [ 3254.976878] exit $rc [ 3255.658372] Lustre: DEBUG MARKER: dmesg [ 3256.095576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 23:30:57 \(1491435057\) [ 3256.286163] Lustre: DEBUG MARKER: == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 23:30:57 (1491435057) [ 3260.911970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3261.201888] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3261.489428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3261.776045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3263.066372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3264.354079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3265.646845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3266.933109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3268.220131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3268.506141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3268.794469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3269.078417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3269.884978] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3270.557156] Lustre: DEBUG MARKER: rc=0; [ 3270.557156] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3270.557156] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3270.557156] echo $(hostname -s): $val; [ 3270.557156] rc=$val; [ 3270.557156] fi; [ 3270.557156] exit $rc [ 3271.231795] Lustre: DEBUG MARKER: dmesg [ 3271.693480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 23:31:13 \(1491435073\) [ 3271.877434] Lustre: DEBUG MARKER: == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 23:31:13 (1491435073) [ 3273.480209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3273.771584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3275.063164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3276.364932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3277.655162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3278.948364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3287.066469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3287.352397] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3287.643176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3287.939798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3289.228257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3290.520212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3291.808350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3293.098667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3294.388860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3294.678742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3294.964705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3295.250850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3296.061622] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3296.728356] Lustre: DEBUG MARKER: rc=0; [ 3296.728356] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3296.728356] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3296.728356] echo $(hostname -s): $val; [ 3296.728356] rc=$val; [ 3296.728356] fi; [ 3296.728356] exit $rc [ 3297.421182] Lustre: DEBUG MARKER: dmesg [ 3297.893498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 23:31:39 \(1491435099\) [ 3298.085906] Lustre: DEBUG MARKER: == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 23:31:39 (1491435099) [ 3298.996403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000402:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [ 3299.294742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3299.586719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3299.882574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000402:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [ 3302.708170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3302.996247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3303.284835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3303.572843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3304.865041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3306.154344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3307.441311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3308.726214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3310.012833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3310.299359] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3310.590332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3310.877568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3311.680216] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3312.330338] Lustre: DEBUG MARKER: rc=0; [ 3312.330338] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3312.330338] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3312.330338] echo $(hostname -s): $val; [ 3312.330338] rc=$val; [ 3312.330338] fi; [ 3312.330338] exit $rc [ 3313.014824] Lustre: DEBUG MARKER: dmesg [ 3313.455938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12e: Check forbidden restore ====================================================== 23:31:54 \(1491435114\) [ 3313.645379] Lustre: DEBUG MARKER: == sanity-hsm test 12e: Check forbidden restore ====================================================== 23:31:54 (1491435114) [ 3314.560688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3314.851119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3317.737609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3318.027116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3318.316554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3318.603396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3319.895881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3321.193962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3322.483378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3323.778389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3325.067421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3325.354247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3325.641289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3325.929805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3326.732398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3327.377075] Lustre: DEBUG MARKER: rc=0; [ 3327.377075] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3327.377075] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3327.377075] echo $(hostname -s): $val; [ 3327.377075] rc=$val; [ 3327.377075] fi; [ 3327.377075] exit $rc [ 3328.052774] Lustre: DEBUG MARKER: dmesg [ 3328.493659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12f: Restore a released file explicitly =========================================== 23:32:09 \(1491435129\) [ 3328.674155] Lustre: DEBUG MARKER: == sanity-hsm test 12f: Restore a released file explicitly =========================================== 23:32:09 (1491435129) [ 3329.594603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3329.883982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3330.194893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3330.487084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3333.445232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3333.732971] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3334.022342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3334.307435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3335.594088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3336.881544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3338.167129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3339.455340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3340.742669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3341.027781] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3341.316072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3341.604116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3342.405862] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3343.056813] Lustre: DEBUG MARKER: rc=0; [ 3343.056813] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3343.056813] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3343.056813] echo $(hostname -s): $val; [ 3343.056813] rc=$val; [ 3343.056813] fi; [ 3343.056813] exit $rc [ 3343.739646] Lustre: DEBUG MARKER: dmesg [ 3344.169299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12g: Restore a released file implicitly =========================================== 23:32:25 \(1491435145\) [ 3344.356975] Lustre: DEBUG MARKER: == sanity-hsm test 12g: Restore a released file implicitly =========================================== 23:32:25 (1491435145) [ 3345.284997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3345.576037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3346.074956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3346.365597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3349.193816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3349.485263] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3349.776896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3350.064949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3351.353577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3352.643345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3353.929596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3355.225881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3356.514368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3356.802176] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3357.096608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3357.388898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3358.201279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3358.855131] Lustre: DEBUG MARKER: rc=0; [ 3358.855131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3358.855131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3358.855131] echo $(hostname -s): $val; [ 3358.855131] rc=$val; [ 3358.855131] fi; [ 3358.855131] exit $rc [ 3359.535668] Lustre: DEBUG MARKER: dmesg [ 3359.970649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 23:32:41 \(1491435161\) [ 3360.156859] Lustre: DEBUG MARKER: == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 23:32:41 (1491435161) [ 3361.067491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3361.359103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3362.066177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3362.357908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3365.170379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3365.459999] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3365.750455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3366.041484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3367.332705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3368.623310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3369.909506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3371.199865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3372.487460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3372.773936] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3373.063876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3373.353406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3374.158531] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3374.812844] Lustre: DEBUG MARKER: rc=0; [ 3374.812844] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3374.812844] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3374.812844] echo $(hostname -s): $val; [ 3374.812844] rc=$val; [ 3374.812844] fi; [ 3374.812844] exit $rc [ 3375.483387] Lustre: DEBUG MARKER: dmesg [ 3375.933514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12m: Archive\/release\/implicit restore ============================================= 23:32:57 \(1491435177\) [ 3376.118615] Lustre: DEBUG MARKER: == sanity-hsm test 12m: Archive/release/implicit restore ============================================= 23:32:57 (1491435177) [ 3377.027090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3377.318227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3380.402483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3380.689408] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3380.977703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3381.262697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3382.550189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3383.838115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3385.124765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3386.411747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3387.699800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3387.989347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3388.278328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3388.565635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3389.368905] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3390.030996] Lustre: DEBUG MARKER: rc=0; [ 3390.030996] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3390.030996] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3390.030996] echo $(hostname -s): $val; [ 3390.030996] rc=$val; [ 3390.030996] fi; [ 3390.030996] exit $rc [ 3390.703613] Lustre: DEBUG MARKER: dmesg [ 3391.145992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12n: Import\/implicit restore\/release ============================================== 23:33:12 \(1491435192\) [ 3391.333266] Lustre: DEBUG MARKER: == sanity-hsm test 12n: Import/implicit restore/release ============================================== 23:33:12 (1491435192) [ 3396.092788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3396.383159] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3396.671775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3396.959268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3398.246725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3399.540815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3400.831337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3402.117725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3403.407212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3403.696276] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3403.983771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3404.270952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3405.074704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3405.734631] Lustre: DEBUG MARKER: rc=0; [ 3405.734631] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3405.734631] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3405.734631] echo $(hostname -s): $val; [ 3405.734631] rc=$val; [ 3405.734631] fi; [ 3405.734631] exit $rc [ 3406.413319] Lustre: DEBUG MARKER: dmesg [ 3406.853492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 23:33:28 \(1491435208\) [ 3407.037663] Lustre: DEBUG MARKER: == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 23:33:28 (1491435208) [ 3407.960765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3408.253865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3408.553593] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x152 [ 3408.844941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 3409.186936] Lustre: *** cfs_fail_loc=152, val=0*** [ 3409.334866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3409.625907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3409.916450] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 3410.209659] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3410.498834] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 3410.788121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3411.075421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3411.362818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 3421.803405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3422.094102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3424.922073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3425.210242] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3425.497878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3425.787422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3427.082676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3428.372760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3429.660028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3430.947410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3432.235806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3432.523152] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3432.812697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3433.099423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3433.907080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3434.562931] Lustre: DEBUG MARKER: rc=0; [ 3434.562931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3434.562931] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3434.562931] echo $(hostname -s): $val; [ 3434.562931] rc=$val; [ 3434.562931] fi; [ 3434.562931] exit $rc [ 3435.244066] Lustre: DEBUG MARKER: dmesg [ 3435.666603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 23:33:56 \(1491435236\) [ 3435.855431] Lustre: DEBUG MARKER: == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 23:33:56 (1491435236) [ 3436.778674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3437.069178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3441.068582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3441.357080] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3441.645079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3441.933435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3443.228861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3444.515873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3445.802097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3447.087410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3448.383136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3448.673846] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3448.965064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3449.250793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3450.061920] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3450.728476] Lustre: DEBUG MARKER: rc=0; [ 3450.728476] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3450.728476] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3450.728476] echo $(hostname -s): $val; [ 3450.728476] rc=$val; [ 3450.728476] fi; [ 3450.728476] exit $rc [ 3451.402874] Lustre: DEBUG MARKER: dmesg [ 3451.887906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 23:34:13 \(1491435253\) [ 3452.070245] Lustre: DEBUG MARKER: == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 23:34:13 (1491435253) [ 3452.241838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 3452.893366] Lustre: lustre-MDT0000: Connection restored to e2ce2e05-a1b3-0c7a-22f6-b0410d9ccdfc (at 10.9.5.238@tcp) [ 3452.895509] Lustre: Skipped 30 previous similar messages [ 3454.630861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3454.922098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3456.213873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3457.507457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3475.269471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3475.558425] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3475.849676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3476.136984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3477.431852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3478.719664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3480.011682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3481.304633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3482.596944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3482.884978] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3483.176138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3483.468088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3485.075887] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3485.731328] Lustre: DEBUG MARKER: rc=0; [ 3485.731328] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3485.731328] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3485.731328] echo $(hostname -s): $val; [ 3485.731328] rc=$val; [ 3485.731328] fi; [ 3485.731328] exit $rc [ 3486.412165] Lustre: DEBUG MARKER: dmesg [ 3486.858897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 13: Recursively import and restore a directory ==================================== 23:34:48 \(1491435288\) [ 3487.043424] Lustre: DEBUG MARKER: == sanity-hsm test 13: Recursively import and restore a directory ==================================== 23:34:48 (1491435288) [ 3536.239861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3536.529372] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3536.818891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3537.107141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3538.398686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3539.686884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3540.974174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3542.263628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3543.553774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3543.843848] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3544.134351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3544.418932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3545.264368] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3545.926168] Lustre: DEBUG MARKER: rc=0; [ 3545.926168] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3545.926168] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3545.926168] echo $(hostname -s): $val; [ 3545.926168] rc=$val; [ 3545.926168] fi; [ 3545.926168] exit $rc [ 3546.611605] Lustre: DEBUG MARKER: dmesg [ 3547.071947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 23:35:48 \(1491435348\) [ 3547.256155] Lustre: DEBUG MARKER: == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 23:35:48 (1491435348) [ 3548.659928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3548.948567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3550.246170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3551.547285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3556.717387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3557.005701] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3557.292842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3557.579849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3558.869387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3560.165189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3561.452419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3562.741151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3564.033393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3564.319375] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3564.605732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3564.891815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3565.700555] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3566.377924] Lustre: DEBUG MARKER: rc=0; [ 3566.377924] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3566.377924] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3566.377924] echo $(hostname -s): $val; [ 3566.377924] rc=$val; [ 3566.377924] fi; [ 3566.377924] exit $rc [ 3567.055835] Lustre: DEBUG MARKER: dmesg [ 3567.510161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 15: Rebind a list of files ======================================================== 23:36:08 \(1491435368\) [ 3567.704843] Lustre: DEBUG MARKER: == sanity-hsm test 15: Rebind a list of files ======================================================== 23:36:08 (1491435368) [ 3572.150119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 3572.440483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 3573.732103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 3575.026494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 3589.924066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3590.211866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3590.501261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3590.791216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3592.082051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3593.370580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3594.657880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3595.946318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3597.238643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3597.526023] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3597.814677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3598.102534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3598.921491] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3599.574638] Lustre: DEBUG MARKER: rc=0; [ 3599.574638] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3599.574638] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3599.574638] echo $(hostname -s): $val; [ 3599.574638] rc=$val; [ 3599.574638] fi; [ 3599.574638] exit $rc [ 3600.255710] Lustre: DEBUG MARKER: dmesg [ 3600.716873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 16: Test CT bandwith control option =============================================== 23:36:42 \(1491435402\) [ 3600.908495] Lustre: DEBUG MARKER: == sanity-hsm test 16: Test CT bandwith control option =============================================== 23:36:42 (1491435402) [ 3602.305495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3602.596112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3603.892137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3605.189716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3606.480681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3607.776170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3609.070598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3610.369503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3611.671272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3612.968229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3614.271460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3615.567241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3616.860296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3618.153944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3619.446969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3620.739584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3622.037890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3624.858611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3625.152155] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3625.442874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3625.730158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3627.019550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3628.307882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3629.603465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3630.901708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3632.196162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3632.487315] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3632.782592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3633.072920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3633.875587] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3634.527989] Lustre: DEBUG MARKER: rc=0; [ 3634.527989] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3634.527989] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3634.527989] echo $(hostname -s): $val; [ 3634.527989] rc=$val; [ 3634.527989] fi; [ 3634.527989] exit $rc [ 3635.217450] Lustre: DEBUG MARKER: dmesg [ 3635.660390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 20: Release is not permitted ====================================================== 23:37:16 \(1491435436\) [ 3635.856412] Lustre: DEBUG MARKER: == sanity-hsm test 20: Release is not permitted ====================================================== 23:37:16 (1491435436) [ 3636.094515] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3636.753461] Lustre: DEBUG MARKER: rc=0; [ 3636.753461] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3636.753461] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3636.753461] echo $(hostname -s): $val; [ 3636.753461] rc=$val; [ 3636.753461] fi; [ 3636.753461] exit $rc [ 3637.443710] Lustre: DEBUG MARKER: dmesg [ 3637.883115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 21: Simple release tests ========================================================== 23:37:19 \(1491435439\) [ 3638.062079] Lustre: DEBUG MARKER: == sanity-hsm test 21: Simple release tests ========================================================== 23:37:19 (1491435439) [ 3639.555295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3639.884124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [ 3640.264843] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [ 3640.620747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3640.914398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3642.207045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3643.497796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3643.898244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x44:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3644.192300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000401:0x44:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3644.556606] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [ 3645.194136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3648.079706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3648.378819] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3648.665631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3648.948766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3650.241737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3651.530902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3652.816900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3654.105266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3654.393962] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3654.686066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3654.974701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3655.772438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3656.437856] Lustre: DEBUG MARKER: rc=0; [ 3656.437856] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3656.437856] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3656.437856] echo $(hostname -s): $val; [ 3656.437856] rc=$val; [ 3656.437856] fi; [ 3656.437856] exit $rc [ 3657.128858] Lustre: DEBUG MARKER: dmesg [ 3657.572863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 22: Could not swap a release file ================================================= 23:37:38 \(1491435458\) [ 3657.766931] Lustre: DEBUG MARKER: == sanity-hsm test 22: Could not swap a release file ================================================= 23:37:38 (1491435458) [ 3659.058138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3659.352251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3660.645717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3661.941949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3665.189065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3665.492720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3665.782647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3666.072884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3667.361749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3668.650220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3669.938417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3671.224456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3671.512605] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3671.801747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3672.089286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3672.900798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3673.558648] Lustre: DEBUG MARKER: rc=0; [ 3673.558648] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3673.558648] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3673.558648] echo $(hostname -s): $val; [ 3673.558648] rc=$val; [ 3673.558648] fi; [ 3673.558648] exit $rc [ 3674.246221] Lustre: DEBUG MARKER: dmesg [ 3674.682370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 23: Release does not change a\/mtime \(utime\) ======================================= 23:37:56 \(1491435476\) [ 3674.875878] Lustre: DEBUG MARKER: == sanity-hsm test 23: Release does not change a/mtime (utime) ======================================= 23:37:56 (1491435476) [ 3676.149928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3676.440308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3677.731874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3679.029062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3681.880609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3682.169106] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3682.457888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3682.746822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3684.039973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3685.328654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3686.619216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3687.908105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3689.196326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3689.483655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3689.771613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3690.060515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3690.871247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3691.527097] Lustre: DEBUG MARKER: rc=0; [ 3691.527097] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3691.527097] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3691.527097] echo $(hostname -s): $val; [ 3691.527097] rc=$val; [ 3691.527097] fi; [ 3691.527097] exit $rc [ 3692.213756] Lustre: DEBUG MARKER: dmesg [ 3692.657772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24a: Archive, release, and restore does not change a\/mtime \(i\/o\) ================== 23:38:13 \(1491435493\) [ 3692.847303] Lustre: DEBUG MARKER: == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ================== 23:38:13 (1491435493) [ 3695.207443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3695.501448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3696.794608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3698.086843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3698.409126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3698.701081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3699.992616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3701.283412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3704.112429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3704.399855] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3704.690206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3704.980533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3706.270872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3707.559377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3708.853724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3710.146138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3711.434051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3711.720664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3712.008942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3712.293380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3713.288650] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3713.958289] Lustre: DEBUG MARKER: rc=0; [ 3713.958289] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3713.958289] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3713.958289] echo $(hostname -s): $val; [ 3713.958289] rc=$val; [ 3713.958289] fi; [ 3713.958289] exit $rc [ 3714.646344] Lustre: DEBUG MARKER: dmesg [ 3715.119376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 23:38:36 \(1491435516\) [ 3715.293935] Lustre: DEBUG MARKER: == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 23:38:36 (1491435516) [ 3716.738551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3717.031326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3718.324130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3719.616662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3719.941563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3720.232628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3721.524302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3722.823250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3727.982585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3728.273146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3728.564183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3728.852250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3730.140820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3731.432073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3732.718515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3734.005392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3735.296144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3735.586748] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3735.875936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3736.163623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3736.967792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3737.631150] Lustre: DEBUG MARKER: rc=0; [ 3737.631150] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3737.631150] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3737.631150] echo $(hostname -s): $val; [ 3737.631150] rc=$val; [ 3737.631150] fi; [ 3737.631150] exit $rc [ 3738.318580] Lustre: DEBUG MARKER: dmesg [ 3738.771559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24c: check that user,group,other request masks work =============================== 23:39:00 \(1491435540\) [ 3738.964634] Lustre: DEBUG MARKER: == sanity-hsm test 24c: check that user,group,other request masks work =============================== 23:39:00 (1491435540) [ 3739.865052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.user_request_mask [ 3740.154123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.group_request_mask [ 3740.444662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.other_request_mask [ 3741.228794] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=archive [ 3741.996643] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=archive [ 3742.764409] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=archive [ 3745.601458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3745.888843] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3746.175287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3746.463326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3747.755659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3749.042333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3750.332496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3751.620791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3752.910700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3753.210759] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3753.499626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3753.785789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3754.555426] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=RESTORE [ 3754.847330] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=RESTORE [ 3755.136731] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=RESTORE [ 3755.460652] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3756.123405] Lustre: DEBUG MARKER: rc=0; [ 3756.123405] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3756.123405] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3756.123405] echo $(hostname -s): $val; [ 3756.123405] rc=$val; [ 3756.123405] fi; [ 3756.123405] exit $rc [ 3756.802931] Lustre: DEBUG MARKER: dmesg [ 3757.242223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 23:39:18 \(1491435558\) [ 3757.431639] Lustre: DEBUG MARKER: == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 23:39:18 (1491435558) [ 3759.760252] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3760.052371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3761.343102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3762.635919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3762.944836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3763.236171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3764.531603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3765.830414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 3770.882082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3771.183526] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3771.475794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3771.762914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3773.058828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3774.361070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3775.655560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3776.944548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3778.231762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3778.519745] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3778.807608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3779.091460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3780.738172] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3781.404341] Lustre: DEBUG MARKER: rc=0; [ 3781.404341] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3781.404341] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3781.404341] echo $(hostname -s): $val; [ 3781.404341] rc=$val; [ 3781.404341] fi; [ 3781.404341] exit $rc [ 3782.085941] Lustre: DEBUG MARKER: dmesg [ 3782.641863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 23:39:43 \(1491435583\) [ 3782.833311] Lustre: DEBUG MARKER: == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 23:39:43 (1491435583) [ 3784.153653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3784.445355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3785.738321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3787.030724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3792.217974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3792.505154] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3792.796056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3793.086686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3794.381157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3795.675422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3796.963080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3798.253123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3799.548271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3799.838642] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3800.130035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3800.419145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3801.233095] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3801.897570] Lustre: DEBUG MARKER: rc=0; [ 3801.897570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3801.897570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3801.897570] echo $(hostname -s): $val; [ 3801.897570] rc=$val; [ 3801.897570] fi; [ 3801.897570] exit $rc [ 3802.586044] Lustre: DEBUG MARKER: dmesg [ 3803.029117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 23:40:04 \(1491435604\) [ 3803.223518] Lustre: DEBUG MARKER: == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 23:40:04 (1491435604) [ 3804.147722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3804.438253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3807.439459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3807.729115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3808.024618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3808.314470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3809.606096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3810.894868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3812.182938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3813.481194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3814.770344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3815.059831] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3815.350393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3815.640059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3816.446538] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3817.116408] Lustre: DEBUG MARKER: rc=0; [ 3817.116408] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3817.116408] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3817.116408] echo $(hostname -s): $val; [ 3817.116408] rc=$val; [ 3817.116408] fi; [ 3817.116408] exit $rc [ 3817.795472] Lustre: DEBUG MARKER: dmesg [ 3818.250258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25a: Restore lost file \(HS_LOST flag\) from import ================================= 23:40:19 \(1491435619\) [ 3818.442168] Lustre: DEBUG MARKER: == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ================================= 23:40:19 (1491435619) [ 3822.601356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3822.891096] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3823.183745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3823.470328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3824.759825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3826.057122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3826.346921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3826.639133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3826.925589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3827.732155] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3828.386315] Lustre: DEBUG MARKER: rc=0; [ 3828.386315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3828.386315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3828.386315] echo $(hostname -s): $val; [ 3828.386315] rc=$val; [ 3828.386315] fi; [ 3828.386315] exit $rc [ 3829.059182] Lustre: DEBUG MARKER: dmesg [ 3829.510469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25b: Restore lost file \(HS_LOST flag\) after release =============================== 23:40:30 \(1491435630\) [ 3829.699929] Lustre: DEBUG MARKER: == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release =============================== 23:40:30 (1491435630) [ 3830.629715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3830.920222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3833.765576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3834.055068] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3834.344457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3834.632612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3835.931946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3837.220462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3838.514023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3839.800984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3841.090346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3841.379626] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3841.669223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3841.958578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3842.769204] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3843.431466] Lustre: DEBUG MARKER: rc=0; [ 3843.431466] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3843.431466] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3843.431466] echo $(hostname -s): $val; [ 3843.431466] rc=$val; [ 3843.431466] fi; [ 3843.431466] exit $rc [ 3844.110429] Lustre: DEBUG MARKER: dmesg [ 3844.556785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 26: Remove the archive of a valid file ============================================ 23:40:45 \(1491435645\) [ 3844.752833] Lustre: DEBUG MARKER: == sanity-hsm test 26: Remove the archive of a valid file ============================================ 23:40:45 (1491435645) [ 3847.710117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3848.000692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3849.296230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3850.587156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3851.880305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3853.173547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3854.469668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3855.764067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3857.055919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3858.349514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3859.641506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3860.932694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3862.232899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3863.528707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3864.833732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3866.132756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3867.433813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3868.727752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3870.025077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3871.339870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3872.643578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3873.936453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3875.230941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3876.525217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3877.822067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3879.125457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3880.422115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3881.719922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3883.014431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3884.308393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3885.599712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3885.896467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 3886.189051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 3889.021919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3889.321878] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3889.645688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3889.933579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3891.222977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3892.512118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3893.800137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3895.091473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3896.378590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3896.668703] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3896.960708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3897.247679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3898.056543] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3898.723029] Lustre: DEBUG MARKER: rc=0; [ 3898.723029] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3898.723029] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3898.723029] echo $(hostname -s): $val; [ 3898.723029] rc=$val; [ 3898.723029] fi; [ 3898.723029] exit $rc [ 3899.407110] Lustre: DEBUG MARKER: dmesg [ 3899.864652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27a: Remove the archive of an imported file \(Operation not permitted\) ============= 23:41:41 \(1491435701\) [ 3900.077641] Lustre: DEBUG MARKER: == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ============= 23:41:41 (1491435701) [ 3906.551255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3906.845459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3907.145199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3907.435685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3908.729854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3910.024699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3911.310309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3912.600520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3913.887501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3915.175595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3916.462271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3917.748972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3918.036973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3918.325646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3918.609965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3919.424389] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3920.074095] Lustre: DEBUG MARKER: rc=0; [ 3920.074095] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3920.074095] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3920.074095] echo $(hostname -s): $val; [ 3920.074095] rc=$val; [ 3920.074095] fi; [ 3920.074095] exit $rc [ 3920.751172] Lustre: DEBUG MARKER: dmesg [ 3921.194064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27b: Remove the archive of a relased file \(Operation not permitted\) =============== 23:42:02 \(1491435722\) [ 3921.380129] Lustre: DEBUG MARKER: == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) =============== 23:42:02 (1491435722) [ 3924.778193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3925.069095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3926.359716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3927.652254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3928.943302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3930.232399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3931.521985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3932.819813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3934.114930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3935.410869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3936.705752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3938.003143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3939.301038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3940.597861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3941.892308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3943.189901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3944.488559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3945.785174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3947.077633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3948.373487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3949.667139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3950.962451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3952.260352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3953.554500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3954.854944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3956.158686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3957.450868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3958.746353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3960.046694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3961.342714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3962.638358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3965.496483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3965.785891] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 3966.075747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3966.363782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3967.649515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3968.944050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3970.235448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3971.530110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3972.818905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3973.111150] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 3973.401855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3973.688770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 3974.493484] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3975.151562] Lustre: DEBUG MARKER: rc=0; [ 3975.151562] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3975.151562] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3975.151562] echo $(hostname -s): $val; [ 3975.151562] rc=$val; [ 3975.151562] fi; [ 3975.151562] exit $rc [ 3975.839720] Lustre: DEBUG MARKER: dmesg [ 3976.274072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 28: Concurrent archive\/file remove ================================================ 23:42:57 \(1491435777\) [ 3976.462160] Lustre: DEBUG MARKER: == sanity-hsm test 28: Concurrent archive/file remove ================================================ 23:42:57 (1491435777) [ 3978.854246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3979.146766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3980.439907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3981.734117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3983.040657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3984.336388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3985.633913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3986.928531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3988.227567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3989.520928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3990.820482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3992.119829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3993.413672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3994.706294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3996.001330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3997.298234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3998.591084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 3999.884703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4001.189489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4002.483446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4003.785281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4005.089237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4006.400053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4007.694028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4008.990837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4010.285124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4011.583372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4012.887161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4014.184064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4015.483560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4015.797356] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 4016.103259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4016.407057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4016.719875] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4017.025964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4017.323748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4017.611848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4017.905076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4020.746090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4021.038701] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4021.329664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4021.617318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4022.913447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4024.227889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4025.518885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4026.812865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4027.103291] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4027.393874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4027.679948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4028.508780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4029.162334] Lustre: DEBUG MARKER: rc=0; [ 4029.162334] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4029.162334] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4029.162334] echo $(hostname -s): $val; [ 4029.162334] rc=$val; [ 4029.162334] fi; [ 4029.162334] exit $rc [ 4029.859706] Lustre: DEBUG MARKER: dmesg [ 4030.315671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 23:43:51 \(1491435831\) [ 4030.499941] Lustre: DEBUG MARKER: == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 23:43:51 (1491435831) [ 4033.940688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4034.237545] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4034.527131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4034.813613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4036.099841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4037.384864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4037.674702] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4037.963686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4038.252654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4039.059432] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4039.733900] Lustre: DEBUG MARKER: rc=0; [ 4039.733900] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4039.733900] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4039.733900] echo $(hostname -s): $val; [ 4039.733900] rc=$val; [ 4039.733900] fi; [ 4039.733900] exit $rc [ 4040.417276] Lustre: DEBUG MARKER: dmesg [ 4040.868711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29b: Archive\/delete\/remove by FID from the archive. =============================== 23:44:02 \(1491435842\) [ 4041.064539] Lustre: DEBUG MARKER: == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. =============================== 23:44:02 (1491435842) [ 4042.412433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4042.703472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4043.995684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4044.295471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4044.592551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4047.435282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4047.723882] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4048.012668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4048.300682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4049.591993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4050.890785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4052.186581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4053.482201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4054.770678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4055.059563] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4055.349890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4055.636326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4056.459080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4057.125942] Lustre: DEBUG MARKER: rc=0; [ 4057.125942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4057.125942] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4057.125942] echo $(hostname -s): $val; [ 4057.125942] rc=$val; [ 4057.125942] fi; [ 4057.125942] exit $rc [ 4057.808173] Lustre: DEBUG MARKER: dmesg [ 4058.264669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29c: Archive\/delete\/remove by FID, using a file list. ============================= 23:44:19 \(1491435859\) [ 4058.451215] Lustre: DEBUG MARKER: == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ============================= 23:44:19 (1491435859) [ 4061.054576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4061.348367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4062.642266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4063.932761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4064.226082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4064.518492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4064.809038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4065.099534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4065.407773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4065.703177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4066.002802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4066.296221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4066.588357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4066.879897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 4069.695334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4069.985418] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4070.275630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4070.566581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4071.854367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4073.145294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4074.433879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4075.721382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4076.011957] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4076.301052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4076.587761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4077.403207] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4078.061505] Lustre: DEBUG MARKER: rc=0; [ 4078.061505] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4078.061505] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4078.061505] echo $(hostname -s): $val; [ 4078.061505] rc=$val; [ 4078.061505] fi; [ 4078.061505] exit $rc [ 4078.755995] Lustre: DEBUG MARKER: dmesg [ 4079.202216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 23:44:40 \(1491435880\) [ 4079.387385] Lustre: DEBUG MARKER: == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 23:44:40 (1491435880) [ 4079.595282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [ 4079.784649] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [ 4079.996964] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4080.652453] Lustre: DEBUG MARKER: rc=0; [ 4080.652453] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4080.652453] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4080.652453] echo $(hostname -s): $val; [ 4080.652453] rc=$val; [ 4080.652453] fi; [ 4080.652453] exit $rc [ 4081.343883] Lustre: DEBUG MARKER: dmesg [ 4081.798467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30a: Restore at exec \(import case\) ================================================ 23:44:43 \(1491435883\) [ 4081.980780] Lustre: DEBUG MARKER: == sanity-hsm test 30a: Restore at exec (import case) ================================================ 23:44:43 (1491435883) [ 4083.623985] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 4084.415473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 4087.252840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4087.540350] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4087.827500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4088.115094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4089.404278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4090.692309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4091.985359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4093.274834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4094.567099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4094.861251] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4095.150222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4095.437716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4096.269848] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4096.912600] Lustre: DEBUG MARKER: rc=0; [ 4096.912600] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4096.912600] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4096.912600] echo $(hostname -s): $val; [ 4096.912600] rc=$val; [ 4096.912600] fi; [ 4096.912600] exit $rc [ 4097.595324] Lustre: DEBUG MARKER: dmesg [ 4098.052927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30b: Restore at exec \(release case\) =============================================== 23:44:59 \(1491435899\) [ 4098.228828] Lustre: DEBUG MARKER: == sanity-hsm test 30b: Restore at exec (release case) =============================================== 23:44:59 (1491435899) [ 4099.174122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4099.464759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4099.765824] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 4100.688751] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 4103.574859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4103.865757] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4104.154807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4104.441851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4105.731371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4107.018963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4108.305510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4109.594047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4110.883772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4111.169528] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4111.456694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4111.742969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4112.544255] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4113.212212] Lustre: DEBUG MARKER: rc=0; [ 4113.212212] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4113.212212] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4113.212212] echo $(hostname -s): $val; [ 4113.212212] rc=$val; [ 4113.212212] fi; [ 4113.212212] exit $rc [ 4113.888334] Lustre: DEBUG MARKER: dmesg [ 4114.380788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30c: Update during exec of released file must fail ================================ 23:45:15 \(1491435915\) [ 4114.570615] Lustre: DEBUG MARKER: == sanity-hsm test 30c: Update during exec of released file must fail ================================ 23:45:15 (1491435915) [ 4115.512300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4115.802465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4116.115630] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 4126.886576] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 4129.704691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4129.994771] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4130.283990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4130.571340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4131.865294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4133.167201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4134.455970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4135.746960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4137.033313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4137.320343] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4137.610765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4137.898202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4138.716776] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4139.392260] Lustre: DEBUG MARKER: rc=0; [ 4139.392260] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4139.392260] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4139.392260] echo $(hostname -s): $val; [ 4139.392260] rc=$val; [ 4139.392260] fi; [ 4139.392260] exit $rc [ 4140.081653] Lustre: DEBUG MARKER: dmesg [ 4140.556523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31a: Import a large file and check size during restore ============================ 23:45:41 \(1491435941\) [ 4140.749503] Lustre: DEBUG MARKER: == sanity-hsm test 31a: Import a large file and check size during restore ============================ 23:45:41 (1491435941) [ 4145.022532] Lustre: HSM agent 43d2d38d-8855-c518-88d6-7518291a0b16 already registered [ 4145.050835] LustreError: 20966:0:(mdt_coordinator.c:1390:mdt_hsm_update_request_state()) lustre-MDT0000: Progress on [0x200000403:0x82:0x0] for cookie 0x58e581a3 does not match request FID [0x200000404:0xd3:0x0] nor data FID [0x200000403:0x81:0x0] [ 4187.767550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4188.060355] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4188.352831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4188.642806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4189.930378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4191.222710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4192.511623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4193.798022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4195.085413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4196.371808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4196.661146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4196.948419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4197.239072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4198.062624] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4198.720966] Lustre: DEBUG MARKER: rc=0; [ 4198.720966] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4198.720966] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4198.720966] echo $(hostname -s): $val; [ 4198.720966] rc=$val; [ 4198.720966] fi; [ 4198.720966] exit $rc [ 4199.407519] Lustre: DEBUG MARKER: dmesg [ 4199.908465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 23:46:41 \(1491436001\) [ 4200.102128] Lustre: DEBUG MARKER: == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 23:46:41 (1491436001) [ 4204.962156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4205.252788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4206.546137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4207.838801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4209.136543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4210.431211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4211.722789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4213.017883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4214.307856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4215.599130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4216.892150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4218.186649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4219.486934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4220.794233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4222.087948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4223.382887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4224.677739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4225.974418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4227.269624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4228.564913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4229.858673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4231.158454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4232.466040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4233.769393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4235.069194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4236.365920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4237.663689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4238.965885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4240.263071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4241.561292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4242.861923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4285.765516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4286.058898] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4286.365557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4286.656998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4287.951408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4289.243216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4290.534674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4290.824744] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4291.112642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4291.400318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4292.216748] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4292.882697] Lustre: DEBUG MARKER: rc=0; [ 4292.882697] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4292.882697] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4292.882697] echo $(hostname -s): $val; [ 4292.882697] rc=$val; [ 4292.882697] fi; [ 4292.882697] exit $rc [ 4293.567681] Lustre: DEBUG MARKER: dmesg [ 4294.028862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 23:48:15 \(1491436095\) [ 4294.214716] Lustre: DEBUG MARKER: == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 23:48:15 (1491436095) [ 4298.762436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4299.056752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4300.347274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4301.638568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4302.939666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4304.241421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4305.531575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4306.823768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4308.122194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4309.420733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4310.714423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4312.012460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4313.311704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4314.612805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4315.912979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4317.217201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4318.515266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4319.806768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4321.101556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4322.400727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4323.696997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4324.988921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4326.282368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4327.579035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4328.873691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4330.168700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4331.463848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4332.761661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4375.656920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4375.946128] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4376.237736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4376.526281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4377.815705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4379.113604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4380.402110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4381.690884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4382.981878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4384.271471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4385.560771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4386.848495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4387.147649] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4387.438713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4387.733931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4388.549593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4389.227087] Lustre: DEBUG MARKER: rc=0; [ 4389.227087] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4389.227087] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4389.227087] echo $(hostname -s): $val; [ 4389.227087] rc=$val; [ 4389.227087] fi; [ 4389.227087] exit $rc [ 4389.925268] Lustre: DEBUG MARKER: dmesg [ 4390.383688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 33: Kill a restore waiting process ================================================ 23:49:51 \(1491436191\) [ 4390.575282] Lustre: DEBUG MARKER: == sanity-hsm test 33: Kill a restore waiting process ================================================ 23:49:51 (1491436191) [ 4394.555166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4394.848758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4396.145389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4397.444861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4398.743187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4400.041198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4401.337659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4402.642837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4403.946999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4405.241575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4406.533624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4407.831514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4409.128865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4410.425601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4411.721837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4413.016146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4414.316048] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4415.608368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4416.900879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4418.200737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4419.492924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4420.795461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4422.088646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4423.386073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4424.682999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4425.978069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4427.272054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4428.567077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4429.865296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4431.163184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4432.460391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4432.776226] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 4433.069390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4433.358239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4433.649411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 4443.944079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 4444.233380] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=100 [ 4444.526273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4444.818797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4446.130371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4446.425562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4447.722216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4449.018117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4450.315099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4451.612238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4452.906974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4454.207902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4455.505884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4456.800543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4458.093209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4459.386144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4460.689107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4461.984456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4463.278782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4464.577178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4465.873269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4467.168492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4468.464907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4469.760154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4471.054640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4472.347977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4473.646532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4474.943722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000402:0x33:0x0' | egrep 'WAITING|STARTED' [ 4475.237826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4475.531622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x33:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 4475.826424] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [ 4478.638567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4478.929790] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4479.219779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4479.509126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4480.796922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4482.092101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4483.384239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4484.675619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4484.964276] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4485.254594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4485.541511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4486.350234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4487.014342] Lustre: DEBUG MARKER: rc=0; [ 4487.014342] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4487.014342] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4487.014342] echo $(hostname -s): $val; [ 4487.014342] rc=$val; [ 4487.014342] fi; [ 4487.014342] exit $rc [ 4487.712380] Lustre: DEBUG MARKER: dmesg [ 4488.159481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 34: Remove file during restore ==================================================== 23:51:29 \(1491436289\) [ 4488.346470] Lustre: DEBUG MARKER: == sanity-hsm test 34: Remove file during restore ==================================================== 23:51:29 (1491436289) [ 4491.694266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4491.987106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4493.282476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4494.574495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4495.873620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4497.164411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4498.456204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4499.748475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4501.041885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4502.333150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4503.627422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4504.925309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4506.225894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4507.530065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4508.828910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4510.130630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4511.422488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4512.720114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4514.011992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4515.307819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4516.601161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4517.892944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4519.187664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4520.484812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4521.784966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4523.079715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4524.373967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4525.668857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4526.968942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4528.259644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4529.553354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4529.878108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4530.172530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4530.474053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4530.770840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4531.067945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4531.360841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4532.655198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4533.958643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4535.269675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4536.578670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4537.878612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4539.181687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4540.477063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4541.770789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4543.067553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4544.365602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4545.672334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4546.969023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4548.263057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4549.560267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4550.858502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4552.150856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4553.445138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4554.741229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4556.040304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4557.334571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4558.628711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4559.922636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4561.215733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4562.510042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4563.804636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4565.100994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4566.400729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4567.735169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4570.597263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4570.884322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4571.176847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4571.463184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4572.751097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4574.037894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4575.327212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4576.616365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4577.904552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4578.195194] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4578.486516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4578.776097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4579.594222] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4580.271363] Lustre: DEBUG MARKER: rc=0; [ 4580.271363] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4580.271363] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4580.271363] echo $(hostname -s): $val; [ 4580.271363] rc=$val; [ 4580.271363] fi; [ 4580.271363] exit $rc [ 4580.960578] Lustre: DEBUG MARKER: dmesg [ 4581.416921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 35: Overwrite file during restore ================================================= 23:53:02 \(1491436382\) [ 4581.613478] Lustre: DEBUG MARKER: == sanity-hsm test 35: Overwrite file during restore ================================================= 23:53:02 (1491436382) [ 4584.639633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4584.931673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4586.221379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4587.512818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4588.806905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4590.105490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4591.403073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4592.696427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4593.990412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4595.282707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4596.574754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4597.873305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4599.163627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4600.454954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4601.747281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4603.041575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4604.348171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4605.643473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4606.937004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4608.236720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4609.529931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4610.822378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4612.119316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4613.413791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4614.715621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4616.007947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4617.306656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4618.604395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4619.898825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4621.193099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4622.488032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4622.809790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4623.101115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4623.405656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4623.697561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4623.991330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4624.282460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4625.576384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4626.874032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4628.166102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4629.459788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4630.754283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4632.048900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4633.344513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4634.637101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4635.941524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4637.235881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4638.532641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4639.845833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4641.138912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4642.432304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4643.729796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4645.025804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4646.323770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4647.619564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4648.915749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4650.219585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4651.513109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4652.808064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4654.103469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4655.396773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4656.695848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4657.991198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4659.295112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4660.589992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4663.447986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4663.738322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4664.028642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4664.315647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4665.604648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4666.898470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4668.188831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4669.477838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4670.764947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4671.054423] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4671.345355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4671.632171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4672.436381] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4673.119373] Lustre: DEBUG MARKER: rc=0; [ 4673.119373] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4673.119373] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4673.119373] echo $(hostname -s): $val; [ 4673.119373] rc=$val; [ 4673.119373] fi; [ 4673.119373] exit $rc [ 4673.815159] Lustre: DEBUG MARKER: dmesg [ 4674.282100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 36: Move file during restore ====================================================== 23:54:35 \(1491436475\) [ 4674.471727] Lustre: DEBUG MARKER: == sanity-hsm test 36: Move file during restore ====================================================== 23:54:35 (1491436475) [ 4677.034618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4677.323289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4678.613167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4679.905424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4681.199021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4682.491548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4683.783496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4685.079805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4686.375979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4687.665793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4688.963310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4690.255656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4691.549920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4692.851241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4694.151636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4695.447466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4696.740980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4698.033335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4699.329590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4700.626295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4701.923384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4703.215090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4704.509619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4705.809378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4707.114401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4708.418046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4709.711439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4711.004517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4712.303843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4713.597966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4714.897459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4715.222310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4715.514726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4715.813738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4716.104430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4716.406558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4716.697394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4717.994325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4719.291598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4720.585160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4721.878689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4723.174404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4724.468953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4725.762074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4727.057549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4728.352433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4729.645050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4730.940049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4732.236211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4733.531041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4734.824865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4736.125290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4737.427806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4738.726635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4740.023226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4741.319137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4742.614706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4743.916659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4745.215378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4746.510674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4747.808461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4749.098664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4750.393833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4751.687809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4752.989301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 4755.861028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4756.150943] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4756.439519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4756.727377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4758.018118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4759.307313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4760.601264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4761.893390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4763.190197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4763.493897] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4763.783099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4764.069283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4764.900970] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4765.574154] Lustre: DEBUG MARKER: rc=0; [ 4765.574154] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4765.574154] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4765.574154] echo $(hostname -s): $val; [ 4765.574154] rc=$val; [ 4765.574154] fi; [ 4765.574154] exit $rc [ 4766.288114] Lustre: DEBUG MARKER: dmesg [ 4766.822921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 37: re-archive a dirty file ======================================================= 23:56:08 \(1491436568\) [ 4767.010049] Lustre: DEBUG MARKER: == sanity-hsm test 37: re-archive a dirty file ======================================================= 23:56:08 (1491436568) [ 4769.718831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4770.008893] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4770.298150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4770.584462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4771.871515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4773.162478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4774.451724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4774.743061] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4775.029722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4775.315705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4777.127179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4777.419220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4778.710527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4780.001299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4780.301327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 4792.861272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4793.152615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4794.449750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4797.273763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4797.568760] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4797.858439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4798.146200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4799.433276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4800.728651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4802.025088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4803.311516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4804.598910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4804.888111] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4805.175611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4805.462871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4806.265549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4806.948799] Lustre: DEBUG MARKER: rc=0; [ 4806.948799] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4806.948799] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4806.948799] echo $(hostname -s): $val; [ 4806.948799] rc=$val; [ 4806.948799] fi; [ 4806.948799] exit $rc [ 4807.646484] Lustre: DEBUG MARKER: dmesg [ 4808.120584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 40: Parallel archive requests ===================================================== 23:56:49 \(1491436609\) [ 4808.311071] Lustre: DEBUG MARKER: == sanity-hsm test 40: Parallel archive requests ===================================================== 23:56:49 (1491436609) [ 4813.803360] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 4814.096217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4814.383165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4814.670279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 4825.940511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 4826.275663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 4827.626620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 4830.452872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4830.742204] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4831.031703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4831.319766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4832.612546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4833.904724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4835.192884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4836.479230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4837.767399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4838.055574] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4838.346263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4838.634804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4839.532584] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4840.190464] Lustre: DEBUG MARKER: rc=0; [ 4840.190464] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4840.190464] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4840.190464] echo $(hostname -s): $val; [ 4840.190464] rc=$val; [ 4840.190464] fi; [ 4840.190464] exit $rc [ 4840.870349] Lustre: DEBUG MARKER: dmesg [ 4841.335868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 23:57:22 \(1491436642\) [ 4841.520080] Lustre: DEBUG MARKER: == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 23:57:22 (1491436642) [ 4842.434768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000406:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4842.729080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000406:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4843.034866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client e0779b89-1f8d-2da1-023d-bf01db52859a [ 4843.179734] Lustre: 9580:0:(genops.c:1527:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e0779b89-1f8d-2da1-023d-bf01db52859a at adminstrative request [ 4844.315626] Lustre: lustre-MDT0000: Connection restored to 6a04c9a8-6892-d84a-f4e7-7a2f1fa7b691 (at 10.9.5.237@tcp) [ 4844.317903] Lustre: Skipped 2 previous similar messages [ 4848.134394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4848.425720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4848.716562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4849.002047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4850.290271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4851.577153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4852.864584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4853.155921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4853.446559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4853.733435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4854.542631] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4855.212164] Lustre: DEBUG MARKER: rc=0; [ 4855.212164] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4855.212164] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4855.212164] echo $(hostname -s): $val; [ 4855.212164] rc=$val; [ 4855.212164] fi; [ 4855.212164] exit $rc [ 4855.914802] Lustre: DEBUG MARKER: dmesg [ 4856.428082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 23:57:37 \(1491436657\) [ 4856.615244] Lustre: DEBUG MARKER: == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 23:57:37 (1491436657) [ 4857.546717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000408:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4857.839355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000408:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4858.144609] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client e0779b89-1f8d-2da1-023d-bf01db52859a [ 4858.292027] Lustre: 10832:0:(genops.c:1527:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e0779b89-1f8d-2da1-023d-bf01db52859a at adminstrative request [ 4863.247841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4863.545579] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4863.836788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4864.125382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4865.413892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4866.708459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4867.996871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4868.285779] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4868.574700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4868.861797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4869.673810] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4870.340431] Lustre: DEBUG MARKER: rc=0; [ 4870.340431] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4870.340431] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4870.340431] echo $(hostname -s): $val; [ 4870.340431] rc=$val; [ 4870.340431] fi; [ 4870.340431] exit $rc [ 4871.024109] Lustre: DEBUG MARKER: dmesg [ 4871.485437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 54: Write during an archive cancels it ============================================ 23:57:52 \(1491436672\) [ 4871.679911] Lustre: DEBUG MARKER: == sanity-hsm test 54: Write during an archive cancels it ============================================ 23:57:52 (1491436672) [ 4874.929138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4875.222171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4875.521643] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 4875.836843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4876.136723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4877.430019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4878.728894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4880.027110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4881.319739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4882.617858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4883.913354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4885.212260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4886.505179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4887.797018] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4889.090395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4890.382400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4891.686155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4892.977845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4894.278258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4895.571709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4896.864303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4898.160232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4899.460360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4900.759649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4902.056978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4903.354174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4904.663248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4905.956921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4907.251384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4908.548125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4909.844111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4911.135650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4912.431169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4912.736215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 4915.563491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4915.854638] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4916.148298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4916.437628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4917.730886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4919.026710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4920.316365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4921.604571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4922.890203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4923.180670] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4923.475641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4923.762805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4924.587824] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4925.271436] Lustre: DEBUG MARKER: rc=0; [ 4925.271436] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4925.271436] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4925.271436] echo $(hostname -s): $val; [ 4925.271436] rc=$val; [ 4925.271436] fi; [ 4925.271436] exit $rc [ 4925.963396] Lustre: DEBUG MARKER: dmesg [ 4926.420472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 23:58:47 \(1491436727\) [ 4926.603328] Lustre: DEBUG MARKER: == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 23:58:47 (1491436727) [ 4929.715114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4930.007821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4930.309537] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 4930.620862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4930.914027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4931.212453] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 4934.034114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4934.322795] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4934.614059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4934.902781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4936.190245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4937.476552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4938.765450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4940.052089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4941.343559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4941.642674] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4941.948370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4942.237709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4943.063124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4943.738078] Lustre: DEBUG MARKER: rc=0; [ 4943.738078] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4943.738078] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4943.738078] echo $(hostname -s): $val; [ 4943.738078] rc=$val; [ 4943.738078] fi; [ 4943.738078] exit $rc [ 4944.421697] Lustre: DEBUG MARKER: dmesg [ 4944.890058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 56: Setattr during an archive is ok =============================================== 23:59:06 \(1491436746\) [ 4945.072945] Lustre: DEBUG MARKER: == sanity-hsm test 56: Setattr during an archive is ok =============================================== 23:59:06 (1491436746) [ 4949.071066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4949.362976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4949.719132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4950.011946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4951.307769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4952.602790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4953.902665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4955.203874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4956.498488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4957.791212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4959.088896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4960.390082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4961.682299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4962.980206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4964.273146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4965.568694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4966.863075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4968.162835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4969.455898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4970.758442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4972.061484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4973.362602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4974.657948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4975.953201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4977.248582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4978.550413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4979.847248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4981.143866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4982.438447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4983.739557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4985.039499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4986.339777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 4989.176841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4989.466460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 4989.758316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4990.047659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4991.342107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4992.631024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4993.920665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4995.207901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4996.500635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4996.791742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 4997.082845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4997.372726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 4998.206877] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4998.876658] Lustre: DEBUG MARKER: rc=0; [ 4998.876658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4998.876658] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4998.876658] echo $(hostname -s): $val; [ 4998.876658] rc=$val; [ 4998.876658] fi; [ 4998.876658] exit $rc [ 4999.562236] Lustre: DEBUG MARKER: dmesg [ 5000.029630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 00:00:01 \(1491436801\) [ 5000.213194] Lustre: DEBUG MARKER: == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 00:00:01 (1491436801) [ 5002.088445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5002.383757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5003.686614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5004.983246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5008.125407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5008.428026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5011.254165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5011.542648] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5011.834073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5012.122757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5013.413054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5014.704285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5015.999380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5017.287160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5018.573194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5018.859265] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5019.147754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5019.433144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5020.249932] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5020.929663] Lustre: DEBUG MARKER: rc=0; [ 5020.929663] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5020.929663] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5020.929663] echo $(hostname -s): $val; [ 5020.929663] rc=$val; [ 5020.929663] fi; [ 5020.929663] exit $rc [ 5021.620569] Lustre: DEBUG MARKER: dmesg [ 5022.103329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 00:00:23 \(1491436823\) [ 5022.291183] Lustre: DEBUG MARKER: == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 00:00:23 (1491436823) [ 5023.248812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5023.540396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5024.093719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5024.436380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5024.730233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5025.313959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1cd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5025.657761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5025.947978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5026.519319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1ce:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5029.379463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5029.669664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5029.959623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5030.244775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5031.532406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5032.821591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5034.114789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5035.401424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5036.689868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5036.977936] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5037.266457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5037.552420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5038.382945] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5039.051512] Lustre: DEBUG MARKER: rc=0; [ 5039.051512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5039.051512] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5039.051512] echo $(hostname -s): $val; [ 5039.051512] rc=$val; [ 5039.051512] fi; [ 5039.051512] exit $rc [ 5039.733633] Lustre: DEBUG MARKER: dmesg [ 5040.186183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 00:00:41 \(1491436841\) [ 5040.376807] Lustre: DEBUG MARKER: == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 00:00:41 (1491436841) [ 5040.553090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 5041.615455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5041.907438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5044.736159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5045.023994] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5045.311925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5045.600081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5046.891860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5048.178071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5049.466516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5050.756684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5052.052984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5052.344246] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5052.633932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5052.921387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5053.753798] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5054.439730] Lustre: DEBUG MARKER: rc=0; [ 5054.439730] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5054.439730] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5054.439730] echo $(hostname -s): $val; [ 5054.439730] rc=$val; [ 5054.439730] fi; [ 5054.439730] exit $rc [ 5055.127202] Lustre: DEBUG MARKER: dmesg [ 5055.589757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 60: Changing progress update interval from default ================================ 00:00:56 \(1491436856\) [ 5055.780751] Lustre: DEBUG MARKER: == sanity-hsm test 60: Changing progress update interval from default ================================ 00:00:56 (1491436856) [ 5058.482372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5058.770448] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5059.057205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5059.344528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5060.638483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5061.931208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5063.220044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5063.506861] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5063.793301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5064.073678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5066.361664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [ 5066.646137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [ 5067.424712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000402:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [ 5068.712168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000402:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [ 5070.000589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000402:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [ 5071.294161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000402:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [ 5072.588909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000402:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [ 5078.338699] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5081.169986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5081.459706] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5081.744180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5082.027256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5083.309414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5084.596308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5085.878998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5087.162485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5087.444776] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5087.728559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5088.008347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5088.865708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5089.541927] Lustre: DEBUG MARKER: rc=0; [ 5089.541927] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5089.541927] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5089.541927] echo $(hostname -s): $val; [ 5089.541927] rc=$val; [ 5089.541927] fi; [ 5089.541927] exit $rc [ 5090.211361] Lustre: DEBUG MARKER: dmesg [ 5090.663162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 70: Copytool logs JSON register\/unregister events to FIFO ========================= 00:01:31 \(1491436891\) [ 5090.851340] Lustre: DEBUG MARKER: == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================= 00:01:31 (1491436891) [ 5093.566002] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5093.848254] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5094.131822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5094.412871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5095.698637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5096.982409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5098.264811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5098.545783] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5098.827930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5099.108560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5102.105958] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5102.390008] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [ 5102.674176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [ 5105.505038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5105.787700] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5106.078199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5106.358814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5107.645874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5108.939387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5109.223343] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5109.506526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5109.788559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5111.449894] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5112.112417] Lustre: DEBUG MARKER: rc=0; [ 5112.112417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5112.112417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5112.112417] echo $(hostname -s): $val; [ 5112.112417] rc=$val; [ 5112.112417] fi; [ 5112.112417] exit $rc [ 5112.803425] Lustre: DEBUG MARKER: dmesg [ 5113.278444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 00:01:54 \(1491436914\) [ 5113.461113] Lustre: DEBUG MARKER: == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 00:01:54 (1491436914) [ 5116.169840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5116.452844] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5116.734849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5117.014768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5118.296893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5119.583833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5119.866414] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5120.148920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5120.427119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5126.499939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5126.784598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5128.073169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5129.361112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5130.654658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5131.939406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5133.226854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5134.516920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5135.809523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5137.105601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5138.391646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5139.684799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5140.979657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5142.268882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5143.555017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5144.846911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5146.133757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5147.420505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5148.708904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5150.004598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5151.291581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5152.578868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5153.865529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5155.154282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5156.448311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5157.742965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5159.032877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5160.323792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5161.611104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5162.904383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5164.195311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5165.496045] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5168.324373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5168.607171] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5168.890860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5169.173952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5170.459697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5171.749775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5173.033972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5174.348587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5174.631005] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5174.915848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5175.195340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5176.612962] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5177.279621] Lustre: DEBUG MARKER: rc=0; [ 5177.279621] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5177.279621] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5177.279621] echo $(hostname -s): $val; [ 5177.279621] rc=$val; [ 5177.279621] fi; [ 5177.279621] exit $rc [ 5177.958280] Lustre: DEBUG MARKER: dmesg [ 5178.428728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 00:02:59 \(1491436979\) [ 5178.620155] Lustre: DEBUG MARKER: == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 00:02:59 (1491436979) [ 5181.334951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5181.616068] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5181.900985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5182.181122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5183.466416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5184.751446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5185.035551] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5185.317802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5185.597304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5191.208518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5191.492891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5192.783691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5194.082455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5195.373443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5196.667353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5197.956419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5199.245771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5200.534915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5201.826764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5203.115165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5204.403178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5205.695058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5206.988453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5207.962966] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5210.781815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5211.066608] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5211.348820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5211.633081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5212.914016] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5214.195348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5215.478375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5216.761423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5217.043934] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5217.325973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5217.607615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5218.927434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5219.596382] Lustre: DEBUG MARKER: rc=0; [ 5219.596382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5219.596382] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5219.596382] echo $(hostname -s): $val; [ 5219.596382] rc=$val; [ 5219.596382] fi; [ 5219.596382] exit $rc [ 5220.283306] Lustre: DEBUG MARKER: dmesg [ 5220.751539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 90: Archive\/restore a file list =================================================== 00:03:42 \(1491437022\) [ 5220.939711] Lustre: DEBUG MARKER: == sanity-hsm test 90: Archive/restore a file list =================================================== 00:03:42 (1491437022) [ 5222.670770] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 5222.953856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5223.235038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5223.519602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 5234.001370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 5234.306467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 5235.092516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 5235.415701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [ 5238.252305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5238.543946] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5238.832164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5239.116278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5240.402727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5241.694218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5242.980227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5244.266389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5245.558829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5245.845124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5246.131322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5246.413713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5247.236746] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5247.920192] Lustre: DEBUG MARKER: rc=0; [ 5247.920192] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5247.920192] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5247.920192] echo $(hostname -s): $val; [ 5247.920192] rc=$val; [ 5247.920192] fi; [ 5247.920192] exit $rc [ 5248.597810] Lustre: DEBUG MARKER: dmesg [ 5249.073251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 100: Set coordinator \/proc tunables =============================================== 00:04:10 \(1491437050\) [ 5249.266392] Lustre: DEBUG MARKER: == sanity-hsm test 100: Set coordinator /proc tunables =============================================== 00:04:10 (1491437050) [ 5249.442704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [ 5249.732573] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=20 [ 5250.020127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [ 5250.307238] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=0 [ 5250.594547] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [ 5250.880095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 5251.165100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=20 [ 5251.450507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [ 5251.736625] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=0 [ 5252.021590] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [ 5252.306682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [ 5252.593297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=7200 [ 5252.880241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [ 5253.164215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=0 [ 5253.451339] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [ 5253.743589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [ 5254.029671] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=6 [ 5254.315392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [ 5254.601866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=0 [ 5254.885796] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=3 [ 5255.172428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 5255.458631] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [ 5255.745646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 5256.033711] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=0 [ 5256.323894] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=1 [ 5256.646488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5257.322298] Lustre: DEBUG MARKER: rc=0; [ 5257.322298] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5257.322298] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5257.322298] echo $(hostname -s): $val; [ 5257.322298] rc=$val; [ 5257.322298] fi; [ 5257.322298] exit $rc [ 5258.024069] Lustre: DEBUG MARKER: dmesg [ 5258.481268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 102: Verify coordinator control =================================================== 00:04:19 \(1491437059\) [ 5258.677472] Lustre: DEBUG MARKER: == sanity-hsm test 102: Verify coordinator control =================================================== 00:04:19 (1491437059) [ 5258.845067] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5259.133434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5259.415211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5259.701674] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5259.988272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5260.272161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5260.561816] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5260.847241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5261.131818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5262.416613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5263.701891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5264.987699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5266.271758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5267.556880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5268.840963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5270.131318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5270.423808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5270.709494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5270.992218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5271.324404] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5271.993829] Lustre: DEBUG MARKER: rc=0; [ 5271.993829] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5271.993829] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5271.993829] echo $(hostname -s): $val; [ 5271.993829] rc=$val; [ 5271.993829] fi; [ 5271.993829] exit $rc [ 5272.682884] Lustre: DEBUG MARKER: dmesg [ 5273.136899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 103: Purge all requests =========================================================== 00:04:34 \(1491437074\) [ 5273.323105] Lustre: DEBUG MARKER: == sanity-hsm test 103: Purge all requests =========================================================== 00:04:34 (1491437074) [ 5274.534097] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 5274.821897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5275.106813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5275.395943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep -v CANCELED | grep -v SUCCEED | grep -v FAILED [ 5278.232408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5278.519676] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5278.806610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5279.090437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5280.376942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5281.665333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5282.950576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5284.236986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5285.521381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5285.805915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5286.092511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5286.376179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5287.201712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5287.870898] Lustre: DEBUG MARKER: rc=0; [ 5287.870898] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5287.870898] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5287.870898] echo $(hostname -s): $val; [ 5287.870898] rc=$val; [ 5287.870898] fi; [ 5287.870898] exit $rc [ 5288.559655] Lustre: DEBUG MARKER: dmesg [ 5289.035371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 104: Copy tool data field ========================================================= 00:04:50 \(1491437090\) [ 5289.226294] Lustre: DEBUG MARKER: == sanity-hsm test 104: Copy tool data field ========================================================= 00:04:50 (1491437090) [ 5293.653338] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5293.940192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5294.223629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5294.515989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep 0x200000402:0x218:0x0 | cut -f16 -d= [ 5294.805048] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5295.097497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5295.381792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5298.204662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5298.493912] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5298.783414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5299.066988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5300.351815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5301.637940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5302.922454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5304.208572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5305.494492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5305.781201] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5306.066714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5306.349817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5307.176381] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5307.850477] Lustre: DEBUG MARKER: rc=0; [ 5307.850477] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5307.850477] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5307.850477] echo $(hostname -s): $val; [ 5307.850477] rc=$val; [ 5307.850477] fi; [ 5307.850477] exit $rc [ 5308.531785] Lustre: DEBUG MARKER: dmesg [ 5308.994220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 105: Restart of coordinator ======================================================= 00:05:10 \(1491437110\) [ 5309.186286] Lustre: DEBUG MARKER: == sanity-hsm test 105: Restart of coordinator ======================================================= 00:05:10 (1491437110) [ 5309.359575] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5309.655315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5309.956543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5310.333451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 5310.619919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5310.910463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5311.194777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5312.480818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5313.771114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5315.068586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5316.354704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5317.639689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5318.927483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5320.216637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5320.503249] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5320.790763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5321.073941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5321.359473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5321.644591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5321.926350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5322.211075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 5322.497752] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5322.784399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5323.070554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5323.355522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [ 5323.645560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5323.927320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5324.251187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5324.931746] Lustre: DEBUG MARKER: rc=0; [ 5324.931746] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5324.931746] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5324.931746] echo $(hostname -s): $val; [ 5324.931746] rc=$val; [ 5324.931746] fi; [ 5324.931746] exit $rc [ 5325.619064] Lustre: DEBUG MARKER: dmesg [ 5326.105048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 106: Copytool register\/unregister ================================================= 00:05:27 \(1491437127\) [ 5326.291713] Lustre: DEBUG MARKER: == sanity-hsm test 106: Copytool register/unregister ================================================= 00:05:27 (1491437127) [ 5328.073159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 43d2d38d-8855-c518-88d6-7518291a0b16 [ 5331.164658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5331.450908] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5331.736712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5332.020776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5333.305233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5333.591664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5333.877859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5334.163845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5334.945444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 43d2d38d-8855-c518-88d6-7518291a0b16 [ 5336.902457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 43d2d38d-8855-c518-88d6-7518291a0b16 [ 5339.735747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5340.020986] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5340.308503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5340.591404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5341.882291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5343.174810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5344.461392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5344.748056] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5345.035367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5345.317991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5346.150716] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5346.818447] Lustre: DEBUG MARKER: rc=0; [ 5346.818447] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5346.818447] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5346.818447] echo $(hostname -s): $val; [ 5346.818447] rc=$val; [ 5346.818447] fi; [ 5346.818447] exit $rc [ 5347.497089] Lustre: DEBUG MARKER: dmesg [ 5347.991423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 00:05:49 \(1491437149\) [ 5348.185557] Lustre: DEBUG MARKER: == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 00:05:49 (1491437149) [ 5349.112098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5349.400520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5349.699142] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5349.985075] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5350.130909] Lustre: Failing over lustre-MDT0000 [ 5353.416647] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.420862] LustreError: Skipped 1 previous similar message [ 5356.268040] Lustre: 17118:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491437151/real 1491437151] req@ffff8800530a5b00 x1563879826232512/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491437157 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5356.289792] Lustre: server umount lustre-MDT0000 complete [ 5356.291749] Lustre: Skipped 2 previous similar messages [ 5356.432948] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5356.723292] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 5356.723292] grep -q ^lustre-mdt1/ /proc/mounts || [ 5356.723292] zpool export lustre-mdt1 [ 5367.163845] Lustre: DEBUG MARKER: hostname [ 5367.493678] Lustre: DEBUG MARKER: modprobe zfs; [ 5367.493678] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 5367.493678] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 5367.978927] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 5368.257185] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 5368.439242] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 5368.441436] Lustre: Skipped 1 previous similar message [ 5368.544211] LustreError: 18586:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [ 5368.580699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.582918] Lustre: Skipped 1 previous similar message [ 5368.741285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5369.025834] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 5369.508899] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5369.797691] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5370.081156] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5370.372646] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 5375.117151] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 5375.490289] Lustre: lustre-MDT0000: Recovery over after 0:01, of 4 clients 4 recovered and 0 were evicted. [ 5376.212788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5376.281022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5376.418729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5376.508955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5377.840375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5378.124406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5380.953332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5381.234753] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5381.518083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5381.797899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5383.080695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5384.367230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5385.659375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5386.943896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5388.222899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5388.505384] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5388.789478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5389.068516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5389.909220] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5390.612482] Lustre: DEBUG MARKER: rc=0; [ 5390.612482] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5390.612482] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5390.612482] echo $(hostname -s): $val; [ 5390.612482] rc=$val; [ 5390.612482] fi; [ 5390.612482] exit $rc [ 5391.298971] Lustre: DEBUG MARKER: dmesg [ 5391.764854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 109: Policy display\/change ======================================================== 00:06:33 \(1491437193\) [ 5391.955559] Lustre: DEBUG MARKER: == sanity-hsm test 109: Policy display/change ======================================================== 00:06:33 (1491437193) [ 5392.129854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5392.411279] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NBR" [ 5392.696690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5392.978294] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NRA" [ 5393.260302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5393.541344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NBR" [ 5393.824392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5394.107559] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NRA" [ 5394.390731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5394.674102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="NRA NBR" [ 5394.957359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5395.239979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5395.523742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="wrong" [ 5395.665165] Lustre: 21415:0:(mdt_coordinator.c:1789:mdt_hsm_policy_seq_write()) lustre-MDT0000: 'wrong' is unknown, supported policies are: [ 5395.809646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [ 5396.092401] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 5396.374221] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [ 5396.697340] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5397.378551] Lustre: DEBUG MARKER: rc=0; [ 5397.378551] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5397.378551] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5397.378551] echo $(hostname -s): $val; [ 5397.378551] rc=$val; [ 5397.378551] fi; [ 5397.378551] exit $rc [ 5398.071453] Lustre: DEBUG MARKER: dmesg [ 5398.563240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110a: Non blocking restore policy \(import case\) =================================== 00:06:39 \(1491437199\) [ 5398.763537] Lustre: DEBUG MARKER: == sanity-hsm test 110a: Non blocking restore policy (import case) =================================== 00:06:39 (1491437199) [ 5400.403274] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [ 5400.702097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5400.986858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5401.271430] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [ 5404.105395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5404.388952] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5404.679047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5404.958836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5406.242567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5407.536739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5408.818977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5410.101618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5411.384173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5411.666669] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5411.951151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5412.232798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5413.059916] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5413.730708] Lustre: DEBUG MARKER: rc=0; [ 5413.730708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5413.730708] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5413.730708] echo $(hostname -s): $val; [ 5413.730708] rc=$val; [ 5413.730708] fi; [ 5413.730708] exit $rc [ 5414.424799] Lustre: DEBUG MARKER: dmesg [ 5414.892159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110b: Non blocking restore policy \(release case\) ================================== 00:06:56 \(1491437216\) [ 5415.089451] Lustre: DEBUG MARKER: == sanity-hsm test 110b: Non blocking restore policy (release case) ================================== 00:06:56 (1491437216) [ 5416.037433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5416.321219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5416.618085] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [ 5416.911286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5417.195867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5417.483017] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [ 5420.304785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5420.594590] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5420.879454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5421.160889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5422.446170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5423.728890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5425.017316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5426.301351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5427.583416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5427.864723] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5428.147498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5428.424619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5429.251140] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5429.934811] Lustre: DEBUG MARKER: rc=0; [ 5429.934811] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5429.934811] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5429.934811] echo $(hostname -s): $val; [ 5429.934811] rc=$val; [ 5429.934811] fi; [ 5429.934811] exit $rc [ 5430.619235] Lustre: DEBUG MARKER: dmesg [ 5431.104758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111a: No retry policy \(import case\), restore will error =========================== 00:07:12 \(1491437232\) [ 5431.294441] Lustre: DEBUG MARKER: == sanity-hsm test 111a: No retry policy (import case), restore will error =========================== 00:07:12 (1491437232) [ 5432.950334] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 5433.726863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5434.013560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5434.302282] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5437.136904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5437.425679] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5437.720585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5438.013790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5439.299253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5440.587611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5441.870818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5443.162785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5444.444701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5444.725531] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5445.009234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5445.290748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5446.118176] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5446.825834] Lustre: DEBUG MARKER: rc=0; [ 5446.825834] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5446.825834] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5446.825834] echo $(hostname -s): $val; [ 5446.825834] rc=$val; [ 5446.825834] fi; [ 5446.825834] exit $rc [ 5447.505335] Lustre: DEBUG MARKER: dmesg [ 5448.035755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111b: No retry policy \(release case\), restore will error ========================== 00:07:29 \(1491437249\) [ 5448.222887] Lustre: DEBUG MARKER: == sanity-hsm test 111b: No retry policy (release case), restore will error ========================== 00:07:29 (1491437249) [ 5449.176474] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [ 5449.470953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5449.760655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5450.540371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5450.828436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5451.116154] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [ 5453.940828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5454.222906] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5454.508584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5454.790395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5456.074192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5457.359189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5458.642569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5459.925049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5461.207560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5461.498068] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5461.792305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5462.081973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5462.913171] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5463.574884] Lustre: DEBUG MARKER: rc=0; [ 5463.574884] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5463.574884] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5463.574884] echo $(hostname -s): $val; [ 5463.574884] rc=$val; [ 5463.574884] fi; [ 5463.574884] exit $rc [ 5464.252029] Lustre: DEBUG MARKER: dmesg [ 5464.732157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 112: State of recorded request ==================================================== 00:07:46 \(1491437266\) [ 5464.949045] Lustre: DEBUG MARKER: == sanity-hsm test 112: State of recorded request ==================================================== 00:07:46 (1491437266) [ 5465.894222] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5466.179902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5466.460380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5466.758334] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5467.047371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5467.329802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5467.612308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5467.894918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5470.724759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5471.006412] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5471.290479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5471.571574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5472.852672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5474.134752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5475.416352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5476.698133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5477.980789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5478.264557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5478.551563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5478.834092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5479.655604] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5480.325787] Lustre: DEBUG MARKER: rc=0; [ 5480.325787] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5480.325787] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5480.325787] echo $(hostname -s): $val; [ 5480.325787] rc=$val; [ 5480.325787] fi; [ 5480.325787] exit $rc [ 5481.019859] Lustre: DEBUG MARKER: dmesg [ 5481.487861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 200: Register\/Cancel archive ====================================================== 00:08:02 \(1491437282\) [ 5481.665433] Lustre: DEBUG MARKER: == sanity-hsm test 200: Register/Cancel archive ====================================================== 00:08:02 (1491437282) [ 5494.212900] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5494.496328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5494.775194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5495.076611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5495.360694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5495.652820] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5495.944405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5496.226649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5496.521649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5496.817672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5497.105995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5497.388371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5500.205114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5500.487278] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5500.770069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5501.050646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5502.338310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5503.624536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5504.907464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5506.196702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5506.478020] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5506.760701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5507.038636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5507.856982] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5508.530702] Lustre: DEBUG MARKER: rc=0; [ 5508.530702] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5508.530702] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5508.530702] echo $(hostname -s): $val; [ 5508.530702] rc=$val; [ 5508.530702] fi; [ 5508.530702] exit $rc [ 5509.217039] Lustre: DEBUG MARKER: dmesg [ 5509.707222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 201: Register\/Cancel restore ====================================================== 00:08:31 \(1491437311\) [ 5509.899149] Lustre: DEBUG MARKER: == sanity-hsm test 201: Register/Cancel restore ====================================================== 00:08:31 (1491437311) [ 5513.840688] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5514.126363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5514.406905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5514.697516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5514.980730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5515.276226] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5515.561575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5515.845139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5516.130517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5516.418225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5516.712741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5517.000958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5519.822133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5520.105407] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5520.388499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5520.669478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5521.952260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5523.233270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5524.514494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5525.804357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5526.087755] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5526.371104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5526.651249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5527.489979] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5528.182473] Lustre: DEBUG MARKER: rc=0; [ 5528.182473] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5528.182473] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5528.182473] echo $(hostname -s): $val; [ 5528.182473] rc=$val; [ 5528.182473] fi; [ 5528.182473] exit $rc [ 5528.859026] Lustre: DEBUG MARKER: dmesg [ 5529.329580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 202: Register\/Cancel remove ======================================================= 00:08:50 \(1491437330\) [ 5529.517919] Lustre: DEBUG MARKER: == sanity-hsm test 202: Register/Cancel remove ======================================================= 00:08:50 (1491437330) [ 5533.933027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5534.217361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5535.503105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5536.789604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5538.077476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5539.364562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5540.656124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5541.950913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5543.239390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5544.526630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5545.815679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5547.101893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5548.390941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5549.679662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5550.975736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5552.264500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5553.551721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5554.841319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5556.134617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5557.436090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5558.725981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5560.013534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5561.303410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5562.597214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5563.891735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5565.187693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5566.481891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5567.771731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5569.064897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5570.365558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5571.656533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5571.945416] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5572.231352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5572.512816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5572.800203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5573.086628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5573.379799] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5573.668315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5573.949968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5574.233737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5574.517828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5577.346409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5577.629763] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5577.914702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5578.196367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5579.485940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5580.779835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5582.064106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5583.345259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5584.631511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5584.913658] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5585.196877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5585.474742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5586.300445] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5586.979799] Lustre: DEBUG MARKER: rc=0; [ 5586.979799] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5586.979799] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5586.979799] echo $(hostname -s): $val; [ 5586.979799] rc=$val; [ 5586.979799] fi; [ 5586.979799] exit $rc [ 5587.668801] Lustre: DEBUG MARKER: dmesg [ 5588.153242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220: Changelog for archive ======================================================== 00:09:49 \(1491437389\) [ 5588.345320] Lustre: DEBUG MARKER: == sanity-hsm test 220: Changelog for archive ======================================================== 00:09:49 (1491437389) [ 5589.285295] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5589.436379] Lustre: lustre-MDD0000: changelog on [ 5589.580340] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5589.874816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5590.157795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5590.453513] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl1 [ 5590.598883] Lustre: lustre-MDD0000: changelog off [ 5593.289044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5593.573489] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5593.855938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5594.138703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5595.426545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5596.708126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5597.989143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5599.273390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5600.560642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5600.840490] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5601.123609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5601.402309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5602.229536] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5602.896891] Lustre: DEBUG MARKER: rc=0; [ 5602.896891] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5602.896891] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5602.896891] echo $(hostname -s): $val; [ 5602.896891] rc=$val; [ 5602.896891] fi; [ 5602.896891] exit $rc [ 5603.577725] Lustre: DEBUG MARKER: dmesg [ 5604.058507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220a: Changelog for failed archive ================================================ 00:10:05 \(1491437405\) [ 5604.255081] Lustre: DEBUG MARKER: == sanity-hsm test 220a: Changelog for failed archive ================================================ 00:10:05 (1491437405) [ 5605.188128] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5605.331165] Lustre: lustre-MDD0000: changelog on [ 5605.473690] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5606.174686] LustreError: 19125:0:(mdt_coordinator.c:1360:mdt_hsm_update_request_state()) lustre-MDT0000: Cannot find running request for cookie 0x58e5875a on fid=[0x200000402:0x221:0x0] [ 5606.319636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5606.606169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5606.901929] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl2 [ 5607.044437] Lustre: lustre-MDD0000: changelog off [ 5609.754710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5610.037250] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5610.320216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5610.600651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5611.886654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5613.172108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5614.457317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5615.739162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5617.026871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5617.307701] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5617.592815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5617.874290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5618.696673] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5619.376158] Lustre: DEBUG MARKER: rc=0; [ 5619.376158] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5619.376158] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5619.376158] echo $(hostname -s): $val; [ 5619.376158] rc=$val; [ 5619.376158] fi; [ 5619.376158] exit $rc [ 5620.063724] Lustre: DEBUG MARKER: dmesg [ 5620.593920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 221: Changelog for archive canceled =============================================== 00:10:21 \(1491437421\) [ 5620.786505] Lustre: DEBUG MARKER: == sanity-hsm test 221: Changelog for archive canceled =============================================== 00:10:21 (1491437421) [ 5631.982808] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5632.129710] Lustre: lustre-MDD0000: changelog on [ 5632.277375] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5632.575734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5632.861987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5633.157685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5633.442794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5633.737391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5634.021215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5635.308163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5636.594870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5637.884200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5639.175989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5640.462874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5641.754269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5643.046352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5644.336415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5645.623784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5646.917455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5648.214046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5649.503424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5650.792777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5652.097178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5653.396334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5654.683756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5655.977102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5657.269933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5658.561662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5659.856952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5661.146107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5662.440713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5663.744750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5666.590531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5666.873978] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5667.159111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5667.441404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5668.725715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5670.014182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5671.298269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5672.585400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5673.868635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5674.152302] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5674.437590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5674.718627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5675.498775] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl3 [ 5675.642851] Lustre: lustre-MDD0000: changelog off [ 5675.836079] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5676.500257] Lustre: DEBUG MARKER: rc=0; [ 5676.500257] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5676.500257] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5676.500257] echo $(hostname -s): $val; [ 5676.500257] rc=$val; [ 5676.500257] fi; [ 5676.500257] exit $rc [ 5677.184054] Lustre: DEBUG MARKER: dmesg [ 5677.750567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222a: Changelog for explicit restore ============================================== 00:11:19 \(1491437479\) [ 5677.932953] Lustre: DEBUG MARKER: == sanity-hsm test 222a: Changelog for explicit restore ============================================== 00:11:19 (1491437479) [ 5679.586085] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5679.728344] Lustre: lustre-MDD0000: changelog on [ 5679.871416] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5680.167077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5680.453871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5683.305221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5683.587129] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5683.881638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5684.171119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5685.454482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5686.741341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5688.030829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5689.313493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5690.595801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5690.877639] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5691.165569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5691.445208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5692.222173] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl4 [ 5692.364820] Lustre: lustre-MDD0000: changelog off [ 5692.557390] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5693.227317] Lustre: DEBUG MARKER: rc=0; [ 5693.227317] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5693.227317] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5693.227317] echo $(hostname -s): $val; [ 5693.227317] rc=$val; [ 5693.227317] fi; [ 5693.227317] exit $rc [ 5693.902900] Lustre: DEBUG MARKER: dmesg [ 5694.374298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222b: Changelog for implicit restore ============================================== 00:11:35 \(1491437495\) [ 5694.564973] Lustre: DEBUG MARKER: == sanity-hsm test 222b: Changelog for implicit restore ============================================== 00:11:35 (1491437495) [ 5695.507755] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5695.649539] Lustre: lustre-MDD0000: changelog on [ 5695.791262] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5696.089954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5696.376687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5696.854365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5697.147219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5699.982977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5700.266669] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5700.550660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5700.831917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5702.118992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5703.404675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5704.687994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5705.971093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5707.258779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5707.544605] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5707.837277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5708.125191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5708.902811] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl5 [ 5709.045794] Lustre: lustre-MDD0000: changelog off [ 5709.227036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5709.912517] Lustre: DEBUG MARKER: rc=0; [ 5709.912517] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5709.912517] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5709.912517] echo $(hostname -s): $val; [ 5709.912517] rc=$val; [ 5709.912517] fi; [ 5709.912517] exit $rc [ 5710.594833] Lustre: DEBUG MARKER: dmesg [ 5711.073636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 00:11:52 \(1491437512\) [ 5711.266506] Lustre: DEBUG MARKER: == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 00:11:52 (1491437512) [ 5712.932947] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5713.076702] Lustre: lustre-MDD0000: changelog on [ 5713.219463] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5714.036973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5714.322264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5717.147297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5717.428764] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5717.710372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5717.989850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5719.276429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5720.558136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5721.846551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5723.132302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5724.416923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5724.704309] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5724.989756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5725.270964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5726.053095] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl6 [ 5726.197402] Lustre: lustre-MDD0000: changelog off [ 5726.388642] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5727.066982] Lustre: DEBUG MARKER: rc=0; [ 5727.066982] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5727.066982] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5727.066982] echo $(hostname -s): $val; [ 5727.066982] rc=$val; [ 5727.066982] fi; [ 5727.066982] exit $rc [ 5727.746334] Lustre: DEBUG MARKER: dmesg [ 5728.227932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 00:12:09 \(1491437529\) [ 5728.419046] Lustre: DEBUG MARKER: == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 00:12:09 (1491437529) [ 5729.354227] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5729.496080] Lustre: lustre-MDD0000: changelog on [ 5729.638720] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5729.933511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5730.217850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5731.011759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5731.309732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5734.196096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5734.476558] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5734.758163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5735.038132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5736.319885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5737.606401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5738.895436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5740.178251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5741.459653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5741.741393] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5742.025514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5742.307827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5743.086459] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl7 [ 5743.230238] Lustre: lustre-MDD0000: changelog off [ 5743.421697] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5744.089213] Lustre: DEBUG MARKER: rc=0; [ 5744.089213] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5744.089213] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5744.089213] echo $(hostname -s): $val; [ 5744.089213] rc=$val; [ 5744.089213] fi; [ 5744.089213] exit $rc [ 5744.774049] Lustre: DEBUG MARKER: dmesg [ 5745.259487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223a: Changelog for restore canceled \(import case\) ================================ 00:12:26 \(1491437546\) [ 5745.452824] Lustre: DEBUG MARKER: == sanity-hsm test 223a: Changelog for restore canceled (import case) ================================ 00:12:26 (1491437546) [ 5749.172836] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5749.455780] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5750.008894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5750.296703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5750.591552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5750.876470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5751.170764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5751.454928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5752.742369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5754.031719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5755.317937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5756.615316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5757.905525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5759.194960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5760.482594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5761.780184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5763.074925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5764.363076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5765.661931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5766.956680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5768.255452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5769.556321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5770.848992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5772.142323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5773.432850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5774.721637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5776.012902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5777.302292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5778.590913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5779.879153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5782.721468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5783.004968] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5783.289987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5783.573893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5784.859678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5786.151839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5787.436705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5788.720891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5790.003737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5790.287873] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5790.571984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5790.853962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5791.666648] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl8 [ 5791.813872] Lustre: lustre-MDD0000: changelog off [ 5792.015875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5792.712914] Lustre: DEBUG MARKER: rc=0; [ 5792.712914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5792.712914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5792.712914] echo $(hostname -s): $val; [ 5792.712914] rc=$val; [ 5792.712914] fi; [ 5792.712914] exit $rc [ 5793.406152] Lustre: DEBUG MARKER: dmesg [ 5793.902694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223b: Changelog for restore canceled \(release case\) =============================== 00:13:15 \(1491437595\) [ 5794.090397] Lustre: DEBUG MARKER: == sanity-hsm test 223b: Changelog for restore canceled (release case) =============================== 00:13:15 (1491437595) [ 5799.930116] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5800.073188] Lustre: lustre-MDD0000: changelog on [ 5800.075265] Lustre: Skipped 1 previous similar message [ 5800.217121] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5800.514460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5800.801074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5802.087067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5803.378211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5804.669595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5805.956947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5807.242933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5808.528524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5809.820537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5811.111938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5812.397912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5813.684861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5814.970908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5816.261100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5817.552053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5818.841847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5820.134321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5821.424898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5822.721293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5824.010583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5825.299910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5826.591259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5827.880389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5829.169062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5830.461586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5831.750596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5833.043883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5834.343351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5835.631631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5836.924278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5838.213439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5839.508441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5839.828171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5840.113403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5840.408888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5840.697940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [ 5840.989087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5841.279932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5842.568897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5843.862073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5845.152379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5846.441524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5847.730647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5849.025123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5850.316146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5851.609852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5852.906771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5854.201704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5855.498519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5856.796616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5858.091960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5859.381305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5860.670376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5861.960722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5863.254189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5864.548680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5865.839702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5867.135985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5868.430799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5869.719150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [ 5872.566727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5872.851373] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5873.135848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5873.413827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5874.693442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5875.976377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5877.264455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5878.546974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5879.829961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5880.113363] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5880.397342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5880.682784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5881.500861] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl9 [ 5881.649916] Lustre: lustre-MDD0000: changelog off [ 5881.849170] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5882.528948] Lustre: DEBUG MARKER: rc=0; [ 5882.528948] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5882.528948] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5882.528948] echo $(hostname -s): $val; [ 5882.528948] rc=$val; [ 5882.528948] fi; [ 5882.528948] exit $rc [ 5883.230252] Lustre: DEBUG MARKER: dmesg [ 5883.730323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224: Changelog for remove ========================================================= 00:14:45 \(1491437685\) [ 5883.933584] Lustre: DEBUG MARKER: == sanity-hsm test 224: Changelog for remove ========================================================= 00:14:45 (1491437685) [ 5884.867681] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5885.009982] Lustre: lustre-MDD0000: changelog on [ 5885.153208] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5885.451402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5885.737067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5886.027296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5886.313612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5889.136449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5889.418566] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5889.700420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5889.981058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5891.270052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5892.563213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5893.849858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5895.130780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5896.411930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5896.691326] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5896.974338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5897.256467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5898.046728] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl10 [ 5898.376290] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5899.060864] Lustre: DEBUG MARKER: rc=0; [ 5899.060864] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5899.060864] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5899.060864] echo $(hostname -s): $val; [ 5899.060864] rc=$val; [ 5899.060864] fi; [ 5899.060864] exit $rc [ 5899.761766] Lustre: DEBUG MARKER: dmesg [ 5900.231256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224a: Changelog for failed remove ================================================= 00:15:01 \(1491437701\) [ 5900.420725] Lustre: DEBUG MARKER: == sanity-hsm test 224a: Changelog for failed remove ================================================= 00:15:01 (1491437701) [ 5901.374366] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5901.663126] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5901.962996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5902.247778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5903.559937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5903.846200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [ 5906.682819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5906.965402] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5907.249692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5907.529843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5908.810353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5910.090489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5911.377023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5912.660870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5913.941079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5914.223678] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5914.505786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5914.784585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5915.564878] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl11 [ 5915.898126] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5916.587180] Lustre: DEBUG MARKER: rc=0; [ 5916.587180] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5916.587180] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5916.587180] echo $(hostname -s): $val; [ 5916.587180] rc=$val; [ 5916.587180] fi; [ 5916.587180] exit $rc [ 5917.284920] Lustre: DEBUG MARKER: dmesg [ 5917.736589] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 225: Changelog for remove canceled ================================================ 00:15:19 \(1491437719\) [ 5917.928331] Lustre: DEBUG MARKER: == sanity-hsm test 225: Changelog for remove canceled ================================================ 00:15:19 (1491437719) [ 5921.372390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5921.658264] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5921.943091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5922.224167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5923.510307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5924.793628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5925.074153] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5925.355867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5925.636532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5926.496998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5927.194887] Lustre: DEBUG MARKER: rc=0; [ 5927.194887] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5927.194887] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5927.194887] echo $(hostname -s): $val; [ 5927.194887] rc=$val; [ 5927.194887] fi; [ 5927.194887] exit $rc [ 5927.880273] Lustre: DEBUG MARKER: dmesg [ 5928.339443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 226: changelog for last rm\/mv with exiting archive ================================ 00:15:29 \(1491437729\) [ 5928.523792] Lustre: DEBUG MARKER: == sanity-hsm test 226: changelog for last rm/mv with exiting archive ================================ 00:15:29 (1491437729) [ 5929.482640] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5929.765514] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5930.061110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5930.345980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5930.645659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5930.933940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5933.805376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5934.087378] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5934.372208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5934.653965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5935.943689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5937.238551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5938.532192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5939.816304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5941.104951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5941.398099] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5941.680957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5941.961515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5942.746142] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl12 [ 5943.080331] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5943.749415] Lustre: DEBUG MARKER: rc=0; [ 5943.749415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5943.749415] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5943.749415] echo $(hostname -s): $val; [ 5943.749415] rc=$val; [ 5943.749415] fi; [ 5943.749415] exit $rc [ 5944.435889] Lustre: DEBUG MARKER: dmesg [ 5944.924134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 00:15:46 \(1491437746\) [ 5945.110032] Lustre: DEBUG MARKER: == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 00:15:46 (1491437746) [ 5946.019330] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [ 5946.306598] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [ 5946.728101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5947.012924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5949.860359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5950.143467] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5950.434747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5950.719706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5952.007677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5953.290925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5954.578425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5955.862995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5957.145407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5957.428229] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5957.712285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5957.994806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5958.772182] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl13 [ 5959.109749] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5959.781420] Lustre: DEBUG MARKER: rc=0; [ 5959.781420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5959.781420] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5959.781420] echo $(hostname -s): $val; [ 5959.781420] rc=$val; [ 5959.781420] fi; [ 5959.781420] exit $rc [ 5960.467964] Lustre: DEBUG MARKER: dmesg [ 5960.937764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 00:16:02 \(1491437762\) [ 5961.135606] Lustre: DEBUG MARKER: == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 00:16:02 (1491437762) [ 5962.172078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5962.455685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5963.740850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 5969.252487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5969.535429] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 5969.818140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5970.099057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5971.395185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5972.680057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5973.963718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5975.247798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5976.529439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5976.810452] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5977.093235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5977.373141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5978.190701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5978.869213] Lustre: DEBUG MARKER: rc=0; [ 5978.869213] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5978.869213] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5978.869213] echo $(hostname -s): $val; [ 5978.869213] rc=$val; [ 5978.869213] fi; [ 5978.869213] exit $rc [ 5979.544793] Lustre: DEBUG MARKER: dmesg [ 5980.025940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 250: Coordinator max request ====================================================== 00:16:21 \(1491437781\) [ 5980.224459] Lustre: DEBUG MARKER: == sanity-hsm test 250: Coordinator max request ====================================================== 00:16:21 (1491437781) [ 5981.139383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [ 5981.424839] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 5981.709363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5981.991106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5996.005842] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 5996.305726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5996.600527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 5997.899491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 5998.190428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 5999.480176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 5999.769811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6001.061388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6001.349358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6002.645097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6002.943220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6004.240189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6004.526772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6005.841932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6006.130362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6007.424139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6007.714597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6009.011762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6009.299863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6010.592050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6010.881314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6012.177385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6012.465504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6013.755271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6014.043840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6015.332179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6015.621652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6016.911329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6017.200585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6018.489189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6018.776506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6020.066822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6020.356165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6021.645595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6021.932571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6023.227669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6023.515672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6024.805308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6025.095374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6026.386907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6026.677570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6027.972151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [ 6028.262097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [ 6031.097277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6031.380907] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6031.666586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6031.949453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6033.232482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6034.513818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6035.799863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6037.084419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6037.369891] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6037.655422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6037.937688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6038.782158] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6039.450138] Lustre: DEBUG MARKER: rc=0; [ 6039.450138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6039.450138] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6039.450138] echo $(hostname -s): $val; [ 6039.450138] rc=$val; [ 6039.450138] fi; [ 6039.450138] exit $rc [ 6040.135855] Lustre: DEBUG MARKER: dmesg [ 6040.616171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 251: Coordinator request timeout ================================================== 00:17:21 \(1491437841\) [ 6040.815732] Lustre: DEBUG MARKER: == sanity-hsm test 251: Coordinator request timeout ================================================== 00:17:21 (1491437841) [ 6049.418124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [ 6049.703794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6049.985241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6050.269890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [ 6050.553584] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=4 [ 6050.836647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [ 6051.119671] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=2 [ 6051.400911] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6051.684762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6051.966136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6052.315716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6052.599935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6057.895543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6058.181687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000402:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6058.472168] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [ 6058.757963] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [ 6061.585755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6061.876217] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6062.160788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6062.439953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6063.726143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6065.010108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6066.293625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6067.575819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6068.857587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6070.140695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6070.420205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6070.702883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6070.982808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6071.805390] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6072.472603] Lustre: DEBUG MARKER: rc=0; [ 6072.472603] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6072.472603] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6072.472603] echo $(hostname -s): $val; [ 6072.472603] rc=$val; [ 6072.472603] fi; [ 6072.472603] exit $rc [ 6073.151582] Lustre: DEBUG MARKER: dmesg [ 6073.635531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 300: On disk coordinator state kept between MDT umount\/mount ====================== 00:17:54 \(1491437874\) [ 6073.829261] Lustre: DEBUG MARKER: == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ====================== 00:17:54 (1491437874) [ 6074.013914] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6074.298696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6074.577701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6075.858333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6077.140946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6078.422829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6079.704869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6080.986245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6081.269592] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0000.hsm_control [ 6081.410548] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm_control in log params [ 6081.556285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6081.836063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6082.127039] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6082.406925] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6082.548936] Lustre: Failing over lustre-MDT0000 [ 6085.134742] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.134749] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.134752] LustreError: Skipped 2 previous similar messages [ 6086.512817] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.516333] LustreError: Skipped 3 previous similar messages [ 6088.688058] Lustre: 10598:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491437883/real 1491437883] req@ffff8800532c7900 x1563879826246256/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491437889 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6088.707233] Lustre: server umount lustre-MDT0000 complete [ 6088.851612] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6089.146126] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6089.146126] grep -q ^lustre-mdt1/ /proc/mounts || [ 6089.146126] zpool export lustre-mdt1 [ 6099.683677] Lustre: DEBUG MARKER: hostname [ 6100.012812] Lustre: DEBUG MARKER: modprobe zfs; [ 6100.012812] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6100.012812] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6100.455303] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6100.733093] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6100.913669] Lustre: MGS: Connection restored to 62a3a904-3d90-8818-fee7-e746a6639271 (at 0@lo) [ 6100.915874] Lustre: Skipped 10 previous similar messages [ 6101.068404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.233912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6101.525292] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 6102.026272] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6102.309290] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6102.593239] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6102.882074] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6109.944735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 6110.358240] Lustre: lustre-MDT0000: Recovery over after 0:01, of 4 clients 4 recovered and 0 were evicted. [ 6110.699300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6110.701078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6110.910216] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6110.926424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6112.242313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6112.524676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6112.808751] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [ 6112.950983] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm_control in log params [ 6133.100961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6133.381861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6133.673707] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6133.954114] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6134.096197] Lustre: Failing over lustre-MDT0000 [ 6134.537617] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.542214] LustreError: Skipped 1 previous similar message [ 6140.129041] Lustre: 13232:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491437934/real 1491437934] req@ffff8800532c7600 x1563879826247344/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491437940 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6140.148687] Lustre: server umount lustre-MDT0000 complete [ 6140.290580] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6140.583780] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6140.583780] grep -q ^lustre-mdt1/ /proc/mounts || [ 6140.583780] zpool export lustre-mdt1 [ 6151.088910] Lustre: DEBUG MARKER: hostname [ 6151.418055] Lustre: DEBUG MARKER: modprobe zfs; [ 6151.418055] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6151.418055] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6151.884125] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6152.167872] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6152.412471] LustreError: 14700:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [ 6152.450676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.617079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6152.903995] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 6153.384904] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6153.666022] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6153.947731] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6154.244177] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6156.960920] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 6157.377923] Lustre: lustre-MDT0000: Recovery over after 0:01, of 4 clients 4 recovered and 0 were evicted. [ 6158.074475] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6158.078827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6158.275620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6158.319300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6159.621493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6159.902117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6160.239261] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6160.914901] Lustre: DEBUG MARKER: rc=0; [ 6160.914901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6160.914901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6160.914901] echo $(hostname -s): $val; [ 6160.914901] rc=$val; [ 6160.914901] fi; [ 6160.914901] exit $rc [ 6161.604580] Lustre: DEBUG MARKER: dmesg [ 6162.098823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 301: HSM tunnable are persistent ================================================== 00:19:23 \(1491437963\) [ 6162.289373] Lustre: DEBUG MARKER: == sanity-hsm test 301: HSM tunnable are persistent ================================================== 00:19:23 (1491437963) [ 6162.470992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6162.761448] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [ 6162.904103] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [ 6163.061167] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6163.343339] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6163.486291] Lustre: Failing over lustre-MDT0000 [ 6166.564048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.568667] LustreError: Skipped 5 previous similar messages [ 6169.573097] Lustre: 16072:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491437964/real 1491437964] req@ffff88005587e100 x1563879826248112/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491437970 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6169.592507] Lustre: server umount lustre-MDT0000 complete [ 6169.735402] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6170.032079] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6170.032079] grep -q ^lustre-mdt1/ /proc/mounts || [ 6170.032079] zpool export lustre-mdt1 [ 6180.432968] Lustre: DEBUG MARKER: hostname [ 6180.756138] Lustre: DEBUG MARKER: modprobe zfs; [ 6180.756138] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6180.756138] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6181.186089] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6181.467344] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6181.749335] LustreError: 17541:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [ 6181.788336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.948817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6182.234994] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 6182.715300] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6182.999332] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6183.282369] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6183.573856] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6186.960732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 6192.371468] Lustre: lustre-MDT0000: Recovery over after 0:06, of 4 clients 4 recovered and 0 were evicted. [ 6193.403902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6193.408760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6193.597311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6193.623239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6194.944837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6195.231344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6195.377904] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [ 6195.568218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6196.261579] Lustre: DEBUG MARKER: rc=0; [ 6196.261579] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6196.261579] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6196.261579] echo $(hostname -s): $val; [ 6196.261579] rc=$val; [ 6196.261579] fi; [ 6196.261579] exit $rc [ 6196.956071] Lustre: DEBUG MARKER: dmesg [ 6197.434965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 00:19:58 \(1491437998\) [ 6197.618005] Lustre: DEBUG MARKER: == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 00:19:58 (1491437998) [ 6197.790219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6198.074403] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6198.358239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6198.638470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6199.920430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6201.201637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6202.488282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6202.785504] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=3 [ 6202.927303] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [ 6203.082940] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6203.362927] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6203.507031] Lustre: Failing over lustre-MDT0000 [ 6206.564542] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.569047] LustreError: Skipped 5 previous similar messages [ 6209.612043] Lustre: 19332:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491438004/real 1491438004] req@ffff88005d70c600 x1563879826249184/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491438010 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6209.631484] Lustre: server umount lustre-MDT0000 complete [ 6209.772762] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6210.064168] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6210.064168] grep -q ^lustre-mdt1/ /proc/mounts || [ 6210.064168] zpool export lustre-mdt1 [ 6220.477940] Lustre: DEBUG MARKER: hostname [ 6220.801357] Lustre: DEBUG MARKER: modprobe zfs; [ 6220.801357] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6220.801357] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6221.271249] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6221.561202] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6221.851400] LustreError: 20801:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [ 6221.887728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.047583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6222.330005] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 6222.810954] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6222.945112] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 6223.103463] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6223.399201] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6223.694793] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6229.397264] Lustre: lustre-MDT0000: Recovery over after 0:07, of 4 clients 4 recovered and 0 were evicted. [ 6230.507548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 6230.507550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 6230.715617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 6230.716637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 6232.035780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6232.316494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6232.599503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6232.884541] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [ 6233.025707] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [ 6233.210434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6233.934874] Lustre: DEBUG MARKER: rc=0; [ 6233.934874] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6233.934874] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6233.934874] echo $(hostname -s): $val; [ 6233.934874] rc=$val; [ 6233.934874] fi; [ 6233.934874] exit $rc [ 6234.647736] Lustre: DEBUG MARKER: dmesg [ 6235.150150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 00:20:36 \(1491438036\) [ 6235.342035] Lustre: DEBUG MARKER: == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 00:20:36 (1491438036) [ 6235.559988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_400 needs \>= 2 MDTs [ 6235.754517] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_400 needs >= 2 MDTs [ 6235.963107] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6236.644673] Lustre: DEBUG MARKER: rc=0; [ 6236.644673] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6236.644673] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6236.644673] echo $(hostname -s): $val; [ 6236.644673] rc=$val; [ 6236.644673] fi; [ 6236.644673] exit $rc [ 6237.332216] Lustre: DEBUG MARKER: dmesg [ 6237.809579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 00:20:39 \(1491438039\) [ 6237.998385] Lustre: DEBUG MARKER: == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 00:20:39 (1491438039) [ 6238.212774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_401 needs \>= 2 MDTs [ 6238.405490] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_401 needs >= 2 MDTs [ 6238.615421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6239.323360] Lustre: DEBUG MARKER: rc=0; [ 6239.323360] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6239.323360] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6239.323360] echo $(hostname -s): $val; [ 6239.323360] rc=$val; [ 6239.323360] fi; [ 6239.323360] exit $rc [ 6240.009696] Lustre: DEBUG MARKER: dmesg [ 6240.508569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 00:20:41 \(1491438041\) [ 6240.701327] Lustre: DEBUG MARKER: == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 00:20:41 (1491438041) [ 6243.416553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6243.698356] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6243.989661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6244.278740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6245.560428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6246.842300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6248.123947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6249.405333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6250.688815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6251.970862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6252.255414] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6252.540079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6252.820473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6255.110556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep uuid [ 6256.028049] Lustre: lustre-MDT0000: Client 06f64bfb-e331-eb0b-d3fb-dd9b8f1621ba (at 10.9.5.238@tcp) reconnecting [ 6256.476611] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6257.161895] Lustre: DEBUG MARKER: rc=0; [ 6257.161895] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6257.161895] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6257.161895] echo $(hostname -s): $val; [ 6257.161895] rc=$val; [ 6257.161895] fi; [ 6257.161895] exit $rc [ 6257.841774] Lustre: DEBUG MARKER: dmesg [ 6260.799157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6261.082386] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6261.366952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6261.647376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6262.930947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6263.215442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6263.501788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6263.781068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6264.634047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 00:21:05 \(1491438065\) [ 6264.829317] Lustre: DEBUG MARKER: == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 00:21:05 (1491438065) [ 6265.907346] LustreError: 24331:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent '43d2d38d-8855-c518-88d6-7518291a0b16': rc = -11 [ 6266.053263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [ 6275.912654] LustreError: 24331:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent '43d2d38d-8855-c518-88d6-7518291a0b16': rc = -11 [ 6276.347614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6276.630739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6277.162195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6277.445848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6278.731342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6280.016597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6281.307963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6282.594757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6283.885910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6285.180942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6286.468119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x81:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [ 6289.292793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6289.578673] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6289.863464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6290.144662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6291.425924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6292.710185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6293.991713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6295.273862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6296.555327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6296.836826] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6297.120929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6297.404606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6298.226447] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6298.912163] Lustre: DEBUG MARKER: rc=0; [ 6298.912163] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6298.912163] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6298.912163] echo $(hostname -s): $val; [ 6298.912163] rc=$val; [ 6298.912163] fi; [ 6298.912163] exit $rc [ 6299.595223] Lustre: DEBUG MARKER: dmesg [ 6300.099494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 00:21:41 \(1491438101\) [ 6300.290705] Lustre: DEBUG MARKER: == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 00:21:41 (1491438101) [ 6300.508036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_403 needs \>= 2 MDTs [ 6300.699463] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_403 needs >= 2 MDTs [ 6300.914865] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6301.591533] Lustre: DEBUG MARKER: rc=0; [ 6301.591533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6301.591533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6301.591533] echo $(hostname -s): $val; [ 6301.591533] rc=$val; [ 6301.591533] fi; [ 6301.591533] exit $rc [ 6302.290215] Lustre: DEBUG MARKER: dmesg [ 6302.765005] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 00:21:44 \(1491438104\) [ 6302.952878] Lustre: DEBUG MARKER: == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 00:21:44 (1491438104) [ 6303.159907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_404 needs \>= 2 MDTs [ 6303.346275] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_404 needs >= 2 MDTs [ 6303.561213] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6304.242684] Lustre: DEBUG MARKER: rc=0; [ 6304.242684] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6304.242684] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6304.242684] echo $(hostname -s): $val; [ 6304.242684] rc=$val; [ 6304.242684] fi; [ 6304.242684] exit $rc [ 6304.927574] Lustre: DEBUG MARKER: dmesg [ 6305.423599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 405: archive and release under striped directory ================================== 00:21:46 \(1491438106\) [ 6305.609599] Lustre: DEBUG MARKER: == sanity-hsm test 405: archive and release under striped directory ================================== 00:21:46 (1491438106) [ 6305.818893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_405 needs \>= 2 MDTs [ 6306.003238] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_405 needs >= 2 MDTs [ 6306.233218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6306.906409] Lustre: DEBUG MARKER: rc=0; [ 6306.906409] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6306.906409] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6306.906409] echo $(hostname -s): $val; [ 6306.906409] rc=$val; [ 6306.906409] fi; [ 6306.906409] exit $rc [ 6307.589128] Lustre: DEBUG MARKER: dmesg [ 6308.089689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 00:21:49 \(1491438109\) [ 6308.278620] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 00:21:49 (1491438109) [ 6308.485025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_406 needs \>= 2 MDTs [ 6308.686673] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_406 needs >= 2 MDTs [ 6308.901502] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6309.586706] Lustre: DEBUG MARKER: rc=0; [ 6309.586706] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6309.586706] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6309.586706] echo $(hostname -s): $val; [ 6309.586706] rc=$val; [ 6309.586706] fi; [ 6309.586706] exit $rc [ 6310.292532] Lustre: DEBUG MARKER: dmesg [ 6310.773182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 00:21:52 \(1491438112\) [ 6310.966236] Lustre: DEBUG MARKER: == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 00:21:52 (1491438112) [ 6311.145359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version [ 6313.980834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6314.263788] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6314.548817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6314.829342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6316.113657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6317.396809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6317.681124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6317.966360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6318.246837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6321.772403] Lustre: HSM agent e0779b89-1f8d-2da1-023d-bf01db52859a already registered [ 6322.554111] LustreError: 20773:0:(mdt_coordinator.c:1360:mdt_hsm_update_request_state()) lustre-MDT0000: Cannot find running request for cookie 0x58e58a2e on fid=[0x200000409:0x8d:0x0] [ 6323.168073] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6323.883110] Lustre: DEBUG MARKER: rc=0; [ 6323.883110] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6323.883110] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6323.883110] echo $(hostname -s): $val; [ 6323.883110] rc=$val; [ 6323.883110] fi; [ 6323.883110] exit $rc [ 6324.568537] Lustre: DEBUG MARKER: dmesg [ 6327.542464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6327.826142] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [ 6328.110784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6328.391914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6329.674112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6330.956262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6332.238030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6333.524689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6333.806401] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [ 6334.089733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6334.370902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [ 6335.196425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration 3308 sec ======================================================= 00:22:16 \(1491438136\) [ 6335.388024] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration 3308 sec ======================================================= 00:22:16 (1491438136) [ 6336.735698] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6337.016756] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6337.158930] Lustre: Failing over lustre-MDT0000 [ 6337.696082] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.696096] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.696098] LustreError: Skipped 4 previous similar messages [ 6343.322039] Lustre: 29777:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491438138/real 1491438138] req@ffff8800663e9800 x1563879826252336/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491438144 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6343.342318] Lustre: server umount lustre-MDT0000 complete [ 6343.489396] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6343.786711] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6343.786711] grep -q ^lustre-mdt1/ /proc/mounts || [ 6343.786711] zpool export lustre-mdt1 [ 6354.235601] Lustre: DEBUG MARKER: hostname [ 6354.568414] Lustre: DEBUG MARKER: modprobe zfs; [ 6354.568414] zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6354.568414] zpool import -f -o cachefile=none -d /dev/lvm-Role_MDS lustre-mdt1 [ 6354.999373] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 [ 6355.276177] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre lustre-mdt1/mdt1 /mnt/lustre-mds1 [ 6355.559949] LustreError: 31243:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [ 6355.611099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6355.775943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6356.111277] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [ 6356.624098] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6356.905694] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6357.187829] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6357.481575] Lustre: DEBUG MARKER: zfs get -H -o value lustre:svname lustre-mdt1/mdt1 2>/dev/null [ 6359.959693] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [ 6365.014160] Lustre: lustre-MDT0000: Recovery over after 0:05, of 4 clients 4 recovered and 0 were evicted. [ 6365.303822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6365.336248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6365.561851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6365.642965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 6387.741601] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6388.024118] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 6389.709618] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 6389.712175] Lustre: Skipped 2 previous similar messages [ 6390.080070] LustreError: 5892:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880053070000 x1563879826264096/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6394.709399] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.714637] LustreError: Skipped 4 previous similar messages [ 6396.350911] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6396.648511] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6396.648511] grep -q ^lustre-mdt1/ /proc/mounts || [ 6396.648511] zpool export lustre-mdt1 [ 6417.917152] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6418.163144] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6418.428007] Lustre: DEBUG MARKER: ! zpool list -H lustre-mdt1 >/dev/null 2>&1 || [ 6418.428007] grep -q ^lustre-mdt1/ /proc/mounts || [ 6418.428007] zpool export lustre-mdt1 Console [trevis-41vm7] joined by on pts/53 at 04-06 06:45. Console [trevis-41vm7] disconnected from at 04-06 09:45. Console [trevis-41vm7] connected to at 04-06 09:45. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.005005] pid_max: default: 32768 minimum: 301 [ 0.006034] Security Framework initialized [ 0.008009] SELinux: Initializing. [ 0.009219] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.013226] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.015792] Mount-cache hash table entries: 4096 [ 0.017379] Initializing cgroup subsys memory [ 0.018020] Initializing cgroup subsys devices [ 0.020009] Initializing cgroup subsys freezer [ 0.021006] Initializing cgroup subsys net_cls [ 0.022010] Initializing cgroup subsys blkio [ 0.024005] Initializing cgroup subsys perf_event [ 0.025006] Initializing cgroup subsys hugetlb [ 0.027005] Initializing cgroup subsys pids [ 0.028005] Initializing cgroup subsys net_prio [ 0.029126] Disabled fast string operations [ 0.031027] mce: CPU supports 10 MCE banks [ 0.032061] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] tlb_flushall_shift: 6 [ 0.036205] Freeing SMP alternatives: 28k freed [ 0.042681] ACPI: Core revision 20130517 [ 0.044855] ACPI: All ACPI Tables successfully acquired [ 0.046029] ftrace: allocating 25811 entries in 101 pages [ 0.056010] smpboot: Max logical packages: 2 [ 0.057005] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.058005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.060293] Enabling x2apic [ 0.061000] Enabled x2apic [ 0.061005] Switched APIC routing to physical x2apic. [ 0.064139] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.065003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.068039] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.070826] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.324055] Brought up 2 CPUs [ 0.324046] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.326008] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.328041] devtmpfs: initialized [ 0.332015] EVM: security.selinux [ 0.333005] EVM: security.ima [ 0.334006] EVM: security.capability [ 0.335820] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.336010] pinctrl core: initialized pinctrl subsystem [ 0.337166] NET: Registered protocol family 16 [ 0.338317] ACPI: bus type PCI registered [ 0.339008] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.340097] PCI: Using configuration type 1 for base access [ 0.344090] ACPI: Added _OSI(Module Device) [ 0.345013] ACPI: Added _OSI(Processor Device) [ 0.346006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.347005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349725] ACPI: Interpreter enabled [ 0.350013] ACPI: (supports S0 S5) [ 0.351005] ACPI: Using IOAPIC for interrupt routing [ 0.352029] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355663] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356011] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.357010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.358021] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359254] acpiphp: Slot [1] registered [ 0.360032] acpiphp: Slot [2] registered [ 0.361024] acpiphp: Slot [3] registered [ 0.362023] acpiphp: Slot [4] registered [ 0.363023] acpiphp: Slot [5] registered [ 0.364022] acpiphp: Slot [6] registered [ 0.365022] acpiphp: Slot [7] registered [ 0.366022] acpiphp: Slot [8] registered [ 0.367030] acpiphp: Slot [9] registered [ 0.368024] acpiphp: Slot [10] registered [ 0.369023] acpiphp: Slot [11] registered [ 0.370022] acpiphp: Slot [12] registered [ 0.371023] acpiphp: Slot [13] registered [ 0.372022] acpiphp: Slot [14] registered [ 0.373022] acpiphp: Slot [15] registered [ 0.374031] acpiphp: Slot [16] registered [ 0.375030] acpiphp: Slot [17] registered [ 0.376022] acpiphp: Slot [18] registered [ 0.377023] acpiphp: Slot [19] registered [ 0.378024] acpiphp: Slot [20] registered [ 0.379025] acpiphp: Slot [21] registered [ 0.380022] acpiphp: Slot [22] registered [ 0.381036] acpiphp: Slot [23] registered [ 0.382024] acpiphp: Slot [24] registered [ 0.383023] acpiphp: Slot [25] registered [ 0.384022] acpiphp: Slot [26] registered [ 0.385032] acpiphp: Slot [27] registered [ 0.386024] acpiphp: Slot [28] registered [ 0.387022] acpiphp: Slot [29] registered [ 0.388030] acpiphp: Slot [30] registered [ 0.389023] acpiphp: Slot [31] registered [ 0.390272] PCI host bridge to bus 0000:00 [ 0.391009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.392009] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.397058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.398006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.399005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.401859] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.402013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.419794] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.421096] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.422914] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.424070] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.425903] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.427347] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.428289] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.429007] vgaarb: loaded [ 0.430005] vgaarb: bridge control possible 0000:00:02.0 [ 0.431231] SCSI subsystem initialized [ 0.432054] ACPI: bus type USB registered [ 0.433041] usbcore: registered new interface driver usbfs [ 0.434027] usbcore: registered new interface driver hub [ 0.435062] usbcore: registered new device driver usb [ 0.436220] PCI: Using ACPI for IRQ routing [ 0.437376] NetLabel: Initializing [ 0.438007] NetLabel: domain hash size = 128 [ 0.439004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.440059] NetLabel: unlabeled traffic allowed by default [ 0.441167] Switched to clocksource kvm-clock [ 0.451979] pnp: PnP ACPI init [ 0.453186] ACPI: bus type PNP registered [ 0.454949] pnp: PnP ACPI: found 5 devices [ 0.456271] ACPI: bus type PNP unregistered [ 0.466223] NET: Registered protocol family 2 [ 0.467950] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.470485] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.472362] TCP: Hash tables configured (established 16384 bind 16384) [ 0.474075] TCP: reno registered [ 0.475250] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.476815] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.478569] NET: Registered protocol family 1 [ 0.479888] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.481356] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.482789] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.484539] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.486359] Unpacking initramfs... [ 4.693278] Freeing initrd memory: 42360k freed [ 4.718789] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.720537] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.722534] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.724106] Initialise system trusted keyring [ 4.725426] audit: initializing netlink socket (disabled) [ 4.726847] type=2000 audit(1491471962.383:1): initialized [ 4.748107] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.749644] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.752321] zpool: loaded [ 4.753401] zbud: loaded [ 4.754712] VFS: Disk quotas dquot_6.5.2 [ 4.755998] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.757745] msgmni has been set to 3357 [ 4.759086] Key type big_key registered [ 4.762023] NET: Registered protocol family 38 [ 4.763347] Key type asymmetric registered [ 4.764612] Asymmetric key parser 'x509' registered [ 4.766048] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.768424] io scheduler noop registered [ 4.769704] io scheduler deadline registered (default) [ 4.771136] io scheduler cfq registered [ 4.772459] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.773921] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.775703] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.778071] ACPI: Power Button [PWRF] [ 4.779615] GHES: HEST is not enabled! [ 4.780940] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.803926] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.805742] Non-volatile memory driver v1.3 [ 4.807041] Linux agpgart interface v0.103 [ 4.808423] crash memory driver: version 1.1 [ 4.809774] rdac: device handler registered [ 4.811101] hp_sw: device handler registered [ 4.812399] emc: device handler registered [ 4.813666] alua: device handler registered [ 4.814936] libphy: Fixed MDIO Bus: probed [ 4.816207] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.817758] ehci-pci: EHCI PCI platform driver [ 4.819039] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.820526] ohci-pci: OHCI PCI platform driver [ 4.821824] uhci_hcd: USB Universal Host Controller Interface driver [ 4.823570] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.824987] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.827271] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.828670] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.830202] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.831794] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.834089] usb usb1: Product: UHCI Host Controller [ 4.835468] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.837037] usb usb1: SerialNumber: 0000:00:01.2 [ 4.838486] hub 1-0:1.0: USB hub found [ 4.839742] hub 1-0:1.0: 2 ports detected [ 4.841142] usbcore: registered new interface driver usbserial [ 4.842716] usbcore: registered new interface driver usbserial_generic [ 4.844309] usbserial: USB Serial support registered for generic [ 4.845871] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.848973] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.850471] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.852061] mousedev: PS/2 mouse device common for all mice [ 4.853910] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.856772] rtc_cmos 00:00: RTC can wake from S4 [ 4.857755] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.858028] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.864303] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.865739] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.867294] cpuidle: using governor menu [ 4.868629] hidraw: raw HID events driver (C) Jiri Kosina [ 4.870141] usbcore: registered new interface driver usbhid [ 4.871598] usbhid: USB HID core driver [ 4.872853] drop_monitor: Initializing network drop monitor service [ 4.874504] TCP: cubic registered [ 4.875656] Initializing XFRM netlink socket [ 4.877038] NET: Registered protocol family 10 [ 4.878773] NET: Registered protocol family 17 [ 4.880535] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.882523] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.884091] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.886856] Loading compiled-in X.509 certificates [ 4.888074] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.890938] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.893956] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.896687] registered taskstats version 1 [ 4.900785] Key type trusted registered [ 4.904165] Key type encrypted registered [ 4.907523] IMA: No TPM chip found, activating TPM-bypass! [ 4.909358] rtc_cmos 00:00: setting system clock to 2017-04-06 09:46:01 UTC (1491471961) [ 4.912528] Freeing unused kernel memory: 1680k freed [ 4.919040] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.940361] systemd[1]: Detected virtualization kvm. [ 4.941914] systemd[1]: Detected architecture x86-64. [ 4.943360] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.945638] systemd[1]: No hostname configured. [ 4.946991] systemd[1]: Set hostname to . [ 4.948480] systemd[1]: Initializing machine ID from KVM UUID. [ 4.975290] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Reached target Swap. [ 4.978937] systemd[1]: Reached target Swap. [ 4.980474] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 4.982516] systemd[1]: Created slice Root Slice. [ 4.983910] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 4.985938] systemd[1]: Listening on udev Kernel Socket. [ 4.987413] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on Journal Socket. [ 4.989520] systemd[1]: Listening on Journal Socket. [ 4.990951] systemd[1]: Starting Journal Socket. [ OK ] Created slice System Slice. [ 4.992992] systemd[1]: Created slice System Slice. [ 4.994434] systemd[1]: Starting System Slice. [ 4.996373] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 5.001768] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 5.005658] systemd[1]: Starting Journal Service... Starting Journal Service... [ OK ] Reached target Local File Systems. [ 5.009879] systemd[1]: Reached target Local File Systems. [ 5.012274] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 5.015316] systemd[1]: Reached target Timers. [ 5.016664] systemd[1]: Starting Timers. [ OK ] Reached target Slices. [ 5.018542] systemd[1]: Reached target Slices. [ 5.019881] systemd[1]: Starting Slices. [ 5.021756] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Encrypted Volumes. [ 5.026025] systemd[1]: Reached target Encrypted Volumes. [ 5.027490] systemd[1]: Starting Encrypted Volumes. [ 5.029862] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ 5.032619] systemd[1]: Listening on udev Control Socket. [ 5.034086] systemd[1]: Starting udev Control Socket. [ OK ] Started Journal Service. [ 5.036965] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 5.085830] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.160026] dracut-cmdline[87]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.292910] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.297628] Loading iSCSI transport class v2.0-870. [ 5.302185] iscsi: registered transport (tcp) [ 5.335328] FDC 0 is a S82078B [ 5.340445] iBFT detected. [ 5.341618] iBFT has incorrect checksum (0x54)! [ 5.349182] md: raid0 personality registered for level 0 [ 5.352852] md: raid1 personality registered for level 1 [ 5.356203] async_tx: api initialized (async) [ 5.358279] xor: automatically using best checksumming function: [ 5.369007] avx : 22320.000 MB/sec [ 5.389007] raid6: sse2x1 gen() 7593 MB/s [ 5.407007] raid6: sse2x2 gen() 9898 MB/s [ 5.425014] raid6: sse2x4 gen() 11460 MB/s [ 5.426304] raid6: using algorithm sse2x4 gen() (11460 MB/s) [ 5.427709] raid6: using ssse3x2 recovery algorithm [ 5.433538] md: raid6 personality registered for level 6 [ 5.434934] md: raid5 personality registered for level 5 [ 5.436352] md: raid4 personality registered for level 4 [ 5.442509] md: raid10 personality registered for level 10 [ 5.445824] md: linear personality registered for level -1 [ 5.451072] device-mapper: uevent: version 1.0.3 [ 5.452487] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.466473] device-mapper: multipath: version 1.11.0 loaded [ 5.469896] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.500600] RPC: Registered named UNIX socket transport module. [ 5.502079] RPC: Registered udp transport module. [ 5.503377] RPC: Registered tcp transport module. [ 5.504664] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... Starting Device-Mapper Multipath Device Controller... [ OK ] Mounted Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket.[ 5.720441] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Reached target Sockets. [ OK ] Started Device-Mapper Multipath Device Controller. Starting Open-iSCSI... [ 5.729514] [drm] Initialized drm 1.1.0 20060810 [ 5.733947] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ OK ] Started Open-iSCSI. Starting dracut initqueue hook... [ 5.750221] AES CTR mode by8 optimization enabled [ 5.758979] scsi host0: ata_piix [ 5.775117] scsi host1: ata_piix [ 5.776351] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.777210] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 5.779569] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 5.782405] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 5.787126] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 %G[ 5.790072] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.793666] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 5.795967] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.797597] [TTM] Initializing pool allocator [ 5.799805] [TTM] Initializing DMA pool allocator [ 5.801437] [drm] fb mappable at 0xF0000000 [ 5.802723] [drm] vram aper at 0xF0000000 [ 5.803976] [drm] size 33554432 [ 5.805225] [drm] fb depth is 24 [ 5.806399] [drm] pitch is 3072 [ 5.809550] 8021q: 802.1Q VLAN Support v1.8 [ 5.810886] alg: No test for crc32 (crc32-pclmul) %G[ 5.812358] fbcon: cirrusdrmfb (fb0) is primary device [ 5.847135] Console: switching to colour frame buffer device 128x48 [ 5.859230] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 5.863024] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 5.864186] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 5.864744] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.881343] vda: vda1 vda2 vda4 [ 5.965472] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.682818] dracut-initqueue[562]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.684375] dracut-initqueue[562]: Dload Upload Total Spent Left Speed [ 8.797470] dracut-initqueue[562]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14606 100 14606 0 0 123k 0 --:--:-- --:--:-- --:--:-- 125k [ 9.332672] dracut-initqueue[562]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.333517] dracut-initqueue[562]: Dload Upload Total Spent Left Speed [ 12.503645] dracut-initqueue[562]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 20 317M 20 63.8M 0 0 95.5M 0 0:00:03 --:--:-- 0:00:03 95.4M 51 317M 51 164M 0 0 98.7M 0 0:00:03 0:00:01 0:00:02 98.7M 83 317M 83 265M 0 0 99.3M 0 0:00:03 0:00:02 0:00:01 99.3M 100 317M 100 317M 0 0 100M 0 0:00:03 0:00:03 --:--:-- 100M [ 12.602299] loop: module loaded [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting dracut pre-mount hook... [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. Starting dracut mount hook... [ OK ] Reached target Initrd File Systems. [ 12.759676] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Swap. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 14.851771] systemd-journald[88]: Received SIGTERM from PID 1 (systemd). [ 15.403730] type=1403 audit(1491471971.993:2): policy loaded auid=4294967295 ses=4294967295 [ 15.409846] systemd[1]: Successfully loaded SELinux policy in 270.353ms. [ 15.514025] systemd[1]: Relabelled /dev and /run in 20.578ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-anaconda\x2dtmux.slice. [ OK ] Listening on udev Control Socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Reached target Timers. [ OK ] Stopped target Initrd Root File System. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... Mounting Temporary Directory... Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Mounting Huge Pages File System... Mounting Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped Journal Service. Starting Journal Service... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Temporary Directory. [ OK ] Started Remount Root and Kernel File Systems. Starting Rebuild Hardware Database... Starting Load/Save Random Seed... Starting Configure read-only root support... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Debug File System. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Load/Save Random Seed. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Configure read-only root support. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... Starting udev Wait for Complete Device Initialization... %G%G[ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... [ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... [ OK ] Reached target Encrypted Volumes. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting Service enabling compressing RAM with zRam... Starting pre-anaconda logging service... Starting firewalld - dynamic firewall daemon... Starting Dump dmesg to /var/log/dmesg... Starting Terminate Plymouth Boot Screen... Starting Wait for Plymouth Boot Screen to Quit... Starting Login Service... %G (BStarting installer, one moment...                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (B [anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (B anaconda 21.48.22.93-1 for CentOS Linux 7 started.  * installation log files are stored in /tmp during the installation  * shell is available on TTY2  * when reporting a bug add logs from /tmp as separate text/plain attachments 09:46:18 Running pre-installation scripts 09:46:20 Not asking for VNC because of an automated install09:46:20 Not asking for VNC because text mode was explicitly asked for in kickstartStarting automated install........................Checking software selectionGenerating updated storage configurationChecking storage configuration...================================================================================ ================================================================================ Installation  1) [x] Language settings 2) [x] Time settings  (English (United States)) (Etc/UTC timezone) 3) [x] Installation source 4) [x] Software selection (http://10.9.0.10/cblr/links/Ce (Custom software selected) ntOS-7.3-x86_64) 6) [x] Kdump 5) [x] Installation Destination (Kdump is enabled) (Custom partitioning selected) 8) [ ] User creation 7) [x] Network configuration (No user will be created) (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Paths. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. Stopping Load/Save Random Seed... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped target Swap. [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting Temporary Directory... Unmounting /mnt/sysimage/proc... Unmounting Configuration File System... Unmounting /mnt/sysimage/dev/shm... Unmounting /mnt/sysimage/sys/fs/selinux... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/run... [ OK ] Unmounted Temporary Directory. [ OK ] Unmounted /mnt/sysimage/proc. [ OK ] Unmounted Configuration File System. [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. [ OK ] Unmounted /mnt/sysimage/run. Unmounting /mnt/sysimage/sys... [ OK ] Unmounted /mnt/sysimage/dev/shm. [ OK ] Unmounted /mnt/sysimage/dev/pts. Unmounting /mnt/sysimage/dev... [ OK ] Unmounted /mnt/sysimage/dev. [ OK ] Unmounted /mnt/sysimage/sys. Unmounting /mnt/sysimage... [ OK ] Deactivated swap /dev/zram1. [ OK ] Deactivated swap /dev/zram0. [ OK ] Unmounted /mnt/sysimage. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Started Restore /run/initramfs. [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 928.902972] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-310-el7-x8664-5.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Apr 5 06:20:37 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=40b010fb-5248-4819-8bdd-3d94a2a34e41 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fac000-0x367cdfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=40b010fb-5248-4819-8bdd-3d94a2a34e41 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694472k/2097140k available (6765k kernel code, 400k absent, 402268k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.005005] pid_max: default: 32768 minimum: 301 [ 0.006029] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.009160] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.012363] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014514] Mount-cache hash table entries: 4096 [ 0.016207] Initializing cgroup subsys memory [ 0.017012] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.020005] Initializing cgroup subsys net_cls [ 0.021005] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.024006] Initializing cgroup subsys hugetlb [ 0.025005] Initializing cgroup subsys pids [ 0.026005] Initializing cgroup subsys net_prio [ 0.028104] Disabled fast string operations [ 0.029020] mce: CPU supports 10 MCE banks [ 0.031056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] tlb_flushall_shift: 6 [ 0.035173] Freeing SMP alternatives: 28k freed [ 0.040816] ACPI: Core revision 20130517 [ 0.042849] ACPI: All ACPI Tables successfully acquired [ 0.044016] ftrace: allocating 25813 entries in 101 pages [ 0.053190] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.055004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.056004] smpboot: Max logical packages: 2 [ 0.058282] Enabling x2apic [ 0.059000] Enabled x2apic [ 0.059004] Switched APIC routing to physical x2apic. [ 0.062704] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.064002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.067025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.069435] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.073036] Brought up 2 CPUs [ 0.073032] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.075005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.076532] devtmpfs: initialized [ 0.078865] EVM: security.selinux [ 0.079005] EVM: security.ima [ 0.080004] EVM: security.capability [ 0.081739] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.082029] pinctrl core: initialized pinctrl subsystem [ 0.083141] NET: Registered protocol family 16 [ 0.084268] ACPI: bus type PCI registered [ 0.085006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.086087] PCI: Using configuration type 1 for base access [ 0.090078] ACPI: Added _OSI(Module Device) [ 0.091013] ACPI: Added _OSI(Processor Device) [ 0.092006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.093010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.095658] ACPI: Interpreter enabled [ 0.096013] ACPI: (supports S0 S5) [ 0.097004] ACPI: Using IOAPIC for interrupt routing [ 0.098017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.101536] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.102010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.103020] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.104018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.105239] acpiphp: Slot [1] registered [ 0.106031] acpiphp: Slot [2] registered [ 0.107023] acpiphp: Slot [3] registered [ 0.108022] acpiphp: Slot [4] registered [ 0.109022] acpiphp: Slot [5] registered [ 0.110022] acpiphp: Slot [6] registered [ 0.111022] acpiphp: Slot [7] registered [ 0.112022] acpiphp: Slot [8] registered [ 0.113029] acpiphp: Slot [9] registered [ 0.114023] acpiphp: Slot [10] registered [ 0.115022] acpiphp: Slot [11] registered [ 0.116022] acpiphp: Slot [12] registered [ 0.117023] acpiphp: Slot [13] registered [ 0.118022] acpiphp: Slot [14] registered [ 0.119022] acpiphp: Slot [15] registered [ 0.120028] acpiphp: Slot [16] registered [ 0.121028] acpiphp: Slot [17] registered [ 0.122022] acpiphp: Slot [18] registered [ 0.123022] acpiphp: Slot [19] registered [ 0.124022] acpiphp: Slot [20] registered [ 0.125022] acpiphp: Slot [21] registered [ 0.126022] acpiphp: Slot [22] registered [ 0.127028] acpiphp: Slot [23] registered [ 0.128022] acpiphp: Slot [24] registered [ 0.129022] acpiphp: Slot [25] registered [ 0.130022] acpiphp: Slot [26] registered [ 0.131022] acpiphp: Slot [27] registered [ 0.132028] acpiphp: Slot [28] registered [ 0.133022] acpiphp: Slot [29] registered [ 0.134022] acpiphp: Slot [30] registered [ 0.135028] acpiphp: Slot [31] registered [ 0.136258] PCI host bridge to bus 0000:00 [ 0.137007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.138006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.139005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.140006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.141006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.143011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.144006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.145005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.146005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.147828] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.148013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.165671] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.167077] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.168909] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.170070] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.171880] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.173296] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.174279] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.175006] vgaarb: loaded [ 0.176005] vgaarb: bridge control possible 0000:00:02.0 [ 0.177204] SCSI subsystem initialized [ 0.178050] ACPI: bus type USB registered [ 0.179037] usbcore: registered new interface driver usbfs [ 0.180017] usbcore: registered new interface driver hub [ 0.181054] usbcore: registered new device driver usb [ 0.182178] PCI: Using ACPI for IRQ routing [ 0.183345] NetLabel: Initializing [ 0.184008] NetLabel: domain hash size = 128 [ 0.185004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.186040] NetLabel: unlabeled traffic allowed by default [ 0.187144] Switched to clocksource kvm-clock [ 0.196158] pnp: PnP ACPI init [ 0.197334] ACPI: bus type PNP registered [ 0.199100] pnp: PnP ACPI: found 5 devices [ 0.200398] ACPI: bus type PNP unregistered [ 0.210200] NET: Registered protocol family 2 [ 0.211856] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.214330] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.216068] TCP: Hash tables configured (established 16384 bind 16384) [ 0.217769] TCP: reno registered [ 0.218948] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.220523] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.222245] NET: Registered protocol family 1 [ 0.223557] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.225028] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.226472] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.228225] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.229981] Unpacking initramfs... [ 0.628039] Freeing initrd memory: 24712k freed [ 0.639849] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.641660] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.643634] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.645220] Initialise system trusted keyring [ 0.646563] audit: initializing netlink socket (disabled) [ 0.647977] type=2000 audit(1491472914.844:1): initialized [ 0.669131] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.670676] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.673383] zpool: loaded [ 0.674462] zbud: loaded [ 0.675774] VFS: Disk quotas dquot_6.5.2 [ 0.677079] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.678831] msgmni has been set to 3357 [ 0.680124] Key type big_key registered [ 0.683207] NET: Registered protocol family 38 [ 0.684530] Key type asymmetric registered [ 0.685780] Asymmetric key parser 'x509' registered [ 0.687180] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.689577] io scheduler noop registered [ 0.690824] io scheduler deadline registered (default) [ 0.692269] io scheduler cfq registered [ 0.693605] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.695043] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.696774] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.699134] ACPI: Power Button [PWRF] [ 0.700496] GHES: HEST is not enabled! [ 0.701815] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.724886] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.726770] Non-volatile memory driver v1.3 [ 0.728074] Linux agpgart interface v0.103 [ 0.729453] crash memory driver: version 1.1 [ 0.730804] rdac: device handler registered [ 0.732189] hp_sw: device handler registered [ 0.748735] emc: device handler registered [ 0.749991] alua: device handler registered [ 0.751273] libphy: Fixed MDIO Bus: probed [ 0.752562] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.754103] ehci-pci: EHCI PCI platform driver [ 0.755380] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.756863] ohci-pci: OHCI PCI platform driver [ 0.758144] uhci_hcd: USB Universal Host Controller Interface driver [ 0.759866] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.761295] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.763596] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.764991] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.766539] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.768194] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.770660] usb usb1: Product: UHCI Host Controller [ 0.772108] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.774604] usb usb1: SerialNumber: 0000:00:01.2 [ 0.776124] hub 1-0:1.0: USB hub found [ 0.777397] hub 1-0:1.0: 2 ports detected [ 0.778887] usbcore: registered new interface driver usbserial [ 0.780468] usbcore: registered new interface driver usbserial_generic [ 0.782156] usbserial: USB Serial support registered for generic [ 0.783802] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.786982] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.788481] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.790170] mousedev: PS/2 mouse device common for all mice [ 0.792228] rtc_cmos 00:00: RTC can wake from S4 [ 0.792239] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.797154] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.801074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.816701] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.818377] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.820020] cpuidle: using governor menu [ 0.821371] hidraw: raw HID events driver (C) Jiri Kosina [ 0.822997] usbcore: registered new interface driver usbhid [ 0.824474] usbhid: USB HID core driver [ 0.825810] drop_monitor: Initializing network drop monitor service [ 0.827458] TCP: cubic registered [ 0.828693] Initializing XFRM netlink socket [ 0.830143] NET: Registered protocol family 10 [ 0.831831] NET: Registered protocol family 17 [ 0.833430] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.835484] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.837136] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.840014] Loading compiled-in X.509 certificates [ 0.841452] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.844207] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.847502] Loaded X.509 cert 'CentOS Linux kernel signing key: 01570b27fec1f165a8c03592eefb397e9114070a' [ 0.850274] registered taskstats version 1 [ 0.854328] Key type trusted registered [ 0.868262] Key type encrypted registered [ 0.871795] IMA: No TPM chip found, activating TPM-bypass! [ 0.873669] rtc_cmos 00:00: setting system clock to 2017-04-06 10:01:54 UTC (1491472914) [ 0.877166] Freeing unused kernel memory: 1680k freed [ 0.883678] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.888165] systemd[1]: Detected virtualization kvm. [ 0.889594] systemd[1]: Detected architecture x86-64. [ 0.890993] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.893323] systemd[1]: Set hostname to . [ OK ] Reached target Timers. [ 0.927725] systemd[1]: Reached target Timers. [ 0.929107] systemd[1]: Starting Timers. [ OK ] Created slice Root Slice. [ 0.931166] systemd[1]: Created slice Root Slice. [ 0.932536] systemd[1]: Starting Root Slice. [ OK ] Reached target Local File Systems. [ 0.934540] systemd[1]: Reached target Local File Systems. [ 0.936052] systemd[1]: Starting Local File Systems. [ OK ] Listening on Journal Socket. [ 0.938203] systemd[1]: Listening on Journal Socket. [ 0.939603] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Control Socket. [ 0.941666] systemd[1]: Listening on udev Control Socket. [ 0.943131] systemd[1]: Starting udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ 0.945280] systemd[1]: Listening on udev Kernel Socket. [ 0.946723] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 0.948781] systemd[1]: Reached target Sockets. [ 0.950140] systemd[1]: Starting Sockets. [ OK ] Created slice System Slice. [ 0.952138] systemd[1]: Created slice System Slice. [ 0.953555] systemd[1]: Starting System Slice. [ 0.955430] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.959448] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 0.962249] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.964812] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Slices. [ 0.970504] systemd[1]: Reached target Slices. [ 0.972107] systemd[1]: Starting Slices. [ 0.974211] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Swap. [ 1.054772] systemd[1]: Reached target Swap. [ 1.056094] systemd[1]: Starting Swap. [ OK ] Started Journal Service. [ 1.058342] systemd[1]: Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... [ 1.151939] scsi host0: ata_piix [ 1.153190] scsi host1: ata_piix [ 1.154342] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.155916] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.158145] [drm] Initialized drm 1.1.0 20060810 [ 1.160520] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.162491] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.166483] FDC 0 is a S82078B [ 1.176822] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.185933] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.187636] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.188269] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.188301] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.203632] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.205186] [TTM] Initializing pool allocator [ 1.206687] [TTM] Initializing DMA pool allocator [ 1.223988] [drm] fb mappable at 0xF0000000 [ 1.225195] vda: vda1 vda2 [ 1.226699] [drm] vram aper at 0xF0000000 %G[ 1.227941] [drm] size 33554432 [ 1.227941] [drm] fb depth is 24 [ 1.227941] [drm] pitch is 3072 [ 1.231696] fbcon: cirrusdrmfb (fb0) is primary device [ 1.261729] Console: switching to colour frame buffer device 128x48 [ 1.276295] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device %G[ OK ] Found device /dev/disk/by-uuid/40b010fb-5248-4819-8bdd-3d94a2a34e41. Starting File System Ch[ 1.301024] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 eck on /dev/...b-5248-4819-8bdd-3d94a2a34e41... [ OK ] Started dracut initqueue hook. [ OK ] Started File System Check on /dev/d...0fb-5248-4819-8bdd-3d94a2a34e41. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 1.377817] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.379463] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. [ 1.640041] tsc: Refined TSC clocksource calibration: 2593.642 MHz Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.785556] systemd-journald[93]: Received SIGTERM from PID 1 (systemd). [ 1.980457] SELinux: Disabled at runtime. [ 2.016028] type=1404 audit(1491472915.642:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.023023] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.024510] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. Starting Create list of required st... nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Reached target Remote File Systems. Mounting POSIX Message Queue File System... [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Listening on udev Control Socket. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ 3.002531] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Started Remount Root and Kernel File Systems. Starting Flush Journal to Persistent Storage... Starting Load/Save Random Seed... Starting Configure read-only root support... [ 3.013242] systemd-journald[348]: Received request to flush runtime journal from PID 1 Starting Rebuild Hardware Database... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Configure read-only root support. [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ 4.152449] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 4.160377] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/24539a54-8fd0-4ebd-9253-933a8514d32a. Activating swap /dev/disk/by-uuid/2...4-8fd0-4ebd-9253-933a8514d32a... [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... Starting Mark the need to relabel after reboot... Starting Tell Plymouth To Write Out Runtime Data... [ 4.208585] AES CTR mode by8 optimization enabled [ 4.208955] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 4.212036] ppdev: user-space parallel port driver %G[ OK ] Activated swap /dev/disk/by-uuid/24539a54-8fd0-4ebd-9253-933a8514d32a. %G[ OK ] Reached target Swap. [ 4.237774] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 4.243033] alg: No test for crc32 (crc32-pclmul) [ 4.245762] intel_rapl: no valid rapl domains found in package 0 [ 4.251238] intel_powerclamp: No package C-state available[ OK ] Started Import network configuration from initramfs. [ OK ] Started Rebuild Journal Catalog. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Mark the need to relabel after reboot. Starting Update is Completed... Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ OK ] Started Update is Completed. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Dump dmesg to /var/log/dmesg... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting OpenSSH Server Key Generation... Starting Login Service... Starting Permit User Sessions... Starting Authorization Manager... Starting NTP client/server... [ OK ] Started D-Bus System Message Bus. [ 4.596113] type=1305 audit(1491472918.222:3): audit_pid=448 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started Permit User Sessions. [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Command Scheduler. Starting Command Scheduler... Starting Terminate Plymouth Boot Screen... [ OK ] Started NTP client/server. [ OK ] Started Login Service. [ OK ] Started Authorization Manager. [ 6.068069] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 6.070062] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-06T10:02:10+00:00] WARN: ***************************************** [2017-04-06T10:02:10+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-06T10:02:10+00:00] WARN: ***************************************** [2017-04-06T10:02:21+00:00] INFO: Forking chef instance to converge... [2017-04-06T10:02:21+00:00] INFO: *** Chef 12.4.3 *** [2017-04-06T10:02:21+00:00] INFO: Chef-client pid: 8124 [2017-04-06T10:02:23+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-06T10:02:23+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-06T10:02:23+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-06T10:02:23+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-06T10:02:23+00:00] INFO: Running start handlers [2017-04-06T10:02:23+00:00] INFO: Start handlers complete. [2017-04-06T10:02:23+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-06T10:02:23+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-06T10:02:23+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-06T10:02:23+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-06T10:02:23+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170406100223.620961 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170406100223.623907 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170406100223.627904 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170406100223.631839 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170406100223.635736 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170406100223.639728 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-06T10:02:23+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170406100223.643753 [2017-04-06T10:02:23+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-06T10:02:23+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-06T10:02:23+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170406100223.656750 [2017-04-06T10:02:23+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-06T10:02:23+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-06T10:02:30+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-06T10:02:30+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170406100230.418398 [2017-04-06T10:02:30+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-06T10:02:30+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-06T10:02:30+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-06T10:03:12+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-06T10:03:12+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-06T10:03:12+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-06T10:03:27+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-06T10:03:27+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-06T10:03:27+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-06T10:03:27+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-06T10:03:27+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-06T10:03:27+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-06T10:03:27+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-06T10:03:27+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-06T10:03:27+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-06T10:03:27+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170406100327.990198 [2017-04-06T10:03:27+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-06T10:03:27+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-06T10:03:27+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-06T10:03:28+00:00] INFO: service[sssd] restarted [2017-04-06T10:03:28+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-06T10:03:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:28+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:28+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-06T10:03:28+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-06T10:03:28+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-06T10:03:28+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-06T10:03:29+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-06T10:03:29+00:00] INFO: service[ntpd] enabled [2017-04-06T10:03:29+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-06T10:03:29+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170406100329.495425 [2017-04-06T10:03:29+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-06T10:03:29+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-06T10:03:29+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-06T10:03:29+00:00] INFO: service[ntpd] restarted [2017-04-06T10:03:29+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-06T10:03:29+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170406100329.800400 [2017-04-06T10:03:29+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-06T10:03:29+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-06T10:03:29+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-06T10:03:30+00:00] INFO: service[ntpd] restarted [2017-04-06T10:03:30+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-06T10:03:30+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-06T10:03:30+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-06T10:03:30+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-06T10:03:30+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-06T10:03:30+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170406100330.594171 [2017-04-06T10:03:30+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-06T10:03:30+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-06T10:03:30+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-06T10:03:30+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170406100330.602463 [2017-04-06T10:03:30+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-06T10:03:30+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-06T10:03:30+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-06T10:03:30+00:00] INFO: service[autofs] restarted [2017-04-06T10:03:30+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-06T10:03:30+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170406100330.942270 [2017-04-06T10:03:30+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-06T10:03:30+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-06T10:03:30+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-06T10:03:31+00:00] INFO: service[autofs] restarted [2017-04-06T10:03:31+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-06T10:03:31+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-06T10:03:31+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-06T10:03:31+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-06T10:03:32+00:00] INFO: service[autofs] restarted [2017-04-06T10:03:32+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-06T10:03:32+00:00] INFO: service[autofs] enabled [2017-04-06T10:03:32+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-06T10:03:32+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-06T10:03:32+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-06T10:03:32+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-06T10:03:33+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-06T10:03:33+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170406100333.246140 [2017-04-06T10:03:33+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-06T10:03:33+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-06T10:03:33+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170406100333.254503 [2017-04-06T10:03:33+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-06T10:03:33+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-06T10:03:33+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-06T10:03:33+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-06T10:03:33+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-06T10:03:33+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170406100333.264718 [2017-04-06T10:03:33+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-06T10:03:33+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-06T10:03:33+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-06T10:03:36+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-06T10:03:36+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-06T10:03:36+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-06T10:03:36+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-06T10:03:37+00:00] INFO: service[munge] restarted [2017-04-06T10:03:37+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-06T10:03:37+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-06T10:03:37+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-06T10:03:37+00:00] INFO: service[munge] restarted [2017-04-06T10:03:37+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-06T10:03:37+00:00] INFO: service[munge] enabled [2017-04-06T10:03:37+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-06T10:03:38+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-06T10:03:38+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-06T10:03:41+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-06T10:03:41+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-06T10:03:41+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-06T10:03:44+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-06T10:03:44+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170406100344.013268 [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-06T10:03:44+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170406100344.018096 [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-06T10:03:44+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-06T10:03:44+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-06T10:03:44+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-06T10:03:44+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:44+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-06T10:03:44+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-06T10:03:44+00:00] INFO: service[xinetd] started [2017-04-06T10:03:44+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-06T10:03:45+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-06T10:03:45+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-06T10:03:45+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-06T10:03:45+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-06T10:03:45+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-06T10:03:45+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170406100345.979461 [2017-04-06T10:03:45+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-06T10:03:45+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-06T10:03:45+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-06T10:03:45+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-06T10:03:45+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-06T10:03:45+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-06T10:03:45+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-06T10:03:45+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-06T10:03:45+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-06T10:03:45+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-06T10:03:45+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-06T10:03:45+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-06T10:03:45+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-06T10:03:45+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-06T10:03:45+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-06T10:03:45+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-06T10:03:45+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:46+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-06T10:03:46+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-06T10:03:46+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-06T10:03:47+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-06T10:03:47+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-06T10:03:48+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-06T10:03:50+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-06T10:03:50+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-06T10:03:50+00:00] INFO: service[rsh.socket] enabled [2017-04-06T10:03:50+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-06T10:03:50+00:00] INFO: service[rsh.socket] started [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-06T10:03:50+00:00] INFO: service[rlogin.socket] enabled [2017-04-06T10:03:50+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-06T10:03:50+00:00] INFO: service[rlogin.socket] started [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-06T10:03:50+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-06T10:03:50+00:00] INFO: execute[Guard resource] ran successfully [2017-04-06T10:03:50+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-06T10:03:50+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-06T10:03:50+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-06T10:03:50+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-06T10:03:50+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-06T10:03:51+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-06T10:03:51+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-06T10:03:51+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-06T10:03:51+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-06T10:03:51+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-06T10:03:51+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-06T10:03:53+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-06T10:03:53+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-06T10:03:53+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-06T10:03:53+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-06T10:03:53+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-06T10:03:53+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-06T10:03:55+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-06T10:03:55+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-06T10:03:55+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.el7 from addon-epel7-x86_64 repository [2017-04-06T10:03:58+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.el7 [2017-04-06T10:03:58+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-06T10:03:59+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-06T10:03:59+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-06T10:03:59+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-06T10:04:00+00:00] INFO: service[rdma] enabled [2017-04-06T10:04:00+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 125.936802] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 125.945913] Loading iSCSI transport class v2.0-870. [ 125.949450] iscsi: registered transport (iser) [ 126.029039] RPC: Registered named UNIX socket transport module. [ 126.029582] RPC: Registered udp transport module. [ 126.030096] RPC: Registered tcp transport module. [ 126.030484] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 126.033072] RPC: Registered rdma transport module. [ 126.033505] RPC: Registered rdma backchannel transport module. [2017-04-06T10:04:00+00:00] INFO: service[rdma] started [2017-04-06T10:04:00+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-06T10:04:00+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-06T10:04:00+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 126.128153] FS-Cache: Loaded [ 126.145258] FS-Cache: Netfs 'nfs' registered for caching [ 126.151891] Key type dns_resolver registered [ 126.172104] NFS: Registering the id_resolver key type [ 126.172596] Key type id_resolver registered [ 126.172984] Key type id_legacy registered [2017-04-06T10:04:00+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-06T10:04:00+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-06T10:04:00+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170406100400.797008 [2017-04-06T10:04:00+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-06T10:04:00+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-06T10:04:00+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-06T10:04:40+00:00] INFO: service[kdump] restarted [2017-04-06T10:04:40+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-06T10:04:40+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-06T10:04:40+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-06T10:04:40+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-06T10:04:41+00:00] INFO: group[quota_usr] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-06T10:04:41+00:00] INFO: user[quota_usr] created [2017-04-06T10:04:41+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-06T10:04:41+00:00] INFO: group[quota_2usr] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-06T10:04:41+00:00] INFO: user[quota_2usr] created [2017-04-06T10:04:41+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-06T10:04:41+00:00] INFO: group[sanityusr] created [2017-04-06T10:04:41+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-06T10:04:41+00:00] INFO: user[sanityusr] created [2017-04-06T10:04:41+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-06T10:04:41+00:00] INFO: group[sanityusr1] created [2017-04-06T10:04:41+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-06T10:04:41+00:00] INFO: user[sanityusr1] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_1] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_2] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_3] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_4] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_5] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_6] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_7] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_8] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_9] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_10] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_11] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_12] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_13] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_14] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-06T10:04:41+00:00] INFO: user[quota15_15] created [2017-04-06T10:04:41+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_16] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_17] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_18] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_19] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_20] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_21] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_22] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_23] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_24] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_25] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_26] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_27] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_28] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_29] created [2017-04-06T10:04:42+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-06T10:04:42+00:00] INFO: user[quota15_30] created [2017-04-06T10:04:42+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-06T10:04:42+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-06T10:04:42+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-06T10:04:42+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-06T10:04:42+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-06T10:04:42+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-06T10:04:42+00:00] INFO: bash[set_hostid] ran successfully [2017-04-06T10:04:42+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-06T10:04:42+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-06T10:04:42+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-06T10:04:42+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-06T10:04:42+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-06T10:04:42+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-06T10:04:42+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-06T10:04:42+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-06T10:04:42+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-06T10:04:42+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-06T10:04:42+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-06T10:04:42+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-06T10:04:42+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-06T10:04:42+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-06T10:04:42+00:00] INFO: user[mpiuser] created [2017-04-06T10:04:42+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-06T10:04:42+00:00] INFO: group[mpiuser] created [2017-04-06T10:04:42+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-06T10:04:42+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-06T10:04:42+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-06T10:04:42+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-06T10:04:42+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-06T10:04:43+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-06T10:04:43+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-06T10:04:43+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-06T10:04:43+00:00] INFO: service[postfix] restarted [2017-04-06T10:04:43+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-06T10:04:43+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-06T10:04:43+00:00] INFO: execute[newaliases] ran successfully [2017-04-06T10:04:43+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-06T10:04:43+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-06T10:04:43+00:00] INFO: service[sshd] restarted [2017-04-06T10:04:43+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-06T10:04:43+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-06T10:04:44+00:00] INFO: service[rsyslog] restarted [2017-04-06T10:04:44+00:00] INFO: Chef Run complete in 141.134348402 seconds [2017-04-06T10:04:44+00:00] INFO: Skipping removal of unused files from the cache [2017-04-06T10:04:44+00:00] INFO: Running report handlers [2017-04-06T10:04:44+00:00] INFO: Creating JSON run report [2017-04-06T10:04:44+00:00] INFO: Report handlers complete [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Postfix Mail Transport Agent. [ OK ] Unmounted /scratch. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Stopped Automounts filesystems on demand. Stopping System Security Services Daemon... [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped target Swap. Stopping Load/Save Random Seed... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ 1828.424779] type=1305 audit(1491474743.001:1088): audit_pid=0 old=448 auid=4294967295 ses=4294967295 res=1 [ 1828.425841] type=1130 audit(1491474743.003:1089): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Security Auditing Service. [ 1828.430867] type=1131 audit(1491474743.007:1090): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1828.434378] type=1130 audit(1491474743.011:1091): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1828.434395] type=1131 audit(1491474743.011:1092): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. [ 1828.447307] type=1130 audit(1491474743.024:1093): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1828.447324] type=1131 audit(1491474743.024:1094): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Import network configuration from initramfs... [ 1828.459654] type=1130 audit(1491474743.036:1095): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped target Local File Systems. Unmounting /run/user/0... [ OK ] Stopped Configure read-only root support. [ 1828.467156] type=1131 audit(1491474743.044:1096): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Configure read-only root support... [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ 1828.486635] type=1130 audit(1491474743.063:1097): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Reached target Shutdown. [ 1828.516946] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 1828.523475] systemd-journald[348]: Received SIGTERM from PID 1 (systemd-shutdow). [ 1828.533361] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 1828.538494] systemd-shutdown[1]: Unmounting file systems. [ 1828.627107] EXT4-fs (vda1): re-mounted. Opts: (null) [ 1828.644016] EXT4-fs (vda1): re-mounted. Opts: (null) [ 1828.646182] systemd-shutdown[1]: All filesystems unmounted. [ 1828.648426] systemd-shutdown[1]: Deactivating swaps. [ 1828.650653] systemd-shutdown[1]: All swaps deactivated. [ 1828.652846] systemd-shutdown[1]: Detaching loop devices. [ 1828.655540] systemd-shutdown[1]: All loop devices detached. [ 1828.657707] systemd-shutdown[1]: Detaching DM devices. [ 1828.659890] systemd-shutdown[1]: All DM devices detached. [ 1828.662990] systemd-shutdown[1]: Rebooting. [ 1828.665781] Restarting system. [ 1828.667574] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-310-el7-x8664-5.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Apr 5 06:20:37 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=40b010fb-5248-4819-8bdd-3d94a2a34e41 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fac000-0x367cdfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=40b010fb-5248-4819-8bdd-3d94a2a34e41 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694472k/2097140k available (6765k kernel code, 400k absent, 402268k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004010] pid_max: default: 32768 minimum: 301 [ 0.006033] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.009081] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.013091] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.015286] Mount-cache hash table entries: 4096 [ 0.016240] Initializing cgroup subsys memory [ 0.018011] Initializing cgroup subsys devices [ 0.019005] Initializing cgroup subsys freezer [ 0.020005] Initializing cgroup subsys net_cls [ 0.022004] Initializing cgroup subsys blkio [ 0.023005] Initializing cgroup subsys perf_event [ 0.025005] Initializing cgroup subsys hugetlb [ 0.026005] Initializing cgroup subsys pids [ 0.027005] Initializing cgroup subsys net_prio [ 0.029104] Disabled fast string operations [ 0.030020] mce: CPU supports 10 MCE banks [ 0.031056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] tlb_flushall_shift: 6 [ 0.036173] Freeing SMP alternatives: 28k freed [ 0.041674] ACPI: Core revision 20130517 [ 0.043817] ACPI: All ACPI Tables successfully acquired [ 0.045016] ftrace: allocating 25813 entries in 101 pages [ 0.053194] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.055004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.056004] smpboot: Max logical packages: 2 [ 0.058288] Enabling x2apic [ 0.059000] Enabled x2apic [ 0.059004] Switched APIC routing to physical x2apic. [ 0.062622] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.064002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.066953] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.068434] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.072036] Brought up 2 CPUs [ 0.072032] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.074005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.075540] devtmpfs: initialized [ 0.077862] EVM: security.selinux [ 0.078005] EVM: security.ima [ 0.079004] EVM: security.capability [ 0.080744] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.081031] pinctrl core: initialized pinctrl subsystem [ 0.082141] NET: Registered protocol family 16 [ 0.083269] ACPI: bus type PCI registered [ 0.084006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.085086] PCI: Using configuration type 1 for base access [ 0.089079] ACPI: Added _OSI(Module Device) [ 0.090013] ACPI: Added _OSI(Processor Device) [ 0.091010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.092005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.094660] ACPI: Interpreter enabled [ 0.095013] ACPI: (supports S0 S5) [ 0.096005] ACPI: Using IOAPIC for interrupt routing [ 0.097016] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.100555] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.101010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.102010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.103018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.104243] acpiphp: Slot [1] registered [ 0.105031] acpiphp: Slot [2] registered [ 0.106023] acpiphp: Slot [3] registered [ 0.107023] acpiphp: Slot [4] registered [ 0.108022] acpiphp: Slot [5] registered [ 0.109022] acpiphp: Slot [6] registered [ 0.110022] acpiphp: Slot [7] registered [ 0.111022] acpiphp: Slot [8] registered [ 0.112029] acpiphp: Slot [9] registered [ 0.113023] acpiphp: Slot [10] registered [ 0.114022] acpiphp: Slot [11] registered [ 0.115022] acpiphp: Slot [12] registered [ 0.116022] acpiphp: Slot [13] registered [ 0.117027] acpiphp: Slot [14] registered [ 0.118022] acpiphp: Slot [15] registered [ 0.119028] acpiphp: Slot [16] registered [ 0.120028] acpiphp: Slot [17] registered [ 0.121022] acpiphp: Slot [18] registered [ 0.122022] acpiphp: Slot [19] registered [ 0.123022] acpiphp: Slot [20] registered [ 0.124022] acpiphp: Slot [21] registered [ 0.125022] acpiphp: Slot [22] registered [ 0.126028] acpiphp: Slot [23] registered [ 0.127024] acpiphp: Slot [24] registered [ 0.128022] acpiphp: Slot [25] registered [ 0.129022] acpiphp: Slot [26] registered [ 0.130022] acpiphp: Slot [27] registered [ 0.131029] acpiphp: Slot [28] registered [ 0.132022] acpiphp: Slot [29] registered [ 0.133022] acpiphp: Slot [30] registered [ 0.134028] acpiphp: Slot [31] registered [ 0.135261] PCI host bridge to bus 0000:00 [ 0.136007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.137006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.138006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.139006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.140006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.142016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.143006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.144005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.145005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.146842] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.147013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.170691] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.172077] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.173924] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.175070] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.176897] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.178321] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.179276] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.180012] vgaarb: loaded [ 0.181005] vgaarb: bridge control possible 0000:00:02.0 [ 0.182204] SCSI subsystem initialized [ 0.183051] ACPI: bus type USB registered [ 0.184037] usbcore: registered new interface driver usbfs [ 0.185017] usbcore: registered new interface driver hub [ 0.186050] usbcore: registered new device driver usb [ 0.187179] PCI: Using ACPI for IRQ routing [ 0.188341] NetLabel: Initializing [ 0.189007] NetLabel: domain hash size = 128 [ 0.190004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.191040] NetLabel: unlabeled traffic allowed by default [ 0.192150] Switched to clocksource kvm-clock [ 0.201343] pnp: PnP ACPI init [ 0.202549] ACPI: bus type PNP registered [ 0.204298] pnp: PnP ACPI: found 5 devices [ 0.205610] ACPI: bus type PNP unregistered [ 0.215438] NET: Registered protocol family 2 [ 0.217162] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.219673] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.221421] TCP: Hash tables configured (established 16384 bind 16384) [ 0.223134] TCP: reno registered [ 0.224267] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.225768] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.227388] NET: Registered protocol family 1 [ 0.228690] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.230154] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.231617] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.233369] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.235151] Unpacking initramfs... [ 0.634433] Freeing initrd memory: 24712k freed [ 0.642064] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.643816] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.645725] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.647292] Initialise system trusted keyring [ 0.648610] audit: initializing netlink socket (disabled) [ 0.650056] type=2000 audit(1491474770.986:1): initialized [ 0.670973] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.672504] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.675060] zpool: loaded [ 0.676115] zbud: loaded [ 0.677354] VFS: Disk quotas dquot_6.5.2 [ 0.678602] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.680309] msgmni has been set to 3357 [ 0.681576] Key type big_key registered [ 0.684055] NET: Registered protocol family 38 [ 0.685346] Key type asymmetric registered [ 0.686570] Asymmetric key parser 'x509' registered [ 0.687922] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.690249] io scheduler noop registered [ 0.691472] io scheduler deadline registered (default) [ 0.692861] io scheduler cfq registered [ 0.694174] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.695601] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.697295] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.699571] ACPI: Power Button [PWRF] [ 0.700902] GHES: HEST is not enabled! [ 0.702161] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.725051] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.726918] Non-volatile memory driver v1.3 [ 0.728192] Linux agpgart interface v0.103 [ 0.729561] crash memory driver: version 1.1 [ 0.730887] rdac: device handler registered [ 0.732201] hp_sw: device handler registered [ 0.748901] emc: device handler registered [ 0.750187] alua: device handler registered [ 0.751490] libphy: Fixed MDIO Bus: probed [ 0.752790] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.754337] ehci-pci: EHCI PCI platform driver [ 0.755627] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.757136] ohci-pci: OHCI PCI platform driver [ 0.758427] uhci_hcd: USB Universal Host Controller Interface driver [ 0.760191] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.761612] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.763936] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.765348] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.766887] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.768525] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.770876] usb usb1: Product: UHCI Host Controller [ 0.772265] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.774660] usb usb1: SerialNumber: 0000:00:01.2 [ 0.776110] hub 1-0:1.0: USB hub found [ 0.777352] hub 1-0:1.0: 2 ports detected [ 0.778774] usbcore: registered new interface driver usbserial [ 0.780303] usbcore: registered new interface driver usbserial_generic [ 0.781908] usbserial: USB Serial support registered for generic [ 0.783470] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.786501] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.787967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.789546] mousedev: PS/2 mouse device common for all mice [ 0.791361] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.794158] rtc_cmos 00:00: RTC can wake from S4 [ 0.794766] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.795032] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.801244] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.802823] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.804643] cpuidle: using governor menu [ 0.806268] hidraw: raw HID events driver (C) Jiri Kosina [ 0.808163] usbcore: registered new interface driver usbhid [ 0.809901] usbhid: USB HID core driver [ 0.811417] drop_monitor: Initializing network drop monitor service [ 0.813361] TCP: cubic registered [ 0.814770] Initializing XFRM netlink socket [ 0.816450] NET: Registered protocol family 10 [ 0.818414] NET: Registered protocol family 17 [ 0.820246] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.822043] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.823889] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.826879] Loading compiled-in X.509 certificates [ 0.828352] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.831117] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.834406] Loaded X.509 cert 'CentOS Linux kernel signing key: 01570b27fec1f165a8c03592eefb397e9114070a' [ 0.837221] registered taskstats version 1 [ 0.841229] Key type trusted registered [ 0.844673] Key type encrypted registered [ 0.848079] IMA: No TPM chip found, activating TPM-bypass! [ 0.849924] rtc_cmos 00:00: setting system clock to 2017-04-06 10:32:51 UTC (1491474771) [ 0.853320] Freeing unused kernel memory: 1680k freed [ 0.859654] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.864233] systemd[1]: Detected virtualization kvm. [ 0.865663] systemd[1]: Detected architecture x86-64. [ 0.867140] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.869455] systemd[1]: Set hostname to . [ OK ] Reached target Local File Systems. [ 0.902103] systemd[1]: Reached target Local File Systems. [ 0.903595] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 0.905631] systemd[1]: Reached target Timers. [ 0.906959] systemd[1]: Starting Timers. [ OK ] Reached target Swap. [ 0.908803] systemd[1]: Reached target Swap. [ 0.910122] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 0.912074] systemd[1]: Created slice Root Slice. [ 0.913436] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Control Socket. [ 0.915452] systemd[1]: Listening on udev Control Socket. [ 0.916899] systemd[1]: Starting udev Control Socket. [ OK ] Listening on Journal Socket. [ 0.918980] systemd[1]: Listening on Journal Socket. [ 0.920379] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Kernel Socket. [ 0.922406] systemd[1]: Listening on udev Kernel Socket. [ 0.923850] systemd[1]: Starting udev Kernel Socket. [ OK ] Created slice System Slice. [ 0.925944] systemd[1]: Created slice System Slice. [ 0.927335] systemd[1]: Starting System Slice. [ 0.929153] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.933053] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.935495] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 0.937999] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ 0.940646] systemd[1]: Reached target Slices. [ 0.942044] systemd[1]: Starting Slices. [ OK ] Reached target Sockets. [ 0.945521] systemd[1]: Reached target Sockets. [ 0.946891] systemd[1]: Starting Sockets. [ 0.948697] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ 1.029280] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 1.105723] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.109292] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. Starting dracut initqueue hook... [ 1.120913] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.122390] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.125284] [drm] Initialized drm 1.1.0 20060810 [ 1.127670] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.131428] FDC 0 is a S82078B [ 1.131931] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.136190] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.150637] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.152163] [TTM] Initializing pool allocator [ 1.153457] [TTM] Initializing DMA pool allocator [ 1.155027] [drm] fb mappable at 0xF0000000 [ 1.156267] [drm] vram aper at 0xF0000000 [ 1.157465] [drm] size 33554432 [ 1.158555] [drm] fb depth is 24 [ 1.159659] [drm] pitch is 3072 %G%G[ 1.160908] fbcon: cirrusdrmfb (fb0) is primary device [ 1.195360] vda: vda1 vda2 vda4 [ 1.197293] Console: switching to colour frame buffer device 128x48 [ 1.214563] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.218046] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 1.221939] scsi host0: ata_piix [ 1.222127] scsi host1: ata_piix [ 1.222158] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.222159] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ OK ] Found device /dev/disk/by-uuid/40b010fb-5248-4819-8bdd-3d94a2a34e41. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting File System Check on /dev/...b-5248-4819-8bdd-3d94a2a34e41... [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ OK ] Started File System Check on /dev/d...0fb-5248-4819-8bdd-3d94a2a34e41. Mounting /sysroot... [ 1.305672] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.307073] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.558833] systemd-journald[94]: Received SIGTERM from PID 1 (systemd). [ 1.643058] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ 1.681839] SELinux: Disabled at runtime. [ 1.720027] type=1404 audit(1491474772.370:2): selinux=0 auid=4294967295 ses=4294967295 [ 1.741437] ip_tables: (C) 2000-2006 Netfilter Core Team [ 1.742905] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. Starting Read and set NIS domainname from /etc/sysconfig/network... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on LVM2 poll daemon socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ 1.833623] EXT4-fs (vda1): re-mounted. Opts: (null) [ 1.846038] RPC: Registered named UNIX socket transport module. [ 1.846039] RPC: Registered udp transport module. [ 1.846039] RPC: Registered tcp transport module. [ 1.846040] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Debug File System... Starting Create list of required st... nodes for the current kernel... Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting NFSD configuration filesystem... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Started Journal Service. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Load/Save Random Seed... Starting Configure read-only root support... Starting udev Coldplug all Devices... Starting Flush Journal to Persistent Storage... [ OK ] Started LVM2 metadata daemon. [ 1.883760] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 1.901280] systemd-journald[340]: Received request to flush runtime journal from PID 1 [ 1.954807] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 1.955118] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ 1.986483] AES CTR mode by8 optimization enabled [ 1.989940] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 1.997387] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 2.007308] ppdev: user-space parallel port driver [ 2.028553] alg: No test for crc32 (crc32-pclmul) Starting LVM2 metadata daemon... [ OK ] Started Load/Save Random Seed. [ OK ] Mounted NFSD configuration filesystem. [ OK ] Started Configure read-only root support. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/24539a54-8fd0-4ebd-9253-933a8514d32a. Activating swap /dev/disk/by-uuid/2...4-8fd0-4ebd-9253-933a8514d32a... [ OK ] Activated swap /dev/disk/by-uuid/24539a54-8fd0-4ebd-9253-933a8514d32a. %G[ OK ] Reached target Swap. %G[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Preprocess NFS configuration... Starting Import network configuration from initramfs... [ OK ] Started Preprocess NFS configuration. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ 2.060908] intel_rapl: no valid rapl domains found in package 0 [ 2.103017] intel_powerclamp: No package C-state available[ 2.115924] type=1305 audit(1491474772.765:3): audit_pid=457 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... Mounting RPC Pipe File System... [ OK ] Mounted RPC Pipe File System. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Timers. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting NTP client/server... Starting Dump dmesg to /var/log/dmesg... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting System Security Services Daemon... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting GSSAPI Proxy Daemon... Starting Authorization Manager... Starting Network Manager... [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started NTP client/server. [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Started Authorization Manager. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ 2.355487] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.358821] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Started Hostname Service. [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ 2.501536] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ OK ] Started Login Service. [ 2.541924] Loading iSCSI transport class v2.0-870. [ 2.548961] iscsi: registered transport (iser) [ 2.579838] RPC: Registered rdma transport module. [ 2.581259] RPC: Registered rdma backchannel transport module. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting MUNGE authentication service... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Reached target Network is Online. Mounting /scratch... Starting Xinetd A Powerful Replacement For Inetd... Starting Postfix Mail Transport Agent... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started MUNGE authentication service. [ OK ] Started Notify NFS peers of a restart. [ 8.758644] FS-Cache: Loaded [ 8.765262] FS-Cache: Netfs 'nfs' registered for caching [ 8.770562] Key type dns_resolver registered [ OK ] Started System Logging Service. [ OK ] Started OpenSSH server daemon. [ 8.783088] NFS: Registering the id_resolver key type [ 8.784780] Key type id_resolver registered [ 8.787470] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Postfix Mail Transport Agent. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Permit User Sessions... Starting Crash recovery kernel arming... [ OK ] Started Permit User Sessions. [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Terminate Plymouth Boot Screen... [ OK ] Started Command Scheduler. Starting Command Scheduler... Starting Wait for Plymouth Boot Screen to Quit... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [ 117.169713] device-mapper: uevent: version 1.0.3 [ 117.170586] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 202.595576] libcfs: loading out-of-tree module taints kernel. [ 202.596357] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 202.598805] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 202.601321] alg: No test for adler32 (adler32-zlib) [ 202.601853] alg: No test for crc32 (crc32-table) [ 203.540060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 203.645410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 203.664942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 203.728338] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 203.944633] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 203.944967] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 204.377548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 204.531302] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 204.811837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 204.962517] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 206.833193] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 207.077512] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 207.347573] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [ 207.926411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.341528] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 209.583811] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 209.829421] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [ 210.307682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.816851] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 225.816851] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 225.816851] if [ $running -ne $mpts ]; then [ 225.816851] echo $(hostname) env are INSANE!; [ 225.816851] exit 1; [ 225.816851] fi [ 226.337849] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 226.337849] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 226.337849] if [ $running -ne $mpts ]; then [ 226.337849] echo $(hostname) env are INSANE!; [ 226.337849] exit 1; [ 226.337849] fi [ 226.888846] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 226.888846] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 226.888846] if [ $running -ne $mpts ]; then [ 226.888846] echo $(hostname) env are INSANE!; [ 226.888846] exit 1; [ 226.888846] fi [ 227.769585] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 228.022789] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 228.267910] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 228.585700] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 228.742895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.772676] Lustre: Lustre: Build Version: 2.9.55_33_gce75507 [ 228.791904] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 228.793832] LNet: Accept all, port 7988 [ 228.806142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.829087] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 228.879887] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 228.900442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 228.928037] Lustre: lustre-MDT0000: new disk, initializing [ 228.949646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.956246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 229.118358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 229.372873] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 229.858073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 229.861992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 230.018071] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 230.021353] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 230.158576] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 230.410086] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 230.663836] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 230.914738] Lustre: DEBUG MARKER: sync; sync; sync [ 233.945212] Lustre: lustre-MDT0000: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 236.725701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 236.978983] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 238.377556] Lustre: MGS: Connection restored to d912b86c-cb17-d59e-6637-16906a30a2d4 (at 10.9.5.239@tcp) [ 238.432359] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [ 238.434809] Lustre: Skipped 4 previous similar messages [ 238.564478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.469976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 239.626988] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 246.360087] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 246.620845] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 246.873579] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 247.125453] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 247.323742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 247.343922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.360711] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [ 247.363402] Lustre: Skipped 4 previous similar messages [ 247.370531] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [ 247.373083] Lustre: Skipped 1 previous similar message [ 247.401351] Lustre: lustre-MDT0002: new disk, initializing [ 247.448438] Lustre: lustre-MDT0000: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 247.451153] Lustre: Skipped 2 previous similar messages [ 247.458718] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 247.465923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [ 247.468855] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [ 247.651104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 247.908692] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 248.400795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 248.401613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 248.561570] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 248.563851] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 248.706467] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 248.957399] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 249.207887] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 249.463284] Lustre: DEBUG MARKER: sync; sync; sync [ 252.373244] Lustre: lustre-MDT0002: Connection restored to d912b86c-cb17-d59e-6637-16906a30a2d4 (at 10.9.5.239@tcp) [ 252.376161] Lustre: Skipped 1 previous similar message [ 254.784865] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 255.040357] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 256.243652] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [ 256.246367] Lustre: Skipped 4 previous similar messages [ 256.340300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [ 257.260733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 257.434518] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 265.185878] Lustre: MGS: Connection restored to aca590ce-a6d2-d074-cced-1d7516107dd2 (at 10.9.5.244@tcp) [ 265.188736] Lustre: Skipped 4 previous similar messages [ 266.207586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 266.387430] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 270.065117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [ 274.699962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 274.867804] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 279.853205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:1:ost [ 282.693213] Lustre: lustre-MDT0000: Connection restored to aca590ce-a6d2-d074-cced-1d7516107dd2 (at 10.9.5.244@tcp) [ 282.696418] Lustre: Skipped 4 previous similar messages [ 283.629526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 283.790945] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 292.803070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 292.964768] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 301.977657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 302.140314] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 303.065234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000400000400-0x0000000440000400]:4:ost [ 303.068523] Lustre: Skipped 2 previous similar messages [ 310.990063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 311.154374] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 319.594015] Lustre: lustre-MDT0000: Connection restored to aca590ce-a6d2-d074-cced-1d7516107dd2 (at 10.9.5.244@tcp) [ 319.597431] Lustre: Skipped 7 previous similar messages [ 321.233468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 321.415775] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 331.248517] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 331.418383] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 335.582772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000004c0000400-0x0000000500000400]:7:ost [ 335.586155] Lustre: Skipped 2 previous similar messages [ 338.982137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 339.147843] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 341.011181] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 341.322528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 341.480282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 341.615906] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 341.901880] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 350.293892] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 350.984550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.018976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.054138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.069082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.070375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.072981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-06\/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.330446] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.413226] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.433506] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.489766] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.524757] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 351.537692] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500 [ 352.250385] Lustre: DEBUG MARKER: uname -n [ 352.801110] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 353.458917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 353.522196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 353.538175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 353.543211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 353.574811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 353.589996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing yml_node [ 353.780715] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 353.813510] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 353.854199] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 353.986598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || /usr/sbin/lctl lustre_build_version 2>/dev/null || /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 353.994871] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing yml_node [ 354.010974] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 354.020966] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 354.512525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.55_33_gce75507 [ 354.670097] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.55_33_gce75507 [ 354.802823] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 355.111738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.55_33_gce75507 [ 355.268652] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.55_33_gce75507 [ 355.719232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.55_33_gce75507 [ 355.872174] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.55_33_gce75507 [ 356.071005] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: runtests ============----- Thu Apr 6 10:38:47 UTC 2017 [ 356.227093] Lustre: DEBUG MARKER: -----============= acceptance-small: runtests ============----- Thu Apr 6 10:38:47 UTC 2017 [ 356.527639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration -o sec =========================================================== 10:38:47 \(1491475127\) [ 356.678703] Lustre: DEBUG MARKER: == runtests test complete, duration -o sec =========================================================== 10:38:47 (1491475127) [ 357.399780] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 357.422626] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 357.619713] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 357.657428] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 360.167525] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 360.167525] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 360.167525] if [ $running -ne $mpts ]; then [ 360.167525] echo $(hostname) env are INSANE!; [ 360.167525] exit 1; [ 360.167525] fi [ 360.744792] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 360.744792] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 360.744792] if [ $running -ne $mpts ]; then [ 360.744792] echo $(hostname) env are INSANE!; [ 360.744792] exit 1; [ 360.744792] fi [ 361.340351] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 361.340351] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 361.340351] if [ $running -ne $mpts ]; then [ 361.340351] echo $(hostname) env are INSANE!; [ 361.340351] exit 1; [ 361.340351] fi [ 362.224864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 362.484703] Lustre: DEBUG MARKER: cat /proc/mounts [ 363.273090] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 363.529476] Lustre: DEBUG MARKER: cat /proc/mounts [ 368.556794] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 368.868956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 369.024640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 369.155915] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 369.490602] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 370.106681] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.135468] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.140908] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.162219] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.211063] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.362934] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.393061] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.485089] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.531591] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.552608] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 370.850106] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 371.132884] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 371.651074] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 372.230531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test 1: All Runtests ===================================================================== 10:39:03 \(1491475143\) [ 372.381711] Lustre: DEBUG MARKER: == runtests test 1: All Runtests ===================================================================== 10:39:03 (1491475143) [ 372.698067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark touching \/mnt\/lustre at Thu Apr 6 10:39:03 UTC 2017 \(@1491475143\) [ 372.862475] Lustre: DEBUG MARKER: touching /mnt/lustre at Thu Apr 6 10:39:03 UTC 2017 (@1491475143) [ 373.046980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark create an empty file \/mnt\/lustre\/hosts.10942 [ 373.204344] Lustre: DEBUG MARKER: create an empty file /mnt/lustre/hosts.10942 [ 373.394302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.10942 [ 373.551554] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.10942 [ 373.748147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing \/etc\/hosts and \/mnt\/lustre\/hosts.10942 [ 373.895717] Lustre: DEBUG MARKER: comparing /etc/hosts and /mnt/lustre/hosts.10942 [ 374.172244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.10942 to \/mnt\/lustre\/hosts.10942.ren [ 374.324469] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.10942 to /mnt/lustre/hosts.10942.ren [ 374.514282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.10942 again [ 374.669764] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.10942 again [ 374.861800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.10942 [ 375.022960] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.10942 [ 375.230260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.10942 [ 375.380149] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.10942 [ 375.570464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying \/etc\/hosts to \/mnt\/lustre\/hosts.10942.2 [ 375.722834] Lustre: DEBUG MARKER: copying /etc/hosts to /mnt/lustre/hosts.10942.2 [ 375.926793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.10942.2 to 123 bytes [ 376.081920] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.10942.2 to 123 bytes [ 376.747333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark creating \/mnt\/lustre\/d1.runtests [ 376.929366] Lustre: DEBUG MARKER: creating /mnt/lustre/d1.runtests [ 378.415657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark copying 1000 files from \/etc \/bin to \/mnt\/lustre\/d1.runtests\/etc \/bin at Thu Apr 6 10:39:09 UTC 2017 [ 378.566957] Lustre: DEBUG MARKER: copying 1000 files from /etc /bin to /mnt/lustre/d1.runtests/etc /bin at Thu Apr 6 10:39:09 UTC 2017 [ 386.990554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 1000 newly copied files at Thu Apr 6 10:39:17 UTC 2017 [ 387.157938] Lustre: DEBUG MARKER: comparing 1000 newly copied files at Thu Apr 6 10:39:17 UTC 2017 [ 390.079252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark finished at Thu Apr 6 10:39:21 UTC 2017 \(18\) [ 390.225625] Lustre: DEBUG MARKER: finished at Thu Apr 6 10:39:21 UTC 2017 (18) [ 392.010878] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 392.270418] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 393.215101] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005d8cdb00 x1563924888603984/t0(0) o13->lustre-OST0004-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 393.684206] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 393.686886] Lustre: Skipped 1 previous similar message [ 394.794825] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 394.797472] Lustre: Skipped 9 previous similar messages [ 395.582255] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 395.585092] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.631055] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005df2e100 x1563924888604160/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 395.636521] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 396.799046] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005df2e700 x1563924888604288/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 396.804595] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 398.683588] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 398.686271] Lustre: Skipped 3 previous similar messages [ 399.333233] Lustre: server umount lustre-MDT0000 complete [ 399.463759] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 400.582661] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 400.585500] LustreError: Skipped 1 previous similar message [ 400.587831] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 400.593033] Lustre: Skipped 1 previous similar message [ 403.683458] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.688894] LustreError: Skipped 1 previous similar message [ 404.785069] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.790550] LustreError: Skipped 7 previous similar messages [ 406.594704] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 406.862655] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [ 407.311089] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005d1d9800 x1563924888604848/t0(0) o13->lustre-OST0005-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 407.316892] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 408.683532] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.689136] LustreError: Skipped 3 previous similar messages [ 408.697748] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 409.559032] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475171/real 1491475171] req@ffff88005df2e700 x1563924888604384/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491475178 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 409.567149] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 411.797138] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.803222] LustreError: Skipped 7 previous similar messages [ 413.179950] Lustre: server umount lustre-MDT0002 complete [ 413.307197] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 435.272081] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 435.272081] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 435.272081] if [ $running -ne $mpts ]; then [ 435.272081] echo $(hostname) env are INSANE!; [ 435.272081] exit 1; [ 435.272081] fi [ 435.833703] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 435.833703] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 435.833703] if [ $running -ne $mpts ]; then [ 435.833703] echo $(hostname) env are INSANE!; [ 435.833703] exit 1; [ 435.833703] fi [ 436.413193] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 436.413193] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 436.413193] if [ $running -ne $mpts ]; then [ 436.413193] echo $(hostname) env are INSANE!; [ 436.413193] exit 1; [ 436.413193] fi [ 437.329586] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 437.591095] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 437.840353] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 438.087085] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 438.280880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.310739] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 438.314316] Lustre: Skipped 8 previous similar messages [ 438.346296] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.366121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.522485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 438.781134] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 439.272758] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 439.277869] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 439.430279] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 439.432026] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 439.575845] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 439.831710] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 440.083599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 440.342692] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 440.597700] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 442.361043] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475208/real 1491475208] req@ffff88005dde8300 x1563924888605136/t0(0) o38->lustre-MDT0001-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491475213 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 442.776154] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 442.945270] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 444.361044] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475209/real 1491475209] req@ffff88005ddeaa00 x1563924888605328/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491475214 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 444.370308] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 444.634373] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 444.889731] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 445.145166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 445.346897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.421277] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 445.582593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 445.847145] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 446.338487] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 446.340335] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 446.502083] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 446.502336] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 446.651304] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 446.914124] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 447.170066] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 447.430724] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 447.687705] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 450.034341] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 450.197797] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 450.416133] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475216/real 1491475216] req@ffff88005dde8900 x1563924888606192/t0(0) o8->lustre-OST0007-osc-MDT0002@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491475221 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 450.424836] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 453.527434] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 453.687361] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 455.401036] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475216/real 1491475216] req@ffff88005df2f300 x1563924888605920/t0(0) o8->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491475226 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 455.410070] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 456.569958] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 456.721350] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 459.693144] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 459.943362] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 462.885567] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 463.049451] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 466.051978] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 466.225456] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 469.143312] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 469.306287] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 472.351425] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 472.514687] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 475.346866] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 475.525938] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 477.634788] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 477.803985] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 479.234952] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 479.568497] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 479.720027] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 479.857389] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 485.216605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark comparing 1000 previously copied files [ 485.368300] Lustre: DEBUG MARKER: comparing 1000 previously copied files [ 491.906959] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 492.165402] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 493.286555] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 493.289726] Lustre: Skipped 9 previous similar messages [ 495.413268] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 495.416658] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.455105] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880067e66a00 x1563924888611776/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 495.461372] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 498.396321] Lustre: server umount lustre-MDT0000 complete [ 498.533701] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 500.413525] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 500.416713] LustreError: Skipped 1 previous similar message [ 500.419378] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 500.425140] Lustre: Skipped 1 previous similar message [ 503.282480] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.288557] LustreError: Skipped 3 previous similar messages [ 505.675620] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 505.929072] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [ 507.495054] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475271/real 1491475271] req@ffff880067e67900 x1563924888611968/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491475278 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 507.503498] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 507.506395] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.282985] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 508.286040] Lustre: Skipped 14 previous similar messages [ 509.503047] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062a88600 x1563924888612496/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 509.508889] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 512.182202] Lustre: server umount lustre-MDT0002 complete [ 512.310923] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 534.298560] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 534.298560] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 534.298560] if [ $running -ne $mpts ]; then [ 534.298560] echo $(hostname) env are INSANE!; [ 534.298560] exit 1; [ 534.298560] fi [ 534.864342] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 534.864342] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 534.864342] if [ $running -ne $mpts ]; then [ 534.864342] echo $(hostname) env are INSANE!; [ 534.864342] exit 1; [ 534.864342] fi [ 535.443674] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 535.443674] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 535.443674] if [ $running -ne $mpts ]; then [ 535.443674] echo $(hostname) env are INSANE!; [ 535.443674] exit 1; [ 535.443674] fi [ 536.408422] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 536.669978] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 536.920948] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 537.171341] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 537.319339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.371378] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.376330] LustreError: Skipped 20 previous similar messages [ 537.394088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.540567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 537.799200] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 538.285659] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 538.297361] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 538.450634] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 538.456233] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 538.597503] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 538.850499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 539.104661] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 539.362618] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 539.616300] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 541.821732] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 541.987087] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 542.390225] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475308/real 1491475308] req@ffff880067512400 x1563924888612960/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491475313 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 543.394337] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 543.660158] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 543.915348] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 544.169212] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 544.330003] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.533261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 544.794207] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/sbin:/usr/sbin:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre [ 545.288867] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 545.289388] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 545.458935] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 545.459127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 545.605947] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 545.870714] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 546.128349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 546.391360] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 546.648884] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 548.780531] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 548.939086] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 552.213758] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 552.377411] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.132946] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.296938] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 558.076901] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 558.240263] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 561.040311] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 561.228987] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 564.022578] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 564.183439] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 566.980976] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 567.150294] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 569.079257] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 569.082523] Lustre: Skipped 60 previous similar messages [ 570.001263] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 570.165813] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 573.070997] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 573.243949] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 575.344394] Lustre: DEBUG MARKER: /sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 575.507628] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 576.935989] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 577.251208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 577.414097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 577.560160] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 582.900249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/d1.runtests [ 583.061475] Lustre: DEBUG MARKER: removing /mnt/lustre/d1.runtests [ 585.679626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark renaming \/mnt\/lustre\/hosts.10942.ren to \/mnt\/lustre\/hosts.10942 [ 585.885707] Lustre: DEBUG MARKER: renaming /mnt/lustre/hosts.10942.ren to /mnt/lustre/hosts.10942 [ 586.120910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark truncating \/mnt\/lustre\/hosts.10942 [ 586.280223] Lustre: DEBUG MARKER: truncating /mnt/lustre/hosts.10942 [ 586.482193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark removing \/mnt\/lustre\/hosts.10942 [ 586.643868] Lustre: DEBUG MARKER: removing /mnt/lustre/hosts.10942 [ 586.847259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark verifying \/mnt\/lustre\/hosts.10942.2 is 123 bytes [ 587.006831] Lustre: DEBUG MARKER: verifying /mnt/lustre/hosts.10942.2 is 123 bytes [ 587.233514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running createmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 587.401125] Lustre: DEBUG MARKER: running createmany -d /mnt/lustre/d1.runtests/d 100 [ 587.690873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark running unlinkmany -d \/mnt\/lustre\/d1.runtests\/d 100 [ 587.854529] Lustre: DEBUG MARKER: running unlinkmany -d /mnt/lustre/d1.runtests/d 100 [ 588.161925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark done [ 588.326514] Lustre: DEBUG MARKER: done [ 588.763372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 589.031056] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 589.320705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 590.606856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 591.889693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 593.166294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 594.444506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 595.857080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Space was freed: now 373004kB, was 372944kB. [ 596.013654] Lustre: DEBUG MARKER: Space was freed: now 373004kB, was 372944kB. [ 596.204608] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 597.172997] Lustre: DEBUG MARKER: rc=0; [ 597.172997] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 597.172997] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 597.172997] echo $(hostname -s): $val; [ 597.172997] rc=$val; [ 597.172997] fi; [ 597.172997] exit $rc [ 597.777363] Lustre: DEBUG MARKER: dmesg [ 598.159829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == runtests test complete, duration 242 sec ========================================================== 10:42:49 \(1491475369\) [ 598.310860] Lustre: DEBUG MARKER: == runtests test complete, duration 242 sec ========================================================== 10:42:49 (1491475369) [ 598.565712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mmp ============----- Thu Apr 6 10:42:49 UTC 2017 [ 598.720044] Lustre: DEBUG MARKER: -----============= acceptance-small: mmp ============----- Thu Apr 6 10:42:49 UTC 2017 [ 598.999082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration -o sec ================================================================ 10:42:49 \(1491475369\) [ 599.150784] Lustre: DEBUG MARKER: == mmp test complete, duration -o sec ================================================================ 10:42:49 (1491475369) [ 600.152441] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 600.409816] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 601.471089] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005d08db00 x1563924888636448/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 601.475994] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 602.136225] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 602.138886] Lustre: Skipped 15 previous similar messages [ 604.376257] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 604.376654] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.383270] LustreError: Skipped 1 previous similar message [ 606.645181] Lustre: server umount lustre-MDT0000 complete [ 606.779877] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 609.376252] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.381085] LustreError: Skipped 2 previous similar messages [ 613.942557] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 614.209857] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [ 616.463031] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475380/real 1491475380] req@ffff88006292f900 x1563924888636848/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491475387 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 616.470124] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [ 616.472640] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.454843] Lustre: server umount lustre-MDT0002 complete [ 620.580968] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 646.297806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 646.446796] Lustre: DEBUG MARKER: excepting tests: [ 646.608128] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 646.902068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on MDS, enabling it manually... [ 647.050122] Lustre: DEBUG MARKER: MMP is not enabled on MDS, enabling it manually... [ 647.181327] Lustre: DEBUG MARKER: tune2fs -O mmp /dev/lvm-Role_MDS/P1 [ 647.810100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MMP is not enabled on OSS, enabling it manually... [ 647.957952] Lustre: DEBUG MARKER: MMP is not enabled on OSS, enabling it manually... [ 648.380806] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 648.958144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 1: two mounts at the same time =========================================================== 10:43:39 \(1491475419\) [ 649.107760] Lustre: DEBUG MARKER: == mmp test 1: two mounts at the same time =========================================================== 10:43:39 (1491475419) [ 649.295198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_1 failover pair is needed [ 649.447596] Lustre: DEBUG MARKER: SKIP: mmp test_1 failover pair is needed [ 649.626356] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 650.511434] Lustre: DEBUG MARKER: rc=0; [ 650.511434] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 650.511434] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 650.511434] echo $(hostname -s): $val; [ 650.511434] rc=$val; [ 650.511434] fi; [ 650.511434] exit $rc [ 651.076874] Lustre: DEBUG MARKER: dmesg [ 651.454146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 2: one mount delayed by mmp update interval ============================================== 10:43:42 \(1491475422\) [ 651.626077] Lustre: DEBUG MARKER: == mmp test 2: one mount delayed by mmp update interval ============================================== 10:43:42 (1491475422) [ 651.810350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_2 failover pair is needed [ 651.960030] Lustre: DEBUG MARKER: SKIP: mmp test_2 failover pair is needed [ 652.141827] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 653.004422] Lustre: DEBUG MARKER: rc=0; [ 653.004422] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 653.004422] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 653.004422] echo $(hostname -s): $val; [ 653.004422] rc=$val; [ 653.004422] fi; [ 653.004422] exit $rc [ 653.573986] Lustre: DEBUG MARKER: dmesg [ 653.962089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 3: one mount delayed by 2x mmp check interval ============================================ 10:43:44 \(1491475424\) [ 654.111548] Lustre: DEBUG MARKER: == mmp test 3: one mount delayed by 2x mmp check interval ============================================ 10:43:44 (1491475424) [ 654.291821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_3 failover pair is needed [ 654.439901] Lustre: DEBUG MARKER: SKIP: mmp test_3 failover pair is needed [ 654.616410] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 655.469417] Lustre: DEBUG MARKER: rc=0; [ 655.469417] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 655.469417] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 655.469417] echo $(hostname -s): $val; [ 655.469417] rc=$val; [ 655.469417] fi; [ 655.469417] exit $rc [ 656.050698] Lustre: DEBUG MARKER: dmesg [ 656.429642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 4: one mount delayed by \> 2x mmp check interval ========================================== 10:43:47 \(1491475427\) [ 656.577515] Lustre: DEBUG MARKER: == mmp test 4: one mount delayed by > 2x mmp check interval ========================================== 10:43:47 (1491475427) [ 656.775219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_4 failover pair is needed [ 656.917516] Lustre: DEBUG MARKER: SKIP: mmp test_4 failover pair is needed [ 657.102374] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 657.990883] Lustre: DEBUG MARKER: rc=0; [ 657.990883] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 657.990883] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 657.990883] echo $(hostname -s): $val; [ 657.990883] rc=$val; [ 657.990883] fi; [ 657.990883] exit $rc [ 658.572740] Lustre: DEBUG MARKER: dmesg [ 658.942501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 5: mount during unmount of the first filesystem ========================================== 10:43:49 \(1491475429\) [ 659.092235] Lustre: DEBUG MARKER: == mmp test 5: mount during unmount of the first filesystem ========================================== 10:43:49 (1491475429) [ 659.271465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_5 failover pair is needed [ 659.417513] Lustre: DEBUG MARKER: SKIP: mmp test_5 failover pair is needed [ 659.599716] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 660.461780] Lustre: DEBUG MARKER: rc=0; [ 660.461780] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 660.461780] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 660.461780] echo $(hostname -s): $val; [ 660.461780] rc=$val; [ 660.461780] fi; [ 660.461780] exit $rc [ 661.038990] Lustre: DEBUG MARKER: dmesg [ 661.408946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 6: mount after clean unmount ============================================================= 10:43:52 \(1491475432\) [ 661.563727] Lustre: DEBUG MARKER: == mmp test 6: mount after clean unmount ============================================================= 10:43:52 (1491475432) [ 661.751899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_6 failover pair is needed [ 661.903323] Lustre: DEBUG MARKER: SKIP: mmp test_6 failover pair is needed [ 662.084646] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 662.962466] Lustre: DEBUG MARKER: rc=0; [ 662.962466] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 662.962466] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 662.962466] echo $(hostname -s): $val; [ 662.962466] rc=$val; [ 662.962466] fi; [ 662.962466] exit $rc [ 663.531747] Lustre: DEBUG MARKER: dmesg [ 663.905121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 7: mount after reboot ==================================================================== 10:43:54 \(1491475434\) [ 664.049222] Lustre: DEBUG MARKER: == mmp test 7: mount after reboot ==================================================================== 10:43:54 (1491475434) [ 664.231507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: mmp test_7 failover pair is needed [ 664.388143] Lustre: DEBUG MARKER: SKIP: mmp test_7 failover pair is needed [ 664.561934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 665.426787] Lustre: DEBUG MARKER: rc=0; [ 665.426787] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 665.426787] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 665.426787] echo $(hostname -s): $val; [ 665.426787] rc=$val; [ 665.426787] fi; [ 665.426787] exit $rc [ 665.997324] Lustre: DEBUG MARKER: dmesg [ 666.390313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 8: mount during e2fsck =================================================================== 10:43:57 \(1491475437\) [ 666.552666] Lustre: DEBUG MARKER: == mmp test 8: mount during e2fsck =================================================================== 10:43:57 (1491475437) [ 666.682418] Lustre: DEBUG MARKER: debugfs -c -R dump_mmp /dev/lvm-Role_MDS/P1 2>/tmp/mmp.debugfs.msg [ 666.935420] Lustre: DEBUG MARKER: tune2fs -E mmp_update_interval=30 /dev/lvm-Role_MDS/P1 [ 678.210585] Lustre: DEBUG MARKER: e2fsck -fy /dev/lvm-Role_MDS/P1 [ 683.209903] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 685.120900] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 685.361349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 685.598966] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 685.739102] LustreError: 27853:0:(osd_handler.c:7003:osd_mount()) lustre-MDT0000-osd: can't mount /dev/mapper/lvm--Role_MDS-P1: -16 [ 685.742997] LustreError: 27853:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000-osd failed (-16) [ 685.745292] LustreError: 27853:0:(obd_mount.c:199:lustre_start_simple()) lustre-MDT0000-osd setup error -16 [ 685.747657] LustreError: 27853:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -16 [ 685.751696] LustreError: 27853:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-16) [ 685.876086] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 739.652825] Lustre: DEBUG MARKER: tune2fs -E mmp_update_interval=5 /dev/lvm-Role_MDS/P1 [ 935.323676] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 936.231637] Lustre: DEBUG MARKER: rc=0; [ 936.231637] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 936.231637] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 936.231637] echo $(hostname -s): $val; [ 936.231637] rc=$val; [ 936.231637] fi; [ 936.231637] exit $rc [ 936.814901] Lustre: DEBUG MARKER: dmesg [ 937.200695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 9: mount after aborted e2fsck ============================================================ 10:48:28 \(1491475708\) [ 937.347827] Lustre: DEBUG MARKER: == mmp test 9: mount after aborted e2fsck ============================================================ 10:48:28 (1491475708) [ 937.482615] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 937.742187] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 937.988947] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 938.236491] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 938.392571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.412433] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 938.414838] Lustre: Skipped 8 previous similar messages [ 938.443573] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.448265] LustreError: Skipped 20 previous similar messages [ 938.463691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.466080] Lustre: Skipped 1 previous similar message [ 938.611458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 938.867553] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 939.353740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 939.355223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 939.510520] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 939.512789] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 939.656151] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 939.910062] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 940.164286] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 940.423382] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 940.676518] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 942.781295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 942.942363] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 943.459043] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475709/real 1491475709] req@ffff88005d08f300 x1563924888637696/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491475714 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 943.843743] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 944.095147] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 944.224841] Lustre: Failing over lustre-MDT0000 [ 944.239204] LustreError: 28686:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0001-osp-MDT0000:osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 944.244290] LustreError: 28686:0:(lod_sub_object.c:959:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 944.249348] LustreError: 28686:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 getting update log failed: rc = -5 [ 944.254370] LustreError: 28686:0:(lod_dev.c:419:lod_sub_recovery_thread()) Skipped 1 previous similar message [ 950.365448] Lustre: server umount lustre-MDT0000 complete [ 950.494268] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 957.562379] Lustre: DEBUG MARKER: /usr/lib64/lustre/tests/mmp_mark.sh /dev/lvm-Role_MDS/P1 [ 969.901466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_MDS\/P1 on mds1... [ 970.047398] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_MDS/P1 on mds1... [ 970.181318] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 970.438430] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 970.688137] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 970.934464] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 971.077678] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:312: fsck is running on the filesystem [ 971.080598] LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect:312: MMP failure info: last update time: 1491475739, last update node: trevis-41vm7.trevis.hpdd.intel.com, last update device: /dev/lvm-Role_MDS/P1 [ 971.080598] [ 971.088469] LustreError: 30330:0:(osd_handler.c:7003:osd_mount()) lustre-MDT0000-osd: can't mount /dev/mapper/lvm--Role_MDS-P1: -22 [ 971.093403] LustreError: 30330:0:(obd_config.c:574:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 971.096181] LustreError: 30330:0:(obd_mount.c:199:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 971.099020] LustreError: 30330:0:(obd_mount_server.c:1806:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -22 [ 971.103923] LustreError: 30330:0:(obd_mount.c:1502:lustre_fill_super()) Unable to mount (-22) [ 971.229595] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 971.477419] Lustre: DEBUG MARKER: tune2fs -f -E clear-mmp /dev/lvm-Role_MDS/P1 [ 984.060672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_OSS\/P1 on ost1... [ 984.207270] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_OSS/P1 on ost1... [ 985.716974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 986.594652] Lustre: DEBUG MARKER: rc=0; [ 986.594652] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 986.594652] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 986.594652] echo $(hostname -s): $val; [ 986.594652] rc=$val; [ 986.594652] fi; [ 986.594652] exit $rc [ 987.169740] Lustre: DEBUG MARKER: dmesg [ 987.547315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test 10: e2fsck with mounted filesystem ======================================================= 10:49:18 \(1491475758\) [ 987.695138] Lustre: DEBUG MARKER: == mmp test 10: e2fsck with mounted filesystem ======================================================= 10:49:18 (1491475758) [ 987.881133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_MDS\/P1 on mds1... [ 988.028613] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_MDS/P1 on mds1... [ 988.160636] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 988.428480] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 988.688416] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 988.932333] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 989.090522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.161426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.312132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 989.567105] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 990.057223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 990.057721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 990.219986] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 990.220523] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 990.361630] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 990.613876] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 990.867659] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 991.129169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 991.398505] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 991.660122] Lustre: DEBUG MARKER: e2fsck -fn /dev/lvm-Role_MDS/P1 [ 992.073489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Mounting \/dev\/lvm-Role_OSS\/P1 on ost1... [ 992.227458] Lustre: DEBUG MARKER: Mounting /dev/lvm-Role_OSS/P1 on ost1... [ 994.192882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 994.353749] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 995.581570] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 995.834437] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 995.962089] Lustre: Failing over lustre-MDT0000 [ 995.964400] LustreError: 31293:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0001-osp-MDT0000:osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 995.969088] LustreError: 31293:0:(osp_object.c:527:osp_attr_get()) Skipped 2 previous similar messages [ 995.971803] LustreError: 31293:0:(lod_sub_object.c:959:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 995.971819] LustreError: 31294:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0002-osp-MDT0000 getting update log failed: rc = -5 [ 995.971820] LustreError: 31294:0:(lod_dev.c:419:lod_sub_recovery_thread()) Skipped 1 previous similar message [ 995.984307] LustreError: 31293:0:(lod_sub_object.c:959:lod_sub_prep_llog()) Skipped 4 previous similar messages [ 1002.053504] Lustre: server umount lustre-MDT0000 complete [ 1002.179521] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1009.259926] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1010.128340] Lustre: DEBUG MARKER: rc=0; [ 1010.128340] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1010.128340] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1010.128340] echo $(hostname -s): $val; [ 1010.128340] rc=$val; [ 1010.128340] fi; [ 1010.128340] exit $rc [ 1010.699698] Lustre: DEBUG MARKER: dmesg [ 1011.169926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/lvm-Role_MDS\/P1 on mds1 manually... [ 1011.317325] Lustre: DEBUG MARKER: Disabling MMP on /dev/lvm-Role_MDS/P1 on mds1 manually... [ 1011.445744] Lustre: DEBUG MARKER: tune2fs -O ^mmp /dev/lvm-Role_MDS/P1 [ 1022.715991] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 | grep mmp [ 1023.014759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disabling MMP on \/dev\/lvm-Role_OSS\/P1 on ost1 manually... [ 1023.163916] Lustre: DEBUG MARKER: Disabling MMP on /dev/lvm-Role_OSS/P1 on ost1 manually... [ 1034.903739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mmp test complete, duration 436 sec =============================================================== 10:50:05 \(1491475805\) [ 1035.054900] Lustre: DEBUG MARKER: == mmp test complete, duration 436 sec =============================================================== 10:50:05 (1491475805) [ 1037.376540] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 1037.376540] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 1037.376540] if [ $running -ne $mpts ]; then [ 1037.376540] echo $(hostname) env are INSANE!; [ 1037.376540] exit 1; [ 1037.376540] fi [ 1037.920792] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 1037.920792] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 1037.920792] if [ $running -ne $mpts ]; then [ 1037.920792] echo $(hostname) env are INSANE!; [ 1037.920792] exit 1; [ 1037.920792] fi [ 1038.481798] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 1038.481798] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 1038.481798] if [ $running -ne $mpts ]; then [ 1038.481798] echo $(hostname) env are INSANE!; [ 1038.481798] exit 1; [ 1038.481798] fi [ 1039.400688] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 1039.658575] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1039.908771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1040.164129] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1040.330978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.399376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.548300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1040.817796] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 1041.309363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1041.310548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1041.471090] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1041.472653] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1041.613908] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1041.873414] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1042.123306] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1042.379439] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1042.632434] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1044.779183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1044.946756] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1046.357573] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 1046.623082] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 1046.878356] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 1047.133592] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 1047.292928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.501593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1047.758938] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust [ 1048.253129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1048.253830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1048.420246] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1048.420247] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1048.570147] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1048.825225] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1049.088596] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1049.351484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 1049.612459] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1051.768281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1051.922352] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1055.210928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1055.366363] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1058.106217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1058.267633] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1061.059309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1061.232559] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1064.063528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1064.224755] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1067.064360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1067.224387] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1070.028617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1070.197405] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1073.037664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1073.208154] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1076.008347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1076.173684] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1078.259423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1078.421746] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1079.851902] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 1080.164705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 1080.328779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1080.460931] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 1085.812709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-single ============----- Thu Apr 6 10:50:56 UTC 2017 [ 1085.975796] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Apr 6 10:50:56 UTC 2017 [ 1086.276639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test complete, duration -o sec ====================================================== 10:50:57 \(1491475857\) [ 1086.429401] Lustre: DEBUG MARKER: == replay-single test complete, duration -o sec ====================================================== 10:50:57 (1491475857) [ 1086.807772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 61d [ 1086.956948] Lustre: DEBUG MARKER: excepting tests: 61d [ 1087.149253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 44b [ 1087.305909] Lustre: DEBUG MARKER: skipping tests SLOW=no: 44b [ 1087.939469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 1087.939500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 1088.114133] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 1088.201784] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 1090.667215] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 1090.667215] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 1090.667215] if [ $running -ne $mpts ]; then [ 1090.667215] echo $(hostname) env are INSANE!; [ 1090.667215] exit 1; [ 1090.667215] fi [ 1091.242294] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 1091.242294] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 1091.242294] if [ $running -ne $mpts ]; then [ 1091.242294] echo $(hostname) env are INSANE!; [ 1091.242294] exit 1; [ 1091.242294] fi [ 1091.827663] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 1091.827663] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 1091.827663] if [ $running -ne $mpts ]; then [ 1091.827663] echo $(hostname) env are INSANE!; [ 1091.827663] exit 1; [ 1091.827663] fi [ 1092.703311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1092.959929] Lustre: DEBUG MARKER: cat /proc/mounts [ 1093.754783] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 1094.015553] Lustre: DEBUG MARKER: cat /proc/mounts [ 1099.019390] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 1099.322997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 1099.477069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1099.614584] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 1099.948482] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1100.702826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1100.704210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1100.717944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1100.743443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1100.753551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1100.928805] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1101.057653] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1101.084525] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1101.087420] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1101.091193] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1101.376669] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 1101.663865] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1102.179831] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1102.773289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0a: empty replay =============================================================== 10:51:13 \(1491475873\) [ 1102.927530] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =============================================================== 10:51:13 (1491475873) [ 1103.064631] Lustre: DEBUG MARKER: sync; sync; sync [ 1103.909475] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1104.165408] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1104.298500] LustreError: 6929:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1104.336550] Turning device dm-0 (0xfc00000) read-only [ 1104.468097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1104.596315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1104.737322] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1104.993330] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1105.124169] Lustre: Failing over lustre-MDT0000 [ 1105.171561] Removing read-only on unknown block (0xfc00000) [ 1105.178835] Lustre: server umount lustre-MDT0000 complete [ 1105.310421] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1106.855452] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.861335] LustreError: Skipped 15 previous similar messages [ 1107.343190] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1107.343198] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.343199] Lustre: Skipped 2 previous similar messages [ 1107.354464] LustreError: Skipped 2 previous similar messages [ 1113.447036] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491475877/real 1491475877] req@ffff88006292ea00 x1563924888646496/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491475884 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1113.455168] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 47 previous similar messages [ 1113.457983] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1115.625258] Lustre: DEBUG MARKER: hostname [ 1115.926287] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1116.179340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1116.431588] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1116.582254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.463261] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0ca63d to 0x61ac6f141c0caf82 [ 1119.511433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.514039] Lustre: Skipped 1 previous similar message [ 1119.686136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1119.945204] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1120.448684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1120.448727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1120.613795] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1120.614894] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1120.763109] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1121.018908] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1121.274430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1121.539767] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1122.295409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1122.308259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1122.474588] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1122.558807] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1123.786554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1124.496208] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1124.514656] Lustre: lustre-MDT0000: Recovery over after 0:01, of 5 clients 5 recovered and 0 were evicted. [ 1124.721815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1124.830882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1124.894269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1125.027639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1126.440636] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1127.329309] Lustre: DEBUG MARKER: rc=0; [ 1127.329309] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1127.329309] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1127.329309] echo $(hostname -s): $val; [ 1127.329309] rc=$val; [ 1127.329309] fi; [ 1127.329309] exit $rc [ 1127.931531] Lustre: DEBUG MARKER: dmesg [ 1128.346212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0b: ensure object created after recover exists. \(3284\) ========================= 10:51:39 \(1491475899\) [ 1128.506739] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================= 10:51:39 (1491475899) [ 1129.535747] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 1129.538720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1129.543860] Lustre: Skipped 1 previous similar message [ 1141.479297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1141.649857] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1143.066786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 1143.073669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 1143.295211] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 1143.309295] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 1145.558192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 1145.599954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 1145.737623] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 1145.841372] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 1147.230032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1148.108589] Lustre: DEBUG MARKER: rc=0; [ 1148.108589] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1148.108589] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1148.108589] echo $(hostname -s): $val; [ 1148.108589] rc=$val; [ 1148.108589] fi; [ 1148.108589] exit $rc [ 1148.702459] Lustre: DEBUG MARKER: dmesg [ 1149.123395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0c: check replay-barrier ======================================================= 10:52:00 \(1491475920\) [ 1149.284007] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier ======================================================= 10:52:00 (1491475920) [ 1149.422643] Lustre: DEBUG MARKER: sync; sync; sync [ 1150.304425] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1150.561867] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1150.692796] LustreError: 9834:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1150.703137] Turning device dm-0 (0xfc00000) read-only [ 1150.833214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1150.963406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1151.139734] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1151.393553] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1151.525150] Lustre: Failing over lustre-MDT0000 [ 1151.572033] Removing read-only on unknown block (0xfc00000) [ 1151.705395] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1161.497063] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.017156] Lustre: DEBUG MARKER: hostname [ 1162.327246] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1162.583076] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1162.833451] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1162.991549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.501285] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0caf82 to 0x61ac6f141c0cbca2 [ 1167.698705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1167.956546] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1168.452555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1168.452943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1168.613823] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1168.616709] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1168.761353] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1168.786807] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1169.021239] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1169.283219] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1169.547881] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1169.713065] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 1172.541768] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1174.713148] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1179.713140] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 1184.713201] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 1189.712897] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:45 [ 1199.713098] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:35 [ 1199.718935] Lustre: Skipped 1 previous similar message [ 1219.713132] Lustre: lustre-MDT0000: Denying connection for new client 11fcc07d-7898-b54f-f4ec-4d256211bf5c(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:15 [ 1219.719311] Lustre: Skipped 3 previous similar messages [ 1234.911065] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1234.913907] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1234.973803] Lustre: lustre-MDT0000: Recovery over after 1:06, of 5 clients 4 recovered and 1 was evicted. [ 1241.161161] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1242.066942] Lustre: DEBUG MARKER: rc=0; [ 1242.066942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1242.066942] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1242.066942] echo $(hostname -s): $val; [ 1242.066942] rc=$val; [ 1242.066942] fi; [ 1242.066942] exit $rc [ 1242.684198] Lustre: DEBUG MARKER: dmesg [ 1243.067073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0d: expired recovery with no clients =========================================== 10:53:34 \(1491476014\) [ 1243.223382] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients =========================================== 10:53:34 (1491476014) [ 1243.355749] Lustre: DEBUG MARKER: sync; sync; sync [ 1244.203589] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1244.459172] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1244.591533] LustreError: 11805:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1244.632320] Turning device dm-0 (0xfc00000) read-only [ 1244.760037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1244.889264] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1245.058485] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1245.311602] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1245.441528] Lustre: Failing over lustre-MDT0000 [ 1245.488713] Removing read-only on unknown block (0xfc00000) [ 1245.495797] Lustre: server umount lustre-MDT0000 complete [ 1245.498008] Lustre: Skipped 1 previous similar message [ 1245.627529] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1247.541217] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1247.543955] LustreError: Skipped 3 previous similar messages [ 1247.546142] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.550786] Lustre: Skipped 3 previous similar messages [ 1254.639052] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.949982] Lustre: DEBUG MARKER: hostname [ 1256.256604] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1256.511369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1256.765040] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1256.923361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.643227] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cbca2 to 0x61ac6f141c0cc283 [ 1260.707118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.709516] Lustre: Skipped 1 previous similar message [ 1260.864183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1261.118777] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1261.624687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1261.627400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1261.787428] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1261.787429] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1261.941747] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1262.198912] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1262.455255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1262.716217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1262.881660] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1262.884442] Lustre: lustre-MDT0000: Denying connection for new client d79285a1-022b-efbb-4d47-137374cebe6d(at 10.9.5.237@tcp), waiting for 5 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 1262.889809] Lustre: Skipped 3 previous similar messages [ 1265.689659] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1327.881176] Lustre: lustre-MDT0000: Denying connection for new client d79285a1-022b-efbb-4d47-137374cebe6d(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:01 [ 1327.886875] Lustre: Skipped 12 previous similar messages [ 1329.375056] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1329.377687] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1329.398974] Lustre: lustre-MDT0000: Recovery over after 1:07, of 5 clients 4 recovered and 1 was evicted. [ 1334.329348] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1335.231581] Lustre: DEBUG MARKER: rc=0; [ 1335.231581] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1335.231581] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1335.231581] echo $(hostname -s): $val; [ 1335.231581] rc=$val; [ 1335.231581] fi; [ 1335.231581] exit $rc [ 1335.829272] Lustre: DEBUG MARKER: dmesg [ 1336.223800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 1: simple create =============================================================== 10:55:07 \(1491476107\) [ 1336.373554] Lustre: DEBUG MARKER: == replay-single test 1: simple create =============================================================== 10:55:07 (1491476107) [ 1336.508459] Lustre: DEBUG MARKER: sync; sync; sync [ 1337.349778] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1337.605348] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1337.747037] LustreError: 13779:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1337.787378] Turning device dm-0 (0xfc00000) read-only [ 1337.916532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1338.046180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1338.188205] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1338.445628] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1338.577944] Lustre: Failing over lustre-MDT0000 [ 1338.625439] Removing read-only on unknown block (0xfc00000) [ 1338.758657] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1340.689210] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1340.692100] LustreError: Skipped 1 previous similar message [ 1340.694188] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.699034] Lustre: Skipped 1 previous similar message [ 1347.783057] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1349.072209] Lustre: DEBUG MARKER: hostname [ 1349.382153] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1349.637478] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1349.891395] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1350.061199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.787334] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cc283 to 0x61ac6f141c0cc864 [ 1353.976504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1354.233302] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1354.738459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1354.738460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1354.900362] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1354.901498] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1354.908884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1355.066799] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1355.325304] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1355.585083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1355.850785] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1356.632969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.632969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.835451] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.862554] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1358.821545] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1358.839663] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1359.080921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1359.192098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1359.238234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1359.392334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1360.741613] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1361.624747] Lustre: DEBUG MARKER: rc=0; [ 1361.624747] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1361.624747] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1361.624747] echo $(hostname -s): $val; [ 1361.624747] rc=$val; [ 1361.624747] fi; [ 1361.624747] exit $rc [ 1362.222283] Lustre: DEBUG MARKER: dmesg [ 1362.616418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 2a: touch ====================================================================== 10:55:33 \(1491476133\) [ 1362.769361] Lustre: DEBUG MARKER: == replay-single test 2a: touch ====================================================================== 10:55:33 (1491476133) [ 1362.925590] Lustre: DEBUG MARKER: sync; sync; sync [ 1363.845115] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1364.104979] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1364.246187] LustreError: 16030:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1364.284064] Turning device dm-0 (0xfc00000) read-only [ 1364.413290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1364.543863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1364.697101] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1364.954841] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1365.087202] Lustre: Failing over lustre-MDT0000 [ 1365.136681] Removing read-only on unknown block (0xfc00000) [ 1365.277432] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1366.439392] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.444324] LustreError: Skipped 153 previous similar messages [ 1375.593801] Lustre: DEBUG MARKER: hostname [ 1375.902558] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1376.156448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1376.412763] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1376.572191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.591123] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cc864 to 0x61ac6f141c0cce4c [ 1376.778134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1377.038243] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1377.453258] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1377.541152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1377.551881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1377.705770] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1377.713821] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1377.856214] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1378.129067] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1378.388362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1378.649403] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1379.408418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.421577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.612848] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.694721] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1381.626415] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1381.657238] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1381.865188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1381.994342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1382.020461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1382.185456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1383.516864] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1384.434710] Lustre: DEBUG MARKER: rc=0; [ 1384.434710] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1384.434710] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1384.434710] echo $(hostname -s): $val; [ 1384.434710] rc=$val; [ 1384.434710] fi; [ 1384.434710] exit $rc [ 1385.047129] Lustre: DEBUG MARKER: dmesg [ 1385.462277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 2b: touch ====================================================================== 10:55:56 \(1491476156\) [ 1385.645552] Lustre: DEBUG MARKER: == replay-single test 2b: touch ====================================================================== 10:55:56 (1491476156) [ 1385.801559] Lustre: DEBUG MARKER: sync; sync; sync [ 1386.680860] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1386.937670] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1387.105982] LustreError: 18280:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1387.115298] Turning device dm-0 (0xfc00000) read-only [ 1387.247809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1387.380216] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1387.525671] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1387.782411] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1387.960044] Removing read-only on unknown block (0xfc00000) [ 1388.094465] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1398.403080] Lustre: DEBUG MARKER: hostname [ 1398.706497] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1398.961103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1399.214940] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1399.375368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.391361] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.395726] LustreError: Skipped 1 previous similar message [ 1399.398247] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cce4c to 0x61ac6f141c0cd41f [ 1399.585566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1399.844982] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1400.350786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1400.354741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1400.510046] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1400.516279] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1400.659916] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1400.922043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1401.178714] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1401.442037] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1401.804408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1402.207638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.227292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.427336] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.501232] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1404.431708] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1404.452431] Lustre: lustre-MDT0000: Recovery over after 0:03, of 5 clients 5 recovered and 0 were evicted. [ 1404.638262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1404.781933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1404.794625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1404.980345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1406.327559] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1407.216244] Lustre: DEBUG MARKER: rc=0; [ 1407.216244] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1407.216244] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1407.216244] echo $(hostname -s): $val; [ 1407.216244] rc=$val; [ 1407.216244] fi; [ 1407.216244] exit $rc [ 1407.815317] Lustre: DEBUG MARKER: dmesg [ 1408.217394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3a: replay failed open\(O_DIRECTORY\) ============================================ 10:56:19 \(1491476179\) [ 1408.386102] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ============================================ 10:56:19 (1491476179) [ 1408.523350] Lustre: DEBUG MARKER: sync; sync; sync [ 1409.439492] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1409.712275] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1409.851864] LustreError: 20538:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1409.862907] Turning device dm-0 (0xfc00000) read-only [ 1410.000129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1410.136924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1410.403348] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1410.668665] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1410.799583] Lustre: Failing over lustre-MDT0000 [ 1410.802058] Lustre: Skipped 1 previous similar message [ 1410.850718] Removing read-only on unknown block (0xfc00000) [ 1410.985374] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1421.296705] Lustre: DEBUG MARKER: hostname [ 1421.432050] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491476185/real 1491476185] req@ffff88007ac2ad00 x1563924888678016/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491476192 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1421.439129] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 1421.611824] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1421.863489] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1422.116988] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1422.279650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.441185] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cd41f to 0x61ac6f141c0cda54 [ 1427.628410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1427.887191] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1428.381720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1428.393277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1428.543916] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1428.550581] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1428.693493] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1428.884709] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1428.949836] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1429.206112] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1429.470913] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1430.237307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.237333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.461117] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.464350] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1432.476947] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1432.497791] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1432.805277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1432.824214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1433.012273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1433.080516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1434.458530] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1435.335372] Lustre: DEBUG MARKER: rc=0; [ 1435.335372] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1435.335372] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1435.335372] echo $(hostname -s): $val; [ 1435.335372] rc=$val; [ 1435.335372] fi; [ 1435.335372] exit $rc [ 1435.941029] Lustre: DEBUG MARKER: dmesg [ 1436.321328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3b: replay failed open -ENOMEM ================================================= 10:56:47 \(1491476207\) [ 1436.481447] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ================================================= 10:56:47 (1491476207) [ 1436.616224] Lustre: DEBUG MARKER: sync; sync; sync [ 1437.450853] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1437.710553] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1437.858909] Turning device dm-0 (0xfc00000) read-only [ 1437.990673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1438.120510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1438.253475] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000114 [ 1438.386452] Lustre: *** cfs_fail_loc=114, val=0*** [ 1438.514113] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 1438.777117] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1439.033200] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1439.209001] Removing read-only on unknown block (0xfc00000) [ 1439.343130] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1449.648572] Lustre: DEBUG MARKER: hostname [ 1449.951681] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1450.206340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1450.457588] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1450.606571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.895372] Lustre: MGS: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [ 1450.897862] Lustre: Skipped 229 previous similar messages [ 1455.895201] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.238@tcp (not set up) [ 1455.899207] Lustre: Skipped 14 previous similar messages [ 1456.066500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1456.322861] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1456.814211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1456.818438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1456.979984] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1456.980633] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1457.126048] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1457.382174] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1457.639985] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1457.901759] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1458.657612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.660809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.884866] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.898096] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1461.162871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1461.167599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1461.393635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1461.393660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1462.766944] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1463.662220] Lustre: DEBUG MARKER: rc=0; [ 1463.662220] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1463.662220] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1463.662220] echo $(hostname -s): $val; [ 1463.662220] rc=$val; [ 1463.662220] fi; [ 1463.662220] exit $rc [ 1464.254527] Lustre: DEBUG MARKER: dmesg [ 1464.675288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3c: replay failed open -ENOMEM ================================================= 10:57:15 \(1491476235\) [ 1464.825097] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ================================================= 10:57:15 (1491476235) [ 1464.959059] Lustre: DEBUG MARKER: sync; sync; sync [ 1465.843793] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1466.114957] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1466.256928] LustreError: 25187:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1466.259590] LustreError: 25187:0:(osd_handler.c:2176:osd_ro()) Skipped 1 previous similar message [ 1466.298648] Turning device dm-0 (0xfc00000) read-only [ 1466.437642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1466.573939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1466.706623] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000128 [ 1466.841408] Lustre: *** cfs_fail_loc=128, val=0*** [ 1466.968991] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 1467.238715] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1467.494550] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1467.668506] Removing read-only on unknown block (0xfc00000) [ 1467.802254] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1470.913154] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1470.913230] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.913231] Lustre: Skipped 9 previous similar messages [ 1470.922045] LustreError: Skipped 14 previous similar messages [ 1478.105378] Lustre: DEBUG MARKER: hostname [ 1478.405978] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1478.659638] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1478.913188] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1479.063040] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1479.067025] LustreError: Skipped 2 previous similar messages [ 1479.073918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.069295] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cdf8d to 0x61ac6f141c0ce56e [ 1485.073530] Lustre: Skipped 1 previous similar message [ 1485.275089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1485.538096] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1486.038596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1486.039732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1486.199744] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1486.200817] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1486.346332] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1486.600925] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1486.858028] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1487.117372] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1487.882310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.882323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.903938] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1487.906697] Lustre: Skipped 1 previous similar message [ 1488.113838] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1488.135892] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1490.106739] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1490.112282] LustreError: Skipped 1 previous similar message [ 1490.127798] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [ 1490.130895] Lustre: Skipped 1 previous similar message [ 1490.376080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1490.456683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1490.538688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1490.703783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1492.024125] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1492.916193] Lustre: DEBUG MARKER: rc=0; [ 1492.916193] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1492.916193] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1492.916193] echo $(hostname -s): $val; [ 1492.916193] rc=$val; [ 1492.916193] fi; [ 1492.916193] exit $rc [ 1493.515568] Lustre: DEBUG MARKER: dmesg [ 1493.907993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 4a: \|x\| 10 open\(O_CREAT\)s ====================================================== 10:57:44 \(1491476264\) [ 1494.066769] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ====================================================== 10:57:44 (1491476264) [ 1494.226930] Lustre: DEBUG MARKER: sync; sync; sync [ 1495.110423] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1495.384400] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1495.530303] Turning device dm-0 (0xfc00000) read-only [ 1495.668081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1495.807172] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1495.973875] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1496.241202] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1496.371432] Lustre: Failing over lustre-MDT0000 [ 1496.373505] Lustre: Skipped 2 previous similar messages [ 1496.421870] Removing read-only on unknown block (0xfc00000) [ 1496.553571] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1506.861543] Lustre: DEBUG MARKER: hostname [ 1507.162926] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1507.416108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1507.668952] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1507.828680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.332450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1513.590901] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1514.079573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1514.082485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1514.237462] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1514.240162] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1514.382658] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1514.641668] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1514.896942] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1515.155642] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1515.908231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1515.911319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.105692] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.145275] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1518.400958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1518.434143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1518.633437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1518.637074] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1520.051839] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1520.943922] Lustre: DEBUG MARKER: rc=0; [ 1520.943922] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1520.943922] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1520.943922] echo $(hostname -s): $val; [ 1520.943922] rc=$val; [ 1520.943922] fi; [ 1520.943922] exit $rc [ 1521.536115] Lustre: DEBUG MARKER: dmesg [ 1521.940110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 4b: \|x\| rm 10 files ============================================================ 10:58:12 \(1491476292\) [ 1522.102959] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ============================================================ 10:58:12 (1491476292) [ 1522.254513] Lustre: DEBUG MARKER: sync; sync; sync [ 1523.094843] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1523.351643] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1523.504431] Turning device dm-0 (0xfc00000) read-only [ 1523.633512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1523.764120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1523.919761] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1524.173904] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1524.351487] Removing read-only on unknown block (0xfc00000) [ 1524.358515] Lustre: server umount lustre-MDT0000 complete [ 1524.360677] Lustre: Skipped 7 previous similar messages [ 1524.489906] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1534.800903] Lustre: DEBUG MARKER: hostname [ 1535.112217] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1535.366993] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1535.618903] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1535.767701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.196995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.199313] Lustre: Skipped 7 previous similar messages [ 1541.350112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1541.606606] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1542.110224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1542.110266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1542.272760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1542.275928] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1542.422848] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1542.679987] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1542.935775] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1543.200687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1543.961641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1543.988972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.237274] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.246505] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.493564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1546.496845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1546.708820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1546.723342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1548.117987] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1549.010076] Lustre: DEBUG MARKER: rc=0; [ 1549.010076] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1549.010076] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1549.010076] echo $(hostname -s): $val; [ 1549.010076] rc=$val; [ 1549.010076] fi; [ 1549.010076] exit $rc [ 1549.619966] Lustre: DEBUG MARKER: dmesg [ 1550.031043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 5: \|x\| 220 open\(O_CREAT\) ======================================================= 10:58:41 \(1491476321\) [ 1550.195089] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) ======================================================= 10:58:41 (1491476321) [ 1550.341940] Lustre: DEBUG MARKER: sync; sync; sync [ 1551.236999] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1551.496759] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1551.637791] LustreError: 32095:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1551.640516] LustreError: 32095:0:(osd_handler.c:2176:osd_ro()) Skipped 2 previous similar messages [ 1551.677618] Turning device dm-0 (0xfc00000) read-only [ 1551.807703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1551.939987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1552.448923] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1552.708655] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1552.887069] Removing read-only on unknown block (0xfc00000) [ 1553.020635] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1563.326653] Lustre: DEBUG MARKER: hostname [ 1563.631324] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1563.886696] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1564.141371] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1564.300629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.199390] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0cf868 to 0x61ac6f141c0d1c5d [ 1570.203743] Lustre: Skipped 2 previous similar messages [ 1570.398929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1570.529160] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1570.531821] Lustre: Skipped 2 previous similar messages [ 1570.663276] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1571.156831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1571.158459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1571.319787] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1571.321420] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1571.463874] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1571.719703] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1571.974759] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1572.234076] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1572.995918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1573.012261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1573.218845] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1573.255802] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.236035] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1575.241264] LustreError: Skipped 2 previous similar messages [ 1575.488526] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 1575.491495] Lustre: Skipped 2 previous similar messages [ 1576.487582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1576.600311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1576.738719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1576.948352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1582.134260] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1583.026999] Lustre: DEBUG MARKER: rc=0; [ 1583.026999] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1583.026999] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1583.026999] echo $(hostname -s): $val; [ 1583.026999] rc=$val; [ 1583.026999] fi; [ 1583.026999] exit $rc [ 1583.644272] Lustre: DEBUG MARKER: dmesg [ 1584.061566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 6a: mkdir + contained create =================================================== 10:59:15 \(1491476355\) [ 1584.224460] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create =================================================== 10:59:15 (1491476355) [ 1584.367572] Lustre: DEBUG MARKER: sync; sync; sync [ 1585.345414] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1585.604299] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1585.752378] Turning device dm-0 (0xfc00000) read-only [ 1585.885069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1586.016133] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1586.159094] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1586.417628] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1586.594927] Removing read-only on unknown block (0xfc00000) [ 1586.728052] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1597.038425] Lustre: DEBUG MARKER: hostname [ 1597.343894] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1597.607924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1597.859631] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1598.018454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.288563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1603.546624] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1604.041893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1604.041942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1604.204556] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1604.205249] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1604.350463] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1604.609663] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1604.865283] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1605.125694] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1605.887982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1605.889747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1606.103502] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1606.118313] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1608.368061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1608.508802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1608.565334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1608.679994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1612.052028] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1612.941121] Lustre: DEBUG MARKER: rc=0; [ 1612.941121] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1612.941121] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1612.941121] echo $(hostname -s): $val; [ 1612.941121] rc=$val; [ 1612.941121] fi; [ 1612.941121] exit $rc [ 1613.569869] Lustre: DEBUG MARKER: dmesg [ 1613.991186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 6b: \|X\| rmdir ================================================================== 10:59:44 \(1491476384\) [ 1614.153449] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ================================================================== 10:59:44 (1491476384) [ 1614.301097] Lustre: DEBUG MARKER: sync; sync; sync [ 1615.221799] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1615.482861] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1615.633082] Turning device dm-0 (0xfc00000) read-only [ 1615.768626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1615.905274] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1616.052136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1616.311004] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1616.488623] Removing read-only on unknown block (0xfc00000) [ 1616.622875] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1625.311050] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.315540] LustreError: Skipped 4 previous similar messages [ 1626.941113] Lustre: DEBUG MARKER: hostname [ 1627.245543] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1627.507501] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1627.776422] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1627.938792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.529330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1631.790533] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1632.383785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1632.385229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1632.555138] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1632.555381] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1632.704318] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1632.962730] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1633.218477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1633.480741] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1634.243643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.249817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.482807] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.525568] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1636.724711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1636.811685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1636.882451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1637.038120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1638.408541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1639.295593] Lustre: DEBUG MARKER: rc=0; [ 1639.295593] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1639.295593] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1639.295593] echo $(hostname -s): $val; [ 1639.295593] rc=$val; [ 1639.295593] fi; [ 1639.295593] exit $rc [ 1639.914866] Lustre: DEBUG MARKER: dmesg [ 1640.340698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 7: mkdir \|X\| contained create ================================================== 11:00:11 \(1491476411\) [ 1640.502514] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ================================================== 11:00:11 (1491476411) [ 1640.650594] Lustre: DEBUG MARKER: sync; sync; sync [ 1641.552728] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1641.811536] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1641.947601] Turning device dm-0 (0xfc00000) read-only [ 1642.076935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1642.206631] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1642.351093] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1642.607714] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1642.738100] Lustre: Failing over lustre-MDT0000 [ 1642.740315] Lustre: Skipped 4 previous similar messages [ 1642.788339] Removing read-only on unknown block (0xfc00000) [ 1642.921170] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1653.232124] Lustre: DEBUG MARKER: hostname [ 1653.548875] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1653.806265] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1654.058285] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1654.219023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.426803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1654.687180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1655.208778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1655.219503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1655.370420] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1655.374405] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1655.520712] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1655.777970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1656.035958] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1656.297086] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1657.072626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.088408] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.281865] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.333224] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1659.529937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1659.633566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1659.711569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1659.906148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1661.259914] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1662.163351] Lustre: DEBUG MARKER: rc=0; [ 1662.163351] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1662.163351] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1662.163351] echo $(hostname -s): $val; [ 1662.163351] rc=$val; [ 1662.163351] fi; [ 1662.163351] exit $rc [ 1662.771092] Lustre: DEBUG MARKER: dmesg [ 1663.190838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 8: creat open \|X\| close ======================================================== 11:00:34 \(1491476434\) [ 1663.354773] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ======================================================== 11:00:34 (1491476434) [ 1663.487765] Lustre: DEBUG MARKER: sync; sync; sync [ 1664.330005] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1664.585788] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1664.735094] Turning device dm-0 (0xfc00000) read-only [ 1664.865058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1664.995055] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1665.289450] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1665.545946] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1665.726537] Removing read-only on unknown block (0xfc00000) [ 1665.866888] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1676.187638] Lustre: DEBUG MARKER: hostname [ 1676.501389] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1676.761560] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1677.017468] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1677.167484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.389371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1683.648773] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1684.142133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1684.144806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1684.306040] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1684.306061] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1684.454432] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1684.714524] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1684.971613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1685.233748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1686.001718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.001719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.221613] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.223156] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1688.484065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1688.520732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1688.670511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1688.853405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1690.187928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1691.107359] Lustre: DEBUG MARKER: rc=0; [ 1691.107359] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1691.107359] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1691.107359] echo $(hostname -s): $val; [ 1691.107359] rc=$val; [ 1691.107359] fi; [ 1691.107359] exit $rc [ 1691.722844] Lustre: DEBUG MARKER: dmesg [ 1692.131702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 9: \|X\| create \(same inum\/gen\) ================================================== 11:01:03 \(1491476463\) [ 1692.288842] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ================================================== 11:01:03 (1491476463) [ 1692.419827] Lustre: DEBUG MARKER: sync; sync; sync [ 1693.302043] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1693.550100] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1693.677194] LustreError: 11063:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1693.679914] LustreError: 11063:0:(osd_handler.c:2176:osd_ro()) Skipped 4 previous similar messages [ 1693.689972] Turning device dm-0 (0xfc00000) read-only [ 1693.815570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1693.941718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1694.085208] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1694.335054] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1694.507116] Removing read-only on unknown block (0xfc00000) [ 1694.636551] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1704.939786] Lustre: DEBUG MARKER: hostname [ 1705.245842] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1705.498169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1705.742612] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1705.897667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.319183] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0d8946 to 0x61ac6f141c0d8f66 [ 1711.323346] Lustre: Skipped 4 previous similar messages [ 1711.518358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1711.771155] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1712.027230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1712.029785] Lustre: Skipped 4 previous similar messages [ 1712.261776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1712.261816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1712.428295] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1712.429279] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1712.571496] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1712.821748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1713.070908] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1713.326707] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1714.086306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.115899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.320808] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.346157] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1716.367202] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1716.373033] LustreError: Skipped 4 previous similar messages [ 1716.389195] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 1716.392244] Lustre: Skipped 4 previous similar messages [ 1716.599801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1716.643080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1716.775856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1716.873617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1718.228689] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1719.098713] Lustre: DEBUG MARKER: rc=0; [ 1719.098713] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1719.098713] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1719.098713] echo $(hostname -s): $val; [ 1719.098713] rc=$val; [ 1719.098713] fi; [ 1719.098713] exit $rc [ 1719.687778] Lustre: DEBUG MARKER: dmesg [ 1720.140237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 10: create \|X\| rename unlink =================================================== 11:01:31 \(1491476491\) [ 1720.304570] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink =================================================== 11:01:31 (1491476491) [ 1720.456845] Lustre: DEBUG MARKER: sync; sync; sync [ 1721.281988] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1721.531267] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1721.674237] Turning device dm-0 (0xfc00000) read-only [ 1721.801450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1721.926772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1722.071461] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1722.324319] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1722.497141] Removing read-only on unknown block (0xfc00000) [ 1722.627521] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1732.927717] Lustre: DEBUG MARKER: hostname [ 1733.233429] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1733.484743] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1733.730211] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1733.876115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.892979] LustreError: 11-0: MGC10.9.5.243@tcp: operation obd_ping to node 0@lo failed: rc = -107 [ 1733.895369] LustreError: Skipped 19 previous similar messages [ 1734.082826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1734.332918] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1734.822701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1734.823926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1734.986980] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1734.987256] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1735.132414] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1735.382930] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1735.631742] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1735.885098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1736.630241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.654662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.849729] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.898235] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1740.089676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1740.170190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1740.244282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1740.374965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1741.742339] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1742.627316] Lustre: DEBUG MARKER: rc=0; [ 1742.627316] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1742.627316] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1742.627316] echo $(hostname -s): $val; [ 1742.627316] rc=$val; [ 1742.627316] fi; [ 1742.627316] exit $rc [ 1743.220837] Lustre: DEBUG MARKER: dmesg [ 1743.648640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 11: create open write rename \|X\| create-old-name read ========================== 11:01:54 \(1491476514\) [ 1743.807359] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================== 11:01:54 (1491476514) [ 1743.951799] Lustre: DEBUG MARKER: sync; sync; sync [ 1744.830398] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1745.077363] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1745.225274] Turning device dm-0 (0xfc00000) read-only [ 1745.354723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1745.481849] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1745.628982] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1745.877089] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1746.048900] Removing read-only on unknown block (0xfc00000) [ 1746.180391] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1748.930187] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.935212] Lustre: Skipped 20 previous similar messages [ 1756.492238] Lustre: DEBUG MARKER: hostname [ 1756.798587] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1757.047635] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1757.295065] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1757.450200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.593434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1761.850288] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1762.368942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1762.369454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1762.527950] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1762.527962] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1762.670669] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1762.918983] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1763.167569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1763.429795] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1764.181423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.184971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.404216] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.405940] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.670145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1766.694337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1766.822698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1766.948679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1768.296988] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1769.173124] Lustre: DEBUG MARKER: rc=0; [ 1769.173124] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1769.173124] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1769.173124] echo $(hostname -s): $val; [ 1769.173124] rc=$val; [ 1769.173124] fi; [ 1769.173124] exit $rc [ 1769.754910] Lustre: DEBUG MARKER: dmesg [ 1770.177293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 12: open, unlink \|X\| close ===================================================== 11:02:21 \(1491476541\) [ 1770.333152] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ===================================================== 11:02:21 (1491476541) [ 1770.500252] Lustre: DEBUG MARKER: sync; sync; sync [ 1771.419250] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1771.669819] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1771.814291] Turning device dm-0 (0xfc00000) read-only [ 1771.939965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1772.064078] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1772.206918] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1772.454185] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1772.633846] Removing read-only on unknown block (0xfc00000) [ 1772.765203] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1783.068775] Lustre: DEBUG MARKER: hostname [ 1783.378074] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1783.625718] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1783.870081] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1784.025886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.225890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1784.481097] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1784.969571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1784.976586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1785.125872] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1785.135788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1785.272205] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1785.522527] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1785.772477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1786.025742] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1786.780760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1786.782241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1786.996116] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1787.034401] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1789.331794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1789.483250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1790.237541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1790.384045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1791.712392] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1792.625477] Lustre: DEBUG MARKER: rc=0; [ 1792.625477] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1792.625477] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1792.625477] echo $(hostname -s): $val; [ 1792.625477] rc=$val; [ 1792.625477] fi; [ 1792.625477] exit $rc [ 1793.217298] Lustre: DEBUG MARKER: dmesg [ 1793.620737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 13: open chmod 0 \|x\| write close =============================================== 11:02:44 \(1491476564\) [ 1793.772344] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close =============================================== 11:02:44 (1491476564) [ 1793.918825] Lustre: DEBUG MARKER: sync; sync; sync [ 1794.806296] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1795.055150] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1795.199926] Turning device dm-0 (0xfc00000) read-only [ 1795.325411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1795.450464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1795.588772] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1795.836265] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1796.009947] Removing read-only on unknown block (0xfc00000) [ 1796.140826] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1806.451926] Lustre: DEBUG MARKER: hostname [ 1806.756891] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1807.001961] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1807.246890] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1807.402995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.727151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1811.979813] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1812.498106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1812.499288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1812.661240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1812.663789] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1812.803422] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1813.052920] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1813.300289] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1813.556466] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1814.308696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.319678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.522876] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.550856] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.788553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1816.855450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1817.020496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1817.099466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1818.469772] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1819.342453] Lustre: DEBUG MARKER: rc=0; [ 1819.342453] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1819.342453] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1819.342453] echo $(hostname -s): $val; [ 1819.342453] rc=$val; [ 1819.342453] fi; [ 1819.342453] exit $rc [ 1819.928505] Lustre: DEBUG MARKER: dmesg [ 1820.329347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 14: open\(O_CREAT\), unlink \|X\| close ============================================ 11:03:11 \(1491476591\) [ 1820.483994] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ============================================ 11:03:11 (1491476591) [ 1820.625026] Lustre: DEBUG MARKER: sync; sync; sync [ 1821.553271] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1821.801447] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1821.936407] Turning device dm-0 (0xfc00000) read-only [ 1822.062604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1822.187242] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1822.328985] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1822.578377] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1822.750775] Removing read-only on unknown block (0xfc00000) [ 1822.879965] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1833.187102] Lustre: DEBUG MARKER: hostname [ 1833.499785] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1833.748431] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1833.995095] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1834.149569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.346185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1834.597141] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1835.085307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1835.092046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1835.245591] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1835.247983] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1835.390893] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1835.641840] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1835.889536] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1836.146864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1836.902578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.918327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.108047] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.143071] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.407664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1840.425073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1840.604669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1840.665475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1842.030197] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1842.900625] Lustre: DEBUG MARKER: rc=0; [ 1842.900625] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1842.900625] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1842.900625] echo $(hostname -s): $val; [ 1842.900625] rc=$val; [ 1842.900625] fi; [ 1842.900625] exit $rc [ 1843.487797] Lustre: DEBUG MARKER: dmesg [ 1843.894731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 15: open\(O_CREAT\), unlink \|X\| touch new, close ================================ 11:03:34 \(1491476614\) [ 1844.043914] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ================================ 11:03:34 (1491476614) [ 1844.210303] Lustre: DEBUG MARKER: sync; sync; sync [ 1845.039867] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1845.287068] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1845.430391] Turning device dm-0 (0xfc00000) read-only [ 1845.560083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1845.686772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1845.833275] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1846.080872] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1846.273667] Removing read-only on unknown block (0xfc00000) [ 1846.403795] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1856.705927] Lustre: DEBUG MARKER: hostname [ 1857.007876] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1857.252898] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1857.496950] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1857.641738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.388170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1862.646090] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1863.133830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1863.135747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1863.295248] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1863.297019] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1863.455640] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1863.704835] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1863.952579] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1864.205301] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1864.946925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1864.966028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.180966] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.200945] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1867.460583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1867.551777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1867.645550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1867.758170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1869.097430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1869.980879] Lustre: DEBUG MARKER: rc=0; [ 1869.980879] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1869.980879] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1869.980879] echo $(hostname -s): $val; [ 1869.980879] rc=$val; [ 1869.980879] fi; [ 1869.980879] exit $rc [ 1870.591033] Lustre: DEBUG MARKER: dmesg [ 1870.995426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 16: \|X\| open\(O_CREAT\), unlink, touch new, unlink new ========================== 11:04:01 \(1491476641\) [ 1871.147544] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================== 11:04:01 (1491476641) [ 1871.289108] Lustre: DEBUG MARKER: sync; sync; sync [ 1872.222307] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1872.474337] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1872.618367] Turning device dm-0 (0xfc00000) read-only [ 1872.743524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1872.869163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1873.032663] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1873.279453] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1873.450632] Removing read-only on unknown block (0xfc00000) [ 1873.581024] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1880.624872] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.630329] LustreError: Skipped 791 previous similar messages [ 1883.882613] Lustre: DEBUG MARKER: hostname [ 1884.182934] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1884.429634] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1884.676175] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1884.831661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.848953] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.854907] LustreError: Skipped 9 previous similar messages [ 1885.037209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1885.288554] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1885.782548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1885.785975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1885.941507] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1885.943833] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1886.083545] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1886.332715] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1886.580659] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1886.834465] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1887.587386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.590080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.777189] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.826379] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1890.116531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1890.290669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1891.003321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1891.145948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1892.474537] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1893.364759] Lustre: DEBUG MARKER: rc=0; [ 1893.364759] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1893.364759] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1893.364759] echo $(hostname -s): $val; [ 1893.364759] rc=$val; [ 1893.364759] fi; [ 1893.364759] exit $rc [ 1893.960281] Lustre: DEBUG MARKER: dmesg [ 1894.380128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 17: \|X\| open\(O_CREAT\), \|replay\| close ========================================== 11:04:25 \(1491476665\) [ 1894.535315] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================== 11:04:25 (1491476665) [ 1894.680482] Lustre: DEBUG MARKER: sync; sync; sync [ 1895.533816] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1895.781429] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1895.927268] Turning device dm-0 (0xfc00000) read-only [ 1896.056433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1896.182919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1896.325605] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1896.576027] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1896.749017] Removing read-only on unknown block (0xfc00000) [ 1896.879132] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1907.184232] Lustre: DEBUG MARKER: hostname [ 1907.490311] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1907.746758] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1907.992241] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1908.147095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.752564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1914.014314] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1914.515620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1914.521148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1914.675716] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1914.678730] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1914.820045] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1915.070028] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1915.320469] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1915.574786] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1916.329968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.345649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.507016] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.601970] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1918.892636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1919.061285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1919.747471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1919.915110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1921.245769] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1922.115024] Lustre: DEBUG MARKER: rc=0; [ 1922.115024] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1922.115024] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1922.115024] echo $(hostname -s): $val; [ 1922.115024] rc=$val; [ 1922.115024] fi; [ 1922.115024] exit $rc [ 1922.703342] Lustre: DEBUG MARKER: dmesg [ 1923.214906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 18: open\(O_CREAT\), unlink, touch new, close, touch, unlink ===================== 11:04:54 \(1491476694\) [ 1923.379605] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ===================== 11:04:54 (1491476694) [ 1923.518284] Lustre: DEBUG MARKER: sync; sync; sync [ 1924.398122] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1924.650393] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1924.795284] Turning device dm-0 (0xfc00000) read-only [ 1924.921342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1925.047222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1925.200511] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1925.455630] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1925.584368] Lustre: Failing over lustre-MDT0000 [ 1925.586610] Lustre: Skipped 10 previous similar messages [ 1925.634415] Removing read-only on unknown block (0xfc00000) [ 1925.765300] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1935.839042] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491476699/real 1491476699] req@ffff88007ad33900 x1563924888745776/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491476706 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1935.846347] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [ 1936.065211] Lustre: DEBUG MARKER: hostname [ 1936.361593] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1936.607226] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1936.851480] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1937.007011] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.046264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1942.302741] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1942.815750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1942.818144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1942.973762] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1942.977532] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1943.119095] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1943.371367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1943.620770] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1943.875172] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1944.614300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.627890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.861064] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.861115] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1947.112766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1947.193166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1947.297819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1947.403269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1948.751606] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1949.632457] Lustre: DEBUG MARKER: rc=0; [ 1949.632457] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1949.632457] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1949.632457] echo $(hostname -s): $val; [ 1949.632457] rc=$val; [ 1949.632457] fi; [ 1949.632457] exit $rc [ 1950.254209] Lustre: DEBUG MARKER: dmesg [ 1950.654351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 19: mcreate, open, write, rename =============================================== 11:05:21 \(1491476721\) [ 1950.807751] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename =============================================== 11:05:21 (1491476721) [ 1950.950927] Lustre: DEBUG MARKER: sync; sync; sync [ 1951.851858] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1952.103547] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1952.232099] LustreError: 32683:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1952.234828] LustreError: 32683:0:(osd_handler.c:2176:osd_ro()) Skipped 9 previous similar messages [ 1952.245463] Turning device dm-0 (0xfc00000) read-only [ 1952.373418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1952.505262] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1952.657553] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1952.904914] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1953.076997] Removing read-only on unknown block (0xfc00000) [ 1953.211948] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1963.527364] Lustre: DEBUG MARKER: hostname [ 1963.828640] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1964.080993] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1964.324911] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1964.480845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.681135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1964.932891] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1965.422513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1965.424609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1965.586635] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1965.588750] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1965.728066] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1965.976266] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1966.225609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1966.479236] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1967.224990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.236740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.390088] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.505677] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1969.747318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1969.901682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1970.622793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1970.778843] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1972.107899] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1972.981908] Lustre: DEBUG MARKER: rc=0; [ 1972.981908] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1972.981908] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1972.981908] echo $(hostname -s): $val; [ 1972.981908] rc=$val; [ 1972.981908] fi; [ 1972.981908] exit $rc [ 1973.583733] Lustre: DEBUG MARKER: dmesg [ 1973.988259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20a: \|X\| open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ======= 11:05:44 \(1491476744\) [ 1974.145315] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ======= 11:05:44 (1491476744) [ 1974.274467] Lustre: DEBUG MARKER: sync; sync; sync [ 1975.087092] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1975.334509] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1975.478539] Turning device dm-0 (0xfc00000) read-only [ 1975.604522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1975.728758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1975.875195] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1976.125162] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1976.297499] Removing read-only on unknown block (0xfc00000) [ 1976.426472] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1986.733539] Lustre: DEBUG MARKER: hostname [ 1987.050196] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1987.296062] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1987.539926] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1987.685102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.530333] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c0dcd94 to 0x61ac6f141c0dd401 [ 1992.534645] Lustre: Skipped 10 previous similar messages [ 1992.759082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1993.009162] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1993.498435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1993.500533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1993.663414] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1993.664366] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1993.804605] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1994.055544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1994.305587] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1994.565304] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1995.316182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.365119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.616053] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.717134] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.900606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1995.903437] Lustre: Skipped 10 previous similar messages [ 1997.564266] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1997.569523] LustreError: Skipped 10 previous similar messages [ 1997.599810] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [ 1997.602674] Lustre: Skipped 10 previous similar messages [ 1997.930257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1998.012283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1998.098281] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1998.220787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1999.579956] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2000.457428] Lustre: DEBUG MARKER: rc=0; [ 2000.457428] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2000.457428] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2000.457428] echo $(hostname -s): $val; [ 2000.457428] rc=$val; [ 2000.457428] fi; [ 2000.457428] exit $rc [ 2001.054005] Lustre: DEBUG MARKER: dmesg [ 2001.474076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20b: write, unlink, eviction, replay, \(test mds_cleanup_orphans\) =============== 11:06:12 \(1491476772\) [ 2001.635809] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay, (test mds_cleanup_orphans) =============== 11:06:12 (1491476772) [ 2001.861742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client d79285a1-022b-efbb-4d47-137374cebe6d [ 2002.002360] Lustre: 4553:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d79285a1-022b-efbb-4d47-137374cebe6d at adminstrative request [ 2003.292287] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync 1 [ 2003.568124] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2003.816156] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2004.138367] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2014.448121] Lustre: DEBUG MARKER: hostname [ 2014.754554] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2015.001004] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2015.253577] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2015.400675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.267001] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2021.522633] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2022.042188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2022.050053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2022.206374] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2022.208693] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2022.347642] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2022.609734] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2022.858636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2023.114022] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2023.868260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2023.868703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.080300] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.098617] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2026.371499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2026.389339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2026.597308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2026.608248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2027.974276] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2028.466409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2028.468614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2028.628401] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2028.629967] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2029.038788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 2029.366986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before 373056, after 373056 [ 2029.525442] Lustre: DEBUG MARKER: before 373056, after 373056 [ 2029.710774] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2030.580546] Lustre: DEBUG MARKER: rc=0; [ 2030.580546] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2030.580546] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2030.580546] echo $(hostname -s): $val; [ 2030.580546] rc=$val; [ 2030.580546] fi; [ 2030.580546] exit $rc [ 2031.183522] Lustre: DEBUG MARKER: dmesg [ 2031.588751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20c: check that client eviction does not affect file content =================== 11:06:42 \(1491476802\) [ 2031.764062] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 11:06:42 (1491476802) [ 2031.917257] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client d79285a1-022b-efbb-4d47-137374cebe6d [ 2032.051067] Lustre: 7157:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d79285a1-022b-efbb-4d47-137374cebe6d at adminstrative request [ 2034.498576] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2035.380350] Lustre: DEBUG MARKER: rc=0; [ 2035.380350] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2035.380350] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2035.380350] echo $(hostname -s): $val; [ 2035.380350] rc=$val; [ 2035.380350] fi; [ 2035.380350] exit $rc [ 2035.964305] Lustre: DEBUG MARKER: dmesg [ 2036.377454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 21: \|X\| open\(O_CREAT\), unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 11:06:47 \(1491476807\ [ 2036.528809] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 11:06:47 (1491476807) [ 2036.668020] Lustre: DEBUG MARKER: sync; sync; sync [ 2037.533779] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2037.785332] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2037.930080] Turning device dm-0 (0xfc00000) read-only [ 2038.060234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2038.186917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2038.337507] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2038.596082] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2038.776766] Removing read-only on unknown block (0xfc00000) [ 2038.783843] Lustre: server umount lustre-MDT0000 complete [ 2038.785856] Lustre: Skipped 18 previous similar messages [ 2038.915890] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2049.233057] Lustre: DEBUG MARKER: hostname [ 2049.540512] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2049.789350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2050.041200] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2050.203965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.767273] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 2053.769640] Lustre: Skipped 575 previous similar messages [ 2053.819274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.821750] Lustre: Skipped 18 previous similar messages [ 2053.995193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2054.246921] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2054.751309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2054.751321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2054.911860] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2054.913533] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2055.057396] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2055.304938] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2055.553451] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2055.807137] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2056.557512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2056.559696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2056.780542] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2056.781438] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.061570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2059.084105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2059.277795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2059.346295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2060.725357] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2061.599717] Lustre: DEBUG MARKER: rc=0; [ 2061.599717] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2061.599717] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2061.599717] echo $(hostname -s): $val; [ 2061.599717] rc=$val; [ 2061.599717] fi; [ 2061.599717] exit $rc [ 2062.192249] Lustre: DEBUG MARKER: dmesg [ 2062.635838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 22: open\(O_CREAT\), \|X\| unlink, replay, close \(test mds_cleanup_orphans\) ======== 11:07:13 \(1491476833\) [ 2062.811358] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ======== 11:07:13 (1491476833) [ 2062.949465] Lustre: DEBUG MARKER: sync; sync; sync [ 2063.839994] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2064.090718] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2064.234602] Turning device dm-0 (0xfc00000) read-only [ 2064.359455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2064.484546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2064.625698] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2064.873633] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2065.045991] Removing read-only on unknown block (0xfc00000) [ 2065.174934] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2075.479718] Lustre: DEBUG MARKER: hostname [ 2075.786924] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2076.034464] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2076.278915] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2076.433075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.667195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2076.919259] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2077.408252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2077.410661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2077.568725] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2077.570014] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2077.713919] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2077.962407] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2078.211356] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2078.475183] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2079.218744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.244311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.410116] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.515142] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.675574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2082.783278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2082.834148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2083.005877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2084.342308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2085.210134] Lustre: DEBUG MARKER: rc=0; [ 2085.210134] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2085.210134] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2085.210134] echo $(hostname -s): $val; [ 2085.210134] rc=$val; [ 2085.210134] fi; [ 2085.210134] exit $rc [ 2085.797383] Lustre: DEBUG MARKER: dmesg [ 2086.222765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 23: open\(O_CREAT\), \|X\| unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 11:07:37 \(1491476857\ [ 2086.388565] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 11:07:37 (1491476857) [ 2086.534449] Lustre: DEBUG MARKER: sync; sync; sync [ 2087.424538] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2087.674188] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2087.817545] Turning device dm-0 (0xfc00000) read-only [ 2087.942735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2088.068109] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2088.212977] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2088.462289] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2088.634821] Removing read-only on unknown block (0xfc00000) [ 2088.763572] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2099.068907] Lustre: DEBUG MARKER: hostname [ 2099.378350] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2099.627351] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2099.871907] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2100.027551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.076416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2105.329339] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2105.824332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2105.825676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2105.992955] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2105.998717] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2106.134093] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2106.384399] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2106.633020] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2106.885737] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2107.643657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.682147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.856997] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.909614] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.102612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2110.215763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2110.279349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2110.394912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2111.776708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2112.655133] Lustre: DEBUG MARKER: rc=0; [ 2112.655133] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2112.655133] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2112.655133] echo $(hostname -s): $val; [ 2112.655133] rc=$val; [ 2112.655133] fi; [ 2112.655133] exit $rc [ 2113.262655] Lustre: DEBUG MARKER: dmesg [ 2113.700686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 24: open\(O_CREAT\), replay, unlink, close \(test mds_cleanup_orphans\) ============ 11:08:04 \(1491476884\) [ 2113.894278] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ============ 11:08:04 (1491476884) [ 2114.032692] Lustre: DEBUG MARKER: sync; sync; sync [ 2114.882874] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2115.143105] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2115.283971] Turning device dm-0 (0xfc00000) read-only [ 2115.415711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2115.547332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2115.690492] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2115.949486] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2116.130208] Removing read-only on unknown block (0xfc00000) [ 2116.261291] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2126.568052] Lustre: DEBUG MARKER: hostname [ 2126.866419] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2127.114092] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2127.358057] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2127.513241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.206298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2133.458107] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2133.942936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2133.945375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2134.103635] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2134.105624] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2134.246914] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2134.496493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2134.745412] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2134.999386] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2135.745746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.764405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.974138] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2135.994066] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.254431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2138.276710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2138.476680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2138.548990] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2139.942204] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2140.829673] Lustre: DEBUG MARKER: rc=0; [ 2140.829673] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2140.829673] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2140.829673] echo $(hostname -s): $val; [ 2140.829673] rc=$val; [ 2140.829673] fi; [ 2140.829673] exit $rc [ 2141.419951] Lustre: DEBUG MARKER: dmesg [ 2141.862120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 25: open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ============ 11:08:32 \(1491476912\) [ 2142.021164] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ============ 11:08:32 (1491476912) [ 2142.162051] Lustre: DEBUG MARKER: sync; sync; sync [ 2142.987921] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2143.238429] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2143.382303] Turning device dm-0 (0xfc00000) read-only [ 2143.507893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2143.633193] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2143.771785] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2144.017866] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2144.190514] Removing read-only on unknown block (0xfc00000) [ 2144.322090] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2154.626672] Lustre: DEBUG MARKER: hostname [ 2154.929023] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2155.176369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2155.423740] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2155.568850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.386152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2161.636905] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2162.143325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2162.152775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2162.311742] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2162.312560] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2162.454328] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2162.703435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2162.951049] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2163.205954] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2163.948350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2163.954560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.214303] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.231943] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2166.532518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2166.544568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2166.705644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2166.710867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2168.110867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2169.006370] Lustre: DEBUG MARKER: rc=0; [ 2169.006370] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2169.006370] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2169.006370] echo $(hostname -s): $val; [ 2169.006370] rc=$val; [ 2169.006370] fi; [ 2169.006370] exit $rc [ 2169.597254] Lustre: DEBUG MARKER: dmesg [ 2170.009582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 26: \|X\| open\(O_CREAT\), unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 11:09:01 \(14 [ 2170.168089] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 11:09:01 (1491476941) [ 2170.306423] Lustre: DEBUG MARKER: sync; sync; sync [ 2171.172481] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2171.430701] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2171.577674] Turning device dm-0 (0xfc00000) read-only [ 2171.703536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2171.829309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2171.986038] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2172.232326] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2172.403977] Removing read-only on unknown block (0xfc00000) [ 2172.534638] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2182.838068] Lustre: DEBUG MARKER: hostname [ 2183.134089] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2183.383478] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2183.629683] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2183.785635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.285300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2189.536835] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2190.027320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2190.030652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2190.187390] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2190.190123] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2190.328327] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2190.578722] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2190.827202] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2191.080755] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2191.841926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2191.846384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2192.036048] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2192.103195] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2194.367219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2194.526650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2195.309954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2195.470264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2196.797550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2197.675337] Lustre: DEBUG MARKER: rc=0; [ 2197.675337] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2197.675337] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2197.675337] echo $(hostname -s): $val; [ 2197.675337] rc=$val; [ 2197.675337] fi; [ 2197.675337] exit $rc [ 2198.273705] Lustre: DEBUG MARKER: dmesg [ 2198.719223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 27: \|X\| open\(O_CREAT\), unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 11:09:29 \(1491476969\) [ 2198.884728] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 11:09:29 (1491476969) [ 2199.031382] Lustre: DEBUG MARKER: sync; sync; sync [ 2199.899110] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2200.148625] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2200.293880] Turning device dm-0 (0xfc00000) read-only [ 2200.422147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2200.547197] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2200.700661] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2200.949556] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2201.121795] Removing read-only on unknown block (0xfc00000) [ 2201.253219] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2211.558525] Lustre: DEBUG MARKER: hostname [ 2211.856174] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2212.103585] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2212.348167] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2212.496813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.464879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2217.717778] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2218.207142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2218.208258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2218.367343] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2218.368330] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2218.509302] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2218.759644] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2219.012076] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2219.270403] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2220.019128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.041680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.214866] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.281053] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.492289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2222.574799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2222.681058] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2222.782796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2224.146319] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2225.028561] Lustre: DEBUG MARKER: rc=0; [ 2225.028561] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2225.028561] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2225.028561] echo $(hostname -s): $val; [ 2225.028561] rc=$val; [ 2225.028561] fi; [ 2225.028561] exit $rc [ 2225.627141] Lustre: DEBUG MARKER: dmesg [ 2226.092728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 28: open\(O_CREAT\), \|X\| unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 11:09:57 \(14 [ 2226.269132] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 11:09:57 (1491476997) [ 2226.427500] Lustre: DEBUG MARKER: sync; sync; sync [ 2227.320411] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2227.569860] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2227.714337] Turning device dm-0 (0xfc00000) read-only [ 2227.839919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2227.963771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2228.109655] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2228.358486] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2228.532095] Removing read-only on unknown block (0xfc00000) [ 2228.661287] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2238.972595] Lustre: DEBUG MARKER: hostname [ 2239.273691] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2239.523559] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2239.770273] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2239.924581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.476925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2245.728443] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2246.216968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2246.223478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2246.378240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2246.379794] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2246.521379] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2246.772367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2247.023623] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2247.276040] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2248.065924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.111635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.262182] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.422822] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2250.557880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2250.687347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2250.718129] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2250.892511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2252.239732] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2253.132256] Lustre: DEBUG MARKER: rc=0; [ 2253.132256] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2253.132256] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2253.132256] echo $(hostname -s): $val; [ 2253.132256] rc=$val; [ 2253.132256] fi; [ 2253.132256] exit $rc [ 2253.724452] Lustre: DEBUG MARKER: dmesg [ 2254.157232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 29: open\(O_CREAT\), \|X\| unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 11:10:25 \(1491477025\) [ 2254.307308] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 11:10:25 (1491477025) [ 2254.453252] Lustre: DEBUG MARKER: sync; sync; sync [ 2255.344325] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2255.602915] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2255.737763] Turning device dm-0 (0xfc00000) read-only [ 2255.863242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2255.987881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2256.136159] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2256.387600] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2256.560728] Removing read-only on unknown block (0xfc00000) [ 2256.691403] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2260.308161] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 2260.310887] LustreError: Skipped 48 previous similar messages [ 2266.996818] Lustre: DEBUG MARKER: hostname [ 2267.299660] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2267.557444] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2267.804783] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2267.959022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.159190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2268.410574] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2268.900969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2268.902600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2269.063104] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2269.066622] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2269.204847] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2269.455324] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2269.702956] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2269.958124] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2270.703165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.705260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.891736] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.959452] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.163303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2275.257734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2275.374961] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2275.475369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2276.850388] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2277.743540] Lustre: DEBUG MARKER: rc=0; [ 2277.743540] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2277.743540] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2277.743540] echo $(hostname -s): $val; [ 2277.743540] rc=$val; [ 2277.743540] fi; [ 2277.743540] exit $rc [ 2278.345926] Lustre: DEBUG MARKER: dmesg [ 2278.768507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 30: open\(O_CREAT\) two, unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 11:10:49 \(1491477049\) [ 2278.934884] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 11:10:49 (1491477049) [ 2279.105052] Lustre: DEBUG MARKER: sync; sync; sync [ 2279.930824] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2280.192472] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2280.342775] Turning device dm-0 (0xfc00000) read-only [ 2280.473134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2280.605178] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2280.750087] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2281.007030] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2281.187662] Removing read-only on unknown block (0xfc00000) [ 2281.318519] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2283.007196] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2283.012085] Lustre: Skipped 38 previous similar messages [ 2291.629128] Lustre: DEBUG MARKER: hostname [ 2291.928925] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2292.176859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2292.422134] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2292.573505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.644084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2296.896038] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2297.401089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2297.402822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2297.558382] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2297.561309] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2297.700177] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2297.950399] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2298.201447] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2298.454726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2299.200971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2299.212255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2299.411026] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2299.457260] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2301.716065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2301.720148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2301.942858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2301.982445] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2303.353696] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2304.243531] Lustre: DEBUG MARKER: rc=0; [ 2304.243531] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2304.243531] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2304.243531] echo $(hostname -s): $val; [ 2304.243531] rc=$val; [ 2304.243531] fi; [ 2304.243531] exit $rc [ 2304.846732] Lustre: DEBUG MARKER: dmesg [ 2305.275296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 31: open\(O_CREAT\) two, unlink one, \|X\| unlink one, close two \(test mds_cleanup_orphans\) ====================================================================================================== 11:11:16 \(14914 [ 2305.448548] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ====================================================================================================== 11:11:16 (1491477076) [ 2305.608349] Lustre: DEBUG MARKER: sync; sync; sync [ 2306.511036] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2306.760107] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2306.905052] Turning device dm-0 (0xfc00000) read-only [ 2307.035256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2307.160616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2307.302005] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2307.550265] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2307.724702] Removing read-only on unknown block (0xfc00000) [ 2307.854540] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2318.165267] Lustre: DEBUG MARKER: hostname [ 2318.470465] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2318.722182] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2318.969910] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2319.126199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.319657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2319.573406] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2320.062313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2320.063442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2320.226088] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2320.226089] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2320.368320] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2320.618208] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2320.867796] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2321.120945] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2321.868243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2321.897600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2322.105514] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2322.134902] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2324.415492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2324.579333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2325.342887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2325.493576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2326.816875] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2327.709840] Lustre: DEBUG MARKER: rc=0; [ 2327.709840] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2327.709840] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2327.709840] echo $(hostname -s): $val; [ 2327.709840] rc=$val; [ 2327.709840] fi; [ 2327.709840] exit $rc [ 2328.309631] Lustre: DEBUG MARKER: dmesg [ 2328.721373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 32: close\(\) notices client eviction\; close\(\) after client eviction ============= 11:11:39 \(1491477099\) [ 2328.877053] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ============= 11:11:39 (1491477099) [ 2329.030803] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client d79285a1-022b-efbb-4d47-137374cebe6d [ 2329.164233] Lustre: 31286:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting d79285a1-022b-efbb-4d47-137374cebe6d at adminstrative request [ 2331.612932] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2332.500968] Lustre: DEBUG MARKER: rc=0; [ 2332.500968] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2332.500968] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2332.500968] echo $(hostname -s): $val; [ 2332.500968] rc=$val; [ 2332.500968] fi; [ 2332.500968] exit $rc [ 2333.101818] Lustre: DEBUG MARKER: dmesg [ 2333.581564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 33a: fid seq shouldn\'t be reused after abort recovery ========================== 11:11:44 \(1491477104\) [ 2333.735872] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================== 11:11:44 (1491477104) [ 2333.891137] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2334.150102] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2334.316197] Turning device dm-0 (0xfc00000) read-only [ 2334.447319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2334.579045] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2334.713992] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2334.964022] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2335.134636] Removing read-only on unknown block (0xfc00000) [ 2335.265082] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2335.561811] Lustre: DEBUG MARKER: hostname [ 2335.839551] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2336.086748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2336.331966] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2336.487746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.553424] LustreError: 32262:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2336.555874] LustreError: 32262:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2336.558255] Lustre: 32310:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2336.560702] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2336.709055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2336.966867] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2337.458727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2337.460230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2337.618667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2337.618980] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2337.765765] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2338.014314] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2338.261975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2338.515447] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2341.397842] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2342.286696] Lustre: DEBUG MARKER: rc=0; [ 2342.286696] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2342.286696] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2342.286696] echo $(hostname -s): $val; [ 2342.286696] rc=$val; [ 2342.286696] fi; [ 2342.286696] exit $rc [ 2342.886706] Lustre: DEBUG MARKER: dmesg [ 2343.329472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 33b: test fid seq allocation =================================================== 11:11:54 \(1491477114\) [ 2343.475264] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation =================================================== 11:11:54 (1491477114) [ 2343.610453] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1311 [ 2343.870729] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2344.126643] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2344.332309] Turning device dm-0 (0xfc00000) read-only [ 2344.463874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2344.591315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2344.724242] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2344.974148] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2345.145091] Removing read-only on unknown block (0xfc00000) [ 2345.275931] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2345.575560] Lustre: DEBUG MARKER: hostname [ 2345.849539] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2346.095953] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2346.340792] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2346.496169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.556046] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2346.571801] LustreError: 1787:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2346.574476] LustreError: 1787:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2346.577079] Lustre: 1835:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2346.579987] Lustre: 1835:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2346.582927] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2346.742721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2346.993787] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2347.483312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2347.484289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2347.642669] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2347.647307] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2347.789608] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2348.037694] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2348.285981] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2348.538906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2351.184100] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2351.413704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2352.290884] Lustre: DEBUG MARKER: rc=0; [ 2352.290884] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2352.290884] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2352.290884] echo $(hostname -s): $val; [ 2352.290884] rc=$val; [ 2352.290884] fi; [ 2352.290884] exit $rc [ 2352.889256] Lustre: DEBUG MARKER: dmesg [ 2353.308045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 34: abort recovery before client does replay \(test mds_cleanup_orphans\) ======== 11:12:04 \(1491477124\) [ 2353.460969] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ======== 11:12:04 (1491477124) [ 2353.599022] Lustre: DEBUG MARKER: sync; sync; sync [ 2354.489873] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2354.738563] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2354.883075] Turning device dm-0 (0xfc00000) read-only [ 2355.014430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2355.140855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2355.274094] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2355.523242] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2355.693339] Removing read-only on unknown block (0xfc00000) [ 2355.823147] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2356.122509] Lustre: DEBUG MARKER: hostname [ 2356.394477] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2356.640595] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2356.884129] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2357.039169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.106474] LustreError: 3712:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2357.109159] LustreError: 3712:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2357.111822] Lustre: 3760:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2357.114631] Lustre: 3760:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2357.117296] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2357.270974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2357.529177] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2358.015271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2358.018345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2358.171134] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2358.178209] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2358.316891] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2358.566816] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2358.815036] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2359.068054] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2361.943560] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2362.853236] Lustre: DEBUG MARKER: rc=0; [ 2362.853236] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2362.853236] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2362.853236] echo $(hostname -s): $val; [ 2362.853236] rc=$val; [ 2362.853236] fi; [ 2362.853236] exit $rc [ 2363.447468] Lustre: DEBUG MARKER: dmesg [ 2363.883378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 35: test recovery from llog for unlink op ====================================== 11:12:14 \(1491477134\) [ 2364.038948] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ====================================== 11:12:14 (1491477134) [ 2364.175609] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [ 2364.306264] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2364.308490] LustreError: 3915:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800682d3000 x1563925365765680/t210453397508(0) o36->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 624/456 e 0 to 0 dl 1491477140 ref 1 fl Interpret:/0/0 rc 0/0 [ 2366.450051] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2366.703164] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2367.070141] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2367.378271] Lustre: DEBUG MARKER: hostname [ 2367.657584] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2367.905112] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2368.148905] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2368.303220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.368258] LustreError: 5407:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2368.370919] LustreError: 5407:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2368.373728] Lustre: 5455:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2368.376457] Lustre: 5455:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2368.379034] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2368.526125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2368.776911] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2369.275521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2369.276101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2369.447307] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2369.448337] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2369.593394] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2369.850810] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2370.106480] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2370.371930] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2373.187544] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2374.056318] Lustre: DEBUG MARKER: rc=0; [ 2374.056318] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2374.056318] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2374.056318] echo $(hostname -s): $val; [ 2374.056318] rc=$val; [ 2374.056318] fi; [ 2374.056318] exit $rc [ 2374.646342] Lustre: DEBUG MARKER: dmesg [ 2375.071770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 36: don\'t resend cancel ======================================================== 11:12:26 \(1491477146\) [ 2375.226795] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ======================================================== 11:12:26 (1491477146) [ 2375.359913] Lustre: DEBUG MARKER: sync; sync; sync [ 2376.254408] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2376.516082] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2376.667093] Turning device dm-0 (0xfc00000) read-only [ 2376.797530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2376.929951] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2377.078777] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2377.332402] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2377.504563] Removing read-only on unknown block (0xfc00000) [ 2377.633906] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2387.935095] Lustre: DEBUG MARKER: hostname [ 2388.233174] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2388.486435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2388.734837] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2388.891574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.089241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2389.343569] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2389.836403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2389.842427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2389.999437] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2390.004972] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2390.149675] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2390.396947] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2390.643970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2390.897458] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2394.151540] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2395.030095] Lustre: DEBUG MARKER: rc=0; [ 2395.030095] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2395.030095] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2395.030095] echo $(hostname -s): $val; [ 2395.030095] rc=$val; [ 2395.030095] fi; [ 2395.030095] exit $rc [ 2395.616944] Lustre: DEBUG MARKER: dmesg [ 2396.049741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 37: abort recovery before client does replay \(test mds_cleanup_orphans for directories\) ====================================================================================================== 11:12:47 \(149147716 [ 2396.204055] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ====================================================================================================== 11:12:47 (1491477167) [ 2396.347158] Lustre: DEBUG MARKER: sync; sync; sync [ 2397.182570] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2397.433584] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2397.581555] Turning device dm-0 (0xfc00000) read-only [ 2397.711679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2397.835292] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2397.964274] Lustre: DEBUG MARKER: dmesg -c [ 2398.241956] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2398.489752] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2398.661796] Removing read-only on unknown block (0xfc00000) [ 2398.791548] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2399.092924] Lustre: DEBUG MARKER: hostname [ 2399.374965] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2399.633972] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2399.884851] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2400.045999] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.063670] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.068017] LustreError: Skipped 21 previous similar messages [ 2400.119360] LustreError: 9251:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2400.121915] LustreError: 9251:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2400.124727] Lustre: 9299:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2400.127211] Lustre: 9299:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2400.129590] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2400.278399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2400.536175] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2401.036474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2401.039753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2401.201772] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2401.202116] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2401.344273] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2401.595164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2401.844430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2402.098900] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2404.848819] Lustre: DEBUG MARKER: dmesg [ 2405.175589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2406.065793] Lustre: DEBUG MARKER: rc=0; [ 2406.065793] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2406.065793] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2406.065793] echo $(hostname -s): $val; [ 2406.065793] rc=$val; [ 2406.065793] fi; [ 2406.065793] exit $rc [ 2406.660055] Lustre: DEBUG MARKER: dmesg [ 2407.060744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 38: test recovery from unlink llog \(test llog_gen_rec\) ========================= 11:12:58 \(1491477178\) [ 2407.221738] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================= 11:12:58 (1491477178) [ 2408.521524] Lustre: DEBUG MARKER: sync; sync; sync [ 2409.397266] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2409.646867] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2409.780056] Turning device dm-0 (0xfc00000) read-only [ 2409.906038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2410.033976] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2410.175252] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2410.426662] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2410.601633] Removing read-only on unknown block (0xfc00000) [ 2410.731710] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2421.037087] Lustre: DEBUG MARKER: hostname [ 2421.346972] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2421.595692] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2421.841622] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2421.986766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.847862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2428.101993] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2428.596041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2428.599624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2428.759219] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2428.760781] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2428.902848] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2429.155584] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2429.404841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2429.660255] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2430.407390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2430.425619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2430.633770] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2430.666277] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2432.945090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2433.094815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2433.901396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2434.050220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2437.831205] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2438.706294] Lustre: DEBUG MARKER: rc=0; [ 2438.706294] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2438.706294] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2438.706294] echo $(hostname -s): $val; [ 2438.706294] rc=$val; [ 2438.706294] fi; [ 2438.706294] exit $rc [ 2439.314347] Lustre: DEBUG MARKER: dmesg [ 2439.714457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 39: test recovery from unlink llog \(test llog_gen_rec\) ========================= 11:13:30 \(1491477210\) [ 2439.869444] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================= 11:13:30 (1491477210) [ 2440.865154] Lustre: DEBUG MARKER: sync; sync; sync [ 2441.758330] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2442.008929] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2442.153678] Turning device dm-0 (0xfc00000) read-only [ 2442.280158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2442.405421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2442.994690] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2443.242399] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2443.368923] Lustre: Failing over lustre-MDT0000 [ 2443.371163] Lustre: Skipped 21 previous similar messages [ 2443.440730] Removing read-only on unknown block (0xfc00000) [ 2443.570623] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2453.879412] Lustre: DEBUG MARKER: hostname [ 2454.181169] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2454.428921] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2454.673840] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2454.829137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.043720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2455.297603] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2455.820795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.821538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.985556] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.987960] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2456.131785] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2456.391341] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.644220] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.909174] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2457.659936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2457.678523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2457.859807] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2457.899955] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2461.125040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2461.191869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2461.335519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2461.417600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2465.226060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2466.121563] Lustre: DEBUG MARKER: rc=0; [ 2466.121563] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2466.121563] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2466.121563] echo $(hostname -s): $val; [ 2466.121563] rc=$val; [ 2466.121563] fi; [ 2466.121563] exit $rc [ 2466.712166] Lustre: DEBUG MARKER: dmesg [ 2467.142099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 11:13:58 \(1491477238\) [ 2467.302723] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 11:13:58 (1491477238) [ 2467.488006] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2467.650297] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2467.858596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2468.754519] Lustre: DEBUG MARKER: rc=0; [ 2468.754519] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2468.754519] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2468.754519] echo $(hostname -s): $val; [ 2468.754519] rc=$val; [ 2468.754519] fi; [ 2468.754519] exit $rc [ 2469.354695] Lustre: DEBUG MARKER: dmesg [ 2469.767998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 11:14:00 \(1491477240\) [ 2469.927348] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 11:14:00 (1491477240) [ 2470.105123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2470.105123] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2470.105123] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2470.370680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2470.370680] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2470.370680] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2470.633474] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2470.884139] Lustre: DEBUG MARKER: lctl get_param -n devices [ 2471.131957] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 12 deactivate [ 2471.256357] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2471.396588] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 12 activate [ 2471.704460] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2472.582956] Lustre: DEBUG MARKER: rc=0; [ 2472.582956] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2472.582956] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2472.582956] echo $(hostname -s): $val; [ 2472.582956] rc=$val; [ 2472.582956] fi; [ 2472.582956] exit $rc [ 2473.172393] Lustre: DEBUG MARKER: dmesg [ 2473.581729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 42: recovery after ost failure ================================================= 11:14:04 \(1491477244\) [ 2473.736451] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ================================================= 11:14:04 (1491477244) [ 2478.579927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2491.626452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2491.798686] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2533.028538] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2533.811463] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2534.695689] Lustre: DEBUG MARKER: rc=0; [ 2534.695689] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2534.695689] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2534.695689] echo $(hostname -s): $val; [ 2534.695689] rc=$val; [ 2534.695689] fi; [ 2534.695689] exit $rc [ 2535.292762] Lustre: DEBUG MARKER: dmesg [ 2535.717721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 43: mds osc import failure during recovery\; don\'t LBUG ========================= 11:15:06 \(1491477306\) [ 2535.875452] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================= 11:15:06 (1491477306) [ 2536.036654] Lustre: DEBUG MARKER: sync; sync; sync [ 2536.937097] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2537.183933] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2537.320958] LustreError: 16506:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 2537.323300] LustreError: 16506:0:(osd_handler.c:2176:osd_ro()) Skipped 19 previous similar messages [ 2537.360110] Turning device dm-0 (0xfc00000) read-only [ 2537.485039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2537.610071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2538.005985] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2538.256384] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2538.428925] Removing read-only on unknown block (0xfc00000) [ 2538.559271] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2539.875286] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.879853] LustreError: Skipped 789 previous similar messages [ 2547.047043] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491477310/real 1491477310] req@ffff88005ef77900 x1563924888879312/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491477317 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2547.053967] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 2548.859995] Lustre: DEBUG MARKER: hostname [ 2549.171542] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2549.426506] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2549.671177] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2549.827974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.056150] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c101c34 to 0x61ac6f141c1197ae [ 2553.060558] Lustre: Skipped 20 previous similar messages [ 2553.293949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2553.548599] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2554.042418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2554.046900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2554.204872] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2554.205736] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2554.346082] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2554.598360] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2554.849262] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2555.108137] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2555.956423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2555.959503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.186919] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.211165] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2557.059780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 2557.062877] Lustre: Skipped 15 previous similar messages [ 2558.133076] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2558.138465] LustreError: Skipped 26 previous similar messages [ 2558.154414] Lustre: lustre-MDT0000: Recovery over after 0:01, of 5 clients 5 recovered and 0 were evicted. [ 2558.157668] Lustre: Skipped 15 previous similar messages [ 2558.412401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2558.458424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2558.626984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2558.688102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2565.169077] LustreError: 17089:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2570.070775] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2570.959475] Lustre: DEBUG MARKER: rc=0; [ 2570.959475] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2570.959475] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2570.959475] echo $(hostname -s): $val; [ 2570.959475] rc=$val; [ 2570.959475] fi; [ 2570.959475] exit $rc [ 2571.550301] Lustre: DEBUG MARKER: dmesg [ 2571.965808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 44a: race in target handle connect ============================================= 11:15:42 \(1491477342\) [ 2572.149584] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ============================================= 11:15:42 (1491477342) [ 2572.310189] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 2572.559673] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 2572.857058] Lustre: DEBUG MARKER: lctl set_param at_max=40 [ 2573.126567] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2573.376662] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2573.504808] LustreError: 1389:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2582.955203] LustreError: 1390:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2582.958025] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2582.959290] LustreError: 1389:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2582.959300] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2583.097294] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2583.348805] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2583.478318] LustreError: 1390:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2592.955184] LustreError: 1391:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2592.957609] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2592.957609] LustreError: 1390:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2592.957609] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2593.092191] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2593.344064] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2593.475182] LustreError: 1391:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2602.955416] LustreError: 18806:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2602.957674] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2602.960148] LustreError: 1391:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2603.088314] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2603.341829] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2603.476897] LustreError: 18806:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2612.955216] LustreError: 3915:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2612.957544] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2612.959867] Lustre: Skipped 1 previous similar message [ 2612.960428] LustreError: 18806:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2612.960437] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2613.092474] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2613.345700] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2613.474549] LustreError: 3915:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2622.955152] LustreError: 1390:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2622.957636] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2622.957640] LustreError: 3915:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2622.957649] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2623.091063] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2623.340681] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2623.471142] LustreError: 1390:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2632.955331] LustreError: 1391:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2632.957697] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2632.957701] LustreError: 1390:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2632.957710] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2633.091185] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2633.341992] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2643.081211] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2643.334145] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2643.462546] LustreError: 1391:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2643.464877] LustreError: 1391:0:(libcfs_fail.h:165:cfs_race()) Skipped 1 previous similar message [ 2652.955183] LustreError: 1389:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2652.957880] LustreError: 1389:0:(libcfs_fail.h:170:cfs_race()) Skipped 1 previous similar message [ 2652.960196] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2652.961068] LustreError: 1391:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2652.961070] LustreError: 1391:0:(libcfs_fail.h:168:cfs_race()) Skipped 1 previous similar message [ 2652.961081] Lustre: lustre-MDT0000: Export ffff880063fd4400 already connecting from 10.9.5.237@tcp [ 2652.970330] Lustre: Skipped 2 previous similar messages [ 2653.100135] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2653.348245] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2662.955155] Lustre: lustre-MDT0000: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [ 2662.958033] Lustre: Skipped 532 previous similar messages [ 2663.090097] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2663.339453] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2673.084879] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2673.380658] Lustre: DEBUG MARKER: lctl set_param at_max=600 [ 2673.703228] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2674.574951] Lustre: DEBUG MARKER: rc=0; [ 2674.574951] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2674.574951] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2674.574951] echo $(hostname -s): $val; [ 2674.574951] rc=$val; [ 2674.574951] fi; [ 2674.574951] exit $rc [ 2675.169642] Lustre: DEBUG MARKER: dmesg [ 2675.569045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_44b skipping SLOW test 44b [ 2675.733831] Lustre: DEBUG MARKER: SKIP: replay-single test_44b skipping SLOW test 44b [ 2675.943330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 44c: race in target handle connect ============================================= 11:17:26 \(1491477446\) [ 2676.099830] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ============================================= 11:17:26 (1491477446) [ 2676.249043] Lustre: DEBUG MARKER: sync; sync; sync [ 2677.053212] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2677.301463] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2677.436134] Turning device dm-0 (0xfc00000) read-only [ 2677.561888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2677.688723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2677.881168] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000712 [ 2678.133143] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2678.381559] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2678.553260] Removing read-only on unknown block (0xfc00000) [ 2678.559914] Lustre: server umount lustre-MDT0000 complete [ 2678.561688] Lustre: Skipped 19 previous similar messages [ 2678.686931] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2678.983967] Lustre: DEBUG MARKER: hostname [ 2679.256684] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2679.505642] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2679.749333] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2679.905352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.958201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.960365] Lustre: Skipped 19 previous similar messages [ 2679.973195] LustreError: 21396:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2679.975513] LustreError: 21396:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2679.978746] Lustre: 21444:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2679.981208] Lustre: 21444:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2679.983510] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2680.132394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2680.382720] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2680.869148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2680.873231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2681.033098] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2681.033773] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2681.189479] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2681.438291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2681.685133] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2681.940144] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2684.713210] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2684.963940] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2685.297803] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2693.441109] LustreError: 1388:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [ 2695.601789] Lustre: DEBUG MARKER: hostname [ 2695.908785] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2696.166181] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2696.416997] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2696.572594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.640462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2699.900639] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2700.397141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.398629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.555974] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.559845] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.699178] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2700.947975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2701.198663] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2701.455430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2702.198300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2702.214988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2702.386865] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2702.468966] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2705.680078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2705.754328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2705.858164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2705.984822] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2707.369485] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2708.264661] Lustre: DEBUG MARKER: rc=0; [ 2708.264661] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2708.264661] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2708.264661] echo $(hostname -s): $val; [ 2708.264661] rc=$val; [ 2708.264661] fi; [ 2708.264661] exit $rc [ 2708.857778] Lustre: DEBUG MARKER: dmesg [ 2709.280373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 45: Handle failed close ======================================================== 11:18:00 \(1491477480\) [ 2709.445776] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ======================================================== 11:18:00 (1491477480) [ 2709.467354] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 2709.470119] Lustre: Skipped 3 previous similar messages [ 2710.674538] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2711.559112] Lustre: DEBUG MARKER: rc=0; [ 2711.559112] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2711.559112] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2711.559112] echo $(hostname -s): $val; [ 2711.559112] rc=$val; [ 2711.559112] fi; [ 2711.559112] exit $rc [ 2712.149433] Lustre: DEBUG MARKER: dmesg [ 2712.593052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 46: Don\'t leak file handle after open resend \(3325\) ============================ 11:18:03 \(1491477483\) [ 2712.752439] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ============================ 11:18:03 (1491477483) [ 2712.915344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x122 [ 2713.045987] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2713.048135] LustreError: 1401:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800778ebc00 x1563925365938464/t0(0) o700->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 264/248 e 0 to 0 dl 1491477489 ref 1 fl Interpret:/0/0 rc 0/0 [ 2720.174215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2720.432742] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2720.686660] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2721.011135] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2731.315922] Lustre: DEBUG MARKER: hostname [ 2731.614523] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2731.860574] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2732.104108] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2732.259528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.461731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2732.711045] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2733.198688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2733.199428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2733.360217] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2733.362597] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2733.504813] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2733.756846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2734.006213] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2734.261917] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2735.016755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2735.019181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2735.211167] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2735.257225] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2737.556579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2737.713407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2738.455058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2738.612747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2739.931972] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2740.806548] Lustre: DEBUG MARKER: rc=0; [ 2740.806548] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2740.806548] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2740.806548] echo $(hostname -s): $val; [ 2740.806548] rc=$val; [ 2740.806548] fi; [ 2740.806548] exit $rc [ 2741.402307] Lustre: DEBUG MARKER: dmesg [ 2741.822831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 47: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 11:18:32 \(1491477512\) [ 2741.976557] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) =========================== 11:18:32 (1491477512) [ 2754.981957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2755.156214] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2756.605741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2756.640026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2756.821214] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2756.885299] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2758.053734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2758.157453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2758.209192] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2758.377740] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2821.140681] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2822.024194] Lustre: DEBUG MARKER: rc=0; [ 2822.024194] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2822.024194] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2822.024194] echo $(hostname -s): $val; [ 2822.024194] rc=$val; [ 2822.024194] fi; [ 2822.024194] exit $rc [ 2822.624923] Lustre: DEBUG MARKER: dmesg [ 2823.026836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 48: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 11:19:54 \(1491477594\) [ 2823.192158] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) =========================== 11:19:54 (1491477594) [ 2823.349174] Lustre: DEBUG MARKER: sync; sync; sync [ 2824.200657] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2824.450909] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2824.594964] Turning device dm-0 (0xfc00000) read-only [ 2824.720572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2824.846672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2825.015557] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2825.264251] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2825.435283] Removing read-only on unknown block (0xfc00000) [ 2825.566268] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2835.875531] Lustre: DEBUG MARKER: hostname [ 2836.174945] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2836.425780] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2836.676118] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2836.835481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.295568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2840.547405] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2841.038301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2841.038464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2841.202802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2841.203037] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2841.344183] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2841.593085] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2841.841178] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2842.095214] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2845.182821] LustreError: 27839:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0002-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2845.536148] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2846.412416] Lustre: DEBUG MARKER: rc=0; [ 2846.412416] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2846.412416] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2846.412416] echo $(hostname -s): $val; [ 2846.412416] rc=$val; [ 2846.412416] fi; [ 2846.412416] exit $rc [ 2847.017675] Lustre: DEBUG MARKER: dmesg [ 2847.521966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 50: Double OSC recovery, don\'t LASSERT \(3812\) ================================== 11:20:18 \(1491477618\) [ 2847.674053] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ================================== 11:20:18 (1491477618) [ 2847.819428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2847.819428] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2847.819428] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2848.092529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2848.092529] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2848.092529] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2848.357151] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2848.609186] Lustre: DEBUG MARKER: lctl get_param -n devices [ 2848.859133] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 11 recover [ 2849.115844] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 11 recover [ 2854.428893] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2855.314125] Lustre: DEBUG MARKER: rc=0; [ 2855.314125] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2855.314125] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2855.314125] echo $(hostname -s): $val; [ 2855.314125] rc=$val; [ 2855.314125] fi; [ 2855.314125] exit $rc [ 2855.912877] Lustre: DEBUG MARKER: dmesg [ 2856.320262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 52: time out lock replay \(3764\) ================================================ 11:20:27 \(1491477627\) [ 2856.473753] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ================================================ 11:20:27 (1491477627) [ 2856.605369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2856.605369] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2856.605369] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2856.881642] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000157 [ 2857.138360] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2857.388073] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2857.702958] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2868.001753] Lustre: DEBUG MARKER: hostname [ 2868.307473] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2868.553357] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2868.797626] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2868.944147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.444808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2873.695532] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2874.198473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2874.198535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2874.361536] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2874.362897] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2874.505768] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2874.756832] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2875.002264] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2875.256076] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2876.011707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.023572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.201219] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.266650] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2878.283876] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2878.286284] LustreError: 30303:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880077faa100 x1563925365950304/t0(0) o101->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 328/344 e 1 to 0 dl 1491477663 ref 1 fl Complete:/40/0 rc 0/0 [ 2894.047036] Lustre: lustre-MDT0000: Recovery already passed deadline 0:47. If you do not want to wait more, please abort the recovery by force. [ 2894.534601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 18 sec [ 2894.626698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 18 sec [ 2894.705411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 18 sec [ 2894.821871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 18 sec [ 2896.149816] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 2896.471818] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2897.359392] Lustre: DEBUG MARKER: rc=0; [ 2897.359392] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2897.359392] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2897.359392] echo $(hostname -s): $val; [ 2897.359392] rc=$val; [ 2897.359392] fi; [ 2897.359392] exit $rc [ 2897.968245] Lustre: DEBUG MARKER: dmesg [ 2898.375127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53a: \|X\| close request while two MDC requests in flight ======================== 11:21:09 \(1491477669\) [ 2898.539471] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ======================== 11:21:09 (1491477669) [ 2899.693781] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 2899.829364] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2899.960283] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2900.224272] Lustre: DEBUG MARKER: sync; sync; sync [ 2900.779306] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2901.037135] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2901.177034] Turning device dm-0 (0xfc00000) read-only [ 2901.305998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2901.434915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2901.576679] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2901.837570] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2902.013449] Removing read-only on unknown block (0xfc00000) [ 2902.147196] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2903.283219] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 2903.285855] LustreError: Skipped 51 previous similar messages [ 2903.288011] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.292573] Lustre: Skipped 39 previous similar messages [ 2912.463596] Lustre: DEBUG MARKER: hostname [ 2912.781586] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2913.030100] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2913.275353] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2913.420335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.491423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2918.749151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2919.250851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2919.251709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2919.413682] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2919.415773] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2919.557445] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2919.808722] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2920.058142] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2920.314365] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2921.062542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2921.066619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2921.264564] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2921.316159] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2923.592302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2923.745277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2924.521633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2924.668940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2926.011882] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2926.896754] Lustre: DEBUG MARKER: rc=0; [ 2926.896754] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2926.896754] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2926.896754] echo $(hostname -s): $val; [ 2926.896754] rc=$val; [ 2926.896754] fi; [ 2926.896754] exit $rc [ 2927.496625] Lustre: DEBUG MARKER: dmesg [ 2927.915903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53b: \|X\| open request while two MDC requests in flight ========================= 11:21:38 \(1491477698\) [ 2928.067642] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================= 11:21:38 (1491477698) [ 2928.221155] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 2928.352740] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2929.476471] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2929.729629] Lustre: DEBUG MARKER: sync; sync; sync [ 2930.211862] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2930.461972] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2930.639969] Turning device dm-0 (0xfc00000) read-only [ 2930.769963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2930.895492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2931.033401] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2931.280914] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2931.452364] Removing read-only on unknown block (0xfc00000) [ 2931.582177] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2941.886472] Lustre: DEBUG MARKER: hostname [ 2942.196778] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2942.450844] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2942.698218] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2942.854592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.812730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2946.066258] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2946.557537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2946.557545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2946.726052] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2946.733756] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2946.880678] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2947.136467] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2947.385580] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2947.639508] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2948.385239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.390323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.597205] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.610519] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2950.856712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2950.948264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2951.067503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2951.172969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2952.576572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2953.460091] Lustre: DEBUG MARKER: rc=0; [ 2953.460091] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2953.460091] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2953.460091] echo $(hostname -s): $val; [ 2953.460091] rc=$val; [ 2953.460091] fi; [ 2953.460091] exit $rc [ 2954.056326] Lustre: DEBUG MARKER: dmesg [ 2954.483579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53c: \|X\| open request and close request while two MDC requests in flight ======= 11:22:05 \(1491477725\) [ 2954.632687] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ======= 11:22:05 (1491477725) [ 2954.781188] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 2954.911524] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2956.032645] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 2956.287942] Lustre: DEBUG MARKER: sync; sync; sync [ 2956.619414] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2956.879513] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2957.014621] Turning device dm-0 (0xfc00000) read-only [ 2957.145968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2957.278261] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2957.423155] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2957.682557] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2957.861812] Removing read-only on unknown block (0xfc00000) [ 2957.996531] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2968.305053] Lustre: DEBUG MARKER: hostname [ 2968.614304] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2968.861949] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2969.109104] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2969.269386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.580782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2973.832684] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2974.331325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2974.334682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2974.493447] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2974.496617] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2974.636316] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2974.886785] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2975.135284] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2975.392239] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2980.560816] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2980.882602] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2981.780338] Lustre: DEBUG MARKER: rc=0; [ 2981.780338] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2981.780338] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2981.780338] echo $(hostname -s): $val; [ 2981.780338] rc=$val; [ 2981.780338] fi; [ 2981.780338] exit $rc [ 2982.386034] Lustre: DEBUG MARKER: dmesg [ 2982.806235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53d: close reply while two MDC requests in flight ============================== 11:22:33 \(1491477753\) [ 2982.959968] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ============================== 11:22:33 (1491477753) [ 2984.132235] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 2984.260250] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2984.262456] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2984.264565] LustreError: 1392:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005d84fc00 x1563925365956736/t270582939665(0) o35->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 512/456 e 0 to 0 dl 1491477760 ref 1 fl Interpret:/0/0 rc 0/0 [ 2984.393055] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2984.656087] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2984.902898] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2985.229556] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2995.533798] Lustre: DEBUG MARKER: hostname [ 2995.840047] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2996.087677] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2996.333334] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2996.481680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.680598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2996.931572] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2997.448444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2997.454563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2997.613875] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2997.614166] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2997.753866] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2998.004326] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2998.255413] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2998.509400] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2999.281226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2999.284363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2999.481120] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2999.508407] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3002.735102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3002.820246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3002.901508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3003.066846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3004.453152] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3005.359116] Lustre: DEBUG MARKER: rc=0; [ 3005.359116] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3005.359116] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3005.359116] echo $(hostname -s): $val; [ 3005.359116] rc=$val; [ 3005.359116] fi; [ 3005.359116] exit $rc [ 3005.954326] Lustre: DEBUG MARKER: dmesg [ 3006.367132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53e: \|X\| open reply while two MDC requests in flight =========================== 11:22:57 \(1491477777\) [ 3006.533527] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight =========================== 11:22:57 (1491477777) [ 3006.677945] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 3006.806675] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3006.809062] LustreError: 18806:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880067c16400 x1563925365958144/t274877906960(0) o36->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491477783 ref 1 fl Interpret:/0/0 rc 0/0 [ 3007.928021] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3008.178837] Lustre: DEBUG MARKER: sync; sync; sync [ 3008.503942] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3008.754322] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3008.894562] Turning device dm-0 (0xfc00000) read-only [ 3009.020952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3009.145803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3009.284980] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3009.533959] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3009.704834] Removing read-only on unknown block (0xfc00000) [ 3009.836500] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3018.530046] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.534314] LustreError: Skipped 12 previous similar messages [ 3020.148668] Lustre: DEBUG MARKER: hostname [ 3020.452479] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3020.706971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3020.953418] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3021.108955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.730100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3024.983686] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3025.539791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.541243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.700526] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.702638] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.842891] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3026.097191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3026.345040] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3026.599613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3027.344589] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.369766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.556786] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.605320] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3029.782486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3029.910942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3029.946872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3030.128337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3031.482800] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3032.358990] Lustre: DEBUG MARKER: rc=0; [ 3032.358990] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3032.358990] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3032.358990] echo $(hostname -s): $val; [ 3032.358990] rc=$val; [ 3032.358990] fi; [ 3032.358990] exit $rc [ 3032.961360] Lustre: DEBUG MARKER: dmesg [ 3033.391141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53f: \|X\| open reply and close reply while two MDC requests in flight =========== 11:23:24 \(1491477804\) [ 3033.548823] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight =========== 11:23:24 (1491477804) [ 3033.695731] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 3033.832135] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3033.834644] LustreError: 18806:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005d84f000 x1563925365959888/t279172874256(0) o36->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491477810 ref 1 fl Interpret:/0/0 rc 0/0 [ 3034.952983] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 3035.086173] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3035.216593] Lustre: DEBUG MARKER: sync; sync; sync [ 3035.543731] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3035.791687] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3035.920139] Turning device dm-0 (0xfc00000) read-only [ 3036.046621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3036.172039] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3036.311850] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3036.560726] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3036.733719] Removing read-only on unknown block (0xfc00000) [ 3036.863952] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3047.171921] Lustre: DEBUG MARKER: hostname [ 3047.481747] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3047.735096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3047.982421] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3048.137812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.723811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3053.975318] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3054.461887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3054.472572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3054.625767] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3054.631224] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3054.767491] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3055.014970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3055.263791] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3055.521878] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3060.771175] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3061.104968] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3061.988970] Lustre: DEBUG MARKER: rc=0; [ 3061.988970] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3061.988970] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3061.988970] echo $(hostname -s): $val; [ 3061.988970] rc=$val; [ 3061.988970] fi; [ 3061.988970] exit $rc [ 3062.590942] Lustre: DEBUG MARKER: dmesg [ 3063.010321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53g: \|X\| drop open reply and close request while close and open are both in flight ====================================================================================================== 11:23:53 \(1491477833\) [ 3063.165670] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ====================================================================================================== 11:23:53 (1491477833) [ 3063.313479] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 3063.451304] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3063.453710] Lustre: Skipped 1 previous similar message [ 3063.455945] LustreError: 1390:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005eb35b00 x1563925365961472/t283467841552(0) o36->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491477840 ref 1 fl Interpret:/0/0 rc 0/0 [ 3063.463402] LustreError: 1390:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 1 previous similar message [ 3064.575715] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 3064.706154] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3064.708626] Lustre: Skipped 1 previous similar message [ 3064.834877] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3065.087114] Lustre: DEBUG MARKER: sync; sync; sync [ 3065.379651] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3065.632885] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3065.773543] Turning device dm-0 (0xfc00000) read-only [ 3065.900881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3066.026510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3066.167553] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3066.415422] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3066.543815] Lustre: Failing over lustre-MDT0000 [ 3066.545875] Lustre: Skipped 12 previous similar messages [ 3066.592777] Removing read-only on unknown block (0xfc00000) [ 3066.725875] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3077.045682] Lustre: DEBUG MARKER: hostname [ 3077.359771] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3077.607804] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3077.858877] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3078.004722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.935171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3083.190368] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3083.681488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3083.685825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3083.843690] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3083.848545] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3083.987026] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3084.235201] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3084.482477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3084.734723] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3089.944112] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3090.829849] Lustre: DEBUG MARKER: rc=0; [ 3090.829849] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3090.829849] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3090.829849] echo $(hostname -s): $val; [ 3090.829849] rc=$val; [ 3090.829849] fi; [ 3090.829849] exit $rc [ 3091.424338] Lustre: DEBUG MARKER: dmesg [ 3091.836464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53h: open request and close reply while two MDC requests in flight ============= 11:24:22 \(1491477862\) [ 3091.995531] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ============= 11:24:22 (1491477862) [ 3092.134844] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 3092.266321] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3093.386106] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 3093.515902] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3094.646035] Lustre: DEBUG MARKER: sync; sync; sync [ 3094.945857] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3095.199942] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3095.339295] Turning device dm-0 (0xfc00000) read-only [ 3095.465766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3095.590895] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3095.729503] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3095.977352] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3096.149790] Removing read-only on unknown block (0xfc00000) [ 3096.278369] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3106.582079] Lustre: DEBUG MARKER: hostname [ 3106.884539] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3107.139044] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3107.385540] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3107.536785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.924640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3111.175682] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3111.664767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3111.669019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3111.825777] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3111.831922] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3111.969159] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3112.219929] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3112.466599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3112.725169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3117.911125] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3118.245598] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3119.137122] Lustre: DEBUG MARKER: rc=0; [ 3119.137122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3119.137122] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3119.137122] echo $(hostname -s): $val; [ 3119.137122] rc=$val; [ 3119.137122] fi; [ 3119.137122] exit $rc [ 3119.741383] Lustre: DEBUG MARKER: dmesg [ 3120.156047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 11:24:51 \(1491477891\) [ 3120.323913] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 11:24:51 (1491477891) [ 3120.463130] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012b [ 3120.598401] Lustre: *** cfs_fail_loc=12b, val=2147484428*** [ 3125.724857] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 3126.031425] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3126.922518] Lustre: DEBUG MARKER: rc=0; [ 3126.922518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3126.922518] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3126.922518] echo $(hostname -s): $val; [ 3126.922518] rc=$val; [ 3126.922518] fi; [ 3126.922518] exit $rc [ 3127.520937] Lustre: DEBUG MARKER: dmesg [ 3127.938316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 56: don\'t replay a symlink open request \(3440\) ================================= 11:24:58 \(1491477898\) [ 3128.100137] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ================================= 11:24:58 (1491477898) [ 3128.248228] Lustre: DEBUG MARKER: sync; sync; sync [ 3129.074967] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3129.324909] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3129.467511] Turning device dm-0 (0xfc00000) read-only [ 3129.592154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3129.716254] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3129.856192] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3130.105927] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3130.280422] Removing read-only on unknown block (0xfc00000) [ 3130.411936] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3140.133693] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.138157] LustreError: Skipped 536 previous similar messages [ 3140.724237] Lustre: DEBUG MARKER: hostname [ 3141.023659] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3141.270607] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3141.519135] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3141.664987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.015458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3144.267062] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3144.760125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3144.761618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3144.921542] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3144.923127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3145.065415] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3145.315029] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3145.562947] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3145.816257] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3146.658185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3146.680291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3146.839290] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3146.865188] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3149.128370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3149.132061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3149.343689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3149.376724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3160.740068] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3161.621032] Lustre: DEBUG MARKER: rc=0; [ 3161.621032] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3161.621032] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3161.621032] echo $(hostname -s): $val; [ 3161.621032] rc=$val; [ 3161.621032] fi; [ 3161.621032] exit $rc [ 3162.233826] Lustre: DEBUG MARKER: dmesg [ 3162.648537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 57: test recovery from llog for setattr op ===================================== 11:25:33 \(1491477933\) [ 3162.823879] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ===================================== 11:25:33 (1491477933) [ 3162.965859] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012c [ 3163.226656] Lustre: DEBUG MARKER: sync; sync; sync [ 3164.102605] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3164.351636] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3164.478379] LustreError: 1957:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 3164.481057] LustreError: 1957:0:(osd_handler.c:2176:osd_ro()) Skipped 10 previous similar messages [ 3164.491542] Turning device dm-0 (0xfc00000) read-only [ 3164.617627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3164.745068] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3164.888033] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3165.138193] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3165.319631] Removing read-only on unknown block (0xfc00000) [ 3165.451300] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3175.751906] Lustre: DEBUG MARKER: hostname [ 3176.051157] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3176.300699] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3176.547913] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3176.703308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.720858] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c1208fe to 0x61ac6f141c120ec3 [ 3176.724982] Lustre: Skipped 14 previous similar messages [ 3176.910208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3177.010678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 3177.013057] Lustre: Skipped 13 previous similar messages [ 3177.162821] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3177.655461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3177.655557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3177.816069] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3177.821389] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3177.959085] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3178.210109] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3178.460290] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3178.715724] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3179.471372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3179.484528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3179.631625] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3179.762329] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3181.756289] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3181.761689] LustreError: Skipped 15 previous similar messages [ 3181.776954] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 3181.779916] Lustre: Skipped 13 previous similar messages [ 3181.983889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3182.139545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3182.866095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3183.010552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3185.281329] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 3185.591311] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3186.485717] Lustre: DEBUG MARKER: rc=0; [ 3186.485717] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3186.485717] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3186.485717] echo $(hostname -s): $val; [ 3186.485717] rc=$val; [ 3186.485717] fi; [ 3186.485717] exit $rc [ 3187.087856] Lustre: DEBUG MARKER: dmesg [ 3187.510577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58a: test recovery from llog for setattr op \(test llog_gen_rec\) ================ 11:25:58 \(1491477958\) [ 3187.683652] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ================ 11:25:58 (1491477958) [ 3187.843929] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012c [ 3190.843897] Lustre: DEBUG MARKER: sync; sync; sync [ 3191.758931] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3192.015545] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3192.160892] Turning device dm-0 (0xfc00000) read-only [ 3192.287107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3192.414960] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3192.555452] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3192.805936] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3192.991643] Removing read-only on unknown block (0xfc00000) [ 3193.121492] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3203.434202] Lustre: DEBUG MARKER: hostname [ 3203.752683] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3204.002926] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3204.249203] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3204.404088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.603137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3204.855048] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3205.351315] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3205.351339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3205.514139] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3205.516683] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3205.655980] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3205.906861] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3206.154024] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3206.408969] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3207.161613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3207.162704] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3207.387122] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3207.401363] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3210.703260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3210.719835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3210.912403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3211.017850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3217.026639] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 3218.863869] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3219.763828] Lustre: DEBUG MARKER: rc=0; [ 3219.763828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3219.763828] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3219.763828] echo $(hostname -s): $val; [ 3219.763828] rc=$val; [ 3219.763828] fi; [ 3219.763828] exit $rc [ 3220.706221] Lustre: DEBUG MARKER: dmesg [ 3221.130511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58b: test replay of setxattr op ================================================ 11:26:32 \(1491477992\) [ 3221.287281] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ================================================ 11:26:32 (1491477992) [ 3221.450679] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [ 3221.450679] grep -E -q '(ea_inode|large_xattr)' [ 3221.782594] Lustre: DEBUG MARKER: sync; sync; sync [ 3222.682798] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3222.930578] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3223.075842] Turning device dm-0 (0xfc00000) read-only [ 3223.202338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3223.327792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3223.610785] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3223.859865] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3224.040940] Removing read-only on unknown block (0xfc00000) [ 3224.172038] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3232.839036] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491477995/real 1491477995] req@ffff8800685fcc00 x1563924889001808/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491478002 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3232.846126] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [ 3234.478602] Lustre: DEBUG MARKER: hostname [ 3234.784981] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3235.035168] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3235.280968] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3235.438418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.122617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3239.387062] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3239.903166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3239.905306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3240.066418] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3240.068094] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3240.209283] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3240.458712] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3240.706066] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3240.959425] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3241.712894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.728590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.939146] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.957467] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.302486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3244.303722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3244.464089] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3244.465583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3246.498457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3246.510131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3246.692343] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3246.770637] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3246.901898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3247.043958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3247.070873] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3247.242824] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3247.480555] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3248.367536] Lustre: DEBUG MARKER: rc=0; [ 3248.367536] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3248.367536] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3248.367536] echo $(hostname -s): $val; [ 3248.367536] rc=$val; [ 3248.367536] fi; [ 3248.367536] exit $rc [ 3248.977508] Lustre: DEBUG MARKER: dmesg [ 3249.399187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58c: resend\/reconstruct setxattr op ============================================ 11:27:00 \(1491478020\) [ 3249.570604] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ============================================ 11:27:00 (1491478020) [ 3249.734270] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [ 3249.734270] grep -E -q '(ea_inode|large_xattr)' [ 3255.051660] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x123 [ 3255.189201] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3262.189105] Lustre: lustre-MDT0000: Client d79285a1-022b-efbb-4d47-137374cebe6d (at 10.9.5.237@tcp) reconnecting [ 3262.191904] Lustre: Skipped 2 previous similar messages [ 3262.324493] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3262.590754] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x119 [ 3262.722919] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3262.725216] Lustre: Skipped 1 previous similar message [ 3262.727372] LustreError: 1390:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880061453900 x1563925366214560/t309237645320(0) o36->d79285a1-022b-efbb-4d47-137374cebe6d@10.9.5.237@tcp:-1/-1 lens 624/440 e 0 to 0 dl 1491478039 ref 1 fl Interpret:/0/0 rc 0/0 [ 3262.734479] LustreError: 1390:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 1 previous similar message [ 3269.722141] Lustre: lustre-MDT0000: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [ 3269.725096] Lustre: Skipped 459 previous similar messages [ 3269.855405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 3270.307853] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3271.182533] Lustre: DEBUG MARKER: rc=0; [ 3271.182533] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3271.182533] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3271.182533] echo $(hostname -s): $val; [ 3271.182533] rc=$val; [ 3271.182533] fi; [ 3271.182533] exit $rc [ 3271.782557] Lustre: DEBUG MARKER: dmesg [ 3272.253479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 11:27:23 \(1491478043\) [ 3272.409648] Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 11:27:23 (1491478043) [ 3286.151799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3286.324248] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3287.767452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3287.767932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3287.960412] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3287.999541] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3290.201649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3290.310501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3290.381675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3290.501919] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3291.857624] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3292.106322] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3292.306602] Lustre: server umount lustre-MDT0000 complete [ 3292.308886] Lustre: Skipped 16 previous similar messages [ 3292.436066] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3302.740924] Lustre: DEBUG MARKER: hostname [ 3303.053051] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3303.303628] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3303.548250] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3303.703766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.892326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.894891] Lustre: Skipped 16 previous similar messages [ 3307.064972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3307.317371] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3307.811095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3307.811788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3307.973340] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3307.974407] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3308.119579] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3308.368823] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3308.615249] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3308.872282] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3309.614754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3309.649105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3309.797366] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3309.923372] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3312.158851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3312.321879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3313.064250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3313.220112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3334.816759] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3335.719995] Lustre: DEBUG MARKER: rc=0; [ 3335.719995] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3335.719995] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3335.719995] echo $(hostname -s): $val; [ 3335.719995] rc=$val; [ 3335.719995] fi; [ 3335.719995] exit $rc [ 3336.331497] Lustre: DEBUG MARKER: dmesg [ 3336.796956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 60: test llog post recovery init vs llog unlink ================================ 11:28:27 \(1491478107\) [ 3336.967959] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ================================ 11:28:27 (1491478107) [ 3337.309547] Lustre: DEBUG MARKER: sync; sync; sync [ 3338.167501] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3338.427359] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3338.571876] Turning device dm-0 (0xfc00000) read-only [ 3338.697491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3338.824127] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3339.075698] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3339.324630] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3339.505425] Removing read-only on unknown block (0xfc00000) [ 3339.635766] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3349.942466] Lustre: DEBUG MARKER: hostname [ 3350.245752] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3350.494697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3350.745279] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3350.910051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.102506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3355.358712] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3355.846229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3355.847231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3356.004890] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3356.018258] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3356.159729] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3356.420976] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3356.677602] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3356.932573] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3357.686894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3357.702977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3357.896462] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3357.939925] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3360.222439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3360.376515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3361.168522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3361.326107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3362.773587] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3363.687630] Lustre: DEBUG MARKER: rc=0; [ 3363.687630] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3363.687630] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3363.687630] echo $(hostname -s): $val; [ 3363.687630] rc=$val; [ 3363.687630] fi; [ 3363.687630] exit $rc [ 3364.293066] Lustre: DEBUG MARKER: dmesg [ 3364.727548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 11:28:55 \(1491478135\) [ 3364.882628] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 11:28:55 (1491478135) [ 3383.185726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3383.364620] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3407.274445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3407.486878] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3409.008266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3409.008267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3409.227655] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3409.227657] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3411.514238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3411.514274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3411.731809] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3411.743285] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 3443.473147] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3444.408302] Lustre: DEBUG MARKER: rc=0; [ 3444.408302] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3444.408302] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3444.408302] echo $(hostname -s): $val; [ 3444.408302] rc=$val; [ 3444.408302] fi; [ 3444.408302] exit $rc [ 3445.026516] Lustre: DEBUG MARKER: dmesg [ 3445.456534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 11:30:16 \(1491478216\) [ 3445.613404] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 11:30:16 (1491478216) [ 3445.752119] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013a [ 3446.010576] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3446.258336] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3446.583004] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3456.891567] Lustre: DEBUG MARKER: hostname [ 3457.197659] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3457.445080] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3457.690332] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3457.846684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.047101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3458.300739] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3458.811170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3458.813804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3458.977059] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3458.977297] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3459.121507] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3459.372463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3459.621116] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3459.878136] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3470.141530] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3470.389592] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3470.749879] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3481.062071] Lustre: DEBUG MARKER: hostname [ 3481.362135] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3481.608697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3481.853203] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3481.997750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.319118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3486.577061] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3487.064420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3487.069976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3487.231094] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3487.231519] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3487.374124] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3487.622089] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3487.872102] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3488.129685] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3488.882443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.885469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.077308] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.146762] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3491.400057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3491.565190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3492.319815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3492.480335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3493.837766] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3494.744919] Lustre: DEBUG MARKER: rc=0; [ 3494.744919] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3494.744919] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3494.744919] echo $(hostname -s): $val; [ 3494.744919] rc=$val; [ 3494.744919] fi; [ 3494.744919] exit $rc [ 3495.362230] Lustre: DEBUG MARKER: dmesg [ 3495.873605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 11:31:06 \(1491478266\) [ 3496.044995] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 11:31:06 (1491478266) [ 3510.943418] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 3510.946362] LustreError: Skipped 49 previous similar messages [ 3510.948848] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3510.954399] Lustre: Skipped 36 previous similar messages [ 3519.468970] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3519.654225] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3521.101937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3521.127062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3521.288795] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3521.401768] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3521.507175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [ 3521.682610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [ 3521.688727] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [ 3521.893410] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [ 3523.550991] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3524.452720] Lustre: DEBUG MARKER: rc=0; [ 3524.452720] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3524.452720] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3524.452720] echo $(hostname -s): $val; [ 3524.452720] rc=$val; [ 3524.452720] fi; [ 3524.452720] exit $rc [ 3525.063429] Lustre: DEBUG MARKER: dmesg [ 3525.478350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_61d skipping ALWAYS excluded test 61d [ 3525.630375] Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d [ 3525.861440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 62: don\'t mis-drop resent replay =============================================== 11:31:36 \(1491478296\) [ 3526.014911] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay =============================================== 11:31:36 (1491478296) [ 3526.155277] Lustre: DEBUG MARKER: sync; sync; sync [ 3526.971085] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3527.223622] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3527.367281] Turning device dm-0 (0xfc00000) read-only [ 3527.493495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3527.618327] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3527.776089] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000707 [ 3528.034139] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3528.283137] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3528.455026] Removing read-only on unknown block (0xfc00000) [ 3528.587638] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3538.892194] Lustre: DEBUG MARKER: hostname [ 3539.198722] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3539.459083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3539.712136] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3539.867850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.538122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3544.790705] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3545.276798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3545.284528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3545.446770] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3545.452645] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3545.598595] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3545.838899] Lustre: *** cfs_fail_loc=707, val=0*** [ 3545.851650] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3546.101520] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3546.360885] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3547.124648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3547.149915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3547.322927] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3547.422039] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3552.846923] Lustre: lustre-MDT0000: Recovery already passed deadline 0:59. If you do not want to wait more, please abort the recovery by force. [ 3553.561759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 3553.768101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 3553.818067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 3553.956027] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 3555.265075] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3555.623184] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3556.562107] Lustre: DEBUG MARKER: rc=0; [ 3556.562107] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3556.562107] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3556.562107] echo $(hostname -s): $val; [ 3556.562107] rc=$val; [ 3556.562107] fi; [ 3556.562107] exit $rc [ 3557.181699] Lustre: DEBUG MARKER: dmesg [ 3557.684871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 65a: AT: verify early replies ================================================== 11:32:08 \(1491478328\) [ 3557.839992] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ================================================== 11:32:08 (1491478328) [ 3558.019448] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3558.544023] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3559.073052] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3559.321517] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3579.952422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3580.210790] Lustre: DEBUG MARKER: lctl set_param fail_val=6000 [ 3580.458329] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3580.592729] LustreError: 1389:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 3586.595035] LustreError: 1389:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3587.059672] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3596.415627] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3597.319451] Lustre: DEBUG MARKER: rc=0; [ 3597.319451] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3597.319451] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3597.319451] echo $(hostname -s): $val; [ 3597.319451] rc=$val; [ 3597.319451] fi; [ 3597.319451] exit $rc [ 3597.923170] Lustre: DEBUG MARKER: dmesg [ 3598.377264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 65b: AT: verify early replies on packed reply \/ bulk =========================== 11:32:49 \(1491478369\) [ 3598.535252] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk =========================== 11:32:49 (1491478369) [ 3598.703868] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3599.245274] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3599.782238] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3600.037335] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3620.555612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3628.006548] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3628.348671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3629.227926] Lustre: DEBUG MARKER: rc=0; [ 3629.227926] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3629.227926] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3629.227926] echo $(hostname -s): $val; [ 3629.227926] rc=$val; [ 3629.227926] fi; [ 3629.227926] exit $rc [ 3629.831310] Lustre: DEBUG MARKER: dmesg [ 3630.293984] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 11:33:21 \(1491478401\) [ 3630.452681] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 11:33:21 (1491478401) [ 3630.611955] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3631.136199] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3631.668487] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3631.917486] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3652.436463] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=5000 [ 3652.686840] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3652.814790] LustreError: 3915:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3657.817030] LustreError: 3915:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3657.989556] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10000 [ 3658.239939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3658.368215] LustreError: 3915:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3668.370036] LustreError: 3915:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3668.551239] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 3677.916059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3678.810235] Lustre: DEBUG MARKER: rc=0; [ 3678.810235] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3678.810235] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3678.810235] echo $(hostname -s): $val; [ 3678.810235] rc=$val; [ 3678.810235] fi; [ 3678.810235] exit $rc [ 3679.408641] Lustre: DEBUG MARKER: dmesg [ 3679.833590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 66b: AT: verify net latency adjusts ============================================ 11:34:10 \(1491478450\) [ 3679.994762] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ============================================ 11:34:10 (1491478450) [ 3680.161368] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3680.685050] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3681.206886] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3681.454562] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3714.052578] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3714.938948] Lustre: DEBUG MARKER: rc=0; [ 3714.938948] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3714.938948] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3714.938948] echo $(hostname -s): $val; [ 3714.938948] rc=$val; [ 3714.938948] fi; [ 3714.938948] exit $rc [ 3715.533314] Lustre: DEBUG MARKER: dmesg [ 3715.989231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 67a: AT: verify slow request processing doesn\'t induce reconnects ============== 11:34:46 \(1491478486\) [ 3716.152212] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ============== 11:34:46 (1491478486) [ 3716.320484] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3716.850322] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3717.377287] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3717.626457] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3739.047411] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3739.965133] Lustre: DEBUG MARKER: rc=0; [ 3739.965133] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3739.965133] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3739.965133] echo $(hostname -s): $val; [ 3739.965133] rc=$val; [ 3739.965133] fi; [ 3739.965133] exit $rc [ 3740.571595] Lustre: DEBUG MARKER: dmesg [ 3741.007501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 67b: AT: verify instant slowdown doesn\'t induce reconnects ===================== 11:35:11 \(1491478511\) [ 3741.165101] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ===================== 11:35:11 (1491478511) [ 3741.324068] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3741.846976] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3742.373628] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3742.622590] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3763.142605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3763.142605] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3763.142605] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3763.407868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3763.407868] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3763.407868] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3763.671114] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3763.922982] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 3764.177663] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 3765.660125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark phase 2 [ 3765.815534] Lustre: DEBUG MARKER: phase 2 [ 3767.096462] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3767.992054] Lustre: DEBUG MARKER: rc=0; [ 3767.992054] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3767.992054] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3767.992054] echo $(hostname -s): $val; [ 3767.992054] rc=$val; [ 3767.992054] fi; [ 3767.992054] exit $rc [ 3768.594842] Lustre: DEBUG MARKER: dmesg [ 3769.120990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 68: AT: verify slowing locks =================================================== 11:35:40 \(1491478540\) [ 3769.284524] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks =================================================== 11:35:40 (1491478540) [ 3769.448106] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3769.970796] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3770.490125] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3770.738557] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3836.493937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3837.393304] Lustre: DEBUG MARKER: rc=0; [ 3837.393304] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3837.393304] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3837.393304] echo $(hostname -s): $val; [ 3837.393304] rc=$val; [ 3837.393304] fi; [ 3837.393304] exit $rc [ 3838.003900] Lustre: DEBUG MARKER: dmesg [ 3838.496216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70a: check multi client t-f ==================================================== 11:36:49 \(1491478609\) [ 3838.656234] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ==================================================== 11:36:49 (1491478609) [ 3839.810119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3839.968370] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3842.013792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3842.915975] Lustre: DEBUG MARKER: rc=0; [ 3842.915975] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3842.915975] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3842.915975] echo $(hostname -s): $val; [ 3842.915975] rc=$val; [ 3842.915975] fi; [ 3842.915975] exit $rc [ 3843.532823] Lustre: DEBUG MARKER: dmesg [ 3843.983157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70b: dbench 4mdts recovery\; 2 clients ========================================== 11:36:54 \(1491478614\) [ 3844.140663] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 4mdts recovery; 2 clients ========================================== 11:36:54 (1491478614) [ 3845.303218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3845.466974] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3852.884879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=11348 ... [ 3853.050158] Lustre: DEBUG MARKER: Started rundbench load pid=11348 ... [ 3854.497895] Lustre: DEBUG MARKER: sync; sync; sync [ 3855.517557] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3855.768119] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3855.906221] LustreError: 26614:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 3855.908668] LustreError: 26614:0:(osd_handler.c:2176:osd_ro()) Skipped 4 previous similar messages [ 3855.954788] Turning device dm-0 (0xfc00000) read-only [ 3856.082582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3856.207342] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3857.449353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds1 1 times [ 3857.611672] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3857.755527] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3858.009299] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3858.161048] Lustre: Failing over lustre-MDT0000 [ 3858.163030] Lustre: Skipped 10 previous similar messages [ 3858.170460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.213688] Removing read-only on unknown block (0xfc00000) [ 3858.345540] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3859.391327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.396029] LustreError: Skipped 321 previous similar messages [ 3868.654497] Lustre: DEBUG MARKER: hostname [ 3868.889060] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491478630/real 1491478630] req@ffff8800676c6a00 x1563924889131312/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491478637 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3868.898199] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 3868.900646] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.904838] LustreError: Skipped 12 previous similar messages [ 3868.965481] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3869.211877] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3869.457551] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3869.604412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.706625] Lustre: MGS: Connection restored to d912b86c-cb17-d59e-6637-16906a30a2d4 (at 10.9.5.239@tcp) [ 3872.709255] Lustre: Skipped 147 previous similar messages [ 3874.906194] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c1706fc to 0x61ac6f141c17a8b9 [ 3874.910728] Lustre: Skipped 7 previous similar messages [ 3875.111453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3875.361864] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3875.950071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3875.951726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3876.126160] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3876.128852] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3876.267839] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3876.521912] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3876.770742] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3877.025683] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3877.715154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 3877.717968] Lustre: Skipped 7 previous similar messages [ 3877.770646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3877.784137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3878.051613] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3878.068750] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3879.942330] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3879.948103] LustreError: Skipped 7 previous similar messages [ 3880.023395] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [ 3880.026635] Lustre: Skipped 7 previous similar messages [ 3880.283268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3880.328622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3880.583888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3880.656868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3886.452261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds2 2 times [ 3886.636109] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3899.699176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3899.866861] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3901.517721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.528572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.769522] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.771285] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3904.101750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3904.108339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3904.328761] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3904.381497] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3907.053366] Lustre: DEBUG MARKER: sync; sync; sync [ 3908.220170] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 3908.524546] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 3908.707045] Turning device dm-2 (0xfc00002) read-only [ 3908.875381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 3909.020784] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 3910.246932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds3 3 times [ 3910.457363] Lustre: DEBUG MARKER: test_70b fail mds3 3 times [ 3910.695841] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 3910.983216] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 3911.130829] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 3911.133597] Lustre: Skipped 1 previous similar message [ 3911.503046] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800779d5800 x1563924889169616/t0(0) o13->lustre-OST0007-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 3911.508609] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [ 3912.584141] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [ 3912.587311] Lustre: Skipped 8 previous similar messages [ 3916.753462] Removing read-only on unknown block (0xfc00002) [ 3916.760630] Lustre: server umount lustre-MDT0002 complete [ 3916.762889] Lustre: Skipped 5 previous similar messages [ 3916.888578] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3927.177048] Lustre: DEBUG MARKER: hostname [ 3927.488975] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 3927.740599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 3927.990638] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 3928.135905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.189664] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 3928.192556] Lustre: Skipped 5 previous similar messages [ 3928.358696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3928.610708] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3929.104222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3929.107030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3929.261490] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3929.265673] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3929.420166] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3929.670851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3929.918847] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3930.174338] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 3930.924642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3930.924644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3931.138687] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3931.186193] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3934.438831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3934.525346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3934.635721] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3934.807798] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3940.707513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds4 4 times [ 3940.931118] Lustre: DEBUG MARKER: test_70b fail mds4 4 times [ 3959.757771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3959.921021] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3961.568218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3961.581760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3961.794154] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3961.811774] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3965.120981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3965.143551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3965.389019] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3965.489122] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3995.980980] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3996.887162] Lustre: DEBUG MARKER: rc=0; [ 3996.887162] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3996.887162] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3996.887162] echo $(hostname -s): $val; [ 3996.887162] rc=$val; [ 3996.887162] fi; [ 3996.887162] exit $rc [ 3997.499454] Lustre: DEBUG MARKER: dmesg [ 3997.997679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70c: tar 4mdts recovery ======================================================== 11:39:28 \(1491478768\) [ 3998.162494] Lustre: DEBUG MARKER: == replay-single test 70c: tar 4mdts recovery ======================================================== 11:39:28 (1491478768) [ 3999.441683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3999.682371] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4132.425449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70c fail mds4 1 times [ 4132.587310] Lustre: DEBUG MARKER: test_70c fail mds4 1 times [ 4138.186807] LustreError: 11-0: lustre-MDT0003-osp-MDT0000: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 4138.189704] LustreError: Skipped 9 previous similar messages [ 4138.192055] Lustre: lustre-MDT0003-osp-MDT0000: Connection to lustre-MDT0003 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4138.197269] Lustre: Skipped 10 previous similar messages [ 4145.731573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4145.917242] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4147.630531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 4147.632896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 4147.873575] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 4147.886112] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 4154.207270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4154.240920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4154.431836] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4154.448408] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4275.807932] Lustre: DEBUG MARKER: sync; sync; sync [ 4277.160954] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4277.466221] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4277.691195] Turning device dm-0 (0xfc00000) read-only [ 4277.833836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4277.971657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4288.214463] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70c fail mds1 2 times [ 4288.391744] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4288.554800] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4288.823144] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4288.971172] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 4289.024019] Removing read-only on unknown block (0xfc00000) [ 4289.154592] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4294.176912] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491479064 with bad export cookie 7038122449836204427 [ 4294.731604] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1491479065 with bad export cookie 7038122449834737829 [ 4299.466440] Lustre: DEBUG MARKER: hostname [ 4299.778680] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4300.033272] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4300.282177] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4300.440244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.288288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4306.543405] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4307.044531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4307.056524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4307.242318] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4307.246161] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4307.388399] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4307.637591] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4307.888626] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4308.150349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4308.910226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.912257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4309.201492] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4309.233721] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4312.584864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4312.591239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4312.767312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4312.862308] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4365.081026] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4366.010670] Lustre: DEBUG MARKER: rc=0; [ 4366.010670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4366.010670] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4366.010670] echo $(hostname -s): $val; [ 4366.010670] rc=$val; [ 4366.010670] fi; [ 4366.010670] exit $rc [ 4366.645559] Lustre: DEBUG MARKER: dmesg [ 4367.087629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70d: mkdir\/rmdir striped dir 4mdts recovery ==================================== 11:45:38 \(1491479138\) [ 4367.268531] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 4mdts recovery ==================================== 11:45:38 (1491479138) [ 4368.441733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4368.609937] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4488.827605] Lustre: DEBUG MARKER: sync; sync; sync [ 4489.905132] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 4490.226235] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 4490.380039] LustreError: 2081:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0002 read-only *** [ 4490.383087] LustreError: 2081:0:(osd_handler.c:2176:osd_ro()) Skipped 2 previous similar messages [ 4490.406285] Turning device dm-2 (0xfc00002) read-only [ 4490.574557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 4490.748715] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 4500.989389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70d fail mds3 1 times [ 4501.155167] Lustre: DEBUG MARKER: test_70d fail mds3 1 times [ 4501.355169] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 4501.617584] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 4501.757897] Lustre: Failing over lustre-MDT0002 [ 4501.760018] Lustre: Skipped 2 previous similar messages [ 4501.770504] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.237@tcp (stopping) [ 4501.815556] Removing read-only on unknown block (0xfc00002) [ 4501.843484] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.848227] LustreError: Skipped 112 previous similar messages [ 4501.949996] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4512.247715] Lustre: DEBUG MARKER: hostname [ 4512.554917] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 4512.809939] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 4513.061992] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 4513.206947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4513.242954] Lustre: lustre-MDT0000: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 4513.245370] Lustre: Skipped 82 previous similar messages [ 4513.420149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4513.675161] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4514.193842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4514.196162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4514.358630] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4514.370295] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4514.510814] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4514.769934] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4514.856342] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 4514.859023] Lustre: Skipped 2 previous similar messages [ 4515.018819] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4515.272772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 4516.030544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4516.036961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4516.237825] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4516.277224] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4518.480364] Lustre: lustre-MDT0002: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 4518.483351] Lustre: Skipped 2 previous similar messages [ 4519.538738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4519.579668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4519.753139] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4519.835703] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4641.197584] Lustre: DEBUG MARKER: sync; sync; sync [ 4642.369172] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 4642.717681] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 4642.882448] Turning device dm-2 (0xfc00002) read-only [ 4643.018484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 4643.163535] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 4653.411434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70d fail mds3 2 times [ 4653.584244] Lustre: DEBUG MARKER: test_70d fail mds3 2 times [ 4653.783664] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 4654.093680] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 4654.285388] Removing read-only on unknown block (0xfc00002) [ 4654.292749] Lustre: server umount lustre-MDT0002 complete [ 4654.295114] Lustre: Skipped 2 previous similar messages [ 4654.422950] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4659.407420] LustreError: 3693:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491479430 with bad export cookie 7038122449837982945 [ 4659.412799] LustreError: 3693:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4664.714986] Lustre: DEBUG MARKER: hostname [ 4665.018779] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 4665.268051] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 4665.517347] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 4665.674077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.751401] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 4665.754320] Lustre: Skipped 2 previous similar messages [ 4665.901202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4666.156077] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4666.650477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4666.656587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4666.818338] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4666.819744] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4666.965555] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4667.217997] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4667.469201] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4667.726955] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 4668.486132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4668.486139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4668.724823] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4668.737749] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4671.998911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4672.078848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4672.245571] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4672.261942] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4673.669060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4674.594670] Lustre: DEBUG MARKER: rc=0; [ 4674.594670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4674.594670] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4674.594670] echo $(hostname -s): $val; [ 4674.594670] rc=$val; [ 4674.594670] fi; [ 4674.594670] exit $rc [ 4675.209518] Lustre: DEBUG MARKER: dmesg [ 4675.778255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70e: rename cross-MDT with random fails ======================================== 11:50:46 \(1491479446\) [ 4675.938348] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ======================================== 11:50:46 (1491479446) [ 4677.163604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4677.408270] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4797.695978] Lustre: DEBUG MARKER: sync; sync; sync [ 4798.683735] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4798.976133] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4799.184330] Turning device dm-0 (0xfc00000) read-only [ 4799.320697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4799.462941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4809.695406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70e fail mds1 1 times [ 4809.873787] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4810.042665] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4810.364245] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4810.518040] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 4810.578768] Removing read-only on unknown block (0xfc00000) [ 4810.709386] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4810.750221] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 4810.752708] LustreError: Skipped 4 previous similar messages [ 4810.754792] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.759364] Lustre: Skipped 5 previous similar messages [ 4818.255080] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491479581/real 1491479581] req@ffff88004eade100 x1563924892105024/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491479588 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4818.262359] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4818.264841] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.269144] LustreError: Skipped 1 previous similar message [ 4821.017246] Lustre: DEBUG MARKER: hostname [ 4821.321417] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4821.570329] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4821.817139] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4821.974222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.271219] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141c4184e7 to 0x61ac6f141c864dec [ 4824.275638] Lustre: Skipped 1 previous similar message [ 4824.519690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4824.772801] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4825.266380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4825.268382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4825.432475] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4825.432487] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4825.573558] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4825.823677] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4826.071496] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4826.332760] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4827.101742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.104594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.331682] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.333488] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4829.344247] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4829.349864] LustreError: Skipped 1 previous similar message [ 4832.641946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4832.656125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4832.821151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4832.991221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4954.300679] Lustre: DEBUG MARKER: sync; sync; sync [ 4955.335740] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4955.627635] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4955.833115] Turning device dm-0 (0xfc00000) read-only [ 4955.975389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4956.115048] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4966.344935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70e fail mds1 2 times [ 4966.528331] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 4966.713791] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4966.999037] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4967.202232] Removing read-only on unknown block (0xfc00000) [ 4967.338304] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4977.651902] Lustre: DEBUG MARKER: hostname [ 4977.961521] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4978.213490] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4978.464487] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4978.621285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.572178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4982.827987] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4983.388422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4983.392846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4983.564846] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4983.565931] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4983.710611] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4983.961962] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4984.210506] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4984.465777] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4985.219511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4985.248978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4985.435194] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4985.459456] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4990.764025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4990.786037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4990.975795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4991.014798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4992.413190] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4993.347228] Lustre: DEBUG MARKER: rc=0; [ 4993.347228] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4993.347228] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4993.347228] echo $(hostname -s): $val; [ 4993.347228] rc=$val; [ 4993.347228] fi; [ 4993.347228] exit $rc [ 4993.973452] Lustre: DEBUG MARKER: dmesg [ 4994.414196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70f: OSS O_DIRECT recovery with 2 clients ====================================== 11:56:05 \(1491479765\) [ 4994.578338] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 2 clients ====================================== 11:56:05 (1491479765) [ 4995.767178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4995.920952] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5008.128295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 1 times [ 5008.366501] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5021.362138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5021.532228] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5023.052756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5023.052758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5023.279184] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5023.294034] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5024.623400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5024.625483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5024.863205] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5024.955652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5036.772579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 2 times [ 5036.933157] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5049.937723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5050.109464] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5051.600319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5051.625834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5051.890313] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5051.904109] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5054.216023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5054.221277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5054.438598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5054.596162] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5058.848089] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5059.756445] Lustre: DEBUG MARKER: rc=0; [ 5059.756445] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5059.756445] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5059.756445] echo $(hostname -s): $val; [ 5059.756445] rc=$val; [ 5059.756445] fi; [ 5059.756445] exit $rc [ 5060.379286] Lustre: DEBUG MARKER: dmesg [ 5060.855427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 71a: mkdir\/rmdir striped dir with 2 mdts recovery ============================== 11:57:11 \(1491479831\) [ 5061.023858] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ============================== 11:57:11 (1491479831) [ 5062.206407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5062.367112] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5185.055105] Lustre: DEBUG MARKER: sync; sync; sync [ 5186.032482] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5186.353735] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5186.528948] LustreError: 11807:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 5186.531626] LustreError: 11807:0:(osd_handler.c:2176:osd_ro()) Skipped 3 previous similar messages [ 5186.582773] Turning device dm-0 (0xfc00000) read-only [ 5186.740195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5186.923784] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5197.184374] Lustre: DEBUG MARKER: /usr/sbin/lctl mark fail mds2 mds1 1 times [ 5197.369170] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5198.589397] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5198.839609] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5198.967522] Lustre: Failing over lustre-MDT0000 [ 5198.969659] Lustre: Skipped 3 previous similar messages [ 5198.991485] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 5199.167060] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880065156d00 x1563924892805040/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 5199.172195] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 5201.055115] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004e438f00 x1563924892805088/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 5201.060255] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 5204.761456] Removing read-only on unknown block (0xfc00000) [ 5204.897074] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5205.468427] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.473153] LustreError: Skipped 125 previous similar messages [ 5270.806085] Lustre: lustre-MDT0002: Connection restored to d912b86c-cb17-d59e-6637-16906a30a2d4 (at 10.9.5.239@tcp) [ 5270.808652] Lustre: Skipped 91 previous similar messages [ 5337.302141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5338.292315] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5355.422818] Lustre: DEBUG MARKER: hostname [ 5355.736439] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5355.995347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5356.253210] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5356.415499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.251891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.254564] Lustre: Skipped 2 previous similar messages [ 5370.431857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5370.697171] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5370.834865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 5370.838915] Lustre: Skipped 3 previous similar messages [ 5371.206266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5371.210337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5371.368318] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5371.373164] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5371.518795] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5371.775090] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5372.038453] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5372.301748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5373.094698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.096423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.259159] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5373.432850] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5387.176418] Lustre: lustre-MDT0000: Recovery over after 0:16, of 5 clients 5 recovered and 0 were evicted. [ 5387.179706] Lustre: Skipped 3 previous similar messages [ 5387.646283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec [ 5387.747282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec [ 5387.876926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec [ 5387.929100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec [ 5388.144805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5388.221873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5388.378939] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5388.401196] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5389.828030] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5390.767969] Lustre: DEBUG MARKER: rc=0; [ 5390.767969] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5390.767969] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5390.767969] echo $(hostname -s): $val; [ 5390.767969] rc=$val; [ 5390.767969] fi; [ 5390.767969] exit $rc [ 5391.384818] Lustre: DEBUG MARKER: dmesg [ 5391.837467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 73a: open\(O_CREAT\), unlink, replay, reconnect before open replay, close ======== 12:02:42 \(1491480162\) [ 5392.003670] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ======== 12:02:42 (1491480162) [ 5392.154571] Lustre: DEBUG MARKER: sync; sync; sync [ 5393.041411] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5393.301374] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5393.440070] Turning device dm-0 (0xfc00000) read-only [ 5393.570489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5393.702112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5393.835041] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000302 [ 5394.099572] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5394.357134] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5395.237266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5395.239664] Lustre: Skipped 20 previous similar messages [ 5397.231079] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880068a56100 x1563924892878224/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 5397.235953] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 5399.752633] Removing read-only on unknown block (0xfc00000) [ 5399.759586] Lustre: server umount lustre-MDT0000 complete [ 5399.761642] Lustre: Skipped 3 previous similar messages [ 5399.891925] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5410.206793] Lustre: DEBUG MARKER: hostname [ 5410.519283] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5410.776685] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5411.031474] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5411.190222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.548751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5413.818186] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5414.329237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5414.333031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5414.493745] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5414.493837] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5414.639287] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5414.895892] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5415.152109] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5415.416431] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5416.187347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.187349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.369490] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.455763] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5417.876950] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5424.897856] Lustre: lustre-MDT0000: Recovery already passed deadline 0:55. If you do not want to wait more, please abort the recovery by force. [ 5425.646552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5425.744650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5425.808053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5425.942391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5427.363928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5428.304944] Lustre: DEBUG MARKER: rc=0; [ 5428.304944] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5428.304944] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5428.304944] echo $(hostname -s): $val; [ 5428.304944] rc=$val; [ 5428.304944] fi; [ 5428.304944] exit $rc [ 5428.950330] Lustre: DEBUG MARKER: dmesg [ 5429.415298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 73b: open\(O_CREAT\), unlink, replay, reconnect at open_replay reply, close ====== 12:03:20 \(1491480200\) [ 5429.581989] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ====== 12:03:20 (1491480200) [ 5429.732714] Lustre: DEBUG MARKER: sync; sync; sync [ 5430.577887] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5430.837183] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5430.990560] Turning device dm-0 (0xfc00000) read-only [ 5431.128499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5431.259320] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5431.392302] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000030c [ 5431.658687] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5431.920229] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5432.099368] Removing read-only on unknown block (0xfc00000) [ 5432.236828] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5433.320191] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 5433.320478] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.320480] Lustre: Skipped 13 previous similar messages [ 5433.329531] LustreError: Skipped 15 previous similar messages [ 5440.583047] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491480203/real 1491480203] req@ffff880068a54900 x1563924892944096/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491480210 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5440.590060] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 5440.592537] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.596883] LustreError: Skipped 3 previous similar messages [ 5442.572480] Lustre: DEBUG MARKER: hostname [ 5442.883073] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5443.137269] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5443.393764] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5443.614998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5446.599254] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc34dc6 to 0x61ac6f141cc35410 [ 5446.603410] Lustre: Skipped 3 previous similar messages [ 5446.897669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5447.163097] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5447.658360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5447.661516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5447.849050] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5447.852880] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5447.999073] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5448.256032] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5448.513409] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5448.777996] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5449.573415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.573424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.785280] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.814376] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5451.638048] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5451.643635] LustreError: Skipped 3 previous similar messages [ 5454.080185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5454.110832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5454.287385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5454.341784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5455.774428] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5456.675741] Lustre: DEBUG MARKER: rc=0; [ 5456.675741] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5456.675741] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5456.675741] echo $(hostname -s): $val; [ 5456.675741] rc=$val; [ 5456.675741] fi; [ 5456.675741] exit $rc [ 5457.308698] Lustre: DEBUG MARKER: dmesg [ 5457.759798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 74: Ensure applications don\'t fail waiting for OST recovery ==================== 12:03:48 \(1491480228\) [ 5457.935957] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ==================== 12:03:48 (1491480228) [ 5460.091092] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5460.349520] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5460.742864] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5471.056347] Lustre: DEBUG MARKER: hostname [ 5471.367606] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5471.624719] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5471.880774] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5472.030444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.709297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5474.972924] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5475.470767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5475.470908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5475.630919] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5475.637178] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5475.780314] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5476.037309] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5476.302989] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5476.570208] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5476.999693] Lustre: lustre-MDT0000: Denying connection for new client b95864ef-afa7-04ba-4d67-4317c2541e32(at 10.9.5.237@tcp), waiting for 3 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5482.893614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5483.054813] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5484.895864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5485.070062] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5487.369225] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5488.279993] Lustre: DEBUG MARKER: rc=0; [ 5488.279993] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5488.279993] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5488.279993] echo $(hostname -s): $val; [ 5488.279993] rc=$val; [ 5488.279993] fi; [ 5488.279993] exit $rc [ 5488.905786] Lustre: DEBUG MARKER: dmesg [ 5489.382842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ============== 12:04:20 \(1491480260\) [ 5489.541570] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ============== 12:04:20 (1491480260) [ 5489.682636] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5489.832139] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5489.834647] LustreError: 3617:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005ed22850 x1563925217797072/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491480266 ref 1 fl Interpret:/0/0 rc 0/0 [ 5489.952931] Lustre: DEBUG MARKER: sync; sync; sync [ 5490.798025] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5491.059366] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5491.209477] Turning device dm-0 (0xfc00000) read-only [ 5491.339152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5491.468806] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5491.612083] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5491.872413] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5492.056641] Removing read-only on unknown block (0xfc00000) [ 5492.191314] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5502.507875] Lustre: DEBUG MARKER: hostname [ 5502.820184] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5503.077933] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5503.334867] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5503.496646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.753605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5504.019859] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5504.520293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5504.520621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5504.684802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5504.689663] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5504.840117] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5505.099215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5505.358472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5505.625352] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5506.275017] hrtimer: interrupt took 4785358 ns [ 5506.430896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.445428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.642070] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.645293] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5511.926148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5511.964647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5512.142184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5512.172851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5513.649708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5514.564773] Lustre: DEBUG MARKER: rc=0; [ 5514.564773] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5514.564773] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5514.564773] echo $(hostname -s): $val; [ 5514.564773] rc=$val; [ 5514.564773] fi; [ 5514.564773] exit $rc [ 5515.192651] Lustre: DEBUG MARKER: dmesg [ 5515.647843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ============== 12:04:46 \(1491480286\) [ 5515.807989] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ============== 12:04:46 (1491480286) [ 5515.950131] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5516.094894] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5516.097297] LustreError: 3617:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880062d16c50 x1563925217800784/t369367187469(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 2088/4320 e 0 to 0 dl 1491480292 ref 1 fl Interpret:/0/0 rc 0/0 [ 5516.219848] Lustre: DEBUG MARKER: sync; sync; sync [ 5517.118976] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5517.377939] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5517.516282] Turning device dm-0 (0xfc00000) read-only [ 5517.646689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5517.777396] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5537.795873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5537.965728] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5539.680625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5539.692767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5539.920430] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5539.920432] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5542.237332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5542.247900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5542.404348] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5542.496290] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5543.958910] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5544.875237] Lustre: DEBUG MARKER: rc=0; [ 5544.875237] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5544.875237] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5544.875237] echo $(hostname -s): $val; [ 5544.875237] rc=$val; [ 5544.875237] fi; [ 5544.875237] exit $rc [ 5545.494403] Lustre: DEBUG MARKER: dmesg [ 5545.951324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========== 12:05:16 \(1491480316\) [ 5546.110271] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========== 12:05:16 (1491480316) [ 5546.260488] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5546.403725] LustreError: 3617:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880062f97c50 x1563925217805216/t369367187488(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491480323 ref 1 fl Interpret:/0/0 rc 0/0 [ 5546.528717] Lustre: DEBUG MARKER: sync; sync; sync [ 5547.393214] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5547.652994] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5547.914441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5548.045656] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5549.814444] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5550.073923] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5550.255123] Removing read-only on unknown block (0xfc00000) [ 5550.390387] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5560.704986] Lustre: DEBUG MARKER: hostname [ 5561.014792] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5561.270463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5561.525535] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5561.685423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.866892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5567.128630] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5567.642364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5567.652749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5567.803520] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5567.809221] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5567.960997] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5568.219661] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5568.474972] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5568.738290] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5569.509748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5569.511720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5569.729811] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5569.743992] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5573.963297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5574.068265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5574.129287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5574.283732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5588.421858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5588.579241] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5590.266180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5590.268586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5590.445652] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5590.547749] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5592.793926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5592.952153] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5593.689821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5593.848755] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5595.287385] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5596.201741] Lustre: DEBUG MARKER: rc=0; [ 5596.201741] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5596.201741] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5596.201741] echo $(hostname -s): $val; [ 5596.201741] rc=$val; [ 5596.201741] fi; [ 5596.201741] exit $rc [ 5596.822059] Lustre: DEBUG MARKER: dmesg [ 5597.371673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ============ 12:06:08 \(1491480368\) [ 5597.525330] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ============ 12:06:08 (1491480368) [ 5597.666297] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5597.939379] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5597.941807] Lustre: Skipped 1 previous similar message [ 5597.944092] LustreError: 3617:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880062914450 x1563925217812928/t373662154765(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491480374 ref 1 fl Interpret:/0/0 rc 0/0 [ 5600.926804] Lustre: DEBUG MARKER: sync; sync; sync [ 5601.785899] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5602.045262] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5602.193592] Turning device dm-0 (0xfc00000) read-only [ 5602.323613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5602.453071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5604.269866] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5604.528110] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5604.710425] Removing read-only on unknown block (0xfc00000) [ 5604.845453] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5621.890255] Lustre: DEBUG MARKER: hostname [ 5622.199759] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5622.457919] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5622.710313] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5622.869463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.087333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5623.347132] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5623.848295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5623.848297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5624.010698] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5624.010882] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5624.161078] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5624.419845] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5624.677899] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5624.940907] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5637.168940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5637.340938] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5639.120769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5639.126091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5639.315825] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5639.379305] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.568310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5643.676598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5643.726155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5643.875645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5644.009190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5644.169963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5644.181341] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5644.381764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5645.865083] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5646.823501] Lustre: DEBUG MARKER: rc=0; [ 5646.823501] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5646.823501] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5646.823501] echo $(hostname -s): $val; [ 5646.823501] rc=$val; [ 5646.823501] fi; [ 5646.823501] exit $rc [ 5647.443844] Lustre: DEBUG MARKER: dmesg [ 5647.991696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================== 12:06:58 \(1491480418\) [ 5648.151033] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================== 12:06:58 (1491480418) [ 5651.562466] Lustre: DEBUG MARKER: sync; sync; sync [ 5652.451307] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5652.710110] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5652.860147] Turning device dm-0 (0xfc00000) read-only [ 5652.991114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5653.121044] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5653.264770] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5653.525140] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5653.658658] LustreError: 3693:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491480424 with bad export cookie 7038122449845842783 [ 5653.663620] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 5653.665967] Lustre: Skipped 13 previous similar messages [ 5653.712773] Removing read-only on unknown block (0xfc00000) [ 5653.846523] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5664.161713] Lustre: DEBUG MARKER: hostname [ 5664.478989] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5664.748493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5665.006308] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5665.165773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.063801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5671.337907] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5671.841642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5671.847392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5672.008584] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5672.009798] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5672.159903] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5672.430479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5672.687695] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5672.948579] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5673.800390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5673.803165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5674.114595] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5674.148609] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5678.359738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5678.511691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5678.541895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5678.719361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5692.696730] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5693.627033] Lustre: DEBUG MARKER: rc=0; [ 5693.627033] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5693.627033] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5693.627033] echo $(hostname -s): $val; [ 5693.627033] rc=$val; [ 5693.627033] fi; [ 5693.627033] exit $rc [ 5694.246640] Lustre: DEBUG MARKER: dmesg [ 5694.700825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================== 12:07:45 \(1491480465\) [ 5694.857566] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================== 12:07:45 (1491480465) [ 5709.790069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5709.957199] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5711.633536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.645710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.925007] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.978770] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5714.256436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5714.363136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5714.413282] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5714.580468] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5716.021719] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5716.937290] Lustre: DEBUG MARKER: rc=0; [ 5716.937290] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5716.937290] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5716.937290] echo $(hostname -s): $val; [ 5716.937290] rc=$val; [ 5716.937290] fi; [ 5716.937290] exit $rc [ 5717.559869] Lustre: DEBUG MARKER: dmesg [ 5718.014529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 =============== 12:08:08 \(1491480488\) [ 5718.183081] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 =============== 12:08:08 (1491480488) [ 5721.575194] Lustre: DEBUG MARKER: sync; sync; sync [ 5722.468336] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5722.728238] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5722.877557] Turning device dm-0 (0xfc00000) read-only [ 5723.008313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5723.137756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5724.981853] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5725.240568] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5725.419704] Removing read-only on unknown block (0xfc00000) [ 5725.553098] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5735.874111] Lustre: DEBUG MARKER: hostname [ 5736.187754] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5736.443131] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5736.702490] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5736.859577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.066945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5739.325282] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5739.818372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5739.821096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5739.981072] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5739.981073] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5740.126139] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5740.381859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5740.636839] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5740.906454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5741.732728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.733735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.964223] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.965342] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5746.252680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5746.309892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5746.478494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5746.531453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5760.705112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5760.873841] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5762.571111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5762.574026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5762.760941] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5762.830369] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5765.051865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5765.093320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5765.210457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5765.345467] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5766.793468] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5767.700378] Lustre: DEBUG MARKER: rc=0; [ 5767.700378] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5767.700378] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5767.700378] echo $(hostname -s): $val; [ 5767.700378] rc=$val; [ 5767.700378] fi; [ 5767.700378] exit $rc [ 5768.328141] Lustre: DEBUG MARKER: dmesg [ 5768.811670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ======================== 12:08:59 \(1491480539\) [ 5768.971388] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ======================== 12:08:59 (1491480539) [ 5772.373454] Lustre: DEBUG MARKER: sync; sync; sync [ 5773.283035] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5773.541613] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5773.700298] Turning device dm-0 (0xfc00000) read-only [ 5773.838791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5773.977267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5775.791799] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5776.054715] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5776.238620] Removing read-only on unknown block (0xfc00000) [ 5776.374191] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5787.522716] Lustre: DEBUG MARKER: hostname [ 5787.831186] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5788.085717] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5788.342153] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5788.502219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.251173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5792.510794] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5793.097168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5793.098735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5793.264946] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5793.265777] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5793.409499] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5793.666668] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5793.925555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5794.192679] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5806.472643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5806.641752] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5808.472308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5808.472801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5808.639317] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5808.722423] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5812.961918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5812.988270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5813.170079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5813.210492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5813.402150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5813.503101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5813.568973] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5813.704276] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5815.191690] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5816.122241] Lustre: DEBUG MARKER: rc=0; [ 5816.122241] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5816.122241] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5816.122241] echo $(hostname -s): $val; [ 5816.122241] rc=$val; [ 5816.122241] fi; [ 5816.122241] exit $rc [ 5816.743649] Lustre: DEBUG MARKER: dmesg [ 5817.226246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ================== 12:09:48 \(1491480588\) [ 5817.386867] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ================== 12:09:48 (1491480588) [ 5817.551914] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5817.695559] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5817.697949] LustreError: 26336:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880062f96450 x1563925217843056/t390842023950(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1488/4320 e 0 to 0 dl 1491480594 ref 1 fl Interpret:/0/0 rc 0/0 [ 5837.857942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5838.024516] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5839.711460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5839.713193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5839.940261] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5839.942062] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5842.209297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5842.233241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5842.416194] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5842.481269] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5843.874630] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5844.790515] Lustre: DEBUG MARKER: rc=0; [ 5844.790515] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5844.790515] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5844.790515] echo $(hostname -s): $val; [ 5844.790515] rc=$val; [ 5844.790515] fi; [ 5844.790515] exit $rc [ 5845.411208] Lustre: DEBUG MARKER: dmesg [ 5845.880702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ================ 12:10:16 \(1491480616\) [ 5846.031066] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ================ 12:10:16 (1491480616) [ 5846.241356] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5846.502855] Lustre: DEBUG MARKER: sync; sync; sync [ 5847.329714] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5847.588075] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5847.732716] LustreError: 5904:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 5847.735266] LustreError: 5904:0:(osd_handler.c:2176:osd_ro()) Skipped 9 previous similar messages [ 5847.778563] Turning device dm-0 (0xfc00000) read-only [ 5847.909791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5848.041542] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5848.185334] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5848.445892] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5848.578003] Lustre: Failing over lustre-MDT0000 [ 5848.580156] Lustre: Skipped 9 previous similar messages [ 5848.632093] Removing read-only on unknown block (0xfc00000) [ 5848.767152] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5848.971327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.976285] LustreError: Skipped 833 previous similar messages [ 5859.083083] Lustre: DEBUG MARKER: hostname [ 5859.393640] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5859.657824] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5859.911864] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5860.127641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.412179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5865.670306] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5866.169869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5866.173166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5866.331294] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5866.331295] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5866.479628] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5866.737172] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5866.995502] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5867.258956] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5868.019298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5868.021379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5868.200202] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5868.272410] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5872.326774] Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.9.5.243@tcp (at 0@lo) [ 5872.331165] Lustre: Skipped 324 previous similar messages [ 5872.534837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5872.694245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5873.444746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5873.598417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5874.958751] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5875.879595] Lustre: DEBUG MARKER: rc=0; [ 5875.879595] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5875.879595] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5875.879595] echo $(hostname -s): $val; [ 5875.879595] rc=$val; [ 5875.879595] fi; [ 5875.879595] exit $rc [ 5876.494880] Lustre: DEBUG MARKER: dmesg [ 5876.965195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ============ 12:10:47 \(1491480647\) [ 5877.130794] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ============ 12:10:47 (1491480647) [ 5877.287932] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5877.561892] Lustre: DEBUG MARKER: sync; sync; sync [ 5878.407359] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5878.677710] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5878.880261] Turning device dm-0 (0xfc00000) read-only [ 5879.013778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5879.145128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5880.956777] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5881.218699] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5881.400184] Removing read-only on unknown block (0xfc00000) [ 5881.535139] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5891.850656] Lustre: DEBUG MARKER: hostname [ 5892.165553] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5892.422473] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5892.678023] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5892.838665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.472747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5898.734621] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5899.226231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5899.230513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5899.389697] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5899.389705] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5899.537477] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5899.802464] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5900.068423] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5900.339208] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5901.098351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5901.098419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5901.308470] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5901.342667] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5905.601954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5905.656985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5905.814829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5905.864625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5920.020057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5920.185643] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5921.903071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5921.903117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5922.099800] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5922.139487] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5924.317383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5924.473107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5924.485673] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5924.704314] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5926.050567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5926.959223] Lustre: DEBUG MARKER: rc=0; [ 5926.959223] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5926.959223] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5926.959223] echo $(hostname -s): $val; [ 5926.959223] rc=$val; [ 5926.959223] fi; [ 5926.959223] exit $rc [ 5927.592421] Lustre: DEBUG MARKER: dmesg [ 5928.127098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ============== 12:11:39 \(1491480699\) [ 5928.326438] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ============== 12:11:39 (1491480699) [ 5928.521558] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5928.783688] Lustre: DEBUG MARKER: sync; sync; sync [ 5929.623597] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5929.890898] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5930.049404] Turning device dm-0 (0xfc00000) read-only [ 5930.186238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5930.325362] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5932.160699] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5932.419600] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5932.603228] Removing read-only on unknown block (0xfc00000) [ 5932.738022] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5949.879675] Lustre: DEBUG MARKER: hostname [ 5950.194467] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5950.464613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5950.770321] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5950.920684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.541623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5961.801832] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5962.296480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5962.301182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5962.462912] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5962.465320] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5962.609999] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5962.867971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5963.131765] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5963.410393] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5975.668915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5975.831574] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5977.602922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5977.605305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5977.834809] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5977.849043] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5982.074177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5982.167737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5982.231776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5982.411466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5982.452679] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5982.647624] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5982.677033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5982.904151] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5984.246905] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5985.165756] Lustre: DEBUG MARKER: rc=0; [ 5985.165756] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5985.165756] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5985.165756] echo $(hostname -s): $val; [ 5985.165756] rc=$val; [ 5985.165756] fi; [ 5985.165756] exit $rc [ 5985.791718] Lustre: DEBUG MARKER: dmesg [ 5986.252550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ==================== 12:12:37 \(1491480757\) [ 5986.417137] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ==================== 12:12:37 (1491480757) [ 5987.094718] Lustre: DEBUG MARKER: sync; sync; sync [ 5987.998615] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5988.258222] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5988.402929] Turning device dm-0 (0xfc00000) read-only [ 5988.535476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5988.667566] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5988.813219] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5989.073905] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5989.254295] Removing read-only on unknown block (0xfc00000) [ 5989.388142] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5999.703483] Lustre: DEBUG MARKER: hostname [ 6000.014880] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6000.272650] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6000.530382] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6000.693060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.419505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6005.421911] Lustre: Skipped 12 previous similar messages [ 6005.636484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6005.896696] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6006.394872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6006.394918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6006.567936] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6006.569833] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6006.717383] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6006.978830] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6006.979220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 6006.979221] Lustre: Skipped 12 previous similar messages [ 6007.241976] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6007.503734] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6008.275316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6008.275331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6008.447251] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6008.550518] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6012.464764] Lustre: lustre-MDT0000: Recovery over after 0:06, of 5 clients 5 recovered and 0 were evicted. [ 6012.468169] Lustre: Skipped 12 previous similar messages [ 6012.695267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6012.829280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6012.892506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6013.029924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6031.932704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6032.858841] Lustre: DEBUG MARKER: rc=0; [ 6032.858841] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6032.858841] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6032.858841] echo $(hostname -s): $val; [ 6032.858841] rc=$val; [ 6032.858841] fi; [ 6032.858841] exit $rc [ 6033.486327] Lustre: DEBUG MARKER: dmesg [ 6033.944814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ==================== 12:13:24 \(1491480804\) [ 6034.111274] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ==================== 12:13:24 (1491480804) [ 6040.414699] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 6040.414709] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6040.414711] Lustre: Skipped 38 previous similar messages [ 6040.425149] LustreError: Skipped 42 previous similar messages [ 6049.011411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6049.180517] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6050.868524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6050.879784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6051.148728] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6051.201795] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6053.482031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6053.577406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6053.634162] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6053.793139] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6055.150312] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6056.075839] Lustre: DEBUG MARKER: rc=0; [ 6056.075839] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6056.075839] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6056.075839] echo $(hostname -s): $val; [ 6056.075839] rc=$val; [ 6056.075839] fi; [ 6056.075839] exit $rc [ 6056.712191] Lustre: DEBUG MARKER: dmesg [ 6057.193586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ================== 12:13:48 \(1491480828\) [ 6057.354707] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ================== 12:13:48 (1491480828) [ 6057.808196] Lustre: DEBUG MARKER: sync; sync; sync [ 6058.652637] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6058.910247] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6059.058835] Turning device dm-0 (0xfc00000) read-only [ 6059.189640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6059.322283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6061.142206] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6061.414834] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6061.606070] Removing read-only on unknown block (0xfc00000) [ 6061.613308] Lustre: server umount lustre-MDT0000 complete [ 6061.615581] Lustre: Skipped 12 previous similar messages [ 6061.751756] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6072.086802] Lustre: DEBUG MARKER: hostname [ 6072.408329] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6072.414042] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491480836/real 1491480836] req@ffff88005fe8ea00 x1563924893752560/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491480843 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6072.421348] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [ 6072.423839] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.428201] LustreError: Skipped 11 previous similar messages [ 6072.679196] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6072.934367] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6073.095096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.430365] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc3ab6d to 0x61ac6f141cc3b47a [ 6078.434645] Lustre: Skipped 11 previous similar messages [ 6078.706983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6078.975062] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6079.473479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6079.482950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6079.642529] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6079.642963] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6079.787684] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6080.045268] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6080.304831] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6080.567955] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6081.331240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.345968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.644674] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.663213] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6083.470406] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6083.475899] LustreError: Skipped 11 previous similar messages [ 6085.956957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6085.963344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6086.174660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6086.186909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6100.326983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6100.505940] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6102.191900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6102.191990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6102.402939] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6102.456769] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6104.635583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6104.753765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6104.819788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6104.929376] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6106.304090] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6107.228071] Lustre: DEBUG MARKER: rc=0; [ 6107.228071] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6107.228071] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6107.228071] echo $(hostname -s): $val; [ 6107.228071] rc=$val; [ 6107.228071] fi; [ 6107.228071] exit $rc [ 6107.846212] Lustre: DEBUG MARKER: dmesg [ 6108.321820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs =================== 12:14:39 \(1491480879\) [ 6108.478928] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs =================== 12:14:39 (1491480879) [ 6108.938230] Lustre: DEBUG MARKER: sync; sync; sync [ 6109.783676] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6110.045343] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6110.194615] Turning device dm-0 (0xfc00000) read-only [ 6110.332549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6110.472576] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6112.282674] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6112.548861] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6112.729938] Removing read-only on unknown block (0xfc00000) [ 6112.864135] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6124.022677] Lustre: DEBUG MARKER: hostname [ 6124.332030] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6124.587266] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6124.838798] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6124.997774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.490455] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (not set up) [ 6126.678946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6126.939212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6127.435890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6127.437542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6127.600849] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6127.602301] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6127.749102] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6128.006627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6128.267704] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6128.529543] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6140.851240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6141.007307] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6142.755821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6142.759178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6143.003678] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6143.003792] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6147.291432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6147.291434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6147.465501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6147.570709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6147.679669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6147.815457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6147.892728] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6148.017888] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6149.397725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6150.310477] Lustre: DEBUG MARKER: rc=0; [ 6150.310477] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6150.310477] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6150.310477] echo $(hostname -s): $val; [ 6150.310477] rc=$val; [ 6150.310477] fi; [ 6150.310477] exit $rc [ 6150.935758] Lustre: DEBUG MARKER: dmesg [ 6151.413528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 84a: stale open during export disconnect ======================================= 12:15:22 \(1491480922\) [ 6151.569135] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ======================================= 12:15:22 (1491480922) [ 6151.709054] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000144 [ 6151.972081] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client b95864ef-afa7-04ba-4d67-4317c2541e32 [ 6152.107293] Lustre: 21309:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b95864ef-afa7-04ba-4d67-4317c2541e32 at adminstrative request [ 6154.578023] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6155.482525] Lustre: DEBUG MARKER: rc=0; [ 6155.482525] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6155.482525] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6155.482525] echo $(hostname -s): $val; [ 6155.482525] rc=$val; [ 6155.482525] fi; [ 6155.482525] exit $rc [ 6156.103443] Lustre: DEBUG MARKER: dmesg [ 6156.562566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 85a: check the cancellation of unused locks during recovery\(IBITS\) ============= 12:15:27 \(1491480927\) [ 6156.738570] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ============= 12:15:27 (1491480927) [ 6157.342976] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6157.602700] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6157.946272] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6168.256960] Lustre: DEBUG MARKER: hostname [ 6168.579083] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6168.839126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6169.094373] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6169.254879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.808177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6175.069975] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6175.567372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6175.567880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6175.730555] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6175.730580] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6175.878506] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6176.137522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6176.394671] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6176.657605] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6177.419470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6177.432505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6177.622438] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6177.688838] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.922471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6181.960262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6182.137803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6182.177925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6183.560300] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6184.472068] Lustre: DEBUG MARKER: rc=0; [ 6184.472068] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6184.472068] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6184.472068] echo $(hostname -s): $val; [ 6184.472068] rc=$val; [ 6184.472068] fi; [ 6184.472068] exit $rc [ 6185.099091] Lustre: DEBUG MARKER: dmesg [ 6185.570582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 85b: check the cancellation of unused locks during recovery\(EXTENT\) ============ 12:15:56 \(1491480956\) [ 6185.729846] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ============ 12:15:56 (1491480956) [ 6185.876423] Lustre: DEBUG MARKER: lctl pool_new lustre.test_85b [ 6192.141826] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 6192.412642] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 6193.215595] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 6193.477174] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 6196.307757] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_85b lustre-OST0000 [ 6214.648977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6214.823935] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6216.294809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6216.309782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6216.472240] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6216.552588] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6218.708368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6218.845101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6218.874872] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6219.051479] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6220.340162] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_85b lustre-OST0000 [ 6222.619219] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_85b [ 6228.948432] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6229.883547] Lustre: DEBUG MARKER: rc=0; [ 6229.883547] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6229.883547] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6229.883547] echo $(hostname -s): $val; [ 6229.883547] rc=$val; [ 6229.883547] fi; [ 6229.883547] exit $rc [ 6230.513671] Lustre: DEBUG MARKER: dmesg [ 6230.917619] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 6231.261249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 12:16:42 \(1491481002\) [ 6231.422758] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 12:16:42 (1491481002) [ 6231.987846] Lustre: DEBUG MARKER: lctl set_param mdt.lustre-MDT*.exports.clear=0 [ 6232.254767] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6232.513817] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6232.860578] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6233.137206] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6233.392981] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6233.649200] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6233.809319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.096057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6234.359540] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6234.854735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6234.860725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6235.017905] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6235.020410] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6235.167206] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6235.426282] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6235.685737] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6235.948932] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6236.380397] Lustre: lustre-MDT0000: Denying connection for new client 705e2ebb-8a1b-ea0a-049a-00cdb278b4e3(at 10.9.5.237@tcp), waiting for 3 known clients (1 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 6236.385828] Lustre: Skipped 2 previous similar messages [ 6242.279311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6242.450390] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6242.675732] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6243.591266] Lustre: DEBUG MARKER: rc=0; [ 6243.591266] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6243.591266] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6243.591266] echo $(hostname -s): $val; [ 6243.591266] rc=$val; [ 6243.591266] fi; [ 6243.591266] exit $rc [ 6244.215200] Lustre: DEBUG MARKER: dmesg [ 6244.688022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 87a: write replay ============================================================== 12:16:55 \(1491481015\) [ 6244.851609] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ============================================================== 12:16:55 (1491481015) [ 6261.733753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6261.909056] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6263.390943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6263.399695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6263.596932] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6263.629107] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6264.852185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 6264.938411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 6265.014269] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 6265.146539] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 6266.761202] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6267.692352] Lustre: DEBUG MARKER: rc=0; [ 6267.692352] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6267.692352] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6267.692352] echo $(hostname -s): $val; [ 6267.692352] rc=$val; [ 6267.692352] fi; [ 6267.692352] exit $rc [ 6268.350582] Lustre: DEBUG MARKER: dmesg [ 6268.869736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 87b: write replay with changed data \(checksum resend\) ========================== 12:17:19 \(1491481039\) [ 6269.037707] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================== 12:17:19 (1491481039) [ 6285.687763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6285.853074] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6287.335828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6287.349974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6287.556537] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6287.571133] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6289.960843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6289.964245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6290.208344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6290.223400] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 6291.582304] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6292.502880] Lustre: DEBUG MARKER: rc=0; [ 6292.502880] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6292.502880] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6292.502880] echo $(hostname -s): $val; [ 6292.502880] rc=$val; [ 6292.502880] fi; [ 6292.502880] exit $rc [ 6293.131843] Lustre: DEBUG MARKER: dmesg [ 6293.630667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 88: MDS should not assign same objid to different files ======================== 12:17:44 \(1491481064\) [ 6293.791538] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ======================== 12:17:44 (1491481064) [ 6295.643179] Lustre: DEBUG MARKER: sync; sync; sync [ 6296.520517] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6296.777978] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6296.925837] Turning device dm-0 (0xfc00000) read-only [ 6297.058434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6297.189136] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6297.336587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6297.336587] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6297.336587] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6297.626381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6297.626381] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6297.626381] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6297.908892] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 6298.183290] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 6298.457612] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 6298.763610] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 6299.026640] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 6299.295272] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6299.552109] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6299.734399] Removing read-only on unknown block (0xfc00000) [ 6299.869423] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6311.031058] Lustre: DEBUG MARKER: hostname [ 6311.314868] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6311.569656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6311.823510] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6311.982998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.117753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6317.378390] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6317.872907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6317.873341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6318.036849] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6318.041938] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6318.202348] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6318.462844] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6318.730553] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6319.008316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6331.271491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6331.454362] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6334.116906] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 6334.377207] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 6336.487898] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6337.404047] Lustre: DEBUG MARKER: rc=0; [ 6337.404047] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6337.404047] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6337.404047] echo $(hostname -s): $val; [ 6337.404047] rc=$val; [ 6337.404047] fi; [ 6337.404047] exit $rc [ 6338.024029] Lustre: DEBUG MARKER: dmesg [ 6338.515632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 89: no disk space leak on late ost connection ================================== 12:18:29 \(1491481109\) [ 6338.676125] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ================================== 12:18:29 (1491481109) [ 6338.966715] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 6339.228278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 6341.790996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6344.186923] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6344.446776] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6344.789359] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6355.126076] Lustre: DEBUG MARKER: hostname [ 6355.441796] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6355.698758] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6355.953109] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6356.111940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.180640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6361.447746] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6361.956323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6361.957930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6362.117893] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6362.121984] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6362.269610] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6362.530495] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6362.787200] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6363.048670] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6369.705735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6369.879579] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6440.449772] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 6440.713067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 6443.291690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6443.569669] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 6443.867792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6445.156406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6446.441098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6447.734674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6449.012463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6450.302274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6451.673666] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6452.605478] Lustre: DEBUG MARKER: rc=0; [ 6452.605478] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6452.605478] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6452.605478] echo $(hostname -s): $val; [ 6452.605478] rc=$val; [ 6452.605478] fi; [ 6452.605478] exit $rc [ 6453.222228] Lustre: DEBUG MARKER: dmesg [ 6453.739051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 90: lfs find identifies the missing striped file segments ====================== 12:20:24 \(1491481224\) [ 6453.904386] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ====================== 12:20:24 (1491481224) [ 6454.078278] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [ 6454.078278] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [ 6454.338292] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [ 6454.338292] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [ 6467.865184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6468.048966] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6471.692260] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6472.608356] Lustre: DEBUG MARKER: rc=0; [ 6472.608356] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6472.608356] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6472.608356] echo $(hostname -s): $val; [ 6472.608356] rc=$val; [ 6472.608356] fi; [ 6472.608356] exit $rc [ 6473.248360] Lustre: DEBUG MARKER: dmesg [ 6473.723611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 93a: replay + reconnect ======================================================== 12:20:44 \(1491481244\) [ 6473.889191] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ======================================================== 12:20:44 (1491481244) [ 6474.029978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6474.029978] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6474.029978] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6486.797536] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 6486.799851] Lustre: Skipped 284 previous similar messages [ 6487.731764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6487.903416] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6489.367050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6489.377929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6489.577865] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6489.596342] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6532.040825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6532.062871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6532.286766] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6532.300348] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6533.702828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6534.675913] Lustre: DEBUG MARKER: rc=0; [ 6534.675913] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6534.675913] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6534.675913] echo $(hostname -s): $val; [ 6534.675913] rc=$val; [ 6534.675913] fi; [ 6534.675913] exit $rc [ 6535.310666] Lustre: DEBUG MARKER: dmesg [ 6535.897316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 93b: replay + reconnect on mds ================================================= 12:21:46 \(1491481306\) [ 6536.061578] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ================================================= 12:21:46 (1491481306) [ 6536.208540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6536.208540] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6536.208540] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6536.527787] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=80 [ 6536.786200] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x715 [ 6537.054720] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6537.311277] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6537.441292] Lustre: Failing over lustre-MDT0000 [ 6537.443454] Lustre: Skipped 9 previous similar messages [ 6537.646391] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6540.918389] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.923444] LustreError: Skipped 440 previous similar messages [ 6547.965050] Lustre: DEBUG MARKER: hostname [ 6548.279902] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6548.536349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6548.792928] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6548.950628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.783175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6554.045261] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6554.544685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6554.548171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6554.706818] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6554.714228] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6554.856175] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6555.125624] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6555.215038] Lustre: *** cfs_fail_loc=715, val=0*** [ 6555.390853] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6555.656769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6556.434850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6556.443187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6556.652851] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6556.668204] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6558.387037] Lustre: *** cfs_fail_loc=715, val=0*** [ 6558.389417] Lustre: Skipped 1 previous similar message [ 6559.527268] Lustre: *** cfs_fail_loc=715, val=0*** [ 6559.529781] Lustre: Skipped 1 previous similar message [ 6560.603441] LustreError: 3069:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6561.603202] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6561.606119] Lustre: *** cfs_fail_loc=715, val=0*** [ 6564.385636] Lustre: lustre-MDT0000: Recovery already passed deadline 0:55. If you do not want to wait more, please abort the recovery by force. [ 6565.524128] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6566.529026] Lustre: *** cfs_fail_loc=715, val=0*** [ 6566.531466] Lustre: Skipped 3 previous similar messages [ 6567.608278] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6571.390581] Lustre: lustre-MDT0000: Recovery already passed deadline 0:48. If you do not want to wait more, please abort the recovery by force. [ 6571.395726] Lustre: Skipped 1 previous similar message [ 6572.527144] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6572.530102] Lustre: Skipped 1 previous similar message [ 6575.613031] Lustre: *** cfs_fail_loc=715, val=0*** [ 6575.615528] Lustre: Skipped 5 previous similar messages [ 6578.398590] Lustre: lustre-MDT0000: Recovery already passed deadline 0:41. If you do not want to wait more, please abort the recovery by force. [ 6578.403777] Lustre: Skipped 1 previous similar message [ 6579.532129] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6579.535065] Lustre: Skipped 2 previous similar messages [ 6585.406597] Lustre: lustre-MDT0000: Recovery already passed deadline 0:34. If you do not want to wait more, please abort the recovery by force. [ 6585.411706] Lustre: Skipped 1 previous similar message [ 6588.611278] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6588.614241] Lustre: Skipped 3 previous similar messages [ 6592.414569] Lustre: lustre-MDT0000: Recovery already passed deadline 0:27. If you do not want to wait more, please abort the recovery by force. [ 6592.419584] Lustre: Skipped 1 previous similar message [ 6593.423037] Lustre: *** cfs_fail_loc=715, val=0*** [ 6593.425340] Lustre: Skipped 11 previous similar messages [ 6606.422571] Lustre: lustre-MDT0000: Recovery already passed deadline 0:13. If you do not want to wait more, please abort the recovery by force. [ 6606.427481] Lustre: Skipped 3 previous similar messages [ 6607.537120] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6607.539917] Lustre: Skipped 7 previous similar messages [ 6620.429488] Lustre: lustre-MDT0000: Recovery already passed deadline 0:00, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6625.621037] Lustre: *** cfs_fail_loc=715, val=0*** [ 6625.623306] Lustre: Skipped 23 previous similar messages [ 6627.434604] Lustre: lustre-MDT0000: Recovery already passed deadline 0:07, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6627.439599] Lustre: Skipped 1 previous similar message [ 6634.441570] Lustre: lustre-MDT0000: Recovery already passed deadline 0:14, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6634.446620] Lustre: Skipped 1 previous similar message [ 6640.606036] LustreError: 3069:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6640.621923] Lustre: 3069:0:(ldlm_lib.c:2550:target_recovery_thread()) too long recovery - read logs [ 6640.624690] Lustre: lustre-MDT0000: Recovery over after 1:27, of 5 clients 5 recovered and 0 were evicted. [ 6640.624700] LustreError: dumping log to /tmp/lustre-log.1491481411.3069 [ 6640.629770] Lustre: Skipped 6 previous similar messages [ 6641.233084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 84 sec [ 6641.250551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 84 sec [ 6641.447351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 84 sec [ 6641.463648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 84 sec [ 6642.853355] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6643.782631] Lustre: DEBUG MARKER: rc=0; [ 6643.782631] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6643.782631] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6643.782631] echo $(hostname -s): $val; [ 6643.782631] rc=$val; [ 6643.782631] fi; [ 6643.782631] exit $rc [ 6644.416410] Lustre: DEBUG MARKER: dmesg [ 6644.905432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ============ 12:23:35 \(1491481415\) [ 6645.074445] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ============ 12:23:35 (1491481415) [ 6648.524775] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 6648.527397] LustreError: Skipped 31 previous similar messages [ 6648.529527] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6648.534176] Lustre: Skipped 31 previous similar messages [ 6663.841728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6664.017710] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6665.715556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6665.719622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6665.935705] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6665.952845] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6668.192293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6668.270418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6668.355844] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6668.497107] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6670.019119] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6670.958142] Lustre: DEBUG MARKER: rc=0; [ 6670.958142] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6670.958142] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6670.958142] echo $(hostname -s): $val; [ 6670.958142] rc=$val; [ 6670.958142] fi; [ 6670.958142] exit $rc [ 6671.592662] Lustre: DEBUG MARKER: dmesg [ 6672.061417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 100b: DNE: create striped dir, fail MDT0 ======================================= 12:24:03 \(1491481443\) [ 6672.228259] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ======================================= 12:24:03 (1491481443) [ 6672.433004] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 6672.725520] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6672.727755] Lustre: Skipped 3 previous similar messages [ 6672.729907] LustreError: 3915:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f014600 x1563925381278832/t438086664259(0) o36->6356dfa7-c47e-306e-d773-6ece6d91ad39@10.9.5.237@tcp:-1/-1 lens 672/448 e 0 to 0 dl 1491481475 ref 1 fl Interpret:/0/0 rc 0/0 [ 6672.737147] LustreError: 3915:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 3 previous similar messages [ 6672.791196] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6673.047119] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6673.243786] Lustre: server umount lustre-MDT0000 complete [ 6673.246185] Lustre: Skipped 6 previous similar messages [ 6673.373829] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6683.681870] Lustre: DEBUG MARKER: hostname [ 6683.992240] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6684.247913] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6684.504217] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6684.665266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.682954] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6684.687427] LustreError: Skipped 6 previous similar messages [ 6684.690017] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc440a4 to 0x61ac6f141cc44bbe [ 6684.694360] Lustre: Skipped 6 previous similar messages [ 6684.752694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.755098] Lustre: Skipped 7 previous similar messages [ 6684.917207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6685.177160] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6685.671294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6685.673973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6685.837873] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6685.837887] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6685.986231] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6686.245112] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6686.504024] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6686.765651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6687.527418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6687.531400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6687.693285] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6687.796725] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6689.402765] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 6689.405726] Lustre: Skipped 7 previous similar messages [ 6689.727966] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6689.733422] LustreError: Skipped 6 previous similar messages [ 6690.043569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6690.201853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6690.939950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6691.106109] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6692.595562] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6693.531527] Lustre: DEBUG MARKER: rc=0; [ 6693.531527] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6693.531527] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6693.531527] echo $(hostname -s): $val; [ 6693.531527] rc=$val; [ 6693.531527] fi; [ 6693.531527] exit $rc [ 6694.178482] Lustre: DEBUG MARKER: dmesg [ 6694.658894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 101: Shouldn\'t reassign precreated objs to other files after recovery ========== 12:24:25 \(1491481465\) [ 6694.838720] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========== 12:24:25 (1491481465) [ 6695.055087] Lustre: DEBUG MARKER: sync; sync; sync [ 6695.971995] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6696.229917] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6696.373870] LustreError: 7427:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 6696.376614] LustreError: 7427:0:(osd_handler.c:2176:osd_ro()) Skipped 6 previous similar messages [ 6696.416188] Turning device dm-0 (0xfc00000) read-only [ 6696.548660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6696.679628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6698.658765] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6698.916474] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6699.433987] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 6699.436520] Lustre: Skipped 4 previous similar messages [ 6699.823071] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f744000 x1563924894266768/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6699.828284] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [ 6704.745649] Removing read-only on unknown block (0xfc00000) [ 6704.881824] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6705.196869] Lustre: DEBUG MARKER: hostname [ 6705.480185] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6705.740752] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6705.995422] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6706.155237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.222658] LustreError: 8017:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 6706.225699] LustreError: 8017:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6706.228238] Lustre: 8065:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6706.231141] Lustre: 8065:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 6706.231887] LustreError: 8061:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0000-osd getting update log failed: rc = -108 [ 6706.231889] LustreError: 8061:0:(lod_dev.c:419:lod_sub_recovery_thread()) Skipped 2 previous similar messages [ 6706.248889] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 6706.403421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6706.673473] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6707.207519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6707.209892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6707.375704] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6707.377516] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6707.527057] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6707.796631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6708.054143] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6708.318141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6715.824034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6716.748220] Lustre: DEBUG MARKER: rc=0; [ 6716.748220] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6716.748220] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6716.748220] echo $(hostname -s): $val; [ 6716.748220] rc=$val; [ 6716.748220] fi; [ 6716.748220] exit $rc [ 6717.397942] Lustre: DEBUG MARKER: dmesg [ 6717.903670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102a: check resend \(request lost\) with multiple modify RPCs in flight ========== 12:24:48 \(1491481488\) [ 6718.067631] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========== 12:24:48 (1491481488) [ 6718.248077] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [ 6718.396508] Lustre: *** cfs_fail_loc=159, val=0*** [ 6718.399752] Lustre: Skipped 4 previous similar messages [ 6719.528058] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6725.396070] Lustre: lustre-MDT0000: Client 6356dfa7-c47e-306e-d773-6ece6d91ad39 (at 10.9.5.237@tcp) reconnecting [ 6725.398835] Lustre: Skipped 1 previous similar message [ 6725.630230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6726.547137] Lustre: DEBUG MARKER: rc=0; [ 6726.547137] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6726.547137] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6726.547137] echo $(hostname -s): $val; [ 6726.547137] rc=$val; [ 6726.547137] fi; [ 6726.547137] exit $rc [ 6727.168169] Lustre: DEBUG MARKER: dmesg [ 6727.663509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102b: check resend \(reply lost\) with multiple modify RPCs in flight ============ 12:24:58 \(1491481498\) [ 6727.827982] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ============ 12:24:58 (1491481498) [ 6727.996840] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6728.141536] Lustre: *** cfs_fail_loc=15a, val=0*** [ 6729.274088] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6735.360496] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6736.288333] Lustre: DEBUG MARKER: rc=0; [ 6736.288333] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6736.288333] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6736.288333] echo $(hostname -s): $val; [ 6736.288333] rc=$val; [ 6736.288333] fi; [ 6736.288333] exit $rc [ 6736.916514] Lustre: DEBUG MARKER: dmesg [ 6737.389054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102c: check replay w\/o reconstruction with multiple mod RPCs in flight ========= 12:25:08 \(1491481508\) [ 6737.549520] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========= 12:25:08 (1491481508) [ 6737.724363] Lustre: DEBUG MARKER: sync; sync; sync [ 6738.642572] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6738.900421] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6739.048496] Turning device dm-0 (0xfc00000) read-only [ 6739.179440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6739.310828] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6739.445921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6740.718678] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6740.989094] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6741.245437] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6741.422992] Removing read-only on unknown block (0xfc00000) [ 6741.557526] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6751.876631] Lustre: DEBUG MARKER: hostname [ 6752.188821] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6752.445713] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6752.697976] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6752.858063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.082947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6753.342190] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6753.835622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6753.841794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6753.997427] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6754.002766] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6754.164950] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6754.423191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6754.683705] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6754.946536] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6755.709857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6755.709871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6755.931994] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6755.947259] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6758.210862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6758.219869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6758.436723] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6758.438607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6759.856780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6760.782217] Lustre: DEBUG MARKER: rc=0; [ 6760.782217] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6760.782217] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6760.782217] echo $(hostname -s): $val; [ 6760.782217] rc=$val; [ 6760.782217] fi; [ 6760.782217] exit $rc [ 6761.432228] Lustre: DEBUG MARKER: dmesg [ 6761.921387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight =========== 12:25:32 \(1491481532\) [ 6762.099798] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [ 6762.300811] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6762.441367] Lustre: *** cfs_fail_loc=15a, val=0*** [ 6762.443617] Lustre: Skipped 13 previous similar messages [ 6763.570057] Lustre: DEBUG MARKER: sync; sync; sync [ 6763.919641] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6764.188173] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6764.443978] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6764.760118] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6775.072894] Lustre: DEBUG MARKER: hostname [ 6775.378645] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6775.559043] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491481538/real 1491481538] req@ffff88005d1d6100 x1563924894292608/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491481545 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6775.566126] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 33 previous similar messages [ 6775.636710] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6775.892826] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6776.052389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.807077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6782.083138] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6782.576892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6782.583295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6782.738667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6782.751339] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6782.894208] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6783.158361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6783.415350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6783.679358] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6784.439835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6784.441246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6784.616968] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6784.718104] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6786.844355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6786.967837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6787.012245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6787.148554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6788.563173] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6789.483146] Lustre: DEBUG MARKER: rc=0; [ 6789.483146] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6789.483146] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6789.483146] echo $(hostname -s): $val; [ 6789.483146] rc=$val; [ 6789.483146] fi; [ 6789.483146] exit $rc [ 6790.120047] Lustre: DEBUG MARKER: dmesg [ 6790.603415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 103: Check otr_next_id overflow ================================================ 12:26:01 \(1491481561\) [ 6790.756920] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ================================================ 12:26:01 (1491481561) [ 6790.931577] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000162 [ 6791.166199] Lustre: *** cfs_fail_loc=162, val=0*** [ 6791.352922] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6791.613577] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6792.023880] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6802.343724] Lustre: DEBUG MARKER: hostname [ 6802.658779] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6802.916467] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6803.170174] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6803.331791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.562064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6803.821566] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6804.318116] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6804.318660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6804.478989] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6804.483971] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6804.651404] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6804.911193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6805.170212] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6805.433542] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6806.197252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6806.210437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6806.401890] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6806.424683] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6808.664355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6808.733030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6808.824583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6808.966421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6810.312505] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6811.236778] Lustre: DEBUG MARKER: rc=0; [ 6811.236778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6811.236778] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6811.236778] echo $(hostname -s): $val; [ 6811.236778] rc=$val; [ 6811.236778] fi; [ 6811.236778] exit $rc [ 6811.865567] Lustre: DEBUG MARKER: dmesg [ 6812.349730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110a: DNE: create striped dir, fail MDT1 ======================================= 12:26:23 \(1491481583\) [ 6812.515641] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ======================================= 12:26:23 (1491481583) [ 6812.659630] Lustre: DEBUG MARKER: sync; sync; sync [ 6813.537787] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6813.797910] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6813.945505] Turning device dm-0 (0xfc00000) read-only [ 6814.075687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6814.206571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6814.417895] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6814.675488] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6814.854332] Removing read-only on unknown block (0xfc00000) [ 6814.989181] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6819.407803] LustreError: 1377:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491481590 with bad export cookie 7038122449846038545 [ 6819.412779] LustreError: 1377:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6825.313212] Lustre: DEBUG MARKER: hostname [ 6825.625344] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6825.882592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6826.136816] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6826.353305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.856949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6832.115096] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6832.619993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.620095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.783719] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.784791] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.946250] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6833.205265] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6833.459817] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6833.719348] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6834.492426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6834.520119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6834.708839] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6834.735613] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6836.989457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6837.031669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6837.220730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6837.241982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6838.771269] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6839.725568] Lustre: DEBUG MARKER: rc=0; [ 6839.725568] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6839.725568] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6839.725568] echo $(hostname -s): $val; [ 6839.725568] rc=$val; [ 6839.725568] fi; [ 6839.725568] exit $rc [ 6840.363794] Lustre: DEBUG MARKER: dmesg [ 6840.859298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ============================ 12:26:51 \(1491481611\) [ 6841.031448] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ============================ 12:26:51 (1491481611) [ 6841.199579] Lustre: DEBUG MARKER: sync; sync; sync [ 6842.143817] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6842.402832] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6842.584710] Turning device dm-0 (0xfc00000) read-only [ 6842.714212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6842.845087] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6843.053097] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6843.314601] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6843.492528] Removing read-only on unknown block (0xfc00000) [ 6843.627520] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6848.398912] LustreError: 1377:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491481619 with bad export cookie 7038122449846039868 [ 6848.403979] LustreError: 1377:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6853.945446] Lustre: DEBUG MARKER: hostname [ 6854.258752] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6854.516041] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6854.769726] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6854.930337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.293189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6860.553213] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6861.047292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6861.047851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6861.216076] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6861.217603] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6861.375986] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6861.636546] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6861.895755] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6862.159744] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6862.923773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6862.940233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.147826] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.169171] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6927.583040] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 6927.585728] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 6928.670535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [ 6928.830666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [ 6931.476869] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6932.424430] Lustre: DEBUG MARKER: rc=0; [ 6932.424430] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6932.424430] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6932.424430] echo $(hostname -s): $val; [ 6932.424430] rc=$val; [ 6932.424430] fi; [ 6932.424430] exit $rc [ 6933.056645] Lustre: DEBUG MARKER: dmesg [ 6933.570105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110c: DNE: create striped dir, fail MDT2 ======================================= 12:28:24 \(1491481704\) [ 6933.737867] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ======================================= 12:28:24 (1491481704) [ 6948.454314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6948.623788] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6950.328356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6950.328370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6950.533776] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6950.560278] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6952.800662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6952.917435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6953.009326] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6953.156227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6954.642104] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6955.564733] Lustre: DEBUG MARKER: rc=0; [ 6955.564733] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6955.564733] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6955.564733] echo $(hostname -s): $val; [ 6955.564733] rc=$val; [ 6955.564733] fi; [ 6955.564733] exit $rc [ 6956.193425] Lustre: DEBUG MARKER: dmesg [ 6956.688194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ============================ 12:28:47 \(1491481727\) [ 6956.852887] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ============================ 12:28:47 (1491481727) [ 6971.685181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6971.848563] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6973.534084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6973.547816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6973.767233] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6973.767234] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7038.277377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 64 sec [ 7038.433794] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 64 sec [ 7041.154422] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7042.105777] Lustre: DEBUG MARKER: rc=0; [ 7042.105777] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7042.105777] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7042.105777] echo $(hostname -s): $val; [ 7042.105777] rc=$val; [ 7042.105777] fi; [ 7042.105777] exit $rc [ 7042.753526] Lustre: DEBUG MARKER: dmesg [ 7043.274533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client\/MDT1\/MDT2 ========= 12:30:14 \(1491481814\) [ 7043.434583] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========= 12:30:14 (1491481814) [ 7045.272080] Lustre: DEBUG MARKER: sync; sync; sync [ 7046.238873] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7046.502086] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7046.650628] Turning device dm-0 (0xfc00000) read-only [ 7046.782021] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7046.911529] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7047.058106] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7047.313683] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7047.491424] Removing read-only on unknown block (0xfc00000) [ 7047.626018] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7048.280669] LustreError: 1377:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491481818 with bad export cookie 7038122449846043648 [ 7058.777268] Lustre: DEBUG MARKER: hostname [ 7059.088300] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7059.345085] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7059.603129] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7059.820385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.282724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7063.547771] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7064.045892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7064.046335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7064.211143] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7064.213762] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7064.364821] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7064.623768] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7064.883863] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7065.143110] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7077.393813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7077.554405] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7079.260373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7079.262252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7079.466792] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7079.484545] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7081.770770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7081.936289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7148.189539] Lustre: lustre-MDT0001-osp-MDT0002: Connection restored to 10.9.5.239@tcp (at 10.9.5.239@tcp) [ 7148.192463] Lustre: Skipped 311 previous similar messages [ 7148.372338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 66 sec [ 7148.540688] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 66 sec [ 7151.526228] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7152.451194] Lustre: DEBUG MARKER: rc=0; [ 7152.451194] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7152.451194] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7152.451194] echo $(hostname -s): $val; [ 7152.451194] rc=$val; [ 7152.451194] fi; [ 7152.451194] exit $rc [ 7153.084937] Lustre: DEBUG MARKER: dmesg [ 7153.572031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110f: DNE: create striped dir, fail MDT1\/MDT2 ================================== 12:32:04 \(1491481924\) [ 7153.733026] Lustre: DEBUG MARKER: == replay-single test 110f: DNE: create striped dir, fail MDT1/MDT2 ================================== 12:32:04 (1491481924) [ 7153.880050] Lustre: DEBUG MARKER: sync; sync; sync [ 7154.781929] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7155.043160] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7155.183500] Turning device dm-0 (0xfc00000) read-only [ 7155.313695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7155.447792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7157.287882] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7157.550547] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7157.684791] Lustre: Failing over lustre-MDT0000 [ 7157.687015] Lustre: Skipped 8 previous similar messages [ 7157.734756] Removing read-only on unknown block (0xfc00000) [ 7157.869244] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7158.107356] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.112354] LustreError: Skipped 302 previous similar messages [ 7158.532879] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491481929 with bad export cookie 7038122449846044495 [ 7169.035484] Lustre: DEBUG MARKER: hostname [ 7169.350631] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7169.610452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7169.868621] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7170.027090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7170.615840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7170.884151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7171.378678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7171.382484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7171.546047] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7171.547643] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7171.694575] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7171.957235] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7172.214138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7172.476083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7184.728309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7184.890550] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7186.624599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7186.624604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7186.810598] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7186.898362] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7189.065288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7189.173265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7189.237970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7189.369350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7189.579405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7189.650400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7189.783942] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7189.827145] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7191.319340] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7192.237145] Lustre: DEBUG MARKER: rc=0; [ 7192.237145] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7192.237145] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7192.237145] echo $(hostname -s): $val; [ 7192.237145] rc=$val; [ 7192.237145] fi; [ 7192.237145] exit $rc [ 7192.868564] Lustre: DEBUG MARKER: dmesg [ 7193.397191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client\/MDT1\/MDT2 ========= 12:32:44 \(1491481964\) [ 7193.559741] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========= 12:32:44 (1491481964) [ 7193.697908] Lustre: DEBUG MARKER: sync; sync; sync [ 7194.666964] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7194.929142] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7195.070590] Turning device dm-0 (0xfc00000) read-only [ 7195.202176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7195.332566] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7197.298941] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7197.555838] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7197.734854] Removing read-only on unknown block (0xfc00000) [ 7197.869226] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7209.023529] Lustre: DEBUG MARKER: hostname [ 7209.335121] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7209.588182] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7209.842831] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7210.001475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.653700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7213.915120] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7214.411587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7214.413678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7214.584555] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7214.586078] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7214.732339] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7214.992148] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7215.249024] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7215.519159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7227.807099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7227.974497] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7229.708185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7229.709355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7229.881027] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7229.975231] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7285.471052] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 7285.473697] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 7285.500022] Lustre: lustre-MDT0000: Recovery over after 1:11, of 5 clients 4 recovered and 1 was evicted. [ 7285.502911] Lustre: Skipped 8 previous similar messages [ 7286.436327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 56 sec [ 7286.599122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 56 sec [ 7286.795022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7286.957772] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7289.666092] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7290.616566] Lustre: DEBUG MARKER: rc=0; [ 7290.616566] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7290.616566] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7290.616566] echo $(hostname -s): $val; [ 7290.616566] rc=$val; [ 7290.616566] fi; [ 7290.616566] exit $rc [ 7291.258298] Lustre: DEBUG MARKER: dmesg [ 7291.778853] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ======================================= 12:34:22 \(1491482062\) [ 7291.947253] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ======================================= 12:34:22 (1491482062) [ 7292.108201] Lustre: DEBUG MARKER: sync; sync; sync [ 7293.018702] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7293.280306] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7293.432086] Turning device dm-0 (0xfc00000) read-only [ 7293.561713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7293.690942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7293.867761] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7294.126458] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7294.306482] Removing read-only on unknown block (0xfc00000) [ 7294.313593] Lustre: server umount lustre-MDT0000 complete [ 7294.315726] Lustre: Skipped 9 previous similar messages [ 7294.401132] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482065 with bad export cookie 7038122449846048324 [ 7294.446130] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7298.491255] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 7298.491261] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.491262] Lustre: Skipped 28 previous similar messages [ 7298.500888] LustreError: Skipped 34 previous similar messages [ 7304.773392] Lustre: DEBUG MARKER: hostname [ 7305.088839] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7305.348516] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7305.603535] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7305.763504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.785314] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7305.789450] LustreError: Skipped 9 previous similar messages [ 7305.791608] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc695cd to 0x61ac6f141cc69d4b [ 7305.795688] Lustre: Skipped 9 previous similar messages [ 7305.834656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.836975] Lustre: Skipped 9 previous similar messages [ 7306.006814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7306.263825] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7306.760042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7306.761956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7306.923253] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7306.923254] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7307.072770] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7307.331060] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7307.588442] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7307.849118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7308.610159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7308.639453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7308.730134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 7308.732952] Lustre: Skipped 8 previous similar messages [ 7308.793325] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7308.898168] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7310.829609] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 7310.835461] LustreError: Skipped 10 previous similar messages [ 7311.040431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7311.157763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7311.204511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7311.385543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7312.734291] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7313.683594] Lustre: DEBUG MARKER: rc=0; [ 7313.683594] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7313.683594] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7313.683594] echo $(hostname -s): $val; [ 7313.683594] rc=$val; [ 7313.683594] fi; [ 7313.683594] exit $rc [ 7314.340758] Lustre: DEBUG MARKER: dmesg [ 7314.852141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ======================================= 12:34:45 \(1491482085\) [ 7315.024311] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ======================================= 12:34:45 (1491482085) [ 7329.699936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7329.872249] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7331.558754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7331.560847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7331.786532] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7331.787095] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7400.295231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 68 sec [ 7400.453142] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 68 sec [ 7402.997680] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7403.924548] Lustre: DEBUG MARKER: rc=0; [ 7403.924548] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7403.924548] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7403.924548] echo $(hostname -s): $val; [ 7403.924548] rc=$val; [ 7403.924548] fi; [ 7403.924548] exit $rc [ 7404.580769] Lustre: DEBUG MARKER: dmesg [ 7405.085616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client\/MDT1\/MDT2 ========= 12:36:16 \(1491482176\) [ 7405.270426] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========= 12:36:16 (1491482176) [ 7405.494733] Lustre: DEBUG MARKER: sync; sync; sync [ 7406.467924] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7406.727166] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7406.869091] LustreError: 32560:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 7406.871709] LustreError: 32560:0:(osd_handler.c:2176:osd_ro()) Skipped 7 previous similar messages [ 7406.912150] Turning device dm-0 (0xfc00000) read-only [ 7407.043988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7407.175498] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7409.188649] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7409.448377] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7409.630842] Removing read-only on unknown block (0xfc00000) [ 7409.767188] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7417.829035] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491482181/real 1491482181] req@ffff88005f1d8000 x1563924894358384/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491482188 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7417.836595] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 7426.878747] Lustre: DEBUG MARKER: hostname [ 7427.187560] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7427.442161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7427.694127] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7427.854309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.062758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7439.323130] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7439.819420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7439.822858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7439.986383] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7439.987522] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7440.136543] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7440.395381] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7440.654497] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7440.917608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7453.197935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7453.367590] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7455.071192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7455.083751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7455.301218] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7455.302989] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7511.263048] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 7511.265824] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 7511.773321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 56 sec [ 7511.934840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 56 sec [ 7512.128686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7512.294336] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7514.840191] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7515.791236] Lustre: DEBUG MARKER: rc=0; [ 7515.791236] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7515.791236] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7515.791236] echo $(hostname -s): $val; [ 7515.791236] rc=$val; [ 7515.791236] fi; [ 7515.791236] exit $rc [ 7516.431703] Lustre: DEBUG MARKER: dmesg [ 7517.016376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client\/MDT1\/MDT2 ========= 12:38:07 \(1491482287\) [ 7517.179923] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========= 12:38:07 (1491482287) [ 7519.114294] Lustre: DEBUG MARKER: sync; sync; sync [ 7520.064419] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7520.324510] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7520.476579] Turning device dm-0 (0xfc00000) read-only [ 7520.606567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7520.734682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7520.883329] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7521.141719] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7521.319693] Removing read-only on unknown block (0xfc00000) [ 7521.456469] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7522.127433] LustreError: 3693:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482292 with bad export cookie 7038122449846052615 [ 7522.132441] LustreError: 3693:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7532.624157] Lustre: DEBUG MARKER: hostname [ 7532.936718] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7533.195600] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7533.450997] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7533.610954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.946131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7538.203369] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7538.713725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7538.715711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7538.881239] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7538.881269] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7539.028059] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7539.288287] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7539.545198] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7539.808665] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7552.073807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7552.238269] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7553.966457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7553.982202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7554.196272] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7554.210174] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7556.489514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7556.653942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7624.093284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 67 sec [ 7624.247196] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 67 sec [ 7626.798840] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7627.747591] Lustre: DEBUG MARKER: rc=0; [ 7627.747591] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7627.747591] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7627.747591] echo $(hostname -s): $val; [ 7627.747591] rc=$val; [ 7627.747591] fi; [ 7627.747591] exit $rc [ 7628.378099] Lustre: DEBUG MARKER: dmesg [ 7628.892121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1\/MDT2 ================ 12:39:59 \(1491482399\) [ 7629.080156] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ================ 12:39:59 (1491482399) [ 7630.993109] Lustre: DEBUG MARKER: sync; sync; sync [ 7631.989543] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7632.250594] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7632.398194] Turning device dm-0 (0xfc00000) read-only [ 7632.530087] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7632.661388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7632.808466] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7633.067407] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7633.246435] Removing read-only on unknown block (0xfc00000) [ 7633.381206] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7634.038377] LustreError: 30714:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482404 with bad export cookie 7038122449846054379 [ 7634.043305] LustreError: 30714:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7644.537621] Lustre: DEBUG MARKER: hostname [ 7644.847661] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7645.104273] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7645.357083] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7645.505275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.509713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7651.770432] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7652.275729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7652.282714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7652.444486] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7652.444582] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7652.590640] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7652.859840] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7653.134784] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7653.413560] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7665.685600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7665.849701] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7667.560986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7667.577086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7667.832550] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7667.925166] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7670.161860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7670.238371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7670.384801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7670.445565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7670.634249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.672436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.891068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.911768] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7672.303427] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7673.256866] Lustre: DEBUG MARKER: rc=0; [ 7673.256866] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7673.256866] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7673.256866] echo $(hostname -s): $val; [ 7673.256866] rc=$val; [ 7673.256866] fi; [ 7673.256866] exit $rc [ 7673.902875] Lustre: DEBUG MARKER: dmesg [ 7674.400267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1\/MDT2 ================ 12:40:45 \(1491482445\) [ 7674.568553] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ================ 12:40:45 (1491482445) [ 7674.745930] Lustre: DEBUG MARKER: sync; sync; sync [ 7675.720314] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7675.981471] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7676.131054] Turning device dm-0 (0xfc00000) read-only [ 7676.262584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7676.391269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7678.325688] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7678.597786] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7678.784090] Removing read-only on unknown block (0xfc00000) [ 7678.917999] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7695.871068] Lustre: DEBUG MARKER: hostname [ 7696.191529] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7696.445995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7696.699675] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7696.859205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.069763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7697.330668] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7697.827823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7697.832328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7697.988171] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7697.990560] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7698.135975] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7698.395555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7698.654637] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7698.922402] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7711.210542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7711.370175] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7713.088279] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7713.099592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7713.316690] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7713.316701] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7715.568109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7715.634874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7715.733543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7715.849836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7715.979322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7716.114589] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7716.174643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7716.305712] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7717.713056] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7718.666098] Lustre: DEBUG MARKER: rc=0; [ 7718.666098] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7718.666098] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7718.666098] echo $(hostname -s): $val; [ 7718.666098] rc=$val; [ 7718.666098] fi; [ 7718.666098] exit $rc [ 7719.338200] Lustre: DEBUG MARKER: dmesg [ 7719.855509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111g: DNE: unlink striped dir, fail MDT1\/MDT2 ================================== 12:41:30 \(1491482490\) [ 7720.015414] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ================================== 12:41:30 (1491482490) [ 7720.201823] Lustre: DEBUG MARKER: sync; sync; sync [ 7721.056377] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7721.324147] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7721.474651] Turning device dm-0 (0xfc00000) read-only [ 7721.607617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7721.737834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7723.631752] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7723.900110] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7724.077701] Removing read-only on unknown block (0xfc00000) [ 7724.214329] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7724.872467] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482495 with bad export cookie 7038122449846058117 [ 7724.877264] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7735.373906] Lustre: DEBUG MARKER: hostname [ 7735.692947] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7735.950606] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7736.206533] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7736.366147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.112457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7740.379611] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7740.912762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7740.915154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7741.083366] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7741.084413] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7741.248652] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7741.507367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7741.763978] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7742.028480] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7753.346023] Lustre: lustre-MDT0002: Connection restored to d912b86c-cb17-d59e-6637-16906a30a2d4 (at 10.9.5.239@tcp) [ 7753.348735] Lustre: Skipped 237 previous similar messages [ 7754.313551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7754.495958] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7756.192983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7756.216343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7756.429337] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7756.429359] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7758.695957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7758.719812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7758.915917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7758.948306] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7759.192548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7759.205700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7759.373814] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7759.423987] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7760.850630] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7761.766908] Lustre: DEBUG MARKER: rc=0; [ 7761.766908] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7761.766908] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7761.766908] echo $(hostname -s): $val; [ 7761.766908] rc=$val; [ 7761.766908] fi; [ 7761.766908] exit $rc [ 7762.412127] Lustre: DEBUG MARKER: dmesg [ 7762.925327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================= 12:42:13 \(1491482533\) [ 7763.092888] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================= 12:42:13 (1491482533) [ 7763.318541] Lustre: DEBUG MARKER: sync; sync; sync [ 7764.370892] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7764.632934] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7764.811688] Turning device dm-0 (0xfc00000) read-only [ 7764.944155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7765.076830] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7765.257738] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7765.531811] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7765.665502] Lustre: Failing over lustre-MDT0000 [ 7765.667530] Lustre: Skipped 7 previous similar messages [ 7765.716200] Removing read-only on unknown block (0xfc00000) [ 7765.849829] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7768.366126] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.370928] LustreError: Skipped 340 previous similar messages [ 7776.165909] Lustre: DEBUG MARKER: hostname [ 7776.484055] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7776.740651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7776.996237] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7777.154590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.373366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7777.633193] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7778.126477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7778.134225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7778.296762] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7778.299775] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7778.447761] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7778.706484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7778.963725] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7779.225505] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7780.043582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7780.059694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7780.235933] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7780.262397] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7782.501770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7782.598753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7782.739811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7782.787933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7784.284911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7785.210967] Lustre: DEBUG MARKER: rc=0; [ 7785.210967] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7785.210967] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7785.210967] echo $(hostname -s): $val; [ 7785.210967] rc=$val; [ 7785.210967] fi; [ 7785.210967] exit $rc [ 7785.852990] Lustre: DEBUG MARKER: dmesg [ 7786.378029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================= 12:42:37 \(1491482557\) [ 7786.548372] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================= 12:42:37 (1491482557) [ 7801.311477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7801.478823] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7803.165511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7803.168224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7803.354690] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7803.428864] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7805.568058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7805.702791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7805.750568] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7805.900902] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7807.375827] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7808.319570] Lustre: DEBUG MARKER: rc=0; [ 7808.319570] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7808.319570] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7808.319570] echo $(hostname -s): $val; [ 7808.319570] rc=$val; [ 7808.319570] fi; [ 7808.319570] exit $rc [ 7808.976410] Lustre: DEBUG MARKER: dmesg [ 7809.492695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================= 12:43:00 \(1491482580\) [ 7809.655934] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================= 12:43:00 (1491482580) [ 7809.885129] Lustre: DEBUG MARKER: sync; sync; sync [ 7810.878290] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7811.136618] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7811.268588] Turning device dm-2 (0xfc00002) read-only [ 7811.401881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7811.532890] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7811.699205] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7811.967851] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7812.122058] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482582 with bad export cookie 7038122449846064172 [ 7812.126698] LustreError: 1378:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7812.129853] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 7812.132330] Lustre: Skipped 17 previous similar messages [ 7812.145788] Removing read-only on unknown block (0xfc00002) [ 7812.284122] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7822.588220] Lustre: DEBUG MARKER: hostname [ 7822.904956] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7823.162349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7823.417626] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7823.578078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7823.786714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7824.045789] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7824.543135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7824.545598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7824.705593] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7824.706885] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7824.854117] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7825.114229] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7825.374264] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7825.645677] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7826.453994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7826.454078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7826.672447] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7826.674243] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7828.948314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7828.973033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7829.154756] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7829.181420] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7830.649304] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7831.580762] Lustre: DEBUG MARKER: rc=0; [ 7831.580762] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7831.580762] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7831.580762] echo $(hostname -s): $val; [ 7831.580762] rc=$val; [ 7831.580762] fi; [ 7831.580762] exit $rc [ 7832.213933] Lustre: DEBUG MARKER: dmesg [ 7832.710787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================= 12:43:23 \(1491482603\) [ 7832.881405] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================= 12:43:23 (1491482603) [ 7847.651652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7847.830728] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7849.556925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7849.561384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7849.768183] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7849.785677] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7852.025141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7852.083283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7852.238448] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7852.281152] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7853.738364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7854.666217] Lustre: DEBUG MARKER: rc=0; [ 7854.666217] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7854.666217] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7854.666217] echo $(hostname -s): $val; [ 7854.666217] rc=$val; [ 7854.666217] fi; [ 7854.666217] exit $rc [ 7855.303904] Lustre: DEBUG MARKER: dmesg [ 7855.801371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ================================ 12:43:46 \(1491482626\) [ 7855.964814] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ================================ 12:43:46 (1491482626) [ 7856.191448] Lustre: DEBUG MARKER: sync; sync; sync [ 7857.213302] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7857.474241] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7857.621639] Turning device dm-0 (0xfc00000) read-only [ 7857.752261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7857.881183] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7859.851380] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7860.111995] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7860.290195] Removing read-only on unknown block (0xfc00000) [ 7860.424801] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7876.879293] Lustre: DEBUG MARKER: hostname [ 7877.193598] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7877.447846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7877.704096] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7877.865943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.071835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7878.332962] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7878.826902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7878.836790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7878.990556] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7878.996955] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7879.145169] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7879.409348] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7879.666917] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7879.930964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7892.213413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7892.383637] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7894.091750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7894.103623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7894.262851] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7894.398687] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7896.308425] Lustre: lustre-MDT0000: Recovery over after 0:17, of 5 clients 5 recovered and 0 were evicted. [ 7896.311688] Lustre: Skipped 8 previous similar messages [ 7896.524361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7896.652214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7896.683781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7896.872491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7896.924974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7897.142445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7897.179433] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7897.418967] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7898.862757] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7899.798880] Lustre: DEBUG MARKER: rc=0; [ 7899.798880] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7899.798880] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7899.798880] echo $(hostname -s): $val; [ 7899.798880] rc=$val; [ 7899.798880] fi; [ 7899.798880] exit $rc [ 7900.426595] Lustre: DEBUG MARKER: dmesg [ 7900.951283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ================================ 12:44:31 \(1491482671\) [ 7901.117414] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ================================ 12:44:31 (1491482671) [ 7901.323031] Lustre: DEBUG MARKER: sync; sync; sync [ 7902.423639] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7902.683303] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7902.829948] Turning device dm-0 (0xfc00000) read-only [ 7902.961123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7903.090742] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7903.224820] Lustre: DEBUG MARKER: sync; sync; sync [ 7904.054145] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7904.313769] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7904.446496] Turning device dm-2 (0xfc00002) read-only [ 7904.576785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7904.706110] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7904.872561] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7905.133062] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7905.307337] Removing read-only on unknown block (0xfc00000) [ 7905.314488] Lustre: server umount lustre-MDT0000 complete [ 7905.316461] Lustre: Skipped 8 previous similar messages [ 7905.447363] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7905.717141] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7905.971037] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7906.098347] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 [ 7906.100692] LustreError: Skipped 31 previous similar messages [ 7912.145396] Removing read-only on unknown block (0xfc00002) [ 7912.288830] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7922.601088] Lustre: DEBUG MARKER: hostname [ 7922.909721] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7923.160622] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7923.409784] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7923.565116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.636928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.639091] Lustre: Skipped 8 previous similar messages [ 7923.799383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7924.069657] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7924.570775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7924.572861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7924.737139] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7924.739346] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7924.921776] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7925.173681] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7925.428791] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7925.689810] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7928.880549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 7928.883241] Lustre: Skipped 8 previous similar messages [ 7935.984134] Lustre: DEBUG MARKER: hostname [ 7936.299405] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7936.556627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7936.811647] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7936.972049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.180345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7937.441683] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7937.956561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7937.956577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7938.122061] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7938.124871] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7938.274193] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7938.532115] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7938.788704] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7939.049641] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7939.821636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7939.823263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7940.035120] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7940.090028] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7942.326453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7942.379302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7942.498123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7942.583291] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7942.736920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7942.854110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7942.950114] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7943.083204] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7944.523831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7945.465148] Lustre: DEBUG MARKER: rc=0; [ 7945.465148] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7945.465148] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7945.465148] echo $(hostname -s): $val; [ 7945.465148] rc=$val; [ 7945.465148] fi; [ 7945.465148] exit $rc [ 7946.113069] Lustre: DEBUG MARKER: dmesg [ 7946.668059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ================================ 12:45:17 \(1491482717\) [ 7946.872672] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ================================ 12:45:17 (1491482717) [ 7947.074632] Lustre: DEBUG MARKER: sync; sync; sync [ 7948.263163] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7948.536311] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7948.686451] Turning device dm-0 (0xfc00000) read-only [ 7948.816391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7948.947902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7950.777057] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7951.039540] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7951.222309] Removing read-only on unknown block (0xfc00000) [ 7951.355213] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7952.016108] LustreError: 22941:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491482722 with bad export cookie 7038122449846070731 [ 7952.020709] LustreError: 22941:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7952.023658] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.028305] Lustre: Skipped 28 previous similar messages [ 7959.119067] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7959.123335] LustreError: Skipped 7 previous similar messages [ 7962.537935] Lustre: DEBUG MARKER: hostname [ 7962.863109] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7963.117072] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7963.370824] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7963.529692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.125211] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc6ed54 to 0x61ac6f141cc6f963 [ 7965.129196] Lustre: Skipped 7 previous similar messages [ 7965.339901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7965.603132] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7966.100485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7966.101565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7966.258613] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7966.261754] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7966.409821] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7966.665067] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7966.920959] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7967.182343] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7970.174202] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 7970.179213] LustreError: Skipped 7 previous similar messages [ 7979.529714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7979.694910] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7981.399624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7981.406565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7981.613968] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7981.640055] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7983.914274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7983.914275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7984.133624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7984.135764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7984.334386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7984.433987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7984.512173] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7984.644117] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7986.074704] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7987.018229] Lustre: DEBUG MARKER: rc=0; [ 7987.018229] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7987.018229] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7987.018229] echo $(hostname -s): $val; [ 7987.018229] rc=$val; [ 7987.018229] fi; [ 7987.018229] exit $rc [ 7987.670441] Lustre: DEBUG MARKER: dmesg [ 7988.275419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ================================ 12:45:59 \(1491482759\) [ 7988.450851] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ================================ 12:45:59 (1491482759) [ 7990.343302] Lustre: DEBUG MARKER: sync; sync; sync [ 7991.314125] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7991.570231] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7991.713282] Turning device dm-2 (0xfc00002) read-only [ 7991.844052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7991.973555] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7992.974309] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7993.231763] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7993.404323] Removing read-only on unknown block (0xfc00002) [ 7993.538930] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8005.801086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8006.009854] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8017.259532] Lustre: DEBUG MARKER: hostname [ 8017.579155] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8017.833823] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8018.086703] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8018.246622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.450285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8018.709770] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8019.202723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8019.208629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8019.372177] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8019.373262] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8019.522464] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8019.782886] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8020.050655] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8020.312228] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8021.162417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8021.174924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8021.345149] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8021.383685] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8023.675385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8023.700700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8023.849044] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8023.954555] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8024.070639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8024.198492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8024.329354] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8024.407711] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8025.859430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8026.817522] Lustre: DEBUG MARKER: rc=0; [ 8026.817522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8026.817522] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8026.817522] echo $(hostname -s): $val; [ 8026.817522] rc=$val; [ 8026.817522] fi; [ 8026.817522] exit $rc [ 8027.446399] Lustre: DEBUG MARKER: dmesg [ 8028.091852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ================================ 12:46:39 \(1491482799\) [ 8028.266213] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ================================ 12:46:39 (1491482799) [ 8049.297036] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491482808/real 1491482808] req@ffff88006141a700 x1563924894429008/t0(0) o38->lustre-MDT0001-osp-MDT0002@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491482819 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8049.304970] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 8050.656161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8050.813584] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8064.018623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8064.179571] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8065.885557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8065.924089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8066.062232] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8066.208758] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8068.439719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8068.594406] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8068.795421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8068.955470] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8069.313223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8069.467343] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8069.664571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8069.832642] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8071.264068] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8072.197388] Lustre: DEBUG MARKER: rc=0; [ 8072.197388] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8072.197388] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8072.197388] echo $(hostname -s): $val; [ 8072.197388] rc=$val; [ 8072.197388] fi; [ 8072.197388] exit $rc [ 8072.863132] Lustre: DEBUG MARKER: dmesg [ 8073.376891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ================================ 12:47:24 \(1491482844\) [ 8073.542486] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ================================ 12:47:24 (1491482844) [ 8073.780219] Lustre: DEBUG MARKER: sync; sync; sync [ 8074.634934] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8074.892946] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8075.022609] LustreError: 30504:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0002 read-only *** [ 8075.025473] LustreError: 30504:0:(osd_handler.c:2176:osd_ro()) Skipped 11 previous similar messages [ 8075.041687] Turning device dm-2 (0xfc00002) read-only [ 8075.172988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8075.301737] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8077.366643] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8077.625693] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8077.800834] Removing read-only on unknown block (0xfc00002) [ 8077.935792] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8089.071119] Lustre: DEBUG MARKER: hostname [ 8089.385065] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8089.643603] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8089.902458] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8090.064127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.274203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8090.533728] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8091.028435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8091.031494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8091.190106] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8091.196788] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8091.338326] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8091.598479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8091.859537] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8092.123303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8104.384854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8104.550601] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8106.249332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8106.251170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8106.435270] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8106.543453] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8108.726295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8108.800078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8108.892038] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8109.006298] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8109.199645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8109.260156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8109.405346] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8109.473319] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8110.957386] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8111.884231] Lustre: DEBUG MARKER: rc=0; [ 8111.884231] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8111.884231] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8111.884231] echo $(hostname -s): $val; [ 8111.884231] rc=$val; [ 8111.884231] fi; [ 8111.884231] exit $rc [ 8112.519905] Lustre: DEBUG MARKER: dmesg [ 8113.030756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 =============================== 12:48:03 \(1491482883\) [ 8113.192622] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 =============================== 12:48:03 (1491482883) [ 8113.415173] Lustre: DEBUG MARKER: sync; sync; sync [ 8114.451315] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8114.708697] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8114.856660] Turning device dm-0 (0xfc00000) read-only [ 8114.987987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8115.118070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8117.005383] Lustre: DEBUG MARKER: sync; sync; sync [ 8117.913670] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8118.173469] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8118.305283] Turning device dm-2 (0xfc00002) read-only [ 8118.435989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8118.567884] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8118.746066] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8119.007339] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8119.181967] Removing read-only on unknown block (0xfc00000) [ 8119.315320] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8125.831284] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8126.087122] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8132.400389] Removing read-only on unknown block (0xfc00002) [ 8132.543177] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8142.853048] Lustre: DEBUG MARKER: hostname [ 8143.161860] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8143.411820] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8143.658572] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8143.815619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.046888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8144.317755] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8144.826312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8144.828225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8144.997297] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8144.998812] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8145.180287] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8145.434723] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8145.687329] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8145.954447] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8158.182195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8158.343581] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8169.571191] Lustre: DEBUG MARKER: hostname [ 8169.891178] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8170.146028] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8170.399540] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8170.558940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.763884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8171.023110] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8171.520453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8171.524537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8171.684845] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8171.687722] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8171.837837] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8172.098095] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8172.370824] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8172.637618] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8173.424307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8173.428095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8173.655507] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8173.671115] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8175.918350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8175.959656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8176.141837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8176.195074] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8176.368634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.509839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.562147] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.719416] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.819033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.966886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8177.016135] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8177.193055] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8178.642338] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8179.586281] Lustre: DEBUG MARKER: rc=0; [ 8179.586281] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8179.586281] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8179.586281] echo $(hostname -s): $val; [ 8179.586281] rc=$val; [ 8179.586281] fi; [ 8179.586281] exit $rc [ 8180.223383] Lustre: DEBUG MARKER: dmesg [ 8180.731094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 =============================== 12:49:11 \(1491482951\) [ 8180.894207] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 =============================== 12:49:11 (1491482951) [ 8181.097252] Lustre: DEBUG MARKER: sync; sync; sync [ 8182.208097] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8182.484299] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8182.630640] Turning device dm-0 (0xfc00000) read-only [ 8182.766668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8182.896927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8186.355123] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8186.622263] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8186.797441] Removing read-only on unknown block (0xfc00000) [ 8186.932426] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8204.900094] Lustre: DEBUG MARKER: hostname [ 8205.226149] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8205.479681] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8205.736835] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8205.951912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.169836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8206.429712] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8206.925168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8206.928083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8207.086414] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8207.091224] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8207.233353] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8207.492749] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8207.749434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8208.010567] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8220.254268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8220.426600] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8233.544838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8233.714968] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8235.416358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8235.417253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8235.624266] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8235.658069] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8237.894141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8237.963414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8238.160432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8238.205544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8238.428836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8238.551047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8238.621228] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8238.757530] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8238.907829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8239.019258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8239.066409] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8239.224736] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8240.638230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8241.573187] Lustre: DEBUG MARKER: rc=0; [ 8241.573187] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8241.573187] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8241.573187] echo $(hostname -s): $val; [ 8241.573187] rc=$val; [ 8241.573187] fi; [ 8241.573187] exit $rc [ 8242.214434] Lustre: DEBUG MARKER: dmesg [ 8242.755509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 =============================== 12:50:13 \(1491483013\) [ 8242.920060] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 =============================== 12:50:13 (1491483013) [ 8243.149816] Lustre: DEBUG MARKER: sync; sync; sync [ 8244.165736] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8244.424091] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8244.563417] Turning device dm-0 (0xfc00000) read-only [ 8244.692647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8244.822083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8244.956350] Lustre: DEBUG MARKER: sync; sync; sync [ 8245.793271] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8246.050213] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8246.193253] Turning device dm-2 (0xfc00002) read-only [ 8246.325507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8246.455755] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8248.415102] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8248.672212] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8248.848357] Removing read-only on unknown block (0xfc00000) [ 8248.983254] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8249.252830] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8249.507271] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8253.378434] LustreError: 1702:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491483024 with bad export cookie 7038122449846085018 [ 8253.382523] LustreError: 1702:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8255.678582] Removing read-only on unknown block (0xfc00002) [ 8255.825121] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8278.876595] Lustre: DEBUG MARKER: hostname [ 8279.186117] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8279.437434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8279.685612] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8279.830141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8280.064348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8280.323306] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8280.815725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8280.821282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8280.976441] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8280.979026] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8281.133149] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8281.388878] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8281.643004] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8281.900186] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8292.194561] Lustre: DEBUG MARKER: hostname [ 8292.520186] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8292.775395] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8293.028957] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8293.188199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.393035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8293.652206] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8294.146268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8294.154575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8294.309605] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8294.311768] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8294.459459] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8294.717606] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8294.977059] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8295.239183] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8307.520705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8307.703026] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8309.442516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8309.447444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8309.677946] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8309.678940] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8311.966909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8311.966927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8312.190079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8312.193974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8312.403030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8312.511122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8312.576616] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8312.729855] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8312.822518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8312.967880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8313.013840] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8313.201562] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8314.643671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8315.593131] Lustre: DEBUG MARKER: rc=0; [ 8315.593131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8315.593131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8315.593131] echo $(hostname -s): $val; [ 8315.593131] rc=$val; [ 8315.593131] fi; [ 8315.593131] exit $rc [ 8316.233916] Lustre: DEBUG MARKER: dmesg [ 8316.783272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 =============================== 12:51:27 \(1491483087\) [ 8316.946647] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 =============================== 12:51:27 (1491483087) [ 8318.915803] Lustre: DEBUG MARKER: sync; sync; sync [ 8320.074998] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8320.334347] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8320.476075] Turning device dm-2 (0xfc00002) read-only [ 8320.606022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8320.736120] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8323.356975] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8323.613670] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8323.786659] Removing read-only on unknown block (0xfc00002) [ 8323.922266] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8337.016561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8337.180228] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8348.377501] Lustre: DEBUG MARKER: hostname [ 8348.697844] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8348.964740] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8349.222182] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8349.437443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.653218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8349.912488] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8350.409248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8350.415181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8350.575299] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8350.577083] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8350.730524] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8350.987234] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8351.251676] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8351.515675] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8354.490276] Lustre: lustre-MDT0002: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [ 8354.493745] Lustre: Skipped 328 previous similar messages [ 8363.763981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8363.923745] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8365.641859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8365.685598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8365.821622] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8365.924750] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8368.169276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8368.337615] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8368.545646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8368.710149] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8368.910404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8369.076671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8369.087299] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8369.265405] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8369.455043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8369.636804] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8369.835535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8369.987746] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8371.380604] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8372.310487] Lustre: DEBUG MARKER: rc=0; [ 8372.310487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8372.310487] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8372.310487] echo $(hostname -s): $val; [ 8372.310487] rc=$val; [ 8372.310487] fi; [ 8372.310487] exit $rc [ 8372.942613] Lustre: DEBUG MARKER: dmesg [ 8373.459979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 115: failover for create\/unlink striped directory ============================== 12:52:24 \(1491483144\) [ 8373.633791] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ============================== 12:52:24 (1491483144) [ 8388.442232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8388.621137] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8390.307615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8390.308495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8390.524783] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8390.527344] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8392.796119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8392.823237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8393.021655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8393.065028] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8394.823259] Lustre: DEBUG MARKER: sync; sync; sync [ 8395.782026] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8396.038960] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8396.183215] Turning device dm-2 (0xfc00002) read-only [ 8396.314201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8396.442691] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8396.680425] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8396.936693] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8397.065167] Lustre: Failing over lustre-MDT0002 [ 8397.067295] Lustre: Skipped 13 previous similar messages [ 8397.935073] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880060495b00 x1563924894469456/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8397.940245] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 8402.753393] Removing read-only on unknown block (0xfc00002) [ 8402.874076] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8402.879090] LustreError: Skipped 508 previous similar messages [ 8402.889495] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8413.193159] Lustre: DEBUG MARKER: hostname [ 8413.510284] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8413.766181] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8414.019065] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8414.171157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.388513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8414.647899] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8415.139905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8415.143203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8415.306714] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8415.307800] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8415.457130] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8415.712998] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8415.969630] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8416.231856] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8416.990736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8417.003296] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8417.162054] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8417.261440] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8419.524284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8419.680153] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8420.413479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8420.568175] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8436.969867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8437.136977] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8438.821394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8438.821396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8439.015565] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8439.060767] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8441.263761] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8441.436720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8441.496595] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8441.686674] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8443.337729] Lustre: DEBUG MARKER: sync; sync; sync [ 8444.393655] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8444.664377] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8444.813350] Turning device dm-0 (0xfc00000) read-only [ 8444.944064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8445.072027] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8445.379303] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8445.648231] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8445.998891] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 8446.001717] Lustre: Skipped 15 previous similar messages [ 8446.895064] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880067b56400 x1563924894479504/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8446.900902] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 12 previous similar messages [ 8451.749091] Removing read-only on unknown block (0xfc00000) [ 8451.883377] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8462.197532] Lustre: DEBUG MARKER: hostname [ 8462.509504] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8462.762802] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8463.015948] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8463.233768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.650206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8466.909189] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8467.415529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8467.419834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8467.579089] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8467.579090] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8467.728213] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8467.986621] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8468.245595] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8468.508978] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8469.276904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8469.280822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8469.504782] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8469.517773] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8471.789302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8471.792472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8472.015876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8472.017579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8473.756805] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8474.727440] Lustre: DEBUG MARKER: rc=0; [ 8474.727440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8474.727440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8474.727440] echo $(hostname -s): $val; [ 8474.727440] rc=$val; [ 8474.727440] fi; [ 8474.727440] exit $rc [ 8475.371968] Lustre: DEBUG MARKER: dmesg [ 8476.022978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 116a: large update log master MDT recovery ===================================== 12:54:06 \(1491483246\) [ 8476.185083] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ===================================== 12:54:06 (1491483246) [ 8476.334766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8476.334766] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8476.334766] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8476.624746] Lustre: DEBUG MARKER: sync; sync; sync [ 8477.590523] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8477.858200] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8477.995892] Turning device dm-0 (0xfc00000) read-only [ 8478.125190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8478.252286] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8478.386718] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80001702 [ 8478.524722] Lustre: *** cfs_fail_loc=1702, val=0*** [ 8478.789599] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8479.048507] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8479.226221] Removing read-only on unknown block (0xfc00000) [ 8479.360905] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8489.678531] Lustre: DEBUG MARKER: hostname [ 8489.988403] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8490.239058] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8490.490342] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8490.648374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.719214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8494.977902] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8495.480795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8495.483675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8495.639525] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8495.648499] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8495.788358] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8496.042618] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8496.298967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8496.572411] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8497.340493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8497.340528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8497.558934] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8497.612213] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8499.567534] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 8499.570621] Lustre: Skipped 13 previous similar messages [ 8499.808044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8499.906541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8499.988968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8500.124130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8501.514967] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8502.447860] Lustre: DEBUG MARKER: rc=0; [ 8502.447860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8502.447860] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8502.447860] echo $(hostname -s): $val; [ 8502.447860] rc=$val; [ 8502.447860] fi; [ 8502.447860] exit $rc [ 8503.085258] Lustre: DEBUG MARKER: dmesg [ 8503.610845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 116b: large update log slave MDT recovery ====================================== 12:54:34 \(1491483274\) [ 8503.776399] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ====================================== 12:54:34 (1491483274) [ 8503.919218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8503.919218] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8503.919218] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8509.496591] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 8509.499452] LustreError: Skipped 37 previous similar messages [ 8519.010238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8519.175739] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8520.966614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8520.968182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8521.133799] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8521.133815] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8523.364089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8523.440931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8523.537300] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8523.662642] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8525.034568] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8525.972903] Lustre: DEBUG MARKER: rc=0; [ 8525.972903] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8525.972903] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8525.972903] echo $(hostname -s): $val; [ 8525.972903] rc=$val; [ 8525.972903] fi; [ 8525.972903] exit $rc [ 8526.605503] Lustre: DEBUG MARKER: dmesg [ 8527.153526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ================================= 12:54:58 \(1491483298\) [ 8527.316436] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ================================= 12:54:58 (1491483298) [ 8529.493959] Lustre: DEBUG MARKER: sync; sync; sync [ 8530.367658] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8530.628147] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8530.780776] Turning device dm-0 (0xfc00000) read-only [ 8530.916192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8531.044947] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8532.839366] Lustre: DEBUG MARKER: sync; sync; sync [ 8533.662360] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8533.918204] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8534.060422] Turning device dm-2 (0xfc00002) read-only [ 8534.192391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8534.322961] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8536.261725] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8536.518209] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8536.696432] Removing read-only on unknown block (0xfc00000) [ 8536.703474] Lustre: server umount lustre-MDT0000 complete [ 8536.705565] Lustre: Skipped 13 previous similar messages [ 8536.835419] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8537.976522] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8538.245681] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8544.427560] Removing read-only on unknown block (0xfc00002) [ 8544.575183] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8555.721159] Lustre: DEBUG MARKER: hostname [ 8556.035678] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8556.286892] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8556.532775] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8556.689549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.762042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.764131] Lustre: Skipped 13 previous similar messages [ 8556.908817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8557.164447] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8557.655015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8557.657879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8557.815965] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8557.820036] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8557.970939] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8558.227022] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8558.478388] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8558.492212] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 8558.494687] Lustre: Skipped 13 previous similar messages [ 8558.741123] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8570.971610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8571.138799] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8582.350252] Lustre: DEBUG MARKER: hostname [ 8582.670492] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8582.925970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8583.177571] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8583.336037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.538316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8583.800687] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8584.305749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8584.316960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8584.476933] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8584.478446] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8584.632766] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8584.893858] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8585.164044] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8585.429956] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8597.692120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8597.863271] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8599.634333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_u [ 8599.634335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_u [ 8599.885454] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8599.905036] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8602.170165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8602.239810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8602.341889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8602.439862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8602.641572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8602.675655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8602.812632] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8602.939277] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.125086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.171411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.330244] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.467436] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.553884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.738392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.771933] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8603.950476] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8605.318425] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8606.264118] Lustre: DEBUG MARKER: rc=0; [ 8606.264118] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8606.264118] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8606.264118] echo $(hostname -s): $val; [ 8606.264118] rc=$val; [ 8606.264118] fi; [ 8606.264118] exit $rc [ 8606.893657] Lustre: DEBUG MARKER: dmesg [ 8607.412419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 118: invalidate osp update will not cause update log corruption ================ 12:56:18 \(1491483378\) [ 8607.590392] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ================ 12:56:18 (1491483378) [ 8607.745005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8607.745005] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8607.745005] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8608.131096] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1705 [ 8608.267056] Lustre: *** cfs_fail_loc=1705, val=0*** [ 8608.411733] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 8608.666664] Lustre: DEBUG MARKER: sync; sync; sync [ 8608.868694] LustreError: 23808:0:(llog_cat.c:764:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8609.635408] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8609.896102] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8610.035032] Turning device dm-0 (0xfc00000) read-only [ 8610.165129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8610.296674] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8610.464504] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8610.725279] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8611.983074] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880062e33300 x1563924894502480/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8611.987600] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 8613.388197] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8613.392327] Lustre: Skipped 33 previous similar messages [ 8616.740785] Removing read-only on unknown block (0xfc00000) [ 8616.873928] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8625.388058] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8625.392061] LustreError: Skipped 3 previous similar messages [ 8627.193981] Lustre: DEBUG MARKER: hostname [ 8627.509152] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8627.763808] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8628.017701] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8628.181151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.394179] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc76795 to 0x61ac6f141cc77373 [ 8631.398344] Lustre: Skipped 3 previous similar messages [ 8631.638930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8631.900303] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8632.397114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8632.398920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8632.565751] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8632.570786] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8632.714297] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8632.971910] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8633.229324] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8633.490797] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8634.245264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8634.248149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8634.415671] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8634.518296] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8636.430841] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 8636.436728] LustreError: Skipped 3 previous similar messages [ 8636.716649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8636.760406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8636.903943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8637.001948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8638.365388] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8639.298201] Lustre: DEBUG MARKER: rc=0; [ 8639.298201] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8639.298201] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8639.298201] echo $(hostname -s): $val; [ 8639.298201] rc=$val; [ 8639.298201] fi; [ 8639.298201] exit $rc [ 8639.933741] Lustre: DEBUG MARKER: dmesg [ 8640.450518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 119: timeout of normal replay does not cause DNE replay fails ================= 12:56:51 \(1491483411\) [ 8640.620286] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ================= 12:56:51 (1491483411) [ 8640.761624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8640.761624] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8640.761624] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8641.042976] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.recovery_time_hard [ 8641.308861] Lustre: DEBUG MARKER: sync; sync; sync [ 8642.276381] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8642.538249] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8642.686135] Turning device dm-0 (0xfc00000) read-only [ 8642.817046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8642.949067] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8643.157179] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8643.416390] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8643.594449] Removing read-only on unknown block (0xfc00000) [ 8643.732799] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8644.050306] Lustre: DEBUG MARKER: hostname [ 8644.320683] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000714 [ 8644.571804] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=65 [ 8644.839151] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8645.097726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8645.358071] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o recovery_time_hard=60 /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8645.517548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.745537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8646.014580] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8646.507662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8646.512381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8646.683057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8646.683058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8646.833220] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8647.093524] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8647.366580] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8647.643767] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8648.028518] Lustre: 23774:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [ 8648.414262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8648.415092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8648.602200] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8648.680295] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8648.811573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8648.961346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8648.978422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8649.173423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8650.308916] Lustre: 24697:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [ 8650.313961] Lustre: 24697:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 1 previous similar message [ 8650.565220] LustreError: 30553:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 714 sleeping for 65000ms [ 8715.568044] LustreError: 30553:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 714 awake [ 8715.570856] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 8715.573444] Lustre: 30553:0:(ldlm_lib.c:1625:abort_req_replay_queue()) @@@ aborted: req@ffff88005d6c3300 x1563925381605040/t0(554050781195) o36->0cafaf20-c923-c85e-f465-f551940bce00@10.9.5.237@tcp:-1/-1 lens 616/0 e 7 to 0 dl 1491483496 ref 1 fl Complete:/4/ffffffff rc 0/-1 [ 8715.581811] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 8715.585080] Lustre: 30553:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [ 8715.592062] Lustre: 30553:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 2 previous similar messages [ 8715.595025] Lustre: lustre-MDT0000: Denying connection for new client 0cafaf20-c923-c85e-f465-f551940bce00(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 1 evicted) to recover in 21188505:12 [ 8715.618885] Lustre: 30553:0:(ldlm_lib.c:2044:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 8715.622017] Lustre: 30553:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 8715.625050] Lustre: 30553:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 8715.632107] Lustre: 30553:0:(ldlm_lib.c:2550:target_recovery_thread()) too long recovery - read logs [ 8715.635289] LustreError: dumping log to /tmp/lustre-log.1491483486.30553 [ 8726.758881] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.recovery_time_hard=180 [ 8727.199902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8728.139072] Lustre: DEBUG MARKER: rc=0; [ 8728.139072] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8728.139072] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8728.139072] echo $(hostname -s): $val; [ 8728.139072] rc=$val; [ 8728.139072] fi; [ 8728.139072] exit $rc [ 8728.788254] Lustre: DEBUG MARKER: dmesg [ 8729.327620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 120: DNE fail abort should stop both normal and DNE replay ===================== 12:58:20 \(1491483500\) [ 8729.493713] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ===================== 12:58:20 (1491483500) [ 8729.648253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8729.648253] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8729.648253] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8729.927855] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8730.186678] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8730.330220] LustreError: 32247:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 8730.332803] LustreError: 32247:0:(osd_handler.c:2176:osd_ro()) Skipped 13 previous similar messages [ 8730.372140] Turning device dm-0 (0xfc00000) read-only [ 8730.510360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8730.640682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8730.908959] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8731.164365] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8735.695082] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005fe58600 x1563924894517408/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8735.700315] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 8736.742841] Removing read-only on unknown block (0xfc00000) [ 8736.878869] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8737.200351] Lustre: DEBUG MARKER: hostname [ 8737.480213] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8737.736393] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8737.989453] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8738.147362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.220475] LustreError: 371:0:(mdt_handler.c:6050:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 8738.223345] LustreError: 371:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 8738.225979] Lustre: 422:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 8738.228544] Lustre: 422:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 1 previous similar message [ 8738.245860] LustreError: 419:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 getting update log failed: rc = -108 [ 8738.249948] LustreError: 419:0:(lod_dev.c:419:lod_sub_recovery_thread()) Skipped 1 previous similar message [ 8738.252472] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 8738.403829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8738.664873] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8739.166290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8739.166292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8739.322081] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8739.334682] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8739.488115] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8739.754788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8740.023950] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8740.302717] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8743.165559] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8744.082093] Lustre: DEBUG MARKER: rc=0; [ 8744.082093] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8744.082093] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8744.082093] echo $(hostname -s): $val; [ 8744.082093] rc=$val; [ 8744.082093] fi; [ 8744.082093] exit $rc [ 8744.710706] Lustre: DEBUG MARKER: dmesg [ 8745.243796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test complete, duration 7659 sec ==================================================== 12:58:36 \(1491483516\) [ 8745.414880] Lustre: DEBUG MARKER: == replay-single test complete, duration 7659 sec ==================================================== 12:58:36 (1491483516) [ 8749.395629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-quota ============----- Thu Apr 6 12:58:40 UTC 2017 [ 8749.558127] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Thu Apr 6 12:58:40 UTC 2017 [ 8749.829200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test complete, duration -o sec ======================================================= 12:58:40 \(1491483520\) [ 8749.991542] Lustre: DEBUG MARKER: == sanity-quota test complete, duration -o sec ======================================================= 12:58:40 (1491483520) [ 8750.294346] Lustre: DEBUG MARKER: ! debugfs -c -R supported_features | [ 8750.294346] grep -q 'quota' [ 8751.063123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 8751.090164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 8751.257917] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 8751.336255] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 8753.960720] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8753.960720] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8753.960720] if [ $running -ne $mpts ]; then [ 8753.960720] echo $(hostname) env are INSANE!; [ 8753.960720] exit 1; [ 8753.960720] fi [ 8754.539741] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 8754.539741] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 8754.539741] if [ $running -ne $mpts ]; then [ 8754.539741] echo $(hostname) env are INSANE!; [ 8754.539741] exit 1; [ 8754.539741] fi [ 8755.155448] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8755.155448] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8755.155448] if [ $running -ne $mpts ]; then [ 8755.155448] echo $(hostname) env are INSANE!; [ 8755.155448] exit 1; [ 8755.155448] fi [ 8756.041380] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8756.299916] Lustre: DEBUG MARKER: cat /proc/mounts [ 8757.086317] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8757.351044] Lustre: DEBUG MARKER: cat /proc/mounts [ 8762.649245] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 8762.963682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 8763.127272] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8763.271749] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 8763.606264] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 8764.201602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.224835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.247881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.271652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.425071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.495933] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.552831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.645060] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.723502] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8764.886434] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8765.222586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 8765.515779] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8766.034872] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8766.610203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 8766.771975] Lustre: DEBUG MARKER: excepting tests: [ 8767.146967] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8771.568498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 0: Test basic quota performance ================================================= 12:59:02 \(1491483542\) [ 8771.734623] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ================================================= 12:59:02 (1491483542) [ 8772.156218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8772.429278] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8772.702964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8773.980768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8775.313785] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8775.612982] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8782.712266] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8796.831984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8797.095433] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8797.441449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8798.731806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8800.024544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8801.306715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8802.589647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8803.880375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8805.193088] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8806.858443] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8807.793071] Lustre: DEBUG MARKER: rc=0; [ 8807.793071] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8807.793071] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8807.793071] echo $(hostname -s): $val; [ 8807.793071] rc=$val; [ 8807.793071] fi; [ 8807.793071] exit $rc [ 8808.408109] Lustre: DEBUG MARKER: dmesg [ 8808.909695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 1: Block hard limit \(normal use and out of quota\) =============================== 12:59:39 \(1491483579\) [ 8809.067804] Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) =============================== 12:59:39 (1491483579) [ 8809.492651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8810.811523] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8811.104246] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8817.974601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(block hardlimit:10 MB\) [ 8818.126114] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 8818.434931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write... [ 8818.594330] Lustre: DEBUG MARKER: Write... [ 8818.950674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8819.105395] Lustre: DEBUG MARKER: Write out of block quota ... [ 8819.780287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8820.046755] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8820.336757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8821.613566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8822.891779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8824.176723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8825.477656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8826.759215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8828.040605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8829.318432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8830.626046] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8832.329321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8832.484600] Lustre: DEBUG MARKER: -------------------------------------- [ 8832.677807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(block hardlimit:10 MB\) [ 8832.843107] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 8833.074160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write ... [ 8833.234798] Lustre: DEBUG MARKER: Write ... [ 8833.593210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8833.745350] Lustre: DEBUG MARKER: Write out of block quota ... [ 8834.471822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8834.744431] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8835.076939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8836.360647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8837.651889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8838.940888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8840.231057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8841.509451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8842.797552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8844.073190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8845.381530] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8847.088479] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8847.975922] Lustre: DEBUG MARKER: rc=0; [ 8847.975922] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8847.975922] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8847.975922] echo $(hostname -s): $val; [ 8847.975922] rc=$val; [ 8847.975922] fi; [ 8847.975922] exit $rc [ 8848.574138] Lustre: DEBUG MARKER: dmesg [ 8849.081039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 2: File hard limit \(normal use and out of quota\) ================================ 13:00:20 \(1491483620\) [ 8849.245276] Lustre: DEBUG MARKER: == sanity-quota test 2: File hard limit (normal use and out of quota) ================================ 13:00:20 (1491483620) [ 8849.680076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8850.992913] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8851.298710] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8851.566928] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8851.825437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8852.083546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8853.340805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8854.597891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8855.855691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8857.112447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8859.487253] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8859.756134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8860.017486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8861.100036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(inode hardlimit:1024 files\) [ 8861.272380] Lustre: DEBUG MARKER: User quota (inode hardlimit:1024 files) [ 8861.492785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8861.652683] Lustre: DEBUG MARKER: Create 1024 files ... [ 8862.475431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8862.630629] Lustre: DEBUG MARKER: Create out of file quota ... [ 8864.069757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8866.514897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8866.684701] Lustre: DEBUG MARKER: -------------------------------------- [ 8866.881061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(inode hardlimit:1024 files\) [ 8867.043789] Lustre: DEBUG MARKER: Group quota (inode hardlimit:1024 files) [ 8867.344859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8867.503190] Lustre: DEBUG MARKER: Create 1024 files ... [ 8868.217839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8868.386821] Lustre: DEBUG MARKER: Create out of file quota ... [ 8869.867838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8871.441923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8872.734267] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8874.436553] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8875.337489] Lustre: DEBUG MARKER: rc=0; [ 8875.337489] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8875.337489] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8875.337489] echo $(hostname -s): $val; [ 8875.337489] rc=$val; [ 8875.337489] fi; [ 8875.337489] exit $rc [ 8875.943378] Lustre: DEBUG MARKER: dmesg [ 8876.494662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 3: Block soft limit \(start timer, timer goes off, stop timer\) =================== 13:00:47 \(1491483647\) [ 8876.647841] Lustre: DEBUG MARKER: == sanity-quota test 3: Block soft limit (start timer, timer goes off, stop timer) =================== 13:00:47 (1491483647) [ 8876.794985] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8883.995839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8885.328085] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8916.166614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8916.437433] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8916.724627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8918.002046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8919.290156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8920.565925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8921.841199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8923.124608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8924.400375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8925.707823] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8926.732360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8926.998773] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8927.331776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8928.615098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8929.892674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8931.168977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8932.455730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8933.743361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8935.049690] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8937.012285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8938.330945] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8969.182366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8969.450652] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8969.736662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8971.020366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8972.298775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8973.574813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8974.859930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8976.145563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8977.421039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8978.703025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8980.011947] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8981.048785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8981.310866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8981.604620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8982.884689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8984.168620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8985.449928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8986.732775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8988.011069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8989.291129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8990.607717] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8992.367936] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8993.262639] Lustre: DEBUG MARKER: rc=0; [ 8993.262639] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8993.262639] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8993.262639] echo $(hostname -s): $val; [ 8993.262639] rc=$val; [ 8993.262639] fi; [ 8993.262639] exit $rc [ 8993.880506] Lustre: DEBUG MARKER: dmesg [ 8994.393082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 4a: File soft limit \(start timer, timer goes off, stop timer\) =================== 13:02:45 \(1491483765\) [ 8994.566832] Lustre: DEBUG MARKER: == sanity-quota test 4a: File soft limit (start timer, timer goes off, stop timer) =================== 13:02:45 (1491483765) [ 8994.750539] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8995.014853] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8995.273446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8995.528531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8996.569918] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8996.829541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8997.088466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8998.441358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8999.764065] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9000.091325] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9000.799625] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9019.477713] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9020.140213] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9021.070723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9021.338074] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9021.613040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9022.889356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9024.171603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9025.451552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9026.730913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9028.014968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9029.306414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9030.628429] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9031.537924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9031.803858] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9032.147060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9033.424934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9034.703078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9035.984607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9037.279953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9038.565965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9039.873409] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9041.807966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9043.129197] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9043.452854] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9044.110355] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9062.788977] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9063.767802] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9064.690734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9064.962326] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9065.237289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9066.516709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9067.799128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9069.089431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9070.369378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9071.645668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9072.933374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9074.222892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9075.538747] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9076.439308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9076.707750] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9076.997595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9078.273518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9079.553184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9080.838542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9082.125366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9083.411658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9084.707702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9086.025963] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9087.737828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9088.621635] Lustre: DEBUG MARKER: rc=0; [ 9088.621635] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9088.621635] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9088.621635] echo $(hostname -s): $val; [ 9088.621635] rc=$val; [ 9088.621635] fi; [ 9088.621635] exit $rc [ 9089.239224] Lustre: DEBUG MARKER: dmesg [ 9089.762699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 4b: Grace time strings handling ================================================= 13:04:20 \(1491483860\) [ 9089.921359] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ================================================= 13:04:20 (1491483860) [ 9090.338515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9091.726852] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9092.625954] Lustre: DEBUG MARKER: rc=0; [ 9092.625954] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9092.625954] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9092.625954] echo $(hostname -s): $val; [ 9092.625954] rc=$val; [ 9092.625954] fi; [ 9092.625954] exit $rc [ 9093.220747] Lustre: DEBUG MARKER: dmesg [ 9093.717172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 5: Chown & chgrp successfully even out of block\/file quota ====================== 13:04:24 \(1491483864\) [ 9093.874074] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 9094.423528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9095.744435] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9096.044334] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9096.310626] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9096.572378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9096.828491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9097.864111] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9098.121645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9098.379758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9099.414495] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9106.894808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9107.163060] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9107.441856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9108.722329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9110.005376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9111.288405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9112.565383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9113.841720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9115.147052] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9117.891398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9118.783674] Lustre: DEBUG MARKER: rc=0; [ 9118.783674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9118.783674] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9118.783674] echo $(hostname -s): $val; [ 9118.783674] rc=$val; [ 9118.783674] fi; [ 9118.783674] exit $rc [ 9119.409834] Lustre: DEBUG MARKER: dmesg [ 9120.038133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 6: Test dropping acquire request on master ====================================== 13:04:51 \(1491483891\) [ 9120.218194] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ====================================== 13:04:51 (1491483891) [ 9120.999384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9122.329723] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9122.641120] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9130.686230] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9131.386917] Lustre: DEBUG MARKER: lctl set_param fail_val=601 fail_loc=0x513 [ 9131.537450] Lustre: *** cfs_fail_loc=513, val=0*** [ 9131.539003] Lustre: Skipped 10 previous similar messages [ 9132.018479] LustreError: 4367:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1563924927890528 [ 9139.507825] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 9139.509960] Lustre: Skipped 14 previous similar messages [ 9139.511599] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 9139.513526] Lustre: Skipped 211 previous similar messages [ 9140.472109] Lustre: *** cfs_fail_loc=513, val=0*** [ 9140.474105] Lustre: Skipped 85 previous similar messages [ 9140.475873] LustreError: 4367:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1563924927891616 [ 9147.516326] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 9150.473462] LustreError: 4367:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1563924927892192 [ 9157.473331] Lustre: *** cfs_fail_loc=513, val=0*** [ 9157.475009] Lustre: Skipped 91 previous similar messages [ 9157.476733] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 9161.472723] LustreError: 4367:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1563924927892752 [ 9172.842328] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9174.693799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9174.962196] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9175.249675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9176.530853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9177.814766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9179.091414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9180.369213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9181.654158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9182.932098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9184.210631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9185.533883] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9187.228056] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9188.118642] Lustre: DEBUG MARKER: rc=0; [ 9188.118642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9188.118642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9188.118642] echo $(hostname -s): $val; [ 9188.118642] rc=$val; [ 9188.118642] fi; [ 9188.118642] exit $rc [ 9188.730801] Lustre: DEBUG MARKER: dmesg [ 9189.209932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7a: Quota reintegration \(global index\) ========================================== 13:06:00 \(1491483960\) [ 9189.390686] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================== 13:06:00 (1491483960) [ 9189.806919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9191.129153] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9191.456968] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9199.373050] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9207.130082] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9207.871419] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 9207.873757] LustreError: Skipped 16 previous similar messages [ 9223.092172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9223.269937] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9224.314792] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9242.693197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9242.966094] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9243.255291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9244.535673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9245.833404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9247.120384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9248.402852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9249.689348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9250.969414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9252.287576] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9258.873519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9259.049442] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9260.083906] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9260.576199] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9260.580244] Lustre: Skipped 7 previous similar messages [ 9274.324215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9274.593624] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9274.884065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9276.163632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9277.445749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9278.729975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9280.010154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9281.292206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9282.572985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9283.849170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9285.133791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9286.459246] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9288.154516] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9289.054501] Lustre: DEBUG MARKER: rc=0; [ 9289.054501] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9289.054501] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9289.054501] echo $(hostname -s): $val; [ 9289.054501] rc=$val; [ 9289.054501] fi; [ 9289.054501] exit $rc [ 9289.664972] Lustre: DEBUG MARKER: dmesg [ 9290.154722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7b: Quota reintegration \(slave index\) =========================================== 13:07:41 \(1491484061\) [ 9290.327093] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) =========================================== 13:07:41 (1491484061) [ 9290.750045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9292.074020] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9292.396707] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9300.278830] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9307.284401] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0xa02 [ 9307.573474] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9319.440900] Lustre: *** cfs_fail_loc=a02, val=0*** [ 9319.755222] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9322.991963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9323.166481] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9324.217914] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9340.715445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9340.996921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9341.298238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9342.589729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9343.868384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9345.158692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9346.436314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9347.721752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9349.031637] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9350.704230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9351.616656] Lustre: DEBUG MARKER: rc=0; [ 9351.616656] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9351.616656] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9351.616656] echo $(hostname -s): $val; [ 9351.616656] rc=$val; [ 9351.616656] fi; [ 9351.616656] exit $rc [ 9352.237945] Lustre: DEBUG MARKER: dmesg [ 9352.713647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7c: Quota reintegration \(restart mds during reintegration\) ====================== 13:08:43 \(1491484123\) [ 9352.876813] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ====================== 13:08:43 (1491484123) [ 9353.327076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9354.662797] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9354.984411] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9371.114516] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9379.495689] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9379.751309] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 9379.882678] Lustre: Failing over lustre-MDT0000 [ 9379.884445] Lustre: Skipped 7 previous similar messages [ 9380.078484] Lustre: server umount lustre-MDT0000 complete [ 9380.080263] Lustre: Skipped 4 previous similar messages [ 9380.210113] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9380.472726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.476815] LustreError: Skipped 241 previous similar messages [ 9380.799365] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9381.064770] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9381.321391] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9381.579431] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9381.732834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.750572] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9381.754143] LustreError: Skipped 2 previous similar messages [ 9381.756050] LustreError: 23765:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [ 9381.757387] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cc789df to 0x61ac6f141cca1628 [ 9381.757388] Lustre: Skipped 2 previous similar messages [ 9381.791052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.793189] Lustre: Skipped 4 previous similar messages [ 9381.983945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9382.244295] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 9382.744299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9382.745614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9382.912351] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9382.914899] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9383.031130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 9383.033798] Lustre: Skipped 3 previous similar messages [ 9383.063722] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9383.328093] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9383.591235] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9383.857645] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9384.118091] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9384.431605] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9386.786567] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9386.791793] LustreError: Skipped 3 previous similar messages [ 9386.888013] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 9386.890978] Lustre: Skipped 4 previous similar messages [ 9507.665310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9507.937618] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9508.231578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9509.509446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9510.793532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9512.086289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9513.376878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9514.665404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9515.945778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9517.232071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9518.564399] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9520.282084] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9521.188589] Lustre: DEBUG MARKER: rc=0; [ 9521.188589] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9521.188589] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9521.188589] echo $(hostname -s): $val; [ 9521.188589] rc=$val; [ 9521.188589] fi; [ 9521.188589] exit $rc [ 9521.804641] Lustre: DEBUG MARKER: dmesg [ 9522.415277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7d: Quota reintegration \(Transfer index in multiple bulks\) ====================== 13:11:33 \(1491484293\) [ 9522.567665] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ====================== 13:11:33 (1491484293) [ 9522.987127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9524.312673] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9524.610719] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9540.451584] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x608 [ 9540.742191] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9554.231419] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9587.741972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9588.019639] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9588.314819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9589.595376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9590.872959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9592.157051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9593.449165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9594.727579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9596.008824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9597.298962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9598.617971] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9601.364356] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9602.266882] Lustre: DEBUG MARKER: rc=0; [ 9602.266882] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9602.266882] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9602.266882] echo $(hostname -s): $val; [ 9602.266882] rc=$val; [ 9602.266882] fi; [ 9602.266882] exit $rc [ 9602.886660] Lustre: DEBUG MARKER: dmesg [ 9603.377573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7e: Quota reintegration \(inode limits\) ========================================== 13:12:54 \(1491484374\) [ 9603.542858] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================== 13:12:54 (1491484374) [ 9603.971402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9605.288854] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9605.609280] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9605.880895] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9606.145028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9606.406617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9607.466069] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9607.724099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9607.983081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9610.058477] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=none [ 9610.322491] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9610.580114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9610.840294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9612.105057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9613.365365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9614.637122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9615.892931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9616.917582] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9617.176679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9617.435534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9619.333961] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9619.599761] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9619.857817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9620.113710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9621.372596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9622.627727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9623.887912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9625.155902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9626.418405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9627.680787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9628.935666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9630.192473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9631.214418] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9631.474504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9631.734441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9634.219652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9634.380132] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9635.859876] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9636.181565] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9636.443192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9636.443192] grep user | awk '{ print $3 }' [ 9636.701122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9636.701122] grep user | awk '{ print $3 }' [ 9637.746504] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9638.004872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9638.004872] grep user | awk '{ print $3 }' [ 9638.263928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9638.263928] grep user | awk '{ print $3 }' [ 9639.332441] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9639.608990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9639.608990] grep group | awk '{ print $3 }' [ 9639.868201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9639.868201] grep group | awk '{ print $3 }' [ 9640.942897] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9641.202004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9641.202004] grep group | awk '{ print $3 }' [ 9641.462482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9641.462482] grep group | awk '{ print $3 }' [ 9651.542024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9652.839638] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9656.272161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9656.437037] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9657.917974] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9658.224643] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9658.486104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9658.486104] grep user | awk '{ print $3 }' [ 9658.746759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9658.746759] grep user | awk '{ print $3 }' [ 9659.799808] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9660.059434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9660.059434] grep user | awk '{ print $3 }' [ 9660.319384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9660.319384] grep user | awk '{ print $3 }' [ 9661.374701] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9661.634586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9661.634586] grep group | awk '{ print $3 }' [ 9661.896369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9661.896369] grep group | awk '{ print $3 }' [ 9662.963444] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9663.230716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9663.230716] grep group | awk '{ print $3 }' [ 9663.494568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9663.494568] grep group | awk '{ print $3 }' [ 9673.664421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9674.954704] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9676.689138] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9677.586991] Lustre: DEBUG MARKER: rc=0; [ 9677.586991] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9677.586991] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9677.586991] echo $(hostname -s): $val; [ 9677.586991] rc=$val; [ 9677.586991] fi; [ 9677.586991] exit $rc [ 9678.200876] Lustre: DEBUG MARKER: dmesg [ 9678.671246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 8: Run dbench with quota enabled ================================================ 13:14:09 \(1491484449\) [ 9678.835820] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ================================================ 13:14:09 (1491484449) [ 9679.253054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9680.572919] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9680.873576] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9681.139950] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9681.401920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9681.660590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9682.694937] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9682.956092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9683.215095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9684.254954] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9838.700292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9838.972911] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9839.286409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9840.567476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9841.854857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9843.142363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9844.426294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9845.713201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9847.001031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9848.288173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9849.596794] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9852.376772] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9853.297907] Lustre: DEBUG MARKER: rc=0; [ 9853.297907] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9853.297907] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9853.297907] echo $(hostname -s): $val; [ 9853.297907] rc=$val; [ 9853.297907] fi; [ 9853.297907] exit $rc [ 9853.920695] Lustre: DEBUG MARKER: dmesg [ 9854.549924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 9: Block limit larger than 4GB \(b10707\) ========================================= 13:17:05 \(1491484625\) [ 9854.718572] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================= 13:17:05 (1491484625) [ 9854.918868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 14959760 required: 4900000 [ 9855.077616] Lustre: DEBUG MARKER: OST0_SIZE: 14959760 required: 4900000 [ 9855.511334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9856.836532] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9857.133724] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9864.006687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set enough high limit\(block:100G\; file: 1000000\) for user: quota_usr [ 9864.167652] Lustre: DEBUG MARKER: Set enough high limit(block:100G; file: 1000000) for user: quota_usr [ 9864.387074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set enough high limit\(block:100G\; file: 1000000\) for group: quota_usr [ 9864.551331] Lustre: DEBUG MARKER: Set enough high limit(block:100G; file: 1000000) for group: quota_usr [ 9864.854457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write the big file of 4.5G ... [ 9865.019044] Lustre: DEBUG MARKER: Write the big file of 4.5G ... [10072.324717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10072.681318] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10072.988923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10074.287601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10075.564329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10076.841533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10078.118712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10079.426203] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10082.349093] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10083.280812] Lustre: DEBUG MARKER: rc=0; [10083.280812] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10083.280812] if [[ $? -eq 0 && $val -ne 0 ]]; then [10083.280812] echo $(hostname -s): $val; [10083.280812] rc=$val; [10083.280812] fi; [10083.280812] exit $rc [10083.894897] Lustre: DEBUG MARKER: dmesg [10084.379597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 10: Test quota for root user ==================================================== 13:20:55 \(1491484855\) [10084.543356] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ==================================================== 13:20:55 (1491484855) [10084.966402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10086.286315] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10086.598140] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10093.959739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10094.226063] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10094.540744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10095.829568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10097.134732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10098.419298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10099.695893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10100.979603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10102.271946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10103.560196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10104.867463] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10106.539983] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10107.434709] Lustre: DEBUG MARKER: rc=0; [10107.434709] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10107.434709] if [[ $? -eq 0 && $val -ne 0 ]]; then [10107.434709] echo $(hostname -s): $val; [10107.434709] rc=$val; [10107.434709] fi; [10107.434709] exit $rc [10108.039121] Lustre: DEBUG MARKER: dmesg [10108.546186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 11: Chown\/chgrp ignores quota =================================================== 13:21:19 \(1491484879\) [10108.714177] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota =================================================== 13:21:19 (1491484879) [10109.146617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10110.469333] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10110.772709] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [10111.041997] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10111.301876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10111.555609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10112.605328] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10112.870514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10113.126929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10114.504213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10114.770388] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10115.118715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10116.410825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10117.688617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10118.975084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10120.256252] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10121.547991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10122.827231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10124.116347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10125.433304] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10127.124421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10128.016726] Lustre: DEBUG MARKER: rc=0; [10128.016726] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10128.016726] if [[ $? -eq 0 && $val -ne 0 ]]; then [10128.016726] echo $(hostname -s): $val; [10128.016726] rc=$val; [10128.016726] fi; [10128.016726] exit $rc [10128.628978] Lustre: DEBUG MARKER: dmesg [10129.130024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 12a: Block quota rebalancing ==================================================== 13:21:40 \(1491484900\) [10129.307182] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ==================================================== 13:21:40 (1491484900) [10129.723214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10131.040333] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10131.405047] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10161.835340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10162.102588] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10162.378180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10163.658580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10164.950752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10166.232828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10167.517197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10168.799611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10170.107179] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10184.153594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10184.436459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10184.733688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10186.014922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10187.296772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10188.577036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10189.866591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10191.155731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10192.442975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10193.724211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10195.048356] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10196.745785] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10197.659221] Lustre: DEBUG MARKER: rc=0; [10197.659221] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10197.659221] if [[ $? -eq 0 && $val -ne 0 ]]; then [10197.659221] echo $(hostname -s): $val; [10197.659221] rc=$val; [10197.659221] fi; [10197.659221] exit $rc [10198.254243] Lustre: DEBUG MARKER: dmesg [10198.768753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 12b: Inode quota rebalancing ==================================================== 13:22:49 \(1491484969\) [10198.962148] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ==================================================== 13:22:49 (1491484969) [10199.373761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10200.699442] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10201.039989] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=u [10201.321080] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10201.580339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10201.835423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10203.093097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10204.352314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10205.610616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10206.877973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10208.140223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10209.426821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10210.689807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10211.726412] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10211.985931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10212.246580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10216.755361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10218.047294] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10223.146619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10224.440859] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10226.126859] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10227.011134] Lustre: DEBUG MARKER: rc=0; [10227.011134] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10227.011134] if [[ $? -eq 0 && $val -ne 0 ]]; then [10227.011134] echo $(hostname -s): $val; [10227.011134] rc=$val; [10227.011134] fi; [10227.011134] exit $rc [10227.610279] Lustre: DEBUG MARKER: dmesg [10228.105634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 13:23:19 \(1491484999\) [10228.265688] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 13:23:19 (1491484999) [10228.680227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10230.004287] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10230.308336] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10240.885085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10241.150783] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10241.440774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10242.724206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10244.003954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10245.282180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10246.566905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10247.848761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10249.131582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10250.447462] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10251.125732] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10252.075205] Lustre: DEBUG MARKER: rc=0; [10252.075205] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10252.075205] if [[ $? -eq 0 && $val -ne 0 ]]; then [10252.075205] echo $(hostname -s): $val; [10252.075205] rc=$val; [10252.075205] fi; [10252.075205] exit $rc [10252.709232] Lustre: DEBUG MARKER: dmesg [10253.197926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 15: Set over 4T block quota ===================================================== 13:23:44 \(1491485024\) [10253.358291] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ===================================================== 13:23:44 (1491485024) [10253.781487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10255.078588] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10257.852332] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10258.747410] Lustre: DEBUG MARKER: rc=0; [10258.747410] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10258.747410] if [[ $? -eq 0 && $val -ne 0 ]]; then [10258.747410] echo $(hostname -s): $val; [10258.747410] rc=$val; [10258.747410] fi; [10258.747410] exit $rc [10259.345605] Lustre: DEBUG MARKER: dmesg [10259.846347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 17: DQACQ return recoverable error ============================================== 13:23:50 \(1491485030\) [10260.019584] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error ============================================== 13:23:50 (1491485030) [10260.439992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10261.758815] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10262.085940] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10276.452297] Lustre: DEBUG MARKER: lctl set_param fail_val=37 fail_loc=0xa04 [10276.643597] Lustre: *** cfs_fail_loc=a04, val=0*** [10277.686213] Lustre: *** cfs_fail_loc=a04, val=0*** [10278.763404] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10280.094195] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10281.110260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10281.378393] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10281.724113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10283.004496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10284.282086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10285.567393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10286.846200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10288.129224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10289.418915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10290.729307] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10292.636858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10293.954209] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10294.283996] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10302.230441] Lustre: DEBUG MARKER: lctl set_param fail_val=11 fail_loc=0xa04 [10302.414727] Lustre: *** cfs_fail_loc=a04, val=0*** [10304.540495] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10305.882195] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10306.880728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10307.147187] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10307.436697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10308.717885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10309.997940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10311.288543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10312.577403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10313.859319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10315.171250] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10317.087986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10318.412294] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10318.741272] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10326.653992] Lustre: DEBUG MARKER: lctl set_param fail_val=110 fail_loc=0xa04 [10326.839628] Lustre: *** cfs_fail_loc=a04, val=0*** [10326.841115] Lustre: Skipped 1 previous similar message [10328.968596] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10330.312108] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10331.263151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10331.528195] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10331.826408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10333.114897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10334.395647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10335.682446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10336.969908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10338.253156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10339.543401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10340.865169] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10342.819970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10344.148578] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10344.474718] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10352.434755] Lustre: DEBUG MARKER: lctl set_param fail_val=107 fail_loc=0xa04 [10352.617298] Lustre: *** cfs_fail_loc=a04, val=0*** [10352.618568] Lustre: Skipped 1 previous similar message [10354.746906] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10356.081724] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10357.067278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10357.334894] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10357.627366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10358.905265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10360.181553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10361.477573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10362.767189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10364.049270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10365.359498] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10367.060802] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10367.949925] Lustre: DEBUG MARKER: rc=0; [10367.949925] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10367.949925] if [[ $? -eq 0 && $val -ne 0 ]]; then [10367.949925] echo $(hostname -s): $val; [10367.949925] rc=$val; [10367.949925] fi; [10367.949925] exit $rc [10368.554159] Lustre: DEBUG MARKER: dmesg [10369.061309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 18: MDS failover while writing, no watchdog triggered \(b14840\) ================== 13:25:40 \(1491485140\) [10369.228780] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ================== 13:25:40 (1491485140) [10369.932678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10371.252618] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10371.561887] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10387.378511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [10387.553853] Lustre: DEBUG MARKER: User quota (limit: 200m) [10387.821634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(buffered\) ... [10387.977393] Lustre: DEBUG MARKER: Write 100M (buffered) ... [10388.141392] Lustre: DEBUG MARKER: sync; sync; sync [10390.426138] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [10390.736690] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [10390.878171] LustreError: 21109:0:(osd_handler.c:2176:osd_ro()) *** setting lustre-MDT0000 read-only *** [10390.927539] Turning device dm-0 (0xfc00000) read-only [10391.056947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [10391.187434] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10391.385081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [10391.557941] Lustre: DEBUG MARKER: Fail mds for 0 seconds [10391.726291] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10391.983277] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [10392.113205] Lustre: Failing over lustre-MDT0000 [10392.163215] Removing read-only on unknown block (0xfc00000) [10392.169708] Lustre: server umount lustre-MDT0000 complete [10392.297955] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10395.352505] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.356375] LustreError: Skipped 2 previous similar messages [10396.793254] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [10396.793254] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.793254] Lustre: Skipped 8 previous similar messages [10396.801018] LustreError: Skipped 14 previous similar messages [10402.610897] Lustre: DEBUG MARKER: hostname [10402.918658] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10403.177976] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10403.436122] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10403.585588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.605172] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10403.609205] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [10403.611460] Lustre: Skipped 51 previous similar messages [10403.613452] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141cca1628 to 0x61ac6f141d182b50 [10403.655961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10403.830252] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10404.097681] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10404.648225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10404.648260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10404.814140] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10404.814251] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10404.966237] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10405.224607] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10405.483975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10405.747211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10406.512773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10406.537273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10406.700249] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10406.804964] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10407.510873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [10408.650641] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [10408.668715] Lustre: lustre-MDT0000: Recovery over after 0:01, of 5 clients 5 recovered and 0 were evicted. [10408.961818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10409.072433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10409.113581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10409.292826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10410.576720] Lustre: DEBUG MARKER: lctl get_param -n at_max [10410.847517] Lustre: DEBUG MARKER: lctl get_param -n at_max [10411.177924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=18493, time=0, timeout=600\) [10411.342259] Lustre: DEBUG MARKER: (dd_pid=18493, time=0, timeout=600) [10411.937194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10412.204768] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10412.512865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10413.819515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10415.127633] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10417.150167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10418.486526] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10418.788020] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10425.707855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [10425.873090] Lustre: DEBUG MARKER: User quota (limit: 200m) [10426.234619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(directio\) ... [10426.400868] Lustre: DEBUG MARKER: Write 100M (directio) ... [10426.544624] Lustre: DEBUG MARKER: sync; sync; sync [10427.676532] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [10427.952574] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [10428.155076] Turning device dm-0 (0xfc00000) read-only [10428.294152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [10428.436510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10428.859563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [10429.064964] Lustre: DEBUG MARKER: Fail mds for 0 seconds [10429.316613] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10429.586901] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [10429.773329] Removing read-only on unknown block (0xfc00000) [10429.918809] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10440.236902] Lustre: DEBUG MARKER: hostname [10440.548451] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10440.808712] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10440.871040] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491485204/real 1491485204] req@ffff88007829b600 x1563924894914432/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491485211 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10440.877472] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 39 previous similar messages [10441.066225] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10441.227382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.076398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10447.337100] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10447.835865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.836035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.993482] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.999155] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10448.153798] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10448.413708] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10448.672526] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10448.936091] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10449.704747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10449.707765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10449.918375] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10449.969880] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10451.652167] LustreError: 28111:0:(tgt_handler.c:505:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff880079d4ac50 x1563924928464048/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@10.9.5.244@tcp:-1/-1 lens 336/0 e 0 to 0 dl 1491485228 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [10452.261426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10452.291372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10452.451373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10452.466103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10453.786683] Lustre: DEBUG MARKER: lctl get_param -n at_max [10454.046349] Lustre: DEBUG MARKER: lctl get_param -n at_max [10458.416893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=21430, time=4, timeout=600\) [10458.577031] Lustre: DEBUG MARKER: (dd_pid=21430, time=4, timeout=600) [10459.078648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10459.344687] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10459.634356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10460.924305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10462.206688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10463.494150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10464.787508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10466.078300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10467.363359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10468.674163] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10470.658762] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10471.580145] Lustre: DEBUG MARKER: rc=0; [10471.580145] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10471.580145] if [[ $? -eq 0 && $val -ne 0 ]]; then [10471.580145] echo $(hostname -s): $val; [10471.580145] rc=$val; [10471.580145] fi; [10471.580145] exit $rc [10472.205570] Lustre: DEBUG MARKER: dmesg [10472.692825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 19: Updating admin limits doesn\'t zero operational limits\(b14790\) =============== 13:27:23 \(1491485243\) [10472.853661] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) =============== 13:27:23 (1491485243) [10473.280570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10474.606889] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10474.917620] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10487.536659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10487.805822] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10488.098946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10489.383729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10490.666667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10491.958796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10493.256284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10494.541390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10495.828798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10497.107167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10498.424691] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10500.177142] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10501.087214] Lustre: DEBUG MARKER: rc=0; [10501.087214] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10501.087214] if [[ $? -eq 0 && $val -ne 0 ]]; then [10501.087214] echo $(hostname -s): $val; [10501.087214] rc=$val; [10501.087214] fi; [10501.087214] exit $rc [10501.690950] Lustre: DEBUG MARKER: dmesg [10502.178121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 20: Test if setquota specifiers work properly \(b15754\) ========================== 13:27:53 \(1491485273\) [10502.340880] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================== 13:27:53 (1491485273) [10504.666236] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10505.577659] Lustre: DEBUG MARKER: rc=0; [10505.577659] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10505.577659] if [[ $? -eq 0 && $val -ne 0 ]]; then [10505.577659] echo $(hostname -s): $val; [10505.577659] rc=$val; [10505.577659] fi; [10505.577659] exit $rc [10506.179694] Lustre: DEBUG MARKER: dmesg [10506.656666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 21: Setquota while writing & deleting \(b16053\) ================================== 13:27:57 \(1491485277\) [10506.828044] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [10507.250942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10508.579066] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10508.879444] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10515.804628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for user: quota_usr [10515.968398] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [10516.196113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for group: quota_usr [10516.364127] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [10516.606382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 1 times [10516.830285] Lustre: DEBUG MARKER: Set quota for 1 times [10518.171700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 2 times [10519.387047] Lustre: DEBUG MARKER: Set quota for 2 times [10520.938460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 3 times [10521.332517] Lustre: DEBUG MARKER: Set quota for 3 times [10523.129741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 4 times [10523.366313] Lustre: DEBUG MARKER: Set quota for 4 times [10524.711461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 5 times [10525.007450] Lustre: DEBUG MARKER: Set quota for 5 times [10526.386431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 6 times [10526.558079] Lustre: DEBUG MARKER: Set quota for 6 times [10527.912873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 7 times [10528.085103] Lustre: DEBUG MARKER: Set quota for 7 times [10529.550622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 8 times [10529.716944] Lustre: DEBUG MARKER: Set quota for 8 times [10536.066033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 9 times [10536.350842] Lustre: DEBUG MARKER: Set quota for 9 times [10537.587775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 10 times [10537.750081] Lustre: DEBUG MARKER: Set quota for 10 times [10539.261247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 11 times [10539.463650] Lustre: DEBUG MARKER: Set quota for 11 times [10540.966896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 12 times [10541.129153] Lustre: DEBUG MARKER: Set quota for 12 times [10542.657217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 13 times [10542.977888] Lustre: DEBUG MARKER: Set quota for 13 times [10544.465383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 14 times [10544.738581] Lustre: DEBUG MARKER: Set quota for 14 times [10546.086596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 15 times [10546.316593] Lustre: DEBUG MARKER: Set quota for 15 times [10550.074904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10550.343365] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10550.624946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10551.910361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10553.194926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10554.486548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10555.786149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10557.078117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10558.396353] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10561.165596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10562.057778] Lustre: DEBUG MARKER: rc=0; [10562.057778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10562.057778] if [[ $? -eq 0 && $val -ne 0 ]]; then [10562.057778] echo $(hostname -s): $val; [10562.057778] rc=$val; [10562.057778] fi; [10562.057778] exit $rc [10562.674463] Lustre: DEBUG MARKER: dmesg [10563.174282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 22: enable\/disable quota by \'lctl conf_param\' =================================== 13:28:54 \(1491485334\) [10563.330282] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param' =================================== 13:28:54 (1491485334) [10563.490252] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [10563.774453] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10564.049194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10564.321918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10565.588139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10566.850909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10568.114260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10569.376078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10570.635455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10571.893282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10573.152938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10574.409294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10575.449933] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10575.710234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10575.970025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10577.009069] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10584.685172] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10584.945144] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10586.087017] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [10586.088786] Lustre: Skipped 53 previous similar messages [10587.199070] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005ed83600 x1563924894938128/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10587.202843] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [10591.305315] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10598.488431] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10598.742467] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10599.119042] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491485362/real 1491485362] req@ffff880077ead200 x1563924894938368/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491485369 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10599.124660] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10599.263076] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005ed81b00 x1563924894938832/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10599.267359] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [10605.071934] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10627.317570] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10627.317570] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10627.317570] if [ $running -ne $mpts ]; then [10627.317570] echo $(hostname) env are INSANE!; [10627.317570] exit 1; [10627.317570] fi [10627.868386] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [10627.868386] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [10627.868386] if [ $running -ne $mpts ]; then [10627.868386] echo $(hostname) env are INSANE!; [10627.868386] exit 1; [10627.868386] fi [10628.442299] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10628.442299] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10628.442299] if [ $running -ne $mpts ]; then [10628.442299] echo $(hostname) env are INSANE!; [10628.442299] exit 1; [10628.442299] fi [10629.376068] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [10629.635873] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10629.886259] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10630.132839] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10630.278318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.300412] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [10630.302787] Lustre: Skipped 51 previous similar messages [10630.492102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10630.751116] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10631.246289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10631.246686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10631.404722] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10631.405045] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10631.550250] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10631.803975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10632.057986] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10632.317180] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10632.571301] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10634.737097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10634.889796] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10636.305524] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [10636.571256] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [10636.828506] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [10637.084708] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [10637.243339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.485129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10637.745776] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10638.239689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10638.246218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10638.405569] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10638.406162] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10638.554910] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10638.815272] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10639.079079] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10639.342611] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [10639.602640] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10641.789094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10641.954503] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10645.869415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10646.031978] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10648.860822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10649.022209] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10651.960095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10652.132530] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10654.983827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10655.139188] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10658.127974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10658.292087] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10661.143523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10661.397746] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10664.423572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10664.592156] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10667.444072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10667.608108] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10669.712916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10669.875359] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10671.321143] Lustre: DEBUG MARKER: lctl get_param -n timeout [10671.635876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [10671.793882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10671.961644] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [10677.250595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10677.781868] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=none [10678.047118] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10678.305383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10678.564672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10687.145424] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10687.410270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10687.672718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10688.726135] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [10700.983461] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10701.249663] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10701.506733] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [10701.509689] Lustre: Skipped 32 previous similar messages [10701.839065] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880077bac000 x1563924894949840/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10701.845191] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [10705.743066] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880063465800 x1563924894950160/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10705.749008] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [10707.610659] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10714.760893] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10715.018785] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10715.295071] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880063466d00 x1563924894950736/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10721.400202] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10743.208478] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10743.208478] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10743.208478] if [ $running -ne $mpts ]; then [10743.208478] echo $(hostname) env are INSANE!; [10743.208478] exit 1; [10743.208478] fi [10743.779338] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [10743.779338] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [10743.779338] if [ $running -ne $mpts ]; then [10743.779338] echo $(hostname) env are INSANE!; [10743.779338] exit 1; [10743.779338] fi [10744.370105] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10744.370105] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10744.370105] if [ $running -ne $mpts ]; then [10744.370105] echo $(hostname) env are INSANE!; [10744.370105] exit 1; [10744.370105] fi [10745.307621] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [10745.567656] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10745.816589] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10746.063720] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10746.221641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.443237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10746.701610] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10747.195600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10747.196241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10747.357797] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10747.357972] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10747.504019] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10747.758688] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10748.013924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10748.275086] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10748.531226] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10750.668157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10750.832573] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10751.294110] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491485516/real 1491485516] req@ffff880063d11e00 x1563924894951232/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491485521 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10751.301446] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [10752.247113] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [10752.517213] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [10752.773166] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [10753.025798] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [10753.188248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.411941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10753.672468] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10754.168660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10754.172845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10754.333547] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10754.344054] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10754.485205] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10754.741938] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10755.008434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10755.271238] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [10755.530340] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10757.713313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10757.878456] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10761.191753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10761.354417] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10764.192431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10764.363192] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10767.175646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10767.342718] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10770.163921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10770.332763] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10773.138916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10773.308329] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10776.134522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10776.306784] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10779.151760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10779.319236] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10781.229888] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [10781.233149] Lustre: Skipped 60 previous similar messages [10782.175516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10782.346708] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10784.469261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10784.633033] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10786.071927] Lustre: DEBUG MARKER: lctl get_param -n timeout [10786.393427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [10786.546163] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10786.686908] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [10792.032652] Lustre: DEBUG MARKER: lctl set_param debug=+quota [10792.326220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10792.907340] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10793.813924] Lustre: DEBUG MARKER: rc=0; [10793.813924] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10793.813924] if [[ $? -eq 0 && $val -ne 0 ]]; then [10793.813924] echo $(hostname -s): $val; [10793.813924] rc=$val; [10793.813924] fi; [10793.813924] exit $rc [10794.431495] Lustre: DEBUG MARKER: dmesg [10794.955599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 23: Quota should be honored with directIO \(b16125\) ============================== 13:32:45 \(1491485565\) [10795.113333] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ============================== 13:32:45 (1491485565) [10795.334371] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 14959876 required: 6144 [10795.487098] Lustre: DEBUG MARKER: OST0_SIZE: 14959876 required: 6144 [10795.678701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark run for 4MB test file [10795.836804] Lustre: DEBUG MARKER: run for 4MB test file [10796.270192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10797.589614] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10797.877747] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10813.284683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 4 MB\) [10813.436720] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [10813.680433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: trigger EDQUOT with O_DIRECT [10813.835823] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [10814.038289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write half of file [10814.204118] Lustre: DEBUG MARKER: Write half of file [10814.524486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [10814.681933] Lustre: DEBUG MARKER: Write out of block quota ... [10814.961784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: done [10815.124018] Lustre: DEBUG MARKER: Step1: done [10815.327429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: rewrite should succeed [10815.484792] Lustre: DEBUG MARKER: Step2: rewrite should succeed [10815.735123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: done [10815.890460] Lustre: DEBUG MARKER: Step2: done [10816.318996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10816.598490] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10816.888619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10818.174845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10819.461746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10820.742920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10822.065589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10823.352159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10824.660944] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10826.365202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 14959876 required: 61440 [10826.522100] Lustre: DEBUG MARKER: OST0_SIZE: 14959876 required: 61440 [10826.724777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark run for 40MB test file [10826.869631] Lustre: DEBUG MARKER: run for 40MB test file [10827.295806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10828.635940] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10828.937044] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10835.524747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 40 MB\) [10835.677744] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [10835.937605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: trigger EDQUOT with O_DIRECT [10836.092937] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [10836.282634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write half of file [10836.439861] Lustre: DEBUG MARKER: Write half of file [10837.384297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [10837.559380] Lustre: DEBUG MARKER: Write out of block quota ... [10838.642913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: done [10838.798390] Lustre: DEBUG MARKER: Step1: done [10839.003541] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: rewrite should succeed [10839.160997] Lustre: DEBUG MARKER: Step2: rewrite should succeed [10839.416947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: done [10839.570365] Lustre: DEBUG MARKER: Step2: done [10840.002962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10840.286323] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10840.641467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10841.920019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10843.200655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10844.482425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10845.766921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10847.052262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10848.329897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10849.642830] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10851.341791] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10852.223150] Lustre: DEBUG MARKER: rc=0; [10852.223150] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10852.223150] if [[ $? -eq 0 && $val -ne 0 ]]; then [10852.223150] echo $(hostname -s): $val; [10852.223150] rc=$val; [10852.223150] fi; [10852.223150] exit $rc [10852.813454] Lustre: DEBUG MARKER: dmesg [10853.310889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 24: lfs draws an asterix when limit is reached \(b16646\) ========================= 13:33:44 \(1491485624\) [10853.466027] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================= 13:33:44 (1491485624) [10853.877081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10855.198835] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10855.484379] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10862.151187] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10863.041968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10863.324017] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10863.621111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10864.912261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10866.197583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10867.487967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10868.765615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10870.044181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10871.326825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10872.607260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10873.898324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10875.209779] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10876.886794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10877.761997] Lustre: DEBUG MARKER: rc=0; [10877.761997] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10877.761997] if [[ $? -eq 0 && $val -ne 0 ]]; then [10877.761997] echo $(hostname -s): $val; [10877.761997] rc=$val; [10877.761997] fi; [10877.761997] exit $rc [10878.354361] Lustre: DEBUG MARKER: dmesg [10878.842199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27a: lfs quota\/setquota should handle wrong arguments \(b19612\) ================== 13:34:09 \(1491485649\) [10879.010648] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ================== 13:34:09 (1491485649) [10879.196657] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10880.095376] Lustre: DEBUG MARKER: rc=0; [10880.095376] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10880.095376] if [[ $? -eq 0 && $val -ne 0 ]]; then [10880.095376] echo $(hostname -s): $val; [10880.095376] rc=$val; [10880.095376] fi; [10880.095376] exit $rc [10880.679164] Lustre: DEBUG MARKER: dmesg [10881.160766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27b: lfs quota\/setquota should handle user\/group ID \(b20200\) ==================== 13:34:12 \(1491485652\) [10881.321877] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group ID (b20200) ==================== 13:34:12 (1491485652) [10883.610173] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10884.481599] Lustre: DEBUG MARKER: rc=0; [10884.481599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10884.481599] if [[ $? -eq 0 && $val -ne 0 ]]; then [10884.481599] echo $(hostname -s): $val; [10884.481599] rc=$val; [10884.481599] fi; [10884.481599] exit $rc [10885.075885] Lustre: DEBUG MARKER: dmesg [10885.563250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27c: lfs quota should support human-readable output ============================= 13:34:16 \(1491485656\) [10885.723215] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ============================= 13:34:16 (1491485656) [10887.081445] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10887.945493] Lustre: DEBUG MARKER: rc=0; [10887.945493] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10887.945493] if [[ $? -eq 0 && $val -ne 0 ]]; then [10887.945493] echo $(hostname -s): $val; [10887.945493] rc=$val; [10887.945493] fi; [10887.945493] exit $rc [10888.541233] Lustre: DEBUG MARKER: dmesg [10889.017947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27d: lfs setquota should support fraction block limit =========================== 13:34:19 \(1491485659\) [10889.172987] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit =========================== 13:34:19 (1491485659) [10890.395384] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10891.297776] Lustre: DEBUG MARKER: rc=0; [10891.297776] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10891.297776] if [[ $? -eq 0 && $val -ne 0 ]]; then [10891.297776] echo $(hostname -s): $val; [10891.297776] rc=$val; [10891.297776] fi; [10891.297776] exit $rc [10891.909413] Lustre: DEBUG MARKER: dmesg [10892.479076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 30: Hard limit updates should not reset grace times ============================= 13:34:23 \(1491485663\) [10892.640987] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ============================= 13:34:23 (1491485663) [10893.050265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10894.373304] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10894.662633] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10919.545910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10919.819297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10920.117494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10921.408948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10922.693594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10923.983709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10925.264398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10926.545855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10927.826807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10929.105903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10930.413206] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10932.172735] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10933.066616] Lustre: DEBUG MARKER: rc=0; [10933.066616] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10933.066616] if [[ $? -eq 0 && $val -ne 0 ]]; then [10933.066616] echo $(hostname -s): $val; [10933.066616] rc=$val; [10933.066616] fi; [10933.066616] exit $rc [10933.671133] Lustre: DEBUG MARKER: dmesg [10934.190340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 33: Basic usage tracking for user & group ======================================= 13:35:05 \(1491485705\) [10934.360782] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [10934.774107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10936.094897] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10937.442066] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10938.488510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10938.757474] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10939.045704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10940.321616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10941.614469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10942.895935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10944.174981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10945.451248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10946.733356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10948.015999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10949.293285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10950.601461] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10951.313478] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10952.195470] Lustre: DEBUG MARKER: rc=0; [10952.195470] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10952.195470] if [[ $? -eq 0 && $val -ne 0 ]]; then [10952.195470] echo $(hostname -s): $val; [10952.195470] rc=$val; [10952.195470] fi; [10952.195470] exit $rc [10952.782507] Lustre: DEBUG MARKER: dmesg [10953.282740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 34: Usage transfer for user & group ============================================= 13:35:24 \(1491485724\) [10953.439502] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [10953.844332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10955.184304] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10955.676999] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10956.590805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10956.856632] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10957.132645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10958.417546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10959.700658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10960.984712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10962.276907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10963.567276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10964.850528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10966.135730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10967.742802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10968.005826] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10968.349749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10969.625201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10970.907075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10972.190089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10973.476776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10974.763690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10976.041183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10977.322189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10978.924688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10979.190144] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10979.470702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10980.754939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10982.032809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10983.327055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10984.607339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10985.896492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10987.173659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10988.798235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10989.057648] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10989.348284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10990.632698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10991.919657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10993.196112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10994.474946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10995.752949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10997.031492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10998.334489] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10998.980458] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10999.857636] Lustre: DEBUG MARKER: rc=0; [10999.857636] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10999.857636] if [[ $? -eq 0 && $val -ne 0 ]]; then [10999.857636] echo $(hostname -s): $val; [10999.857636] rc=$val; [10999.857636] fi; [10999.857636] exit $rc [11000.449317] Lustre: DEBUG MARKER: dmesg [11000.964684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 35: Usage is still accessible across reboot ===================================== 13:36:11 \(1491485771\) [11001.127055] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ===================================== 13:36:11 (1491485771) [11001.530781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11002.849281] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [11003.294728] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11004.036763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Restart... [11004.191037] Lustre: DEBUG MARKER: Restart... [11005.014180] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11005.269178] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11005.935097] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005de19b00 x1563924894981040/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11005.938700] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [11006.246414] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11006.248731] Lustre: Skipped 32 previous similar messages [11008.055168] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [11008.055242] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11008.055243] Lustre: Skipped 9 previous similar messages [11008.062039] LustreError: Skipped 10 previous similar messages [11011.501250] Lustre: server umount lustre-MDT0000 complete [11011.502915] Lustre: Skipped 5 previous similar messages [11011.632197] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11013.055262] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11013.059243] LustreError: Skipped 120 previous similar messages [11018.756531] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11019.012144] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [11019.463074] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11019.466842] LustreError: Skipped 3 previous similar messages [11023.455053] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004c95c000 x1563924894982016/t0(0) o13->lustre-OST0006-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11023.459845] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 10 previous similar messages [11025.382716] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11052.259214] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11052.259214] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11052.259214] if [ $running -ne $mpts ]; then [11052.259214] echo $(hostname) env are INSANE!; [11052.259214] exit 1; [11052.259214] fi [11052.820312] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11052.820312] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11052.820312] if [ $running -ne $mpts ]; then [11052.820312] echo $(hostname) env are INSANE!; [11052.820312] exit 1; [11052.820312] fi [11053.394410] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11053.394410] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11053.394410] if [ $running -ne $mpts ]; then [11053.394410] echo $(hostname) env are INSANE!; [11053.394410] exit 1; [11053.394410] fi [11056.678323] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11056.937999] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11057.192894] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11057.440504] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11057.595369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.669633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.671910] Lustre: Skipped 5 previous similar messages [11057.821229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11058.081380] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [11058.573544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11058.578221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11058.735698] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11058.736836] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11058.885108] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11059.144554] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11059.401223] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11059.663788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11059.924298] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11062.089762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11062.252385] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11062.665106] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491485828/real 1491485828] req@ffff88005fc95e00 x1563924894982416/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491485833 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11062.672474] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [11063.655773] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11063.924371] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11064.180257] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11064.439151] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11064.600019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.817667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11065.078078] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [11065.580272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11065.581199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11065.742276] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11065.743851] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11065.894201] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11066.153350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11066.414660] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11066.676762] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11066.935831] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11069.099480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11069.264140] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11072.563559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11072.731104] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11075.507431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11075.672584] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11078.505755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11078.669101] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11081.481427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11081.645405] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11083.554730] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [11083.557907] Lustre: Skipped 29 previous similar messages [11084.482554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11084.646135] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11087.482134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11087.737134] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11090.703303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11090.872040] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11093.753514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11093.919651] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11096.046071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11096.201759] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11097.641191] Lustre: DEBUG MARKER: lctl get_param -n timeout [11097.977197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11098.137998] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11098.278824] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11103.708838] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [11103.990278] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11104.505948] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11105.081478] Lustre: DEBUG MARKER: lctl set_param debug=+quota [11105.610721] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11106.574335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11106.845805] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [11107.134686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11108.423374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11109.708534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11110.999690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11112.281134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11113.566380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11114.856070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11116.135894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11117.463193] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11118.187542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11119.083298] Lustre: DEBUG MARKER: rc=0; [11119.083298] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11119.083298] if [[ $? -eq 0 && $val -ne 0 ]]; then [11119.083298] echo $(hostname -s): $val; [11119.083298] rc=$val; [11119.083298] fi; [11119.083298] exit $rc [11119.699477] Lustre: DEBUG MARKER: dmesg [11120.188054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 37: Quota accounted properly for file created by \'lfs setstripe\' ================ 13:38:11 \(1491485891\) [11120.372819] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ================ 13:38:11 (1491485891) [11120.512050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11120.512050] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11120.512050] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11121.062148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11122.384819] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [11123.073286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11123.341978] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [11123.629695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11124.911290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11126.189412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11127.476875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11128.761545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11130.043910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11131.320916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11132.909787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11133.178344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [11133.467259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11134.749197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11136.029385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11137.311417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11138.588688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11139.873920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11141.156372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11142.458235] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11143.107989] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11143.992077] Lustre: DEBUG MARKER: rc=0; [11143.992077] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11143.992077] if [[ $? -eq 0 && $val -ne 0 ]]; then [11143.992077] echo $(hostname -s): $val; [11143.992077] rc=$val; [11143.992077] fi; [11143.992077] exit $rc [11144.582973] Lustre: DEBUG MARKER: dmesg [11145.084929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 38: Quota accounting iterator doesn\'t skip id entries =========================== 13:38:36 \(1491485916\) [11145.243391] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries =========================== 13:38:36 (1491485916) [11145.377438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11145.377438] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11145.377438] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11145.927127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11147.240354] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [11243.184638] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11243.822239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osd-ldiskfs.lustre-MDT0000.quota_slave.acct_user [11253.577689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11253.847245] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [11254.398808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11255.741071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11257.030451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11258.308630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11259.593554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11260.877697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11262.380326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11263.878083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11265.166344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [11266.478997] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [11267.143067] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11268.058613] Lustre: DEBUG MARKER: rc=0; [11268.058613] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11268.058613] if [[ $? -eq 0 && $val -ne 0 ]]; then [11268.058613] echo $(hostname -s): $val; [11268.058613] rc=$val; [11268.058613] fi; [11268.058613] exit $rc [11268.683754] Lustre: DEBUG MARKER: dmesg [11269.182704] Lustre: DEBUG MARKER: lctl set_param debug=-quota [11269.559903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test complete, duration 2520 sec ===================================================== 13:40:40 \(1491486040\) [11269.727266] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 2520 sec ===================================================== 13:40:40 (1491486040) [11269.956816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-scrub ============----- Thu Apr 6 13:40:40 UTC 2017 [11270.113084] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Thu Apr 6 13:40:40 UTC 2017 [11270.384096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test complete, duration -o sec ======================================================= 13:40:41 \(1491486041\) [11270.535516] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration -o sec ======================================================= 13:40:41 (1491486041) [11270.891298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11270.891298] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11270.891298] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11271.481275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11271.481275] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11271.481275] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11272.557674] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11272.813126] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11273.903114] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004bcc7000 x1563924895330352/t0(0) o13->lustre-OST0004-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11273.907750] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [11279.202061] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11286.337238] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11286.592174] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [11300.760055] Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11301.008272] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11321.406199] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11321.654175] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11322.421837] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11322.666276] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11325.482863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11325.482863] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11325.482863] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11325.793683] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11326.335882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11326.356390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [11326.386068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11326.476742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11326.485215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11326.666580] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11326.671441] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11326.822694] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11326.823549] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11326.826485] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [11327.146274] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11327.394445] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11327.674242] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11327.862918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11328.974340] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11329.225265] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11329.482652] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11329.675749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11335.460443] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11335.460443] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11335.460443] if [ $running -ne $mpts ]; then [11335.460443] echo $(hostname) env are INSANE!; [11335.460443] exit 1; [11335.460443] fi [11336.012496] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11336.012496] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11336.012496] if [ $running -ne $mpts ]; then [11336.012496] echo $(hostname) env are INSANE!; [11336.012496] exit 1; [11336.012496] fi [11336.576326] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11336.576326] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11336.576326] if [ $running -ne $mpts ]; then [11336.576326] echo $(hostname) env are INSANE!; [11336.576326] exit 1; [11336.576326] fi [11337.489223] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11337.753328] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11338.003350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11338.247648] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11338.401375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11338.442143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11338.476916] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11338.479047] Lustre: Skipped 4 previous similar messages [11338.491618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [11338.507975] Lustre: lustre-MDT0000: new disk, initializing [11338.537410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11338.693880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11338.951875] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11339.441579] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11339.445166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11339.599962] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11339.602720] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11339.743964] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11339.996784] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11340.250492] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11340.504687] Lustre: DEBUG MARKER: sync; sync; sync [11345.866608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11346.125688] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11347.329297] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [11347.331701] Lustre: Skipped 4 previous similar messages [11347.410885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11348.294982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11348.448743] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11355.171401] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11355.436206] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11355.688136] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11355.939626] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11356.160327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11356.172113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.187506] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [11356.190144] Lustre: Skipped 4 previous similar messages [11356.198337] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [11356.200842] Lustre: Skipped 1 previous similar message [11356.217322] Lustre: lustre-MDT0002: new disk, initializing [11356.240114] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [11356.402412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11356.658207] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11357.151084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11357.152250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11357.311374] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11357.321561] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11357.461271] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11357.715643] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11357.969685] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11358.221788] Lustre: DEBUG MARKER: sync; sync; sync [11363.539969] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11363.795901] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11365.011283] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [11365.014017] Lustre: Skipped 4 previous similar messages [11365.171284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [11365.174049] Lustre: Skipped 1 previous similar message [11366.061336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11366.221990] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11374.868197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11375.025604] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11383.220832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11383.374449] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11391.549909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11391.706615] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11399.938098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11400.102587] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11401.536779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000003c0000400-0x0000000400000400]:3:ost [11401.539977] Lustre: Skipped 3 previous similar messages [11407.442782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11407.609742] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11409.048526] Lustre: DEBUG MARKER: lctl get_param -n timeout [11409.363628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11409.512141] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11409.644973] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11414.925043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11414.925043] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11414.925043] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11415.203984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11415.203984] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11415.203984] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11415.480339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11415.480339] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11415.480339] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11415.754544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11415.754544] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11415.754544] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11416.315312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11416.315312] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11416.315312] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11416.651279] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [11416.808518] Lustre: DEBUG MARKER: excepting tests: [11417.102924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup\/restore case ===================== 13:43:08 \(1491486188\) [11417.258894] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ===================== 13:43:08 (1491486188) [11420.258450] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11423.609968] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11423.888988] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11424.146315] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11424.276511] Lustre: Failing over lustre-MDT0000 [11424.278740] Lustre: Skipped 1 previous similar message [11424.503651] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11425.797348] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11426.048209] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11432.395034] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11439.501387] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11439.759396] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11440.007200] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11440.254672] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11440.411100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.617954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11440.874667] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11441.363143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11441.364905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11441.519479] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11441.524515] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11441.662652] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11441.917788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11442.173094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11442.430103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11442.684149] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11444.834367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11445.007169] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11446.423943] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11446.692306] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11446.950989] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11447.208780] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11447.368282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.572182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11447.842261] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11448.335255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11448.342308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11448.501187] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11448.504064] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11448.648480] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11448.906686] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11448.953595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [11448.956575] Lustre: Skipped 1 previous similar message [11449.167403] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11449.431350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11449.688978] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11451.827520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11451.993954] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11453.430231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11453.430231] awk '/^status/ { print $2 }' [11453.689137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11453.689137] awk '/^status/ { print $2 }' [11454.465670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11454.465670] awk '/^status/ { print $2 }' [11454.724475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11454.724475] awk '/^status/ { print $2 }' [11455.503631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11455.938182] Lustre: lustre-MDT0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [11455.941099] Lustre: Skipped 1 previous similar message [11456.031462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11456.761307] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11457.634827] Lustre: DEBUG MARKER: rc=0; [11457.634827] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11457.634827] if [[ $? -eq 0 && $val -ne 0 ]]; then [11457.634827] echo $(hostname -s): $val; [11457.634827] rc=$val; [11457.634827] fi; [11457.634827] exit $rc [11458.244149] Lustre: DEBUG MARKER: dmesg [11458.723781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ============================ 13:43:49 \(1491486229\) [11458.889076] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ============================ 13:43:49 (1491486229) [11462.172522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11465.490666] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11465.778967] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11466.039891] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11466.386701] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11467.507774] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11467.759933] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11474.089057] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11481.236777] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11481.497719] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11481.750549] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11481.998387] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11482.142577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.347210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11482.603720] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11483.092221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11483.095735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11483.282443] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11483.287930] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11483.436851] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11483.690097] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11483.942563] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11484.199711] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11484.467436] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11486.588837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11486.756505] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11488.190976] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11488.467622] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11488.732596] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11488.985891] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11489.144560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.340386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11489.608137] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11490.113969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11490.120439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11490.274269] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11490.285583] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11490.426254] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11490.683911] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11490.950785] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11491.212488] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11491.470644] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11493.659993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11493.817171] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11495.255560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11495.421118] Lustre: lustre-MDT0000: Denying connection for new client 857d3967-f089-8f23-24bc-a0ff35b5a9fb(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [11495.426934] Lustre: Skipped 1 previous similar message [11500.567360] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x193 [11500.707160] Lustre: *** cfs_fail_loc=193, val=0*** [11500.833362] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [11501.225436] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11501.482112] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11501.815382] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11502.080078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11502.355816] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11502.611705] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11502.862166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11503.022014] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.039823] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d2b6df0 to 0x61ac6f141d2b73c3 [11503.044564] Lustre: Skipped 1 previous similar message [11503.234886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11503.492890] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11503.983893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11503.988212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11504.146100] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11504.147058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11504.296057] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11504.553767] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11504.807151] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11505.065164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11505.320620] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11505.577610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11505.891140] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11506.773393] Lustre: DEBUG MARKER: rc=0; [11506.773393] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11506.773393] if [[ $? -eq 0 && $val -ne 0 ]]; then [11506.773393] echo $(hostname -s): $val; [11506.773393] rc=$val; [11506.773393] fi; [11506.773393] exit $rc [11507.372028] Lustre: DEBUG MARKER: dmesg [11508.020067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove\/recreate case ========== 13:44:38 \(1491486278\) [11508.071932] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11508.076529] LustreError: Skipped 1 previous similar message [11508.196266] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========== 13:44:38 (1491486278) [11509.484627] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11509.484627] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11509.484627] if [ $running -ne $mpts ]; then [11509.484627] echo $(hostname) env are INSANE!; [11509.484627] exit 1; [11509.484627] fi [11510.060986] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11510.060986] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11510.060986] if [ $running -ne $mpts ]; then [11510.060986] echo $(hostname) env are INSANE!; [11510.060986] exit 1; [11510.060986] fi [11510.652725] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11510.652725] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11510.652725] if [ $running -ne $mpts ]; then [11510.652725] echo $(hostname) env are INSANE!; [11510.652725] exit 1; [11510.652725] fi [11511.598038] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11511.865492] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11512.120682] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11512.386379] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11512.649238] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11514.212762] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11514.479178] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11514.735922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11514.987779] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11515.246879] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11522.204182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11522.366937] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11523.796645] Lustre: DEBUG MARKER: lctl get_param -n timeout [11524.108940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11524.273385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11524.405318] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11532.944532] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11536.262800] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11536.553388] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11536.808140] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11537.115776] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11538.263149] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11538.516444] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11544.889375] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11546.002925] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11546.247103] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11546.496022] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11546.633361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11546.758136] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16* [11547.008624] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11548.555459] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11548.818245] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11549.079199] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11549.224025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11549.348669] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16* [11549.598184] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11551.131632] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11551.392098] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11551.640017] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11551.888664] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11552.044281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.264220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11552.520986] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11553.024327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.025774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.181927] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.183966] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.328366] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11553.587448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11553.843360] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11554.098178] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11554.352606] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11556.525549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11556.696930] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11558.114188] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11558.380683] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11558.637277] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11558.891122] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11559.051806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.241705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11559.507330] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11560.003690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11560.007893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11560.166313] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11560.166871] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11560.314384] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11560.574610] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11560.843277] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11561.105507] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11561.359829] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11563.500095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11563.664047] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11565.105119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11565.105119] awk '/^status/ { print $2 }' [11565.363216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11565.363216] awk '/^status/ { print $2 }' [11566.146492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11566.146492] awk '/^status/ { print $2 }' [11566.403870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11566.403870] awk '/^status/ { print $2 }' [11567.086422] Lustre: lustre-MDT0000: Denying connection for new client 00536dc1-bcbf-4dc5-cc21-2f8cbdee59ca(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [11567.092068] Lustre: Skipped 1 previous similar message [11572.390500] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11573.272158] Lustre: DEBUG MARKER: rc=0; [11573.272158] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11573.272158] if [[ $? -eq 0 && $val -ne 0 ]]; then [11573.272158] echo $(hostname -s): $val; [11573.272158] rc=$val; [11573.272158] fi; [11573.272158] exit $rc [11573.871210] Lustre: DEBUG MARKER: dmesg [11574.351383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1c: Auto detect kinds of OI file\(s\) removed\/recreated cases ===================== 13:45:45 \(1491486345\) [11574.509933] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ===================== 13:45:45 (1491486345) [11577.900446] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11581.276533] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11581.569927] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11581.823328] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11582.192933] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11583.362450] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11583.618249] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11589.989525] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11591.100240] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11591.344302] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11591.588048] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11591.714244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11591.839248] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 [11592.087024] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11593.619166] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11593.866593] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11594.110342] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11594.236168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11594.361071] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 [11594.610941] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11596.134878] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11596.395162] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11596.645080] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11596.892032] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11597.047929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11597.262812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11597.519351] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11598.011807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11598.016180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11598.174701] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11598.176287] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11598.323200] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11598.577569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11598.832639] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11599.092363] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11599.347564] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11601.478979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11601.637841] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11603.059646] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11603.323597] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11603.577468] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11603.831951] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11603.981260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.176393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11604.445331] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11604.950492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11604.960831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11605.114452] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11605.115278] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11605.260286] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11605.518761] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11605.788676] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11606.049295] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11606.306570] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11608.457762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11608.629976] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11610.081552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11610.594126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11611.124354] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11611.650609] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11612.164203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11612.164203] awk '/^status/ { print $2 }' [11612.423161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11612.423161] awk '/^status/ { print $2 }' [11613.220729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11613.220729] awk '/^status/ { print $2 }' [11613.495075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11613.495075] awk '/^status/ { print $2 }' [11614.279207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11614.794517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11616.482664] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11616.482664] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11616.482664] if [ $running -ne $mpts ]; then [11616.482664] echo $(hostname) env are INSANE!; [11616.482664] exit 1; [11616.482664] fi [11617.051652] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11617.051652] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11617.051652] if [ $running -ne $mpts ]; then [11617.051652] echo $(hostname) env are INSANE!; [11617.051652] exit 1; [11617.051652] fi [11617.646752] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11617.646752] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11617.646752] if [ $running -ne $mpts ]; then [11617.646752] echo $(hostname) env are INSANE!; [11617.646752] exit 1; [11617.646752] fi [11618.603073] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11618.875696] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11619.147968] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11619.407756] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11619.668400] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11621.211351] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11621.479944] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11621.736684] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11621.991203] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11622.251674] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11629.147851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11629.310458] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11630.738211] Lustre: DEBUG MARKER: lctl get_param -n timeout [11631.046424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11631.200963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11631.343027] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11639.987551] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11643.309804] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11643.604574] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11643.858312] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11644.027003] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [11644.029362] LustreError: Skipped 27 previous similar messages [11644.031105] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11644.034969] Lustre: Skipped 12 previous similar messages [11644.037154] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11644.041135] LustreError: Skipped 157 previous similar messages [11644.149247] Lustre: server umount lustre-MDT0000 complete [11644.151069] Lustre: Skipped 12 previous similar messages [11644.281422] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11645.395944] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11645.649823] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11651.988622] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11659.193648] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11659.443108] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11659.687357] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11659.823132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11659.950418] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 [11660.198652] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 [11660.445718] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 [11660.691849] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 [11660.938112] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 [11661.184312] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11663.677468] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11663.923942] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11664.172600] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11664.310176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11664.437473] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 [11664.685722] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 [11664.934098] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 [11665.182989] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 [11665.428164] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 [11665.675928] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11668.188325] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11668.448924] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11668.698901] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11668.949195] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11669.103989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.169459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11669.171519] Lustre: Skipped 12 previous similar messages [11669.318275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11669.573969] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11670.060855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11670.068653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11670.223342] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11670.232350] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11670.373619] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11670.626118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11670.877525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11671.136448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11671.388590] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11673.505662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11673.668320] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11674.166039] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491486439/real 1491486439] req@ffff88006656c300 x1563924895384096/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491486444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11674.173357] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 41 previous similar messages [11675.079811] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11675.346351] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11675.601045] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11675.865998] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11676.023465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.224077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11676.483069] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11676.974284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11676.977802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11677.134990] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11677.140245] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11677.284309] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11677.541874] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11677.815657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11678.075446] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11678.331261] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11680.468848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11680.639579] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11682.077341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11682.596138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11683.111768] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11683.622247] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11684.131970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11684.131970] awk '/^status/ { print $2 }' [11684.391295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11684.391295] awk '/^status/ { print $2 }' [11684.560553] Lustre: lustre-MDT0000: Connection restored to 10.9.5.239@tcp (at 10.9.5.239@tcp) [11684.563409] Lustre: Skipped 197 previous similar messages [11685.179856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11685.179856] awk '/^status/ { print $2 }' [11685.439634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11685.439634] awk '/^status/ { print $2 }' [11686.215653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11686.753440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11688.458400] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11688.458400] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11688.458400] if [ $running -ne $mpts ]; then [11688.458400] echo $(hostname) env are INSANE!; [11688.458400] exit 1; [11688.458400] fi [11689.036982] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11689.036982] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11689.036982] if [ $running -ne $mpts ]; then [11689.036982] echo $(hostname) env are INSANE!; [11689.036982] exit 1; [11689.036982] fi [11689.633368] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11689.633368] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11689.633368] if [ $running -ne $mpts ]; then [11689.633368] echo $(hostname) env are INSANE!; [11689.633368] exit 1; [11689.633368] fi [11690.576214] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11690.850143] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11691.115323] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11691.369969] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11691.631166] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11693.180327] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11693.464148] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11693.734862] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11694.005851] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11694.275072] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11701.173672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11701.336154] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11702.764443] Lustre: DEBUG MARKER: lctl get_param -n timeout [11703.076084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11703.241774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11703.375619] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11711.942314] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11715.296537] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11715.637697] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11715.894181] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11716.069308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.071353] Lustre: Skipped 62 previous similar messages [11716.249911] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11717.362838] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11717.616264] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11723.951376] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11731.063987] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11731.306858] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11731.553874] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11731.689626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11731.819941] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 [11732.069712] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 [11732.315123] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 [11732.562803] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11734.615131] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11734.869940] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11735.114035] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11735.250063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11735.377620] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 [11735.621533] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 [11735.867492] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 [11736.114421] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11738.133095] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11738.394044] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11738.656144] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11738.916861] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11739.073855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.287771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11739.543409] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11740.025669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11740.029126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11740.188580] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11740.189506] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11740.333191] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11740.589352] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11740.841669] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11741.099254] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11741.352610] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11743.480798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11743.640587] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11745.058957] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11745.325819] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11745.582349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11745.836198] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11745.993291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.192670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11746.451544] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11746.954929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11746.956607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11747.120129] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11747.120131] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11747.267939] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11747.527129] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11747.799841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11748.057345] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11748.313261] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11750.489713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11750.671071] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11752.120197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11752.652828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11753.169363] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11753.679167] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11754.195170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11754.195170] awk '/^status/ { print $2 }' [11754.452947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11754.452947] awk '/^status/ { print $2 }' [11755.255081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11755.255081] awk '/^status/ { print $2 }' [11755.513692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11755.513692] awk '/^status/ { print $2 }' [11756.295720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11756.827213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11757.388270] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11758.329139] Lustre: DEBUG MARKER: rc=0; [11758.329139] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11758.329139] if [[ $? -eq 0 && $val -ne 0 ]]; then [11758.329139] echo $(hostname -s): $val; [11758.329139] rc=$val; [11758.329139] fi; [11758.329139] exit $rc [11758.927593] Lustre: DEBUG MARKER: dmesg [11759.417096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup\/restore case ===================== 13:48:50 \(1491486530\) [11759.577887] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ===================== 13:48:50 (1491486530) [11760.912299] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11760.912299] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11760.912299] if [ $running -ne $mpts ]; then [11760.912299] echo $(hostname) env are INSANE!; [11760.912299] exit 1; [11760.912299] fi [11761.503423] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11761.503423] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11761.503423] if [ $running -ne $mpts ]; then [11761.503423] echo $(hostname) env are INSANE!; [11761.503423] exit 1; [11761.503423] fi [11762.111706] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11762.111706] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11762.111706] if [ $running -ne $mpts ]; then [11762.111706] echo $(hostname) env are INSANE!; [11762.111706] exit 1; [11762.111706] fi [11763.041343] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11763.314583] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11763.571852] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11763.830569] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11764.089409] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11765.655080] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11765.919891] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11766.177295] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11766.432286] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11766.695670] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11773.573457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11773.737350] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11775.175205] Lustre: DEBUG MARKER: lctl get_param -n timeout [11775.495524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11775.661389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11775.796162] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11784.382629] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11787.685856] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11787.979458] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11788.234518] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11788.647042] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11789.753316] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11790.006157] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11796.392194] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11797.566552] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11797.811894] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11798.061720] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11798.311300] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11798.448508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11798.575410] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11798.838160] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11799.110403] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11799.435352] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11799.684110] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11799.959874] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11800.198810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11800.338326] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11800.465314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11800.593993] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11800.867304] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11801.119809] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11801.366363] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11801.720990] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11801.968167] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11806.803072] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11807.048120] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11807.296946] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11807.544801] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11807.683069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11807.806836] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11808.066945] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11808.336063] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11808.661733] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11808.908948] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11809.164231] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11809.355690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11809.495710] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11809.623408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11809.748972] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11810.018637] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11810.269745] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11810.517772] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11810.871199] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11811.120038] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11815.958594] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11816.222059] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11816.474555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11816.721703] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11816.868823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.877571] LustreError: 10363:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from a7c554eb-99bc-a898-4da4-2a149ef49c2a to 06668af6-1b0f-a5c2-4865-32b99ca4f5c0 [11817.092852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11817.347708] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11817.841330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11817.841358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11818.003583] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11818.005141] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11818.153003] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11818.408094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11818.664843] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11818.927463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11819.182243] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11821.322626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11821.480444] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11822.916675] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11823.185149] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11823.444142] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11823.699700] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11823.860999] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.870385] LustreError: 11680:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0002: UUID has been changed from 6c701c28-fb4d-9ceb-48b8-41b767d9ab3f to 01828cd0-1d9c-9500-4e06-503a56d3dfad [11824.082825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11824.342053] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11824.837403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11824.837468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11825.004998] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11825.005107] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11825.151715] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11825.409633] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11825.695279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11825.956368] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11826.214723] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11828.403656] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11828.583138] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11830.077430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11830.077430] awk '/^status/ { print $2 }' [11830.339933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11830.339933] awk '/^status/ { print $2 }' [11831.113214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11831.113214] awk '/^status/ { print $2 }' [11831.373337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11831.373337] awk '/^status/ { print $2 }' [11832.059307] Lustre: lustre-MDT0000: Denying connection for new client 4ff645b4-4b40-ab77-8dc6-a9160b2c4695(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [11837.349538] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11838.289627] Lustre: DEBUG MARKER: rc=0; [11838.289627] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11838.289627] if [[ $? -eq 0 && $val -ne 0 ]]; then [11838.289627] echo $(hostname -s): $val; [11838.289627] rc=$val; [11838.289627] fi; [11838.289627] exit $rc [11838.904120] Lustre: DEBUG MARKER: dmesg [11839.433458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found \(1\) ======================= 13:50:10 \(1491486610\) [11839.605786] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ======================= 13:50:10 (1491486610) [11843.027421] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11846.343732] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11846.641512] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11846.896899] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11847.213347] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11848.335599] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11848.586683] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11854.919869] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11856.056498] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11856.300703] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11856.547112] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11856.790839] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11856.926687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11857.054618] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11857.317920] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11857.597810] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11857.929499] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11858.176445] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11858.458128] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11858.614543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11858.757493] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11858.884199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11859.009268] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11859.787998] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11860.037080] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11860.285105] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11860.604851] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11860.853678] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11865.612847] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11865.860424] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11866.107613] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11866.359976] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11866.499290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11866.626193] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11866.911731] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11867.175897] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11867.499257] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11867.746535] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11868.000346] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11868.191663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11868.331089] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11868.456685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11868.583317] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11868.849445] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11869.100895] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11869.351596] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11869.743281] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11869.989820] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11874.769873] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11875.028946] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11875.394906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11875.975859] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11876.382783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.406695] LustreError: 16687:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 06668af6-1b0f-a5c2-4865-32b99ca4f5c0 to 65bff135-6e0a-8943-45e2-8df905c12c8d [11877.043375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11877.519638] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11878.018472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11878.022569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11878.190063] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11878.190131] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11878.333167] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11878.585136] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11878.837190] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11879.096125] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11879.348666] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11881.513689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11881.680066] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11883.110686] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11883.384815] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11883.640188] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11883.896181] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11884.056697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11884.069491] LustreError: 18002:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0002: UUID has been changed from 01828cd0-1d9c-9500-4e06-503a56d3dfad to d87ea152-c364-b0bd-4613-6812644fc93f [11884.289244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11884.548050] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11885.085126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11885.088780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11885.243825] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11885.250118] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11885.394693] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11885.666439] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11885.940763] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11886.216193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11886.472411] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11888.927195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11889.100804] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11890.532024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11891.045524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11891.466846] Lustre: lustre-MDT0000: Denying connection for new client 423ba0c1-8855-f1a8-cb70-8ed6a2e768df(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [11891.472556] Lustre: Skipped 1 previous similar message [11893.005846] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11896.653221] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11896.937730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11896.937730] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11896.937730] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11897.243087] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [11897.407438] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004284:0x1:0x0]: rc = 0 [11897.435902] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280003ab1:0x2:0x0] with flags 0x52, rc = 0 [11900.625905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11900.625905] awk '/^status/ { print $2 }' [11900.887906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11900.887906] awk '/^status/ { print $2 }' [11901.671439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11901.671439] awk '/^status/ { print $2 }' [11901.934846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11901.934846] awk '/^status/ { print $2 }' [11902.710137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11903.220059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11903.738645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11904.260950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11907.816664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11908.336300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11908.890216] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11909.774420] Lustre: DEBUG MARKER: rc=0; [11909.774420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11909.774420] if [[ $? -eq 0 && $val -ne 0 ]]; then [11909.774420] echo $(hostname -s): $val; [11909.774420] rc=$val; [11909.774420] fi; [11909.774420] exit $rc [11910.360237] Lustre: DEBUG MARKER: dmesg [11910.866653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found \(2\) ======================= 13:51:21 \(1491486681\) [11911.017659] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ======================= 13:51:21 (1491486681) [11914.471048] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11917.783331] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11918.072061] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11918.328628] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11918.676534] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11919.825520] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11920.088926] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11926.490297] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11933.632726] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11933.893859] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11934.155384] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11934.417889] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11934.562872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11934.688634] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11934.953305] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11935.234160] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11935.592305] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11935.838632] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11936.116198] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11936.308939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11936.449704] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11936.577889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11936.706791] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11936.997896] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11937.246040] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11937.494185] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11937.865827] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11938.114355] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11942.851596] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11943.103985] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11943.352425] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11943.598318] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11943.723660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11943.850582] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11944.109693] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11944.379661] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11944.701759] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11944.950148] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11945.203970] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11945.404678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11945.544082] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11945.671091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11945.796205] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11946.063742] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11946.316206] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11946.565171] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11946.897568] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11947.144546] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11951.888981] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11952.146224] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11952.393079] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11952.639608] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11952.794646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.803766] LustreError: 23817:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 65bff135-6e0a-8943-45e2-8df905c12c8d to 37986ea3-9a21-bf8f-4f9d-da7a6818f201 [11953.073364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11953.330058] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11953.821421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11953.823892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11953.983070] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11953.983071] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11954.131520] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11954.386458] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11954.641165] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11954.901116] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11955.153966] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11957.312084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11957.476224] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11958.893439] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11959.162154] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11959.422163] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11959.678502] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11959.838450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11960.079102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11960.337926] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11960.830360] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11960.832317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11960.990869] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11960.994902] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11961.137135] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11961.395384] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11961.665509] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11961.926881] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11962.187472] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11965.359130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11965.521205] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11966.947338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11967.463452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11967.889349] Lustre: lustre-MDT0000: Denying connection for new client 262b51b2-d4b0-735e-6bfa-d7a6703c7e2b(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [11967.895197] Lustre: Skipped 2 previous similar messages [11968.799133] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11973.061928] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11973.337974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11973.337974] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11973.337974] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11973.638739] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=10 [11973.913397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11973.913397] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11973.913397] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11974.215137] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_threshold_rate=10000 [11974.370934] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004a54:0x1:0x0]: rc = 0 [11974.453918] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280004281:0x2:0x0] with flags 0x52, rc = 0 [11977.659397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11977.659397] awk '/^status/ { print $2 }' [11977.919350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11977.919350] awk '/^status/ { print $2 }' [11978.702681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11978.702681] awk '/^status/ { print $2 }' [11978.974809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11978.974809] awk '/^status/ { print $2 }' [11979.757169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11980.279413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11980.795641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11981.054926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11981.822464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11982.077112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11985.877248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11985.877248] awk '/^status/ { print $2 }' [11986.131996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11986.131996] awk '/^status/ { print $2 }' [11986.910533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11986.910533] awk '/^status/ { print $2 }' [11987.184358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11987.184358] awk '/^status/ { print $2 }' [11987.969178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11988.487154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11989.009077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11989.265648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11990.028087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11990.286102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11994.437276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11994.437276] awk '/^status/ { print $2 }' [11994.693714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11994.693714] awk '/^status/ { print $2 }' [11995.461983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11995.461983] awk '/^status/ { print $2 }' [11995.720814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11995.720814] awk '/^status/ { print $2 }' [11996.495104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11997.017977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11997.536626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11997.790913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11998.552901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11998.815743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12002.608118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12003.128993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12003.715735] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12004.597819] Lustre: DEBUG MARKER: rc=0; [12004.597819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12004.597819] if [[ $? -eq 0 && $val -ne 0 ]]; then [12004.597819] echo $(hostname -s): $val; [12004.597819] rc=$val; [12004.597819] fi; [12004.597819] exit $rc [12005.187908] Lustre: DEBUG MARKER: dmesg [12005.693230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found \(3\) ======================= 13:52:56 \(1491486776\) [12005.850275] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ======================= 13:52:56 (1491486776) [12010.788640] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12014.135896] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12014.433066] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12014.686447] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12015.035497] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12016.192180] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12016.444381] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12022.858989] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12029.980428] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12030.227157] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12030.472688] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12030.718903] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12030.855768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12030.981187] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12031.262903] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12031.552647] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12031.901873] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12032.147709] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12032.418973] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12032.605190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12032.741248] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12032.866647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12032.992287] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12033.281134] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12033.532905] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12033.779223] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12034.158527] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12034.404836] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12039.244383] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12039.493253] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12039.754822] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12040.012760] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12040.149891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12040.273103] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12040.541334] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12040.814638] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12041.186289] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12041.433681] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12041.689372] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12041.847306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12041.984220] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12042.109649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12042.233809] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12042.515382] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12042.769449] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12043.015508] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12043.393359] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12043.642191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12048.507075] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12048.768211] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12049.018544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12049.274924] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12049.498508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12049.507837] LustreError: 32716:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 37986ea3-9a21-bf8f-4f9d-da7a6818f201 to 195d2f79-3a07-a5b9-452a-239acf592a49 [12049.512519] LustreError: 32716:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 1 previous similar message [12049.759909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12050.030634] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12050.525235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12050.532730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12050.682917] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12050.684719] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12050.829519] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12051.082866] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12051.336130] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12051.594092] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12051.847039] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12054.019547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12054.188283] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12055.619349] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12055.882401] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12056.138161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12056.391273] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12056.554273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.782679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12057.044073] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12057.535830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12057.537903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12057.700481] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12057.700535] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12057.849745] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12058.096568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12058.099646] Lustre: Skipped 18 previous similar messages [12058.106761] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12058.378569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12058.642666] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12058.901494] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12061.578193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12061.759626] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12063.200533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12063.721143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12064.142551] Lustre: lustre-MDT0000: Denying connection for new client ef7e15af-642e-bb86-ebbc-1a9675737e8b(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [12065.678065] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12065.684716] Lustre: Skipped 1 previous similar message [12065.687440] LustreError: 1678:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200004a54:0x1:0x0]: rc = -78 [12065.692906] LustreError: 1678:0:(osp_object.c:527:osp_attr_get()) Skipped 2 previous similar messages [12065.707874] Lustre: lustre-MDT0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [12065.710784] Lustre: Skipped 17 previous similar messages [12069.323473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12069.598506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12069.598506] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12069.598506] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12069.893156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=2 [12070.166783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12070.166783] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12070.166783] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12070.462775] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_threshold_rate=20 [12070.617349] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200005224:0x1:0x0]: rc = 0 [12070.649689] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280004a52:0x27:0x0]: rc = 0 [12070.705661] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200005223:0x2:0x0] with flags 0x52, rc = 0 [12070.708529] Lustre: Skipped 2 previous similar messages [12073.856646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12073.856646] awk '/^status/ { print $2 }' [12074.114522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12074.114522] awk '/^status/ { print $2 }' [12074.890879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12074.890879] awk '/^status/ { print $2 }' [12075.147195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12075.147195] awk '/^status/ { print $2 }' [12075.926977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12076.439617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12076.957797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12077.217655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12077.992566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12078.247463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12082.057171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12082.057171] awk '/^status/ { print $2 }' [12082.315545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12082.315545] awk '/^status/ { print $2 }' [12083.090049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12083.090049] awk '/^status/ { print $2 }' [12083.350305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12083.350305] awk '/^status/ { print $2 }' [12084.131202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12084.653241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12085.178411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12085.435034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12086.196269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12086.453374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12090.637335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12090.637335] awk '/^status/ { print $2 }' [12090.894922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12090.894922] awk '/^status/ { print $2 }' [12091.674185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12091.674185] awk '/^status/ { print $2 }' [12091.934320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12091.934320] awk '/^status/ { print $2 }' [12092.705150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12093.220257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12093.731525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12093.990377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12094.756719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12095.013579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12098.798314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12099.307389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12099.870482] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12100.756260] Lustre: DEBUG MARKER: rc=0; [12100.756260] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12100.756260] if [[ $? -eq 0 && $val -ne 0 ]]; then [12100.756260] echo $(hostname -s): $val; [12100.756260] rc=$val; [12100.756260] fi; [12100.756260] exit $rc [12101.346756] Lustre: DEBUG MARKER: dmesg [12101.851786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 5: OI scrub state machine ======================================================= 13:54:32 \(1491486872\) [12102.000373] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine ======================================================= 13:54:32 (1491486872) [12102.824128] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12103.077261] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12105.823115] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880077f74f00 x1563924895448256/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [12105.827759] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [12109.427135] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12116.545978] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12116.797417] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [12118.624063] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12118.627772] LustreError: Skipped 2 previous similar messages [12120.847042] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007a54ea00 x1563924895448832/t0(0) o13->lustre-OST0002-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [12120.851510] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [12123.113424] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12133.565060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12133.565060] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12133.565060] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12133.867968] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12134.414356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12134.421971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12134.556287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12134.561058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12134.576259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [12134.792877] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12134.857195] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12134.860881] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12134.893804] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [12134.905790] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12135.240668] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12135.489128] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12135.761961] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12135.948995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12137.001635] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12137.248734] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12137.501683] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12137.695285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12143.336357] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12143.336357] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12143.336357] if [ $running -ne $mpts ]; then [12143.336357] echo $(hostname) env are INSANE!; [12143.336357] exit 1; [12143.336357] fi [12143.883959] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12143.883959] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12143.883959] if [ $running -ne $mpts ]; then [12143.883959] echo $(hostname) env are INSANE!; [12143.883959] exit 1; [12143.883959] fi [12144.450429] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12144.450429] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12144.450429] if [ $running -ne $mpts ]; then [12144.450429] echo $(hostname) env are INSANE!; [12144.450429] exit 1; [12144.450429] fi [12145.371850] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12145.633794] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12145.885291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12146.130310] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12146.284711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12146.326650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12146.355446] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12146.357557] Lustre: Skipped 4 previous similar messages [12146.369419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12146.384519] Lustre: lustre-MDT0000: new disk, initializing [12146.410064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12146.566132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12146.818900] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12147.308989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12147.309937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12147.468254] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12147.469381] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12147.611385] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12147.865158] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12148.120078] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12148.373185] Lustre: DEBUG MARKER: sync; sync; sync [12153.658246] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12153.924892] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12155.199916] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [12155.202312] Lustre: Skipped 4 previous similar messages [12155.265598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12156.197087] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12156.360693] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12163.053115] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12163.328900] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12163.583295] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12163.834733] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12163.992454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12164.028349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12164.042858] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [12164.045536] Lustre: Skipped 4 previous similar messages [12164.053251] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [12164.055797] Lustre: Skipped 1 previous similar message [12164.071364] Lustre: lustre-MDT0002: new disk, initializing [12164.109629] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [12164.266202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12164.532084] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12165.024595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12165.027253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12165.183861] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12165.197759] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12165.336316] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12165.592491] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12165.846379] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12166.102589] Lustre: DEBUG MARKER: sync; sync; sync [12171.436241] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12171.691203] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12172.911607] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [12172.914272] Lustre: Skipped 4 previous similar messages [12172.986769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [12172.990037] Lustre: Skipped 1 previous similar message [12173.874798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12174.029190] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12182.702343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12182.883122] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12191.050393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12191.224986] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12199.429055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12199.597099] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12207.800311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12207.962663] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12208.257767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000003c0000400-0x0000000400000400]:3:ost [12208.260993] Lustre: Skipped 3 previous similar messages [12215.388512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12215.559932] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12216.996566] Lustre: DEBUG MARKER: lctl get_param -n timeout [12217.311207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12217.482904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12217.618596] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12229.164918] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12232.520119] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12232.813944] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12233.070291] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12233.198361] Lustre: Failing over lustre-MDT0000 [12233.200824] Lustre: Skipped 20 previous similar messages [12233.430576] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12234.577664] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12234.830807] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12241.263971] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12248.368659] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12248.613858] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12248.858940] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12249.107039] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12249.244192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12249.370515] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12249.649218] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12249.928611] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12250.301135] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12250.562874] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12250.842823] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12250.999685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12251.138248] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12251.337642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12251.464469] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12251.763313] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12252.018117] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12252.266714] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12252.582335] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12252.828344] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12257.683191] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12257.927717] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12258.174833] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12258.424307] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12258.594722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12258.720749] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12259.000797] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12259.274712] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12259.602600] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12259.850632] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12260.108696] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12260.302249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12260.450434] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12260.584938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12260.719096] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12261.013827] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12261.270839] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12261.517616] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12261.872272] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12262.118604] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12266.974259] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12267.240657] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12267.489051] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12267.733630] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12267.890413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.900239] LustreError: 14551:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 4502be6c-6f94-a0b8-4bbf-1cadbc3fd8ed to 6a4f4bff-6cff-a8bd-4e5e-da3581a4edb3 [12267.904931] LustreError: 14551:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 1 previous similar message [12267.966524] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.971340] LustreError: Skipped 159 previous similar messages [12268.127065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12268.383919] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12268.868047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12268.878302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12269.033044] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12269.034767] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12269.177767] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12269.433005] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12269.687184] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12269.946851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12270.201693] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12272.336645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12272.500901] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12274.233025] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12274.503917] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12274.766883] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12275.020948] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12275.181592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12275.248955] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_connect to node 0@lo failed: rc = -114 [12275.251829] LustreError: Skipped 35 previous similar messages [12275.275567] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [12275.278245] Lustre: Skipped 14 previous similar messages [12275.425364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12275.683466] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12276.196039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12276.196571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12276.358939] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12276.361946] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12276.510800] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12276.772831] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12277.038345] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12277.301877] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12277.561677] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12279.708334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12279.870301] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12281.335896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12281.335896] awk '/^status/ { print $2 }' [12281.598714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12281.598714] awk '/^status/ { print $2 }' [12282.371476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12282.371476] awk '/^status/ { print $2 }' [12282.633264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12282.633264] awk '/^status/ { print $2 }' [12283.410867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12283.941880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12284.568206] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12284.844175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12284.844175] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12284.844175] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12285.141235] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12285.442638] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12285.599073] Lustre: *** cfs_fail_loc=190, val=0*** [12285.601417] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200000404:0x1:0x0]: rc = 0 [12285.647365] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280000404:0x27:0x0]: rc = 0 [12285.677490] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280000404:0x220:0x0] with flags 0x52, rc = 0 [12285.808653] Lustre: MGS: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [12285.811492] Lustre: Skipped 207 previous similar messages [12285.980115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12285.980115] awk '/^status/ { print $2 }' [12286.236630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12286.236630] awk '/^status/ { print $2 }' [12287.010685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12287.010685] awk '/^status/ { print $2 }' [12287.268898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12287.268898] awk '/^status/ { print $2 }' [12288.071020] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x191 [12288.680032] Lustre: *** cfs_fail_loc=191, val=0*** [12288.682277] Lustre: Skipped 10 previous similar messages [12292.356262] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12292.620240] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12292.873818] Lustre: server umount lustre-MDT0000 complete [12292.876047] Lustre: Skipped 15 previous similar messages [12293.006541] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12294.109431] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12294.363906] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12300.543050] Lustre: 18341:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491487065/real 1491487065] req@ffff880061a3f300 x1563924895479728/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491487071 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12300.550399] Lustre: 18341:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [12300.775532] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12301.928702] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12302.203262] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12302.463552] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12302.714816] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12302.960604] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12303.115567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12303.336085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12303.607758] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12304.096985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12304.098241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12304.258185] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12304.258186] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12304.406090] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12304.659574] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12304.917166] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12305.173726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12305.430933] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12307.545685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12307.712631] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12309.134663] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12309.399204] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12309.658213] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12309.911730] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12310.070415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12310.263672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12310.523654] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12311.016917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12311.028242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12311.183782] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12311.189075] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12311.330645] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12311.588849] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12311.856613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12312.128944] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12312.388661] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12314.542950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12314.724214] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12316.160404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12316.160404] awk '/^status/ { print $2 }' [12316.424192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12316.424192] awk '/^status/ { print $2 }' [12317.198624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12317.198624] awk '/^status/ { print $2 }' [12317.455757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12317.455757] awk '/^status/ { print $2 }' [12318.266520] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12318.538829] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12318.903894] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12320.028157] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12320.285828] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12326.659690] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12345.781973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12346.056910] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12346.323233] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12346.581303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12346.828697] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12346.984206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.999347] Lustre: *** cfs_fail_loc=190, val=0*** [12347.001678] Lustre: Skipped 1 previous similar message [12347.198614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12347.454254] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12347.943892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12347.946860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12348.106556] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12348.107065] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12348.249665] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12348.506936] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12348.761212] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12349.021183] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12349.275476] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12350.003083] Lustre: *** cfs_fail_loc=190, val=0*** [12351.405239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12351.566091] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12352.980665] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12353.256812] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12353.526328] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12353.793899] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12353.955767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12354.151483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12354.409286] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12354.915260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12354.919739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12355.078444] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12355.080102] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12355.222581] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12355.480472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12355.739239] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12355.998720] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12356.006297] Lustre: *** cfs_fail_loc=190, val=0*** [12356.008743] Lustre: Skipped 2 previous similar messages [12356.264377] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12358.470204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12358.654852] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12360.085574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12360.085574] awk '/^status/ { print $2 }' [12360.343994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12360.343994] awk '/^status/ { print $2 }' [12361.124131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12361.124131] awk '/^status/ { print $2 }' [12361.385947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12361.385947] awk '/^status/ { print $2 }' [12362.185152] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x192 [12362.464717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12362.464717] awk '/^status/ { print $2 }' [12362.756298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12362.756298] awk '/^status/ { print $2 }' [12364.014238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12364.014238] awk '/^status/ { print $2 }' [12365.011028] Lustre: *** cfs_fail_loc=192, val=0*** [12365.013395] Lustre: Skipped 5 previous similar messages [12365.274113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12365.274113] awk '/^status/ { print $2 }' [12366.042310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12366.042310] awk '/^status/ { print $2 }' [12366.300955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12366.300955] awk '/^status/ { print $2 }' [12367.144295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12367.144295] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12367.144295] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12367.452120] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12367.750359] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12367.912201] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200000404:0x1b8:0x0]: rc = 0 [12367.921374] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280000402:0x2:0x0] with flags 0x52, rc = 0 [12367.924066] Lustre: Skipped 1 previous similar message [12371.038531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12371.038531] awk '/^status/ { print $2 }' [12371.302377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12371.302377] awk '/^status/ { print $2 }' [12372.087930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12372.087930] awk '/^status/ { print $2 }' [12372.347870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12372.347870] awk '/^status/ { print $2 }' [12373.165639] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12373.450281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12373.450281] awk '/^status/ { print $2 }' [12373.707926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12373.707926] awk '/^status/ { print $2 }' [12374.965397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12374.965397] awk '/^status/ { print $2 }' [12375.739361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12375.739361] awk '/^status/ { print $2 }' [12375.999486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12375.999486] awk '/^status/ { print $2 }' [12376.770029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12377.283604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12377.870773] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12378.756357] Lustre: DEBUG MARKER: rc=0; [12378.756357] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12378.756357] if [[ $? -eq 0 && $val -ne 0 ]]; then [12378.756357] echo $(hostname -s): $val; [12378.756357] rc=$val; [12378.756357] fi; [12378.756357] exit $rc [12379.352897] Lustre: DEBUG MARKER: dmesg [12379.977576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 6: OI scrub resumes from last checkpoint ======================================== 13:59:10 \(1491487150\) [12380.143764] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ======================================== 13:59:10 (1491487150) [12391.675617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12395.034889] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12395.322185] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12395.586720] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12395.927190] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12397.056111] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12397.309219] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12403.680333] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12410.849715] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12411.095973] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12411.358384] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12411.619269] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12411.764226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12411.896597] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12412.197079] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12412.495944] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12412.858210] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12413.107542] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12413.384494] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12413.542806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12413.680339] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12413.807573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12413.932779] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12414.228415] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12414.483892] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12414.731890] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12415.079124] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12415.324341] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12420.118791] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12420.367416] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12420.629663] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12420.879628] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12421.006924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12421.133873] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12421.411245] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12421.688498] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12422.010473] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12422.257143] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12422.511884] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12422.709789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12422.851351] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12422.978930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12423.105138] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12423.398735] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12423.655756] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12423.902915] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12424.259272] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12424.506104] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12429.583763] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12429.842020] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12430.089068] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12430.335152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12430.482449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.491958] LustreError: 30171:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 6a4f4bff-6cff-a8bd-4e5e-da3581a4edb3 to 0e982711-94ee-82e4-4779-d8546443a74e [12430.496633] LustreError: 30171:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 1 previous similar message [12430.720571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12430.978381] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12431.467396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12431.473763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12431.636364] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12431.637953] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12431.781413] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12432.036971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12432.291025] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12432.547472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12432.802629] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12434.934747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12435.089270] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12436.511836] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12436.775036] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12437.030117] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12437.283707] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12437.441438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12437.657104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12437.915186] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12438.420303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12438.424208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12438.583063] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12438.585587] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12438.728779] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12438.982329] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12439.241657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12439.505203] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12439.763061] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12442.269342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12442.439789] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12443.922469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12444.433768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12444.863233] Lustre: lustre-MDT0000: Denying connection for new client b8c7c5c8-15e3-fe71-3a52-13ace693d63d(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [12444.869054] Lustre: Skipped 1 previous similar message [12446.332768] LustreError: 31519:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200001b74:0x1:0x0]: rc = -78 [12446.338087] LustreError: 31519:0:(osp_object.c:527:osp_attr_get()) Skipped 1 previous similar message [12446.341392] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x200001b74:0x1:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12446.347112] Lustre: Skipped 5 previous similar messages [12450.045655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12450.321195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12450.321195] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12450.321195] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12450.624619] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12450.922627] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x190 [12451.076309] Lustre: *** cfs_fail_loc=190, val=0*** [12451.078747] Lustre: Skipped 8 previous similar messages [12451.079042] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200001b74:0x1:0x0]: rc = 0 [12451.121252] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280001b70:0x3:0x0] with flags 0x52, rc = 0 [12456.351220] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x192 [12456.633087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12456.633087] awk '/^status/ { print $2 }' [12456.890228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12456.890228] awk '/^status/ { print $2 }' [12458.147719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12458.147719] awk '/^status/ { print $2 }' [12458.928934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12458.928934] awk '/^status/ { print $2 }' [12459.184528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12459.184528] awk '/^status/ { print $2 }' [12459.992511] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12460.154249] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280001b72:0x1b9:0x0]: rc = 0 [12460.168423] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200001b74:0x3d7:0x0]: rc = 0 [12460.437047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12460.437047] awk '/^status/ { print $2 }' [12460.696215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12460.696215] awk '/^status/ { print $2 }' [12461.485691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12461.485691] awk '/^status/ { print $2 }' [12461.753745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12461.753745] awk '/^status/ { print $2 }' [12462.558420] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x191 [12466.837955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12467.351430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12467.872649] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12468.126112] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12468.471169] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12469.572567] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12469.834087] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12476.199685] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12483.402433] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12483.680701] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12483.939365] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12484.188488] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12484.435954] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12484.592563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.607957] Lustre: *** cfs_fail_loc=190, val=0*** [12484.610062] Lustre: Skipped 21 previous similar messages [12484.803372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12485.059644] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12485.552205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12485.552220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12485.713255] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12485.717145] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12485.865437] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12486.118306] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12486.370714] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12486.637119] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12486.891097] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12489.022655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12489.179066] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12490.620573] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12490.892532] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12491.148262] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12491.401217] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12491.547833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12491.748184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12492.008462] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12492.518347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12492.518348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12492.681109] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12492.681317] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12492.829059] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12493.087493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12493.354077] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12493.613417] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12493.870114] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12496.037078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12496.209408] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12497.653500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12497.653500] awk '/^status/ { print $2 }' [12497.922443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12497.922443] awk '/^status/ { print $2 }' [12498.696345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12498.696345] awk '/^status/ { print $2 }' [12498.959697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12498.959697] awk '/^status/ { print $2 }' [12499.734251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12500.268182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12500.817494] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12501.092259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12501.092259] awk '/^status/ { print $2 }' [12501.348644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12501.348644] awk '/^status/ { print $2 }' [12502.606977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12502.606977] awk '/^status/ { print $2 }' [12503.397138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12503.397138] awk '/^status/ { print $2 }' [12503.664531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12503.664531] awk '/^status/ { print $2 }' [12504.433135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12504.954597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12505.525297] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12506.430501] Lustre: DEBUG MARKER: rc=0; [12506.430501] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12506.430501] if [[ $? -eq 0 && $val -ne 0 ]]; then [12506.430501] echo $(hostname -s): $val; [12506.430501] rc=$val; [12506.430501] fi; [12506.430501] exit $rc [12507.033213] Lustre: DEBUG MARKER: dmesg [12507.550764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 7: System is available during OI scrub scanning ================================= 14:01:18 \(1491487278\) [12507.705102] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ================================= 14:01:18 (1491487278) [12509.001609] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12509.001609] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12509.001609] if [ $running -ne $mpts ]; then [12509.001609] echo $(hostname) env are INSANE!; [12509.001609] exit 1; [12509.001609] fi [12509.581466] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12509.581466] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12509.581466] if [ $running -ne $mpts ]; then [12509.581466] echo $(hostname) env are INSANE!; [12509.581466] exit 1; [12509.581466] fi [12510.168169] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12510.168169] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12510.168169] if [ $running -ne $mpts ]; then [12510.168169] echo $(hostname) env are INSANE!; [12510.168169] exit 1; [12510.168169] fi [12511.109614] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12511.381204] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12511.637054] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12511.893437] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12512.154247] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12513.711591] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12513.984960] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12514.239751] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12514.496882] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12514.759360] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12521.634034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12521.790649] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12523.220332] Lustre: DEBUG MARKER: lctl get_param -n timeout [12523.535812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12523.694275] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12523.831807] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12538.486086] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12541.838447] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12542.127029] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12542.381539] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12542.730178] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12543.880266] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12544.132208] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12550.519849] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12557.688190] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12557.932602] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12558.180275] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12558.427106] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12558.552514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12558.680671] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12558.949830] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12559.224979] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12559.580509] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12559.827828] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12560.105728] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12560.264195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12560.401272] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12560.530557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12560.666261] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12560.967076] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12561.219885] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12561.468782] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12561.780535] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12562.028621] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12566.812343] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12567.061593] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12567.307117] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12567.553419] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12567.689677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12567.815053] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12568.083322] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12568.361539] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12568.687088] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12568.932163] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12569.184984] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12569.372325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12569.516612] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12569.641785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12569.766702] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12570.052097] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12570.304962] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12570.552994] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12570.936232] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12571.180730] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12576.043215] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12576.306708] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12576.569223] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12576.818509] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12576.973130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.982154] LustreError: 10767:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 0e982711-94ee-82e4-4779-d8546443a74e to dee99a24-6c4e-8fc2-407d-058eacf28ea1 [12576.986854] LustreError: 10767:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 1 previous similar message [12577.204569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12577.466934] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12577.970106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12577.973661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12578.132058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12578.133072] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12578.277602] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12578.533202] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12578.790043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12579.051039] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12579.304801] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12581.571830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12581.719664] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12583.145401] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12583.417234] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12583.674555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12583.929308] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12584.089953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12584.331265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12584.589366] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12585.095438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12585.095718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12585.263947] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12585.266728] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12585.411195] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12585.671048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12585.928479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12586.195777] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12586.454559] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12588.597711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12588.757208] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12590.277322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12590.791301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12591.210899] Lustre: lustre-MDT0000: Denying connection for new client cd215769-5f22-faed-baed-864ea1127af2(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [12591.216763] Lustre: Skipped 1 previous similar message [12592.695643] LustreError: 12113:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200002b14:0x1:0x0]: rc = -78 [12592.700862] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x200002b14:0x1:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12596.401728] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12596.685569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12596.685569] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12596.685569] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12596.989099] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12597.292701] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12597.454827] Lustre: *** cfs_fail_loc=190, val=0*** [12597.454833] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200002b14:0x1:0x0]: rc = 0 [12597.460216] Lustre: Skipped 10 previous similar messages [12597.506027] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280002b12:0x27:0x0]: rc = 0 [12597.551165] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200002b13:0x3:0x0] with flags 0x52, rc = 0 [12597.554099] Lustre: Skipped 1 previous similar message [12597.708177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12597.708177] awk '/^status/ { print $2 }' [12597.967554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12597.967554] awk '/^status/ { print $2 }' [12598.740728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12598.740728] awk '/^status/ { print $2 }' [12599.000684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12599.000684] awk '/^status/ { print $2 }' [12599.780172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12600.324168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12600.882645] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12601.164121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12601.164121] awk '/^status/ { print $2 }' [12601.442979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12601.442979] awk '/^status/ { print $2 }' [12602.710176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12602.710176] awk '/^status/ { print $2 }' [12603.980853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12603.980853] awk '/^status/ { print $2 }' [12604.779872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12604.779872] awk '/^status/ { print $2 }' [12605.038558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12605.038558] awk '/^status/ { print $2 }' [12605.846356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12606.373445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12606.935577] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12607.830132] Lustre: DEBUG MARKER: rc=0; [12607.830132] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12607.830132] if [[ $? -eq 0 && $val -ne 0 ]]; then [12607.830132] echo $(hostname -s): $val; [12607.830132] rc=$val; [12607.830132] fi; [12607.830132] exit $rc [12608.424750] Lustre: DEBUG MARKER: dmesg [12608.925436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 8: Control OI scrub manually ==================================================== 14:02:59 \(1491487379\) [12609.085293] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ==================================================== 14:02:59 (1491487379) [12615.031234] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12618.354347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12618.676392] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12618.932915] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12619.183204] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.187815] Lustre: Skipped 9 previous similar messages [12619.282755] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12620.452715] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12620.706249] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12627.069395] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12634.210231] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12634.460205] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12634.707481] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12634.957063] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12635.096994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12635.221849] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12635.490099] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12635.763215] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12636.084792] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12636.332283] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12636.609712] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12636.768677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12636.911838] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12637.110563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12637.235402] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12637.514415] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12637.763721] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12638.011917] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12638.348214] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12638.593748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12643.519092] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12643.766240] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12644.012195] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12644.261258] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12644.398418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12644.522932] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12644.786585] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12645.058921] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12645.389026] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12645.651928] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12645.922137] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12646.122687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12646.267093] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12646.392974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12646.519104] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12646.790500] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12647.040683] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12647.290249] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12647.718230] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12647.964797] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12652.768734] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12653.026847] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12653.275656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12653.521872] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12653.676727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.899994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12654.158282] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12654.650676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12654.651878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12654.808153] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12654.812365] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12654.956856] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12655.219183] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12655.471198] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12655.728488] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12655.984926] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12658.110218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12658.275316] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12659.702143] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12659.966518] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12660.224098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12660.477428] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12660.637500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12660.876480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12661.137683] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12661.628778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12661.640954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12661.795797] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12661.803350] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12661.945300] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12662.202733] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12662.226617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12662.229681] Lustre: Skipped 13 previous similar messages [12662.466636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12662.727110] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12662.982941] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12665.630308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12665.797746] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12667.241770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12667.757561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12668.294073] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190 [12668.576641] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 [12669.734830] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x280002b12:0x126:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12669.763869] Lustre: lustre-MDT0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [12669.766729] Lustre: Skipped 12 previous similar messages [12671.135115] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 [12673.657704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12673.657704] awk '/^status/ { print $2 }' [12673.917412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12673.917412] awk '/^status/ { print $2 }' [12674.699222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12674.699222] awk '/^status/ { print $2 }' [12674.961968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12674.961968] awk '/^status/ { print $2 }' [12675.747098] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [12682.140163] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0002 [12683.655744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12683.655744] awk '/^status/ { print $2 }' [12683.913881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12683.913881] awk '/^status/ { print $2 }' [12684.682380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12684.682380] awk '/^status/ { print $2 }' [12684.940671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12684.940671] awk '/^status/ { print $2 }' [12685.715930] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 9 [12686.228788] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 9 [12686.758174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12686.758174] awk '/^status/ { print $2 }' [12687.018075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12687.018075] awk '/^status/ { print $2 }' [12687.806491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12687.806491] awk '/^status/ { print $2 }' [12688.061963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12688.061963] awk '/^status/ { print $2 }' [12688.856978] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12689.135558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12689.135558] awk '/^status/ { print $2 }' [12689.391543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12689.391543] awk '/^status/ { print $2 }' [12690.648704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12690.648704] awk '/^status/ { print $2 }' [12691.424354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12691.424354] awk '/^status/ { print $2 }' [12691.682790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12691.682790] awk '/^status/ { print $2 }' [12692.453637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12692.973306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12693.557094] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12694.452449] Lustre: DEBUG MARKER: rc=0; [12694.452449] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12694.452449] if [[ $? -eq 0 && $val -ne 0 ]]; then [12694.452449] echo $(hostname -s): $val; [12694.452449] rc=$val; [12694.452449] fi; [12694.452449] exit $rc [12695.068295] Lustre: DEBUG MARKER: dmesg [12695.598767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 9: OI scrub speed control ======================================================= 14:04:26 \(1491487466\) [12695.772848] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control ======================================================= 14:04:26 (1491487466) [12697.078474] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12697.078474] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12697.078474] if [ $running -ne $mpts ]; then [12697.078474] echo $(hostname) env are INSANE!; [12697.078474] exit 1; [12697.078474] fi [12697.645226] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12697.645226] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12697.645226] if [ $running -ne $mpts ]; then [12697.645226] echo $(hostname) env are INSANE!; [12697.645226] exit 1; [12697.645226] fi [12698.232088] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12698.232088] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12698.232088] if [ $running -ne $mpts ]; then [12698.232088] echo $(hostname) env are INSANE!; [12698.232088] exit 1; [12698.232088] fi [12699.176103] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12699.443124] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12699.698397] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12699.953409] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12700.211700] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12701.780845] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12702.058646] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12702.312943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12702.567277] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12702.825528] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12709.720417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12709.888735] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12711.315686] Lustre: DEBUG MARKER: lctl get_param -n timeout [12711.625800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12711.781947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12711.912876] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12742.856651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12746.191567] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12746.481378] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12746.751230] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12747.225520] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12748.337118] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12748.604695] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12754.953264] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12762.114822] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12762.361897] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12762.616692] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12762.861593] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12762.988075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12763.113981] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12763.491076] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12763.819903] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12764.174923] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12764.420706] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12764.705198] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12764.862775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12765.002975] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12765.131150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12765.255845] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12765.796739] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12766.178499] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12766.426195] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12766.843721] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12767.091296] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12772.624153] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12772.872243] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12773.117369] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12773.364413] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12773.501613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12773.627782] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12774.003693] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12774.343314] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12774.712415] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12774.962480] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12775.217319] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12775.404693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12775.550272] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12775.746176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12775.872734] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12776.405933] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12776.734765] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12776.982622] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12777.443082] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12777.689400] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12783.070956] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12783.337267] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12783.600257] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12783.856148] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12784.010673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12784.019221] LustreError: 27790:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 991db1e6-d5be-bba2-435b-732d06308c52 to 20fad4c3-b488-9df8-490f-7c29df2a60c4 [12784.023897] LustreError: 27790:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 3 previous similar messages [12784.249123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12784.506114] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12784.995567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12784.999782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12785.156777] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12785.157308] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12785.303507] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12785.557381] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12785.811665] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12786.073601] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12786.327896] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12788.467034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12788.636821] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12790.085939] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12790.349463] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12790.607017] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12790.862537] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12791.012652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12791.216285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12791.488839] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12792.011154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12792.011772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12792.263929] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12792.266437] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12792.424448] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12792.692309] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12792.956881] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12793.218723] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12793.477943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12795.992163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12796.158249] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12797.594967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12798.114603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12798.631422] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 -s 100 [12799.148861] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 -s 100 [12809.673213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12809.673213] awk '/^status/ { print $2 }' [12809.940167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12809.940167] awk '/^status/ { print $2 }' [12810.710670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12810.710670] awk '/^status/ { print $2 }' [12810.972020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12810.972020] awk '/^status/ { print $2 }' [12811.738265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12812.249937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12812.762796] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 8 -s 100 -r [12813.275164] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 8 -s 100 -r [12823.793131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12823.793131] awk '/^status/ { print $2 }' [12824.050268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12824.050268] awk '/^status/ { print $2 }' [12824.821871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12824.821871] awk '/^status/ { print $2 }' [12825.083934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12825.083934] awk '/^status/ { print $2 }' [12825.857307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12826.378397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12826.890970] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0000.lfsck_speed_limit 300 [12827.402818] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0002.lfsck_speed_limit 300 [12837.915129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12838.174865] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0000.lfsck_speed_limit 0 [12838.950620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12839.209366] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0002.lfsck_speed_limit 0 [12839.974781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12839.974781] awk '/^status/ { print $2 }' [12840.230705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12840.230705] awk '/^status/ { print $2 }' [12841.011512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12841.011512] awk '/^status/ { print $2 }' [12841.271618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12841.271618] awk '/^status/ { print $2 }' [12842.095501] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12842.971960] Lustre: DEBUG MARKER: rc=0; [12842.971960] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12842.971960] if [[ $? -eq 0 && $val -ne 0 ]]; then [12842.971960] echo $(hostname -s): $val; [12842.971960] rc=$val; [12842.971960] fi; [12842.971960] exit $rc [12843.562647] Lustre: DEBUG MARKER: dmesg [12844.105127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount \(1\) ============ 14:06:55 \(1491487615\) [12844.258399] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ============ 14:06:55 (1491487615) [12845.547760] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12845.547760] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12845.547760] if [ $running -ne $mpts ]; then [12845.547760] echo $(hostname) env are INSANE!; [12845.547760] exit 1; [12845.547760] fi [12846.112218] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12846.112218] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12846.112218] if [ $running -ne $mpts ]; then [12846.112218] echo $(hostname) env are INSANE!; [12846.112218] exit 1; [12846.112218] fi [12846.699746] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12846.699746] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12846.699746] if [ $running -ne $mpts ]; then [12846.699746] echo $(hostname) env are INSANE!; [12846.699746] exit 1; [12846.699746] fi [12847.644107] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12847.912348] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12848.168148] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12848.423482] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12848.683221] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12850.233078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12850.503306] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12850.758611] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12851.012127] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12851.269544] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12858.126305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12858.284614] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12859.704785] Lustre: DEBUG MARKER: lctl get_param -n timeout [12860.008049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12860.160755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12860.293713] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12899.251945] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12902.574889] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12902.862324] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12903.116831] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12903.245560] Lustre: Failing over lustre-MDT0000 [12903.247322] Lustre: Skipped 15 previous similar messages [12903.348331] Lustre: server umount lustre-MDT0000 complete [12903.350132] Lustre: Skipped 13 previous similar messages [12903.479198] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12904.614335] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12904.868391] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12904.994962] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 [12904.997168] LustreError: Skipped 36 previous similar messages [12905.082478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12905.086717] LustreError: Skipped 135 previous similar messages [12911.062056] Lustre: 1872:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491487675/real 1491487675] req@ffff880066d7ea00 x1563924895667184/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491487681 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12911.068357] Lustre: 1872:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [12911.331872] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12918.488322] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12918.737255] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12918.981197] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12919.225211] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12919.363874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12919.493976] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12919.757386] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12920.038549] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12920.443345] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12920.692874] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12920.969760] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12921.162843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12921.311454] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12921.446296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12921.579881] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12921.855904] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12922.104971] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12922.355282] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12922.715234] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12922.960995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12927.837367] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12928.098789] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12928.348515] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12928.595777] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12928.720650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12928.845622] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12929.106106] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12929.383651] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12929.738275] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12929.986806] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12930.256603] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12930.454508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12930.599255] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12930.726312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12930.853072] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12931.124892] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12931.377262] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12931.626103] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12931.987566] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12932.235480] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12937.084301] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12937.351873] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12937.604796] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12937.860829] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12938.020762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12938.038407] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [12938.042089] Lustre: Skipped 202 previous similar messages [12938.114743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12938.117125] Lustre: Skipped 14 previous similar messages [12938.263080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12938.518936] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12939.033605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12939.036424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12939.194025] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12939.200949] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12939.342711] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12939.612369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12939.872850] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12940.132373] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12940.388535] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12942.609435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12942.769464] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12944.197936] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12944.465871] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12944.721687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12944.975422] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12945.134817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12945.361733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12945.627767] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12946.123423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12946.123681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12946.301651] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12946.303462] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12946.449089] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12946.730273] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12946.989050] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12947.252149] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12947.510724] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12950.057825] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12950.234308] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12951.666571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12952.191309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12952.625266] Lustre: lustre-MDT0000: Denying connection for new client bcb611a7-ce68-319a-a785-379214de7603(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [12952.631088] Lustre: Skipped 1 previous similar message [12954.151334] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12954.158174] LustreError: 6150:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200004284:0x1:0x0]: rc = -78 [12957.814201] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12958.094133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12958.094133] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12958.094133] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12958.401385] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12958.697496] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190 [12958.849365] Lustre: *** cfs_fail_loc=190, val=0*** [12958.851817] Lustre: Skipped 44 previous similar messages [12958.852031] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004284:0x1:0x0]: rc = 0 [12958.887215] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280004282:0x27:0x0]: rc = 0 [12958.943228] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200004281:0x3:0x0] with flags 0x52, rc = 0 [12959.094587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12959.094587] awk '/^status/ { print $2 }' [12959.373203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12959.373203] awk '/^status/ { print $2 }' [12960.154423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12960.154423] awk '/^status/ { print $2 }' [12960.410731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12960.410731] awk '/^status/ { print $2 }' [12961.329371] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12961.587148] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12961.929694] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12963.043316] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12963.305609] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12969.678641] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12976.816427] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12977.088476] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12977.336361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12977.584537] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12977.752956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.007912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12978.261355] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12978.749757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12978.761431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12978.913542] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12978.914024] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12979.060843] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12979.312252] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12979.567366] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12979.842960] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12980.106135] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12982.241154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12982.399869] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12983.826242] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12984.090534] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12984.344408] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12984.598302] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12984.745311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12984.939070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12985.195990] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12985.687643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12985.690943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12985.849575] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12985.857556] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12985.997782] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12986.256673] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12986.529595] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12986.789908] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12987.048455] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12989.181903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12989.348111] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12990.797189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12990.797189] awk '/^status/ { print $2 }' [12991.055253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12991.055253] awk '/^status/ { print $2 }' [12991.841382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12991.841382] awk '/^status/ { print $2 }' [12992.100650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12992.100650] awk '/^status/ { print $2 }' [12992.884405] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12993.139986] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12993.269980] LustreError: 5278:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [12993.276427] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [12993.279317] Lustre: Skipped 19 previous similar messages [12993.284205] LustreError: 11344:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [12993.287086] Lustre: 8665:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [12993.289998] Lustre: 8665:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [12993.302369] LustreError: 8652:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004eb4f300 x1563924895673616/t0(0) o5->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [12993.302701] LustreError: 8654:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0003-osc-MDT0000: cannot cleanup orphans: rc = -5 [12993.313177] LustreError: 8652:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [12994.514622] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12996.689937] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12996.957477] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12997.097459] LustreError: 9944:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200004284:0x1:0x0]: rc = -5 [12997.097486] LustreError: 5278:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [12997.097488] LustreError: 5278:0:(import.c:662:ptlrpc_connect_import()) Skipped 1 previous similar message [12997.103432] LustreError: 11564:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0002: Aborting recovery [12997.111233] Lustre: 9944:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [12997.114095] Lustre: 9944:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [12997.125764] LustreError: 9929:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800649b1200 x1563924895673840/t0(0) o5->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [12997.125831] LustreError: 9927:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0002: cannot cleanup orphans: rc = -5 [12997.125832] LustreError: 9927:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [12997.139442] LustreError: 9929:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [13004.336412] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13011.485688] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13011.746591] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13011.997177] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13012.244971] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13012.401270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13012.610359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13012.865245] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13013.352855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13013.364696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13013.518958] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13013.528044] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13013.668534] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13013.924636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13014.176043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13014.434328] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13014.689273] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13016.806034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13016.973937] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13018.386878] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13018.651672] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13018.908885] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13019.165120] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13019.324806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13019.523737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13019.784556] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13020.274240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13020.285572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13020.453027] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13020.453936] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13020.603397] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13020.861617] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13021.140302] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13021.401624] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [13021.660552] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13023.821241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13023.984625] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13025.414640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13025.414640] awk '/^status/ { print $2 }' [13025.674862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13025.674862] awk '/^status/ { print $2 }' [13026.447499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13026.447499] awk '/^status/ { print $2 }' [13026.707627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13026.707627] awk '/^status/ { print $2 }' [13027.507586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13027.783884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13027.783884] awk '/^status/ { print $2 }' [13028.075134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13028.075134] awk '/^status/ { print $2 }' [13028.849808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13028.849808] awk '/^status/ { print $2 }' [13029.107500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13029.107500] awk '/^status/ { print $2 }' [13029.889262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13030.400302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13030.965637] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13031.887107] Lustre: DEBUG MARKER: rc=0; [13031.887107] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13031.887107] if [[ $? -eq 0 && $val -ne 0 ]]; then [13031.887107] echo $(hostname -s): $val; [13031.887107] rc=$val; [13031.887107] fi; [13031.887107] exit $rc [13032.481157] Lustre: DEBUG MARKER: dmesg [13033.018907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 11: OI scrub skips the new created objects only once ============================ 14:10:04 \(1491487804\) [13033.173275] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ============================ 14:10:04 (1491487804) [13034.491893] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13034.491893] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13034.491893] if [ $running -ne $mpts ]; then [13034.491893] echo $(hostname) env are INSANE!; [13034.491893] exit 1; [13034.491893] fi [13035.075113] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13035.075113] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13035.075113] if [ $running -ne $mpts ]; then [13035.075113] echo $(hostname) env are INSANE!; [13035.075113] exit 1; [13035.075113] fi [13035.677563] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13035.677563] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13035.677563] if [ $running -ne $mpts ]; then [13035.677563] echo $(hostname) env are INSANE!; [13035.677563] exit 1; [13035.677563] fi [13036.640345] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13036.908702] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13037.170315] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13037.429481] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13037.691005] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13039.232114] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13039.505032] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13039.769165] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13040.025274] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13040.283464] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13047.159800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13047.348589] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13048.787777] Lustre: DEBUG MARKER: lctl get_param -n timeout [13049.096178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13049.256074] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13049.401741] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13055.992229] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 3 -r [13056.508551] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 3 -r [13057.028061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13057.028061] awk '/^status/ { print $2 }' [13057.281933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13057.281933] awk '/^status/ { print $2 }' [13058.047936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13058.047936] awk '/^status/ { print $2 }' [13058.309932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13058.309932] awk '/^status/ { print $2 }' [13059.096424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13059.354413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13060.126558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13060.386148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13061.174972] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 6 -r [13061.686510] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 6 -r [13062.196141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13062.196141] awk '/^status/ { print $2 }' [13062.452593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13062.452593] awk '/^status/ { print $2 }' [13063.225131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13063.225131] awk '/^status/ { print $2 }' [13063.477773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13063.477773] awk '/^status/ { print $2 }' [13064.250793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13064.508861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13065.275766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13065.533265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13066.350531] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13067.238506] Lustre: DEBUG MARKER: rc=0; [13067.238506] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13067.238506] if [[ $? -eq 0 && $val -ne 0 ]]; then [13067.238506] echo $(hostname -s): $val; [13067.238506] rc=$val; [13067.238506] fi; [13067.238506] exit $rc [13067.825320] Lustre: DEBUG MARKER: dmesg [13068.348390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 12: OI scrub can rebuild invalid \/O entries ===================================== 14:10:39 \(1491487839\) [13068.505891] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ===================================== 14:10:39 (1491487839) [13069.577777] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [13069.833172] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [13073.379575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13073.541274] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13085.679818] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13086.566627] Lustre: DEBUG MARKER: rc=0; [13086.566627] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13086.566627] if [[ $? -eq 0 && $val -ne 0 ]]; then [13086.566627] echo $(hostname -s): $val; [13086.566627] rc=$val; [13086.566627] fi; [13086.566627] exit $rc [13087.163918] Lustre: DEBUG MARKER: dmesg [13087.676321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 13: OI scrub can rebuild missed \/O entries ====================================== 14:10:58 \(1491487858\) [13087.825532] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ====================================== 14:10:58 (1491487858) [13088.308761] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [13088.564418] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [13092.109652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13092.282929] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13099.150347] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13100.019090] Lustre: DEBUG MARKER: rc=0; [13100.019090] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13100.019090] if [[ $? -eq 0 && $val -ne 0 ]]; then [13100.019090] echo $(hostname -s): $val; [13100.019090] rc=$val; [13100.019090] fi; [13100.019090] exit $rc [13100.608606] Lustre: DEBUG MARKER: dmesg [13101.167574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 14: OI scrub can repair objects under lost+found ================================ 14:11:12 \(1491487872\) [13101.338198] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair objects under lost+found ================================ 14:11:12 (1491487872) [13101.810587] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [13102.066270] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [13105.898218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13106.065571] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13112.117438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13113.031235] Lustre: DEBUG MARKER: rc=0; [13113.031235] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13113.031235] if [[ $? -eq 0 && $val -ne 0 ]]; then [13113.031235] echo $(hostname -s): $val; [13113.031235] rc=$val; [13113.031235] fi; [13113.031235] exit $rc [13113.620560] Lustre: DEBUG MARKER: dmesg [13114.143106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 15: Dryrun mode OI scrub ======================================================== 14:11:25 \(1491487885\) [13114.334485] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ======================================================== 14:11:25 (1491487885) [13114.474649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13114.474649] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13114.474649] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13117.406412] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [13120.748268] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [13121.030192] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13121.297087] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13121.741085] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13122.856405] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13123.125512] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [13129.477241] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13130.596749] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13130.850898] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [13131.097341] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13131.344923] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13131.481850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13131.606762] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [13131.869721] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [13132.154882] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13132.518485] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13132.770859] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13133.048836] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [13133.243395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13133.391539] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13133.526043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13133.659160] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [13133.941351] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [13134.190678] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [13134.437206] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13134.786948] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13135.035529] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [13139.801147] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13140.049370] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [13140.295277] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13140.542944] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [13140.668918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [13140.796926] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [13141.061695] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [13141.337335] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13141.659296] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13141.909099] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13142.166610] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [13142.365735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13142.505122] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [13142.631370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [13142.758339] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [13143.023714] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [13143.279165] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [13143.528122] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13143.917944] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13144.168100] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [13148.890131] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13149.148238] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13149.397936] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13149.650367] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13149.810356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13149.819457] LustreError: 23446:0:(osd_scrub.c:2770:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from a1ed5d51-12fc-a1df-4e35-9b89a659b447 to 60b194a0-5bb2-98eb-4f6f-878068ed5069 [13149.824154] LustreError: 23446:0:(osd_scrub.c:2770:osd_scrub_setup()) Skipped 3 previous similar messages [13150.055711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13150.314198] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13150.804719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13150.808539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13150.962111] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13150.965077] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13151.109475] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13151.364036] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13151.618146] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13151.875959] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13152.131224] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13154.273588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13154.484208] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13155.941803] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13156.207675] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13156.470420] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13156.726856] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13156.886588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.095444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13157.353904] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13157.853402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13157.853404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13158.020057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13158.020067] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13158.167333] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13158.433108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13158.690971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13158.953572] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [13159.214245] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13161.392515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13161.572363] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13163.016272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13163.016272] awk '/^status/ { print $2 }' [13163.277331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13163.277331] awk '/^status/ { print $2 }' [13164.052778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13164.052778] awk '/^status/ { print $2 }' [13164.316409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13164.316409] awk '/^status/ { print $2 }' [13165.109179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13165.501070] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x2000059f7:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [13165.503346] LustreError: 24793:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x2000059f7:0x1:0x0]: rc = -78 [13165.512768] Lustre: Skipped 2 previous similar messages [13165.678566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13166.206952] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 --dryrun [13166.725578] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 --dryrun [13167.247164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13167.247164] awk '/^status/ { print $2 }' [13167.509209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13167.509209] awk '/^status/ { print $2 }' [13168.281853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13168.281853] awk '/^status/ { print $2 }' [13168.541939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13168.541939] awk '/^status/ { print $2 }' [13169.313494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13169.827574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13170.342345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13170.874026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13171.390184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13171.904578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13172.425852] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 10 --dryrun [13172.937515] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 10 --dryrun [13173.465595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13173.465595] awk '/^status/ { print $2 }' [13173.722309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13173.722309] awk '/^status/ { print $2 }' [13174.507523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13174.507523] awk '/^status/ { print $2 }' [13174.772469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13174.772469] awk '/^status/ { print $2 }' [13175.542947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13176.074382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13176.588752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13177.100343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13177.619349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13178.133413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13178.660491] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 15 [13179.185201] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 15 [13179.696703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13179.696703] awk '/^status/ { print $2 }' [13179.952518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13179.952518] awk '/^status/ { print $2 }' [13180.722222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13180.722222] awk '/^status/ { print $2 }' [13180.979679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13180.979679] awk '/^status/ { print $2 }' [13181.748629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13182.258102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13182.783614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13183.298841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13183.816899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13184.331924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13184.853080] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 20 [13185.365745] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 20 [13185.881511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13185.881511] awk '/^status/ { print $2 }' [13186.141331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [13186.141331] awk '/^status/ { print $2 }' [13186.912795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13186.912795] awk '/^status/ { print $2 }' [13187.188767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [13187.188767] awk '/^status/ { print $2 }' [13187.972243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13188.483934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13188.997565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13189.512486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13190.025212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [13190.536983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [13191.097524] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13191.979132] Lustre: DEBUG MARKER: rc=0; [13191.979132] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13191.979132] if [[ $? -eq 0 && $val -ne 0 ]]; then [13191.979132] echo $(hostname -s): $val; [13191.979132] rc=$val; [13191.979132] fi; [13191.979132] exit $rc [13192.566850] Lustre: DEBUG MARKER: dmesg [13193.605871] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13193.861774] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13195.501289] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [13195.501289] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [13195.501289] Lustre: Skipped 2 previous similar messages [13195.506822] Lustre: Skipped 2 previous similar messages [13195.599096] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006695a100 x1563924895701792/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13200.181905] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13207.399279] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13207.669876] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [13208.951067] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13210.623071] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006695b000 x1563924895702432/t0(0) o13->lustre-OST0002-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13210.627370] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [13214.052326] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13232.487831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13232.487831] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13232.487831] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13232.803436] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13233.421866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13233.431614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13233.472877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13233.498078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13233.510166] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13233.722777] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13233.739315] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13233.821457] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13233.827374] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13233.842195] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13234.171115] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13234.422119] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13234.708868] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [13234.923968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13236.052261] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13236.313596] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13236.579075] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [13236.792886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13245.684128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test complete, duration 1975 sec ===================================================== 14:13:36 \(1491488016\) [13245.853461] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 1975 sec ===================================================== 14:13:36 (1491488016) [13246.078809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-lfsck ============----- Thu Apr 6 14:13:37 UTC 2017 [13246.255312] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lfsck ============----- Thu Apr 6 14:13:37 UTC 2017 [13246.527061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test complete, duration -o sec ======================================================= 14:13:37 \(1491488017\) [13246.674902] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration -o sec ======================================================= 14:13:37 (1491488017) [13247.046799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13247.046799] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13247.046799] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13247.604172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13247.604172] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13247.604172] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13248.454785] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13248.702745] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13249.467538] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13249.715183] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13252.547296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13252.547296] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13252.547296] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13252.853352] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13253.440816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13253.499241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13253.517771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13253.523312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13253.559745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13253.766437] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13253.789800] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13253.819331] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13253.880189] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13253.909392] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13254.229376] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13254.474338] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13254.749259] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat /dev/lvm-Role_MDS/P1 [13254.948004] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13256.025438] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13256.269715] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13256.523404] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat [13256.710694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13262.232622] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13262.232622] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13262.232622] if [ $running -ne $mpts ]; then [13262.232622] echo $(hostname) env are INSANE!; [13262.232622] exit 1; [13262.232622] fi [13262.782168] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13262.782168] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13262.782168] if [ $running -ne $mpts ]; then [13262.782168] echo $(hostname) env are INSANE!; [13262.782168] exit 1; [13262.782168] fi [13263.356533] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13263.356533] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13263.356533] if [ $running -ne $mpts ]; then [13263.356533] echo $(hostname) env are INSANE!; [13263.356533] exit 1; [13263.356533] fi [13264.287305] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13264.543750] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13264.794034] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13265.041801] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13265.244463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13265.256031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13265.291252] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13265.293374] Lustre: Skipped 4 previous similar messages [13265.306670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13265.333134] Lustre: lustre-MDT0000: new disk, initializing [13265.355995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13265.519632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13265.782830] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13266.278787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13266.282291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13266.435520] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13266.435566] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13266.582713] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13266.849464] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13267.100071] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13267.353213] Lustre: DEBUG MARKER: sync; sync; sync [13272.629595] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13272.879320] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13274.126647] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [13274.129054] Lustre: Skipped 4 previous similar messages [13274.201825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13275.095754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13275.256468] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13281.997372] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13282.268243] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13282.524818] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13282.791882] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13283.004654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13283.016262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13283.032554] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [13283.035187] Lustre: Skipped 4 previous similar messages [13283.041911] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [13283.044425] Lustre: Skipped 1 previous similar message [13283.062393] Lustre: lustre-MDT0002: new disk, initializing [13283.084624] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [13283.247055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13283.503223] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13283.988754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13283.994385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13284.149906] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13284.151905] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13284.297488] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13284.551542] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13284.804739] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13285.060690] Lustre: DEBUG MARKER: sync; sync; sync [13290.382042] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [13290.641066] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13291.879258] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [13291.881921] Lustre: Skipped 4 previous similar messages [13291.933177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [13291.936053] Lustre: Skipped 1 previous similar message [13292.832572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13293.003583] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13301.664918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13301.836022] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13309.982441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13310.155768] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13318.330320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13318.492368] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13327.261109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13327.425429] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13332.451314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000003c0000400-0x0000000400000400]:3:ost [13332.454551] Lustre: Skipped 3 previous similar messages [13334.836286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13334.997062] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13336.427915] Lustre: DEBUG MARKER: lctl get_param -n timeout [13336.735520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13336.891520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13337.028846] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13342.308361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13342.308361] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13342.308361] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13342.870570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13342.870570] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13342.870570] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13343.199905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [13343.359275] Lustre: DEBUG MARKER: excepting tests: [13343.679364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 0: Control LFSCK manually ======================================================= 14:15:14 \(1491488114\) [13343.857852] Lustre: DEBUG MARKER: == sanity-lfsck test 0: Control LFSCK manually ======================================================= 14:15:14 (1491488114) [13344.474410] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x1600 [13344.747477] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13344.879425] LustreError: 5179:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [13345.008083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13345.265539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13345.518167] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [13347.882036] LustreError: 5179:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [13348.074662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13348.333363] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13348.463384] LustreError: 5564:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 3000ms [13348.592107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13348.846834] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13349.110081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13349.110081] awk '/^status/ { print $2 }' [13349.369171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13349.369171] awk '/^status/ { print $2 }' [13350.626943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13350.626943] awk '/^status/ { print $2 }' [13351.466041] LustreError: 5564:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [13351.892656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13351.892656] awk '/^status/ { print $2 }' [13352.154425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13352.414023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13352.668976] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13352.923143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13352.923143] awk '/^status/ { print $2 }' [13353.181294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13353.181294] awk '/^status/ { print $2 }' [13353.448034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13354.390684] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13354.647087] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13354.831102] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880060cdf300 x1563924895711632/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13354.836151] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [13357.437182] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13357.441917] Lustre: Skipped 9 previous similar messages [13361.001480] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13368.194702] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13368.449173] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [13374.817503] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13391.249525] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13392.103409] Lustre: DEBUG MARKER: rc=0; [13392.103409] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13392.103409] if [[ $? -eq 0 && $val -ne 0 ]]; then [13392.103409] echo $(hostname -s): $val; [13392.103409] rc=$val; [13392.103409] fi; [13392.103409] exit $rc [13392.675539] Lustre: DEBUG MARKER: dmesg [13393.187519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================= 14:16:04 \(1491488164\) [13393.341534] Lustre: DEBUG MARKER: == sanity-lfsck test 1a: LFSCK can find out and repair crashed FID-in-dirent ========================= 14:16:04 (1491488164) [13394.583614] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13394.583614] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13394.583614] if [ $running -ne $mpts ]; then [13394.583614] echo $(hostname) env are INSANE!; [13394.583614] exit 1; [13394.583614] fi [13395.144633] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13395.144633] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13395.144633] if [ $running -ne $mpts ]; then [13395.144633] echo $(hostname) env are INSANE!; [13395.144633] exit 1; [13395.144633] fi [13395.722422] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13395.722422] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13395.722422] if [ $running -ne $mpts ]; then [13395.722422] echo $(hostname) env are INSANE!; [13395.722422] exit 1; [13395.722422] fi [13396.646705] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13396.906980] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13397.154645] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13397.401623] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13397.547570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13397.754005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13398.008232] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13398.500206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13398.503502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13398.662172] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13398.662173] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13398.808175] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13399.064180] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13399.319139] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13399.579110] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13399.838083] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13401.955812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13402.126134] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13403.540647] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13403.804707] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13404.058509] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13404.311111] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13404.470669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13404.664471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13404.919044] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13405.411955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13405.412921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13405.571578] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13405.574903] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13405.718326] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13405.974497] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13406.231646] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13406.491437] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [13406.749865] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13408.917592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13409.082612] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13412.395769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13412.564270] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13415.387172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13415.564162] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13418.368625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13418.537700] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13421.344217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13421.510716] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13423.576554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13423.746928] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13425.180581] Lustre: DEBUG MARKER: lctl get_param -n timeout [13425.494405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13425.657954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13425.789031] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13431.566271] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1501 [13431.709637] Lustre: *** cfs_fail_loc=1501, val=0*** [13431.845310] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13432.337509] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13432.594674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13432.594674] awk '/^status/ { print $2 }' [13432.852564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13432.852564] awk '/^status/ { print $2 }' [13434.112552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13434.112552] awk '/^status/ { print $2 }' [13435.371536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13435.371536] awk '/^status/ { print $2 }' [13436.630113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13436.630113] awk '/^status/ { print $2 }' [13436.894538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13437.156226] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13437.411917] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13437.731378] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13437.996743] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13438.266598] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13438.532303] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13438.785043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13439.035804] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13439.194379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13439.211895] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d43c0ee to 0x61ac6f141d43fb17 [13439.443758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13439.711634] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13440.201249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13440.204172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13440.375569] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13440.375570] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13440.524989] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13440.780571] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13441.035482] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13441.294827] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13441.549376] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13441.711188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [13441.713945] Lustre: Skipped 11 previous similar messages [13441.716194] Lustre: lustre-MDT0000: Denying connection for new client b3a32302-d390-95a2-faf8-533590b7a7d7(at 10.9.5.237@tcp), waiting for 4 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [13441.721429] Lustre: Skipped 1 previous similar message [13444.249697] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13447.702465] Lustre: lustre-MDT0000: Recovery over after 0:06, of 4 clients 4 recovered and 0 were evicted. [13447.705416] Lustre: Skipped 10 previous similar messages [13451.857361] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1505 [13451.996210] Lustre: *** cfs_fail_loc=1505, val=0*** [13452.138625] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13452.441036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13453.340075] Lustre: DEBUG MARKER: rc=0; [13453.340075] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13453.340075] if [[ $? -eq 0 && $val -ne 0 ]]; then [13453.340075] echo $(hostname -s): $val; [13453.340075] rc=$val; [13453.340075] fi; [13453.340075] exit $rc [13453.941889] Lustre: DEBUG MARKER: dmesg [13454.452125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ======================== 14:17:05 \(1491488225\) [13454.604600] Lustre: DEBUG MARKER: == sanity-lfsck test 1b: LFSCK can find out and repair the missing FID-in-LMA ======================== 14:17:05 (1491488225) [13455.223568] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1502 [13455.357531] Lustre: *** cfs_fail_loc=1502, val=0*** [13455.485733] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13456.000836] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1506 [13456.273497] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13456.539232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13456.539232] awk '/^status/ { print $2 }' [13456.801461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13456.801461] awk '/^status/ { print $2 }' [13458.062695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13458.062695] awk '/^status/ { print $2 }' [13458.322589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13458.579943] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13458.843232] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13459.097287] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13459.423487] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13459.689191] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13459.952407] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13460.216748] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13460.471698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13460.725978] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13460.883952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13461.086204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13461.344024] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13461.847231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13461.850229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13462.007266] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13462.010772] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13462.157093] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13462.414456] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13462.670765] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13462.930972] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13463.190632] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13468.505756] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1505 [13468.649591] Lustre: *** cfs_fail_loc=1505, val=0*** [13468.785517] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13469.097240] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13469.974480] Lustre: DEBUG MARKER: rc=0; [13469.974480] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13469.974480] if [[ $? -eq 0 && $val -ne 0 ]]; then [13469.974480] echo $(hostname -s): $val; [13469.974480] rc=$val; [13469.974480] fi; [13469.974480] exit $rc [13470.572983] Lustre: DEBUG MARKER: dmesg [13471.091834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================== 14:17:22 \(1491488242\) [13471.239246] Lustre: DEBUG MARKER: == sanity-lfsck test 2a: LFSCK can find out and repair crashed linkEA entry ========================== 14:17:22 (1491488242) [13471.882565] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [13472.016306] Lustre: *** cfs_fail_loc=1603, val=0*** [13472.145527] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13472.654302] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13472.913276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13472.913276] awk '/^status/ { print $2 }' [13473.170417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13473.170417] awk '/^status/ { print $2 }' [13474.425751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13474.425751] awk '/^status/ { print $2 }' [13475.688965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13475.688965] awk '/^status/ { print $2 }' [13476.957300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13476.957300] awk '/^status/ { print $2 }' [13477.215507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13477.478520] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13477.735245] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13478.065287] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13478.331544] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13478.606694] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13478.879094] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13479.130620] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13479.383181] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13479.540676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13479.737404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13479.996279] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13480.487147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13480.490048] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13480.653078] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13480.653726] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13480.796468] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13481.051377] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13481.309554] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13481.568392] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13481.824472] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13487.234739] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13488.104512] Lustre: DEBUG MARKER: rc=0; [13488.104512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13488.104512] if [[ $? -eq 0 && $val -ne 0 ]]; then [13488.104512] echo $(hostname -s): $val; [13488.104512] rc=$val; [13488.104512] fi; [13488.104512] exit $rc [13488.707073] Lustre: DEBUG MARKER: dmesg [13489.230885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================== 14:17:40 \(1491488260\) [13489.389463] Lustre: DEBUG MARKER: == sanity-lfsck test 2b: LFSCK can find out and remove invalid linkEA entry ========================== 14:17:40 (1491488260) [13490.020113] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [13490.151591] Lustre: *** cfs_fail_loc=1604, val=0*** [13490.280816] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13490.770377] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13491.033658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13491.033658] awk '/^status/ { print $2 }' [13491.293894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13491.293894] awk '/^status/ { print $2 }' [13492.554604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13492.554604] awk '/^status/ { print $2 }' [13493.811422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13493.811422] awk '/^status/ { print $2 }' [13494.071527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13494.334915] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13494.592992] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13494.927523] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13495.196254] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13495.467643] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13495.732815] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13495.989166] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13496.240871] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13496.399671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13496.609985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13496.884664] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13497.398690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13497.400474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13497.561982] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13497.566118] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13497.707369] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13497.973393] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13498.238476] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13498.499798] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13498.773910] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13504.171352] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13505.052322] Lustre: DEBUG MARKER: rc=0; [13505.052322] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13505.052322] if [[ $? -eq 0 && $val -ne 0 ]]; then [13505.052322] echo $(hostname -s): $val; [13505.052322] rc=$val; [13505.052322] fi; [13505.052322] exit $rc [13505.638988] Lustre: DEBUG MARKER: dmesg [13506.175288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================= 14:17:57 \(1491488277\) [13506.330447] Lustre: DEBUG MARKER: == sanity-lfsck test 2c: LFSCK can find out and remove repeated linkEA entry ========================= 14:17:57 (1491488277) [13507.039338] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1605 [13507.358461] Lustre: *** cfs_fail_loc=1605, val=0*** [13507.680918] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13508.638157] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13509.137190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13509.137190] awk '/^status/ { print $2 }' [13509.396092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13509.396092] awk '/^status/ { print $2 }' [13509.654771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13509.916117] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13510.171371] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13510.303282] Lustre: Failing over lustre-MDT0000 [13510.305412] Lustre: Skipped 11 previous similar messages [13510.402760] Lustre: server umount lustre-MDT0000 complete [13510.404977] Lustre: Skipped 15 previous similar messages [13510.419815] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13510.424568] LustreError: Skipped 129 previous similar messages [13510.538143] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13510.805844] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13511.073886] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13511.335300] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13511.452316] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [13511.454747] LustreError: Skipped 91916 previous similar messages [13511.593761] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13511.849711] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13512.010497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13512.211412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13512.469794] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13512.972786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13512.973949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13513.132366] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13513.140636] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13513.284339] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13513.542303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13513.804185] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13514.063627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13514.320187] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13519.703142] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13520.579030] Lustre: DEBUG MARKER: rc=0; [13520.579030] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13520.579030] if [[ $? -eq 0 && $val -ne 0 ]]; then [13520.579030] echo $(hostname -s): $val; [13520.579030] rc=$val; [13520.579030] fi; [13520.579030] exit $rc [13521.169457] Lustre: DEBUG MARKER: dmesg [13521.719816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ================================== 14:18:12 \(1491488292\) [13521.870355] Lustre: DEBUG MARKER: == sanity-lfsck test 2d: LFSCK can recover the missing linkEA entry ================================== 14:18:12 (1491488292) [13522.468321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161d [13522.609123] Lustre: *** cfs_fail_loc=161d, val=0*** [13522.744273] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13523.246416] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13523.522560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13523.522560] awk '/^status/ { print $2 }' [13523.796605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13523.796605] awk '/^status/ { print $2 }' [13525.070452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13525.070452] awk '/^status/ { print $2 }' [13525.330370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13525.595125] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13525.854336] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13526.196917] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13526.467219] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13526.739285] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13527.003466] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13527.260679] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13527.514642] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13527.673696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13527.691064] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d44d193 to 0x61ac6f141d450747 [13527.695213] Lustre: Skipped 4 previous similar messages [13527.886549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13528.143519] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13528.640672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13528.644570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13528.803046] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13528.804836] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13528.951058] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13529.208336] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13529.465098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13529.724507] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13529.981683] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13532.731778] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13532.736966] LustreError: Skipped 4 previous similar messages [13535.370499] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13536.271138] Lustre: DEBUG MARKER: rc=0; [13536.271138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13536.271138] if [[ $? -eq 0 && $val -ne 0 ]]; then [13536.271138] echo $(hostname -s): $val; [13536.271138] rc=$val; [13536.271138] fi; [13536.271138] exit $rc [13536.871561] Lustre: DEBUG MARKER: dmesg [13537.408814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ============================= 14:18:28 \(1491488308\) [13537.558735] Lustre: DEBUG MARKER: == sanity-lfsck test 2e: namespace LFSCK can verify remote object linkEA ============================= 14:18:28 (1491488308) [13537.818601] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [13537.974530] Lustre: *** cfs_fail_loc=1603, val=0*** [13538.127865] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13538.387685] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [13538.671043] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [13538.671043] awk '/^namespace_mdts_completed/ { print $2 }' [13538.931932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13539.245342] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13540.139290] Lustre: DEBUG MARKER: rc=0; [13540.139290] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13540.139290] if [[ $? -eq 0 && $val -ne 0 ]]; then [13540.139290] echo $(hostname -s): $val; [13540.139290] rc=$val; [13540.139290] fi; [13540.139290] exit $rc [13540.722757] Lustre: DEBUG MARKER: dmesg [13541.241668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ===================================== 14:18:32 \(1491488312\) [13541.390991] Lustre: DEBUG MARKER: == sanity-lfsck test 3: LFSCK can verify multiple-linked objects ===================================== 14:18:32 (1491488312) [13542.038866] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [13542.176782] Lustre: *** cfs_fail_loc=1603, val=0*** [13542.305302] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [13542.441829] Lustre: *** cfs_fail_loc=1604, val=0*** [13542.572757] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13542.832505] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13543.090085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13543.090085] awk '/^status/ { print $2 }' [13543.347098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13543.347098] awk '/^status/ { print $2 }' [13543.610535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13543.869795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13544.175968] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13545.062208] Lustre: DEBUG MARKER: rc=0; [13545.062208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13545.062208] if [[ $? -eq 0 && $val -ne 0 ]]; then [13545.062208] echo $(hostname -s): $val; [13545.062208] rc=$val; [13545.062208] fi; [13545.062208] exit $rc [13545.652736] Lustre: DEBUG MARKER: dmesg [13546.180840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup\/restore ============= 14:18:37 \(1491488317\) [13546.340942] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore ============= 14:18:37 (1491488317) [13547.507877] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13547.773301] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13547.968086] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [13547.971396] Lustre: Skipped 28 previous similar messages [13548.148744] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13548.433079] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13548.695455] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [13548.946627] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13549.198804] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13549.339563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13549.468982] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [13549.740814] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [13550.017804] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13550.354349] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13550.618205] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13550.899528] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat /dev/lvm-Role_MDS/P1 [13551.095709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13551.238588] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13551.368810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13551.497998] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [13551.768651] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [13552.023645] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [13552.278527] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13552.596353] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13552.849477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [13553.122567] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13553.392462] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13553.652922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13553.914978] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13554.134759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13554.150372] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [13554.152694] Lustre: Skipped 316 previous similar messages [13554.228110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13554.230413] Lustre: Skipped 17 previous similar messages [13554.392219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13554.651497] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13555.148516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13555.154325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13555.313057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13555.316469] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13555.472470] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13555.729459] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13555.989721] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13556.251794] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13556.512223] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13556.770502] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [13557.028618] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13557.162447] LustreError: 29754:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [13557.291228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13557.291228] awk '/^flags/ { print $2 }' [13557.549286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13557.549286] awk '/^flags/ { print $2 }' [13558.165066] LustreError: 29754:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [13558.814081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13558.814081] awk '/^flags/ { print $2 }' [13559.089371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13559.167067] LustreError: 29754:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [13559.170243] LustreError: 29754:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [13559.371330] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13559.644774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13559.644774] awk '/^status/ { print $2 }' [13559.919527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13559.919527] awk '/^status/ { print $2 }' [13560.173052] LustreError: 29754:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [13560.175775] LustreError: 29754:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [13561.190302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13561.190302] awk '/^status/ { print $2 }' [13561.452695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13561.713405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13561.978770] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13562.236024] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13562.582986] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13562.854946] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13563.140076] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13563.419323] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13563.676965] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13563.929376] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13564.088673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13564.289665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13564.557915] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13565.066601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13565.066665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13565.246876] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13565.248950] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13565.393742] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13565.652187] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13565.907964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13566.166757] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13566.423428] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13576.737617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1505 [13576.875849] Lustre: *** cfs_fail_loc=1505, val=0*** [13576.878085] Lustre: Skipped 1 previous similar message [13577.020416] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13577.331802] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13578.210277] Lustre: DEBUG MARKER: rc=0; [13578.210277] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13578.210277] if [[ $? -eq 0 && $val -ne 0 ]]; then [13578.210277] echo $(hostname -s): $val; [13578.210277] rc=$val; [13578.210277] fi; [13578.210277] exit $rc [13578.805391] Lustre: DEBUG MARKER: dmesg [13579.346189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ======================================= 14:19:10 \(1491488350\) [13579.512962] Lustre: DEBUG MARKER: == sanity-lfsck test 5: LFSCK can handle IGIF object upgrading ======================================= 14:19:10 (1491488350) [13579.778031] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1504 [13579.913966] Lustre: *** cfs_fail_loc=1504, val=0*** [13580.418918] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13581.179214] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13581.444921] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13581.807665] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13582.090862] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13582.344539] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [13582.596605] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13582.851417] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13582.992643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13583.121685] Lustre: DEBUG MARKER: rm -rf /mnt/lustre-brpt/ROOT/.lustre [13583.377948] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [13583.649402] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [13583.936901] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13584.329514] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13584.585039] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13584.867797] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat /dev/lvm-Role_MDS/P1 [13585.027124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13585.167318] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [13585.294479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13585.422882] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [13585.696219] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [13585.950673] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [13586.203198] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [13586.510098] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [13586.763629] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [13587.029336] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13587.292214] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13587.547135] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13587.807042] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13587.967017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13588.191016] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13588.449122] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13588.941821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13588.941860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13589.102853] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13589.106277] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13589.253460] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13589.508992] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13589.764698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13590.029163] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13590.286099] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13590.543186] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [13590.800898] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13590.938955] LustreError: 3160:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [13591.066501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13591.066501] awk '/^flags/ { print $2 }' [13591.327483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13591.327483] awk '/^flags/ { print $2 }' [13591.942057] LustreError: 3160:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [13592.592858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13592.592858] awk '/^flags/ { print $2 }' [13593.860720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13593.860720] awk '/^flags/ { print $2 }' [13595.124987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13595.124987] awk '/^flags/ { print $2 }' [13596.386647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13596.386647] awk '/^flags/ { print $2 }' [13596.647245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13596.903727] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13597.162691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13597.162691] awk '/^status/ { print $2 }' [13597.423217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13597.423217] awk '/^status/ { print $2 }' [13598.688563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13598.688563] awk '/^status/ { print $2 }' [13598.949340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13599.207713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13599.478933] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13599.740207] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13600.084397] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13600.348298] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 [13600.617166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13600.884058] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13601.141098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13601.393300] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13601.551807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13601.749445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13602.006885] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13602.497608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13602.501873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13602.660455] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13602.666349] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13602.809171] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13603.066388] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13603.326675] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13603.589521] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13603.849282] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13609.155785] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1505 [13609.294720] Lustre: *** cfs_fail_loc=1505, val=0*** [13609.296967] Lustre: Skipped 84 previous similar messages [13609.442260] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13609.759260] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13610.651858] Lustre: DEBUG MARKER: rc=0; [13610.651858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13610.651858] if [[ $? -eq 0 && $val -ne 0 ]]; then [13610.651858] echo $(hostname -s): $val; [13610.651858] rc=$val; [13610.651858] fi; [13610.651858] exit $rc [13611.245989] Lustre: DEBUG MARKER: dmesg [13611.788446] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 6a: LFSCK resumes from last checkpoint \(1\) ====================================== 14:19:42 \(1491488382\) [13611.944766] Lustre: DEBUG MARKER: == sanity-lfsck test 6a: LFSCK resumes from last checkpoint (1) ====================================== 14:19:42 (1491488382) [13612.611090] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1600 [13612.868428] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13613.000940] LustreError: 6260:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 sleeping for 1000ms [13613.003878] LustreError: 6260:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [13613.134478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13614.006036] LustreError: 6260:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1600 awake [13614.008569] LustreError: 6260:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [13616.393795] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001608 [13616.651476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13616.651476] awk '/^status/ { print $2 }' [13616.910555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13616.910555] awk '/^status/ { print $2 }' [13617.011051] Lustre: *** cfs_fail_loc=1608, val=0*** [13617.170172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13617.425377] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1600 [13617.680285] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13617.935373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13618.203468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13618.478406] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13618.734199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13618.734199] awk '/^status/ { print $2 }' [13618.990882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13618.990882] awk '/^status/ { print $2 }' [13619.291465] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13620.171727] Lustre: DEBUG MARKER: rc=0; [13620.171727] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13620.171727] if [[ $? -eq 0 && $val -ne 0 ]]; then [13620.171727] echo $(hostname -s): $val; [13620.171727] rc=$val; [13620.171727] fi; [13620.171727] exit $rc [13620.778916] Lustre: DEBUG MARKER: dmesg [13621.302781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 6b: LFSCK resumes from last checkpoint \(2\) ====================================== 14:19:52 \(1491488392\) [13621.460834] Lustre: DEBUG MARKER: == sanity-lfsck test 6b: LFSCK resumes from last checkpoint (2) ====================================== 14:19:52 (1491488392) [13622.130222] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [13622.384923] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13622.641662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13627.901786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001609 [13628.158754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13628.158754] awk '/^status/ { print $2 }' [13628.413929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13628.413929] awk '/^status/ { print $2 }' [13628.516057] Lustre: *** cfs_fail_loc=1609, val=0*** [13628.673173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13628.928756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13629.184922] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [13629.439466] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13629.569813] LustreError: 8207:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 1000ms [13629.572391] LustreError: 8207:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 10 previous similar messages [13629.700793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13629.956680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13630.212006] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13630.465780] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13630.574048] LustreError: 8207:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [13630.576218] LustreError: 8207:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 10 previous similar messages [13630.724724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13630.724724] awk '/^status/ { print $2 }' [13630.982612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13630.982612] awk '/^status/ { print $2 }' [13632.244482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13632.244482] awk '/^status/ { print $2 }' [13632.554618] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13633.448366] Lustre: DEBUG MARKER: rc=0; [13633.448366] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13633.448366] if [[ $? -eq 0 && $val -ne 0 ]]; then [13633.448366] echo $(hostname -s): $val; [13633.448366] rc=$val; [13633.448366] fi; [13633.448366] exit $rc [13634.037425] Lustre: DEBUG MARKER: dmesg [13634.552465] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount \(1\) ================ 14:20:05 \(1491488405\) [13634.708679] Lustre: DEBUG MARKER: == sanity-lfsck test 7a: non-stopped LFSCK should auto restarts after MDS remount (1) ================ 14:20:05 (1491488405) [13635.732786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 [13635.991834] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13636.249860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13639.516938] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13639.773514] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13640.348959] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13640.608226] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13640.862809] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13641.125760] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13641.380041] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13641.633205] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13641.790142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13641.996198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13642.255863] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13642.753045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13642.755886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13642.913778] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13642.915354] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13643.060915] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13643.315773] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13643.573213] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13643.832065] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13644.094261] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13644.353502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13644.353502] awk '/^status/ { print $2 }' [13644.610506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13644.610506] awk '/^status/ { print $2 }' [13645.869702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13645.869702] awk '/^status/ { print $2 }' [13647.131775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13647.131775] awk '/^status/ { print $2 }' [13647.465032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13648.360652] Lustre: DEBUG MARKER: rc=0; [13648.360652] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13648.360652] if [[ $? -eq 0 && $val -ne 0 ]]; then [13648.360652] echo $(hostname -s): $val; [13648.360652] rc=$val; [13648.360652] fi; [13648.360652] exit $rc [13648.943653] Lustre: DEBUG MARKER: dmesg [13649.524241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount \(2\) ================ 14:20:20 \(1491488420\) [13649.672727] Lustre: DEBUG MARKER: == sanity-lfsck test 7b: non-stopped LFSCK should auto restarts after MDS remount (2) ================ 14:20:20 (1491488420) [13650.983915] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13650.983915] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13650.983915] if [ $running -ne $mpts ]; then [13650.983915] echo $(hostname) env are INSANE!; [13650.983915] exit 1; [13650.983915] fi [13651.562330] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13651.562330] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13651.562330] if [ $running -ne $mpts ]; then [13651.562330] echo $(hostname) env are INSANE!; [13651.562330] exit 1; [13651.562330] fi [13652.161305] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13652.161305] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13652.161305] if [ $running -ne $mpts ]; then [13652.161305] echo $(hostname) env are INSANE!; [13652.161305] exit 1; [13652.161305] fi [13653.095256] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13653.366139] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13653.627936] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13653.884245] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13654.145920] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13655.692342] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13655.957862] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13656.215676] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13656.474208] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13656.738479] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13663.605523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13663.764860] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13665.186870] Lustre: DEBUG MARKER: lctl get_param -n timeout [13665.502394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13665.652177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13665.784332] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13671.595413] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [13671.729644] Lustre: *** cfs_fail_loc=1604, val=0*** [13671.731451] Lustre: Skipped 81 previous similar messages [13671.910311] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1602 [13672.167281] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [13672.424026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13672.424026] awk '/^status/ { print $2 }' [13672.680938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13672.680938] awk '/^status/ { print $2 }' [13673.189386] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13673.447232] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13673.604513] LustreError: 12866:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 1000ms [13673.606786] LustreError: 12866:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [13674.608032] LustreError: 12866:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [13674.610150] LustreError: 12866:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 5 previous similar messages [13675.846921] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13676.110635] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13676.372069] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13676.646438] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13676.901027] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13677.156746] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13677.305966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13677.512680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13677.770625] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13678.263948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13678.275715] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13678.428215] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13678.429787] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13678.575713] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13678.835138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13679.099880] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13679.363896] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13679.621443] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13679.879459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13679.879459] awk '/^status/ { print $2 }' [13680.138917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13680.138917] awk '/^status/ { print $2 }' [13681.397621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13681.397621] awk '/^status/ { print $2 }' [13682.655782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13682.655782] awk '/^status/ { print $2 }' [13682.980948] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13683.852819] Lustre: DEBUG MARKER: rc=0; [13683.852819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13683.852819] if [[ $? -eq 0 && $val -ne 0 ]]; then [13683.852819] echo $(hostname -s): $val; [13683.852819] rc=$val; [13683.852819] fi; [13683.852819] exit $rc [13684.445594] Lustre: DEBUG MARKER: dmesg [13685.000866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 8: LFSCK state machine ========================================================== 14:20:55 \(1491488455\) [13685.164904] Lustre: DEBUG MARKER: == sanity-lfsck test 8: LFSCK state machine ========================================================== 14:20:55 (1491488455) [13686.019988] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13686.288588] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13687.391070] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f35fc00 x1563924895771008/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13687.396504] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 9 previous similar messages [13688.651745] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13695.750362] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13696.005787] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [13699.357064] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491488463/real 1491488463] req@ffff88005f35fc00 x1563924895771120/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491488470 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13699.364743] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [13702.410091] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13736.723072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13736.723072] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13736.723072] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13737.032388] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13737.715681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13737.736890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13737.761954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13738.086856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13738.120418] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [13738.130130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13738.259825] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [13738.387047] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [13738.442720] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13738.504796] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [13738.779469] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13739.027036] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13739.304479] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat /dev/lvm-Role_MDS/P1 [13739.498284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13740.578286] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13740.826800] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13741.081718] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --reformat [13741.273353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13746.782834] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13746.782834] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13746.782834] if [ $running -ne $mpts ]; then [13746.782834] echo $(hostname) env are INSANE!; [13746.782834] exit 1; [13746.782834] fi [13747.337861] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13747.337861] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13747.337861] if [ $running -ne $mpts ]; then [13747.337861] echo $(hostname) env are INSANE!; [13747.337861] exit 1; [13747.337861] fi [13747.908418] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13747.908418] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13747.908418] if [ $running -ne $mpts ]; then [13747.908418] echo $(hostname) env are INSANE!; [13747.908418] exit 1; [13747.908418] fi [13748.843957] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13749.103573] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13749.353218] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13749.599288] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13749.753687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13749.791024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13749.822228] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [13749.824407] Lustre: Skipped 4 previous similar messages [13749.836686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [13749.852481] Lustre: lustre-MDT0000: new disk, initializing [13749.878710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13750.034780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13750.291095] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13750.796565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13750.796716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13750.955681] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13750.957245] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13751.102243] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13751.353428] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13751.604238] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13751.856113] Lustre: DEBUG MARKER: sync; sync; sync [13757.169069] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13757.419055] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13758.633898] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [13758.636368] Lustre: Skipped 4 previous similar messages [13759.601398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13759.758995] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13766.565441] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13766.829338] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13767.083602] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13767.333381] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13767.490572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13767.533860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13767.553368] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [13767.555837] Lustre: Skipped 1 previous similar message [13767.571352] Lustre: lustre-MDT0002: new disk, initializing [13767.594026] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [13767.753122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13768.008161] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13768.506690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13768.507597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13768.670205] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13768.672239] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13768.814364] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13769.066967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13769.320062] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13769.573942] Lustre: DEBUG MARKER: sync; sync; sync [13774.905448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [13775.159812] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13776.378492] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [13776.381233] Lustre: Skipped 9 previous similar messages [13777.345450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13777.509540] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13786.090844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13786.254556] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13794.446905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13794.604356] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13802.781086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13802.945960] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13811.109926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13811.273415] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13818.695759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13818.871671] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13820.316035] Lustre: DEBUG MARKER: lctl get_param -n timeout [13820.635893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13820.785796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13820.917355] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13826.890759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13827.146843] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [13827.278941] Lustre: *** cfs_fail_loc=1603, val=0*** [13827.406116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [13827.540119] Lustre: *** cfs_fail_loc=1604, val=0*** [13827.544071] Lustre: Skipped 19 previous similar messages [13827.960021] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x1601 [13828.213712] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13828.342359] LustreError: 20784:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 sleeping for 2000ms [13828.345473] LustreError: 20784:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [13828.475655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13828.732660] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [13830.348045] LustreError: 20784:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1601 awake [13830.350848] LustreError: 20784:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [13830.506240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13830.762629] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13831.017513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13831.273786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80001609 [13831.528685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13831.528685] awk '/^status/ { print $2 }' [13831.787211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13831.787211] awk '/^status/ { print $2 }' [13832.891075] Lustre: *** cfs_fail_loc=1609, val=0*** [13833.049085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13833.049085] awk '/^status/ { print $2 }' [13833.313938] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1600 [13833.570948] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13833.828601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13834.087054] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160a [13835.701036] Lustre: *** cfs_fail_loc=160a, val=0*** [13839.351922] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13839.608247] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13839.931612] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13840.192158] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160b [13840.445308] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13840.712667] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13840.967150] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13841.218504] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13841.375039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13841.393015] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13841.396922] LustreError: Skipped 14 previous similar messages [13841.399045] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d4698af to 0x61ac6f141d46e4ac [13841.402993] Lustre: Skipped 6 previous similar messages [13841.583200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13841.841141] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13842.338765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13842.339259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13842.500035] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13842.502981] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13842.646067] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13842.903267] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13843.156867] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13843.425745] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13843.680901] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13843.948828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13843.948828] awk '/^status/ { print $2 }' [13845.546644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13845.546644] awk '/^status/ { print $2 }' [13846.432426] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13846.437240] LustreError: Skipped 6 previous similar messages [13846.473508] Lustre: *** cfs_fail_loc=160b, val=0*** [13847.089397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13847.089397] awk '/^status/ { print $2 }' [13847.354831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13847.608886] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1601 [13847.866260] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13848.123892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13848.388468] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13848.650362] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13850.195770] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13850.459489] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160b [13850.713567] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13850.984742] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13851.240825] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13851.497953] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13851.657369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13851.674731] LustreError: 17310:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [13851.858283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13852.114236] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13852.612189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13852.616195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13852.774829] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13852.783331] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13852.925285] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13853.184664] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13853.443367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13853.706153] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13853.963554] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13854.219292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13854.219292] awk '/^status/ { print $2 }' [13855.487165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13855.487165] awk '/^status/ { print $2 }' [13856.727010] Lustre: *** cfs_fail_loc=160b, val=0*** [13856.751463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13856.751463] awk '/^status/ { print $2 }' [13857.014212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13857.278314] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13857.538983] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [13857.910314] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13858.182969] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13858.449704] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13858.708162] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13858.962558] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,skip_lfsck /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13859.120396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13859.314367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [13859.576890] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [13860.076136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13860.078508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13860.238868] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13860.239397] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13860.386160] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13860.642025] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13860.902368] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [13861.163898] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [13861.419727] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13861.679813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13861.679813] awk '/^status/ { print $2 }' [13862.945588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13862.945588] awk '/^status/ { print $2 }' [13864.209471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.recovery_status | [13864.209471] awk '/^status/ { print $2 }' [13864.469493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13864.727622] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x1602 [13864.984448] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace [13865.244696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13865.244696] awk '/^status/ { print $2 }' [13865.508058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13865.508058] awk '/^status/ { print $2 }' [13865.773428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13866.029034] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [13866.284021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13866.284021] awk '/^status/ { print $2 }' [13866.543138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13866.543138] awk '/^status/ { print $2 }' [13867.806658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13867.806658] awk '/^status/ { print $2 }' [13868.065697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13868.364890] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13869.261249] Lustre: DEBUG MARKER: rc=0; [13869.261249] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13869.261249] if [[ $? -eq 0 && $val -ne 0 ]]; then [13869.261249] echo $(hostname -s): $val; [13869.261249] rc=$val; [13869.261249] fi; [13869.261249] exit $rc [13869.867242] Lustre: DEBUG MARKER: dmesg [13870.413396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 9a: LFSCK speed control \(1\) ===================================================== 14:24:01 \(1491488641\) [13870.564794] Lustre: DEBUG MARKER: == sanity-lfsck test 9a: LFSCK speed control (1) ===================================================== 14:24:01 (1491488641) [13870.752902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_9a Need MDS version \>= 2.7.50 [13870.905973] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_9a Need MDS version >= 2.7.50 [13871.106067] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13872.004255] Lustre: DEBUG MARKER: rc=0; [13872.004255] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13872.004255] if [[ $? -eq 0 && $val -ne 0 ]]; then [13872.004255] echo $(hostname -s): $val; [13872.004255] rc=$val; [13872.004255] fi; [13872.004255] exit $rc [13872.598813] Lustre: DEBUG MARKER: dmesg [13873.141655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 9b: LFSCK speed control \(2\) ===================================================== 14:24:04 \(1491488644\) [13873.308536] Lustre: DEBUG MARKER: == sanity-lfsck test 9b: LFSCK speed control (2) ===================================================== 14:24:04 (1491488644) [13873.512997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_9b Need MDS version \>= 2.7.50 [13873.665408] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_9b Need MDS version >= 2.7.50 [13873.856086] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13874.752674] Lustre: DEBUG MARKER: rc=0; [13874.752674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13874.752674] if [[ $? -eq 0 && $val -ne 0 ]]; then [13874.752674] echo $(hostname -s): $val; [13874.752674] rc=$val; [13874.752674] fi; [13874.752674] exit $rc [13875.347935] Lustre: DEBUG MARKER: dmesg [13876.038714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 10: System is available during LFSCK scanning =================================== 14:24:07 \(1491488647\) [13876.201723] Lustre: DEBUG MARKER: == sanity-lfsck test 10: System is available during LFSCK scanning =================================== 14:24:07 (1491488647) [13877.511598] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13877.511598] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13877.511598] if [ $running -ne $mpts ]; then [13877.511598] echo $(hostname) env are INSANE!; [13877.511598] exit 1; [13877.511598] fi [13878.088932] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [13878.088932] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [13878.088932] if [ $running -ne $mpts ]; then [13878.088932] echo $(hostname) env are INSANE!; [13878.088932] exit 1; [13878.088932] fi [13878.675157] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [13878.675157] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [13878.675157] if [ $running -ne $mpts ]; then [13878.675157] echo $(hostname) env are INSANE!; [13878.675157] exit 1; [13878.675157] fi [13879.623419] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [13879.888923] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [13880.144964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [13880.399018] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [13880.661048] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13882.213298] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [13882.481592] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [13882.738906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [13882.995983] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [13883.260567] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13890.113188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13890.266537] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13891.694482] Lustre: DEBUG MARKER: lctl get_param -n timeout [13892.004939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13892.150688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13892.285194] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13898.481536] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1603 [13898.623656] Lustre: *** cfs_fail_loc=1603, val=0*** [13903.442467] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1604 [13903.574509] Lustre: *** cfs_fail_loc=1604, val=0*** [13903.576347] Lustre: Skipped 4 previous similar messages [13908.506587] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [13909.030573] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -s 100 [13919.304360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13924.646076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [13924.907522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0000.lfsck_speed_limit 0 [13925.177299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13925.177299] awk '/^status/ { print $2 }' [13925.481872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13925.481872] awk '/^status/ { print $2 }' [13926.771648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13926.771648] awk '/^status/ { print $2 }' [13928.065895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13928.065895] awk '/^status/ { print $2 }' [13929.382688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13929.382688] awk '/^status/ { print $2 }' [13930.669243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13930.669243] awk '/^status/ { print $2 }' [13931.953651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [13931.953651] awk '/^status/ { print $2 }' [13932.271862] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13933.166039] Lustre: DEBUG MARKER: rc=0; [13933.166039] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13933.166039] if [[ $? -eq 0 && $val -ne 0 ]]; then [13933.166039] echo $(hostname -s): $val; [13933.166039] rc=$val; [13933.166039] fi; [13933.166039] exit $rc [13933.766209] Lustre: DEBUG MARKER: dmesg [13934.301663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ============================================= 14:25:05 \(1491488705\) [13934.455966] Lustre: DEBUG MARKER: == sanity-lfsck test 11a: LFSCK can rebuild lost last_id ============================================= 14:25:05 (1491488705) [13948.106215] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [13948.367994] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [13949.343084] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005eb45800 x1563924895937728/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [13949.347935] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [13955.021546] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [13959.164700] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [13959.169250] Lustre: Skipped 37 previous similar messages [13962.187275] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [13962.440212] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [13968.841283] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14045.198139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14045.356816] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14050.840730] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14051.739345] Lustre: DEBUG MARKER: rc=0; [14051.739345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14051.739345] if [[ $? -eq 0 && $val -ne 0 ]]; then [14051.739345] echo $(hostname -s): $val; [14051.739345] rc=$val; [14051.739345] fi; [14051.739345] exit $rc [14052.320543] Lustre: DEBUG MARKER: dmesg [14052.882234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================== 14:27:03 \(1491488823\) [14053.035815] Lustre: DEBUG MARKER: == sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================== 14:27:03 (1491488823) [14054.324186] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14054.324186] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14054.324186] if [ $running -ne $mpts ]; then [14054.324186] echo $(hostname) env are INSANE!; [14054.324186] exit 1; [14054.324186] fi [14054.885831] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14054.885831] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14054.885831] if [ $running -ne $mpts ]; then [14054.885831] echo $(hostname) env are INSANE!; [14054.885831] exit 1; [14054.885831] fi [14055.467771] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14055.467771] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14055.467771] if [ $running -ne $mpts ]; then [14055.467771] echo $(hostname) env are INSANE!; [14055.467771] exit 1; [14055.467771] fi [14056.433776] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14056.693676] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14056.944942] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14057.188410] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14057.344034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14057.561642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14057.815041] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14058.312486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14058.313256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14058.470769] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14058.470783] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14058.614937] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14058.871687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14059.124775] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14059.395384] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14059.654663] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14061.777144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14061.941663] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14063.370550] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14063.641271] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14063.897820] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14064.151268] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14064.308540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14064.503748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14064.766259] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14065.262478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14065.263283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14065.422272] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14065.435943] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14065.576266] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14065.838323] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14066.098477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14066.357556] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14066.616288] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14068.737088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14068.903532] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14079.281118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14079.443456] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14082.272276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14082.436091] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14085.232836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14085.412148] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14087.486513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14087.649576] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14089.080735] Lustre: DEBUG MARKER: lctl get_param -n timeout [14089.398611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14089.552361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14089.690847] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14095.306368] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [14095.565581] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [14095.889834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_seq [14097.415193] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14097.673253] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [14098.005155] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14098.265521] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14098.519129] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14098.779648] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14099.032475] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14099.289275] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14099.447665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14099.649306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14099.923327] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14100.448609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14100.450711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14100.607317] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14100.609314] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14100.759535] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14101.023624] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14101.284933] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14101.547515] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14101.604378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [14101.607293] Lustre: Skipped 14 previous similar messages [14101.806945] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14104.197111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14104.362100] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14104.506601] Lustre: lustre-MDT0000: Recovery over after 0:03, of 4 clients 4 recovered and 0 were evicted. [14104.509506] Lustre: Skipped 14 previous similar messages [14112.917432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14113.075465] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14114.015803] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [14114.019667] LustreError: Skipped 49 previous similar messages [14116.605540] LustreError: 4343:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 [14116.610524] LustreError: 4343:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [14117.618636] LustreError: 4343:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 [14117.974658] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14118.234433] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14119.551067] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006472e400 x1563924895947904/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14119.557055] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [14120.431237] Lustre: server umount lustre-MDT0000 complete [14120.433896] Lustre: Skipped 15 previous similar messages [14120.562225] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14121.977738] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14121.983550] LustreError: Skipped 105 previous similar messages [14127.665427] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14127.920721] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14134.293239] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14150.726137] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14151.684653] Lustre: DEBUG MARKER: rc=0; [14151.684653] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14151.684653] if [[ $? -eq 0 && $val -ne 0 ]]; then [14151.684653] echo $(hostname -s): $val; [14151.684653] rc=$val; [14151.684653] fi; [14151.684653] exit $rc [14152.274026] Lustre: DEBUG MARKER: dmesg [14152.841205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ============================= 14:28:43 \(1491488923\) [14152.994066] Lustre: DEBUG MARKER: == sanity-lfsck test 12a: single command to trigger LFSCK on all devices ============================= 14:28:43 (1491488923) [14154.295823] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14154.295823] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14154.295823] if [ $running -ne $mpts ]; then [14154.295823] echo $(hostname) env are INSANE!; [14154.295823] exit 1; [14154.295823] fi [14154.868244] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14154.868244] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14154.868244] if [ $running -ne $mpts ]; then [14154.868244] echo $(hostname) env are INSANE!; [14154.868244] exit 1; [14154.868244] fi [14155.445417] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14155.445417] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14155.445417] if [ $running -ne $mpts ]; then [14155.445417] echo $(hostname) env are INSANE!; [14155.445417] exit 1; [14155.445417] fi [14156.392136] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14156.652576] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14156.903811] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14157.150495] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14157.305163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14157.322014] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [14157.324121] Lustre: Skipped 263 previous similar messages [14157.375351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14157.377499] Lustre: Skipped 13 previous similar messages [14157.528849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14157.786041] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14159.483329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14159.484699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14159.646032] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14159.646380] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14159.792660] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14160.051211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14160.303846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14160.562801] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14160.818160] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14162.951518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14163.108999] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14164.521945] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14164.785642] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14165.044282] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14165.297931] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14165.458234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14165.673661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14165.929763] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14166.420467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14166.424735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14166.583830] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14166.583830] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14166.730212] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14166.984903] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14167.239294] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14167.498197] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14167.753607] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14169.911259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14170.074126] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14173.337067] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14173.497741] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14176.239178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14176.399249] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14179.187137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14179.342155] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14182.151670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14182.314051] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14184.387044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14184.548524] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14185.981028] Lustre: DEBUG MARKER: lctl get_param -n timeout [14186.299754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14186.458048] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14186.615730] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14192.862381] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -s 1 -r [14193.140737] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 | [14193.140737] awk '/^namespace_mdts_scanning-phase1/ { print $2 }' [14193.400655] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 | [14193.400655] awk '/^namespace_mdts_scanning-phase1/ { print $2 }' [14193.660020] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 -A [14193.938377] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14193.938377] awk '/^namespace_mdts_stopped/ { print $2 }' [14194.198216] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -s 0 -r [14194.459804] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14194.459804] awk '/^namespace_mdts_completed/ { print $2 }' [14194.719082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [14195.024966] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [14195.362692] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [14195.655044] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -A -s 1 -r [14195.930107] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 | [14195.930107] awk '/^layout_mdts_scanning-phase1/ { print $2 }' [14196.193220] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 | [14196.193220] awk '/^layout_mdts_scanning-phase1/ { print $2 }' [14196.454838] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 -A [14196.729650] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [14196.729650] awk '/^layout_mdts_stopped/ { print $2 }' [14198.020785] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -A -s 0 -r [14198.349952] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [14198.349952] awk '/^layout_mdts_completed/ { print $2 }' [14228.683537] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [14229.270432] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [14229.609619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14230.494744] Lustre: DEBUG MARKER: rc=0; [14230.494744] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14230.494744] if [[ $? -eq 0 && $val -ne 0 ]]; then [14230.494744] echo $(hostname -s): $val; [14230.494744] rc=$val; [14230.494744] fi; [14230.494744] exit $rc [14231.111678] Lustre: DEBUG MARKER: dmesg [14231.661320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 12b: auto detect Lustre device ================================================== 14:30:02 \(1491489002\) [14231.821812] Lustre: DEBUG MARKER: == sanity-lfsck test 12b: auto detect Lustre device ================================================== 14:30:02 (1491489002) [14232.415587] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -A -r [14232.683809] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14232.683809] awk '/^namespace_mdts_completed/ { print $2 }' [14238.947469] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [14238.947469] awk '/^layout_mdts_completed/ { print $2 }' [14239.209087] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [14239.466209] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -t layout -r [14239.768318] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14240.651553] Lustre: DEBUG MARKER: rc=0; [14240.651553] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14240.651553] if [[ $? -eq 0 && $val -ne 0 ]]; then [14240.651553] echo $(hostname -s): $val; [14240.651553] rc=$val; [14240.651553] fi; [14240.651553] exit $rc [14241.247542] Lustre: DEBUG MARKER: dmesg [14241.773886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ============================================= 14:30:12 \(1491489012\) [14241.921764] Lustre: DEBUG MARKER: == sanity-lfsck test 13: LFSCK can repair crashed lmm_oi ============================================= 14:30:12 (1491489012) [14242.081837] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x160f [14242.215176] Lustre: *** cfs_fail_loc=160f, val=0*** [14242.346315] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14242.602154] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14242.882753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14242.882753] awk '/^status/ { print $2 }' [14243.152337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14243.152337] awk '/^status/ { print $2 }' [14243.410943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14243.715069] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14244.603200] Lustre: DEBUG MARKER: rc=0; [14244.603200] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14244.603200] if [[ $? -eq 0 && $val -ne 0 ]]; then [14244.603200] echo $(hostname -s): $val; [14244.603200] rc=$val; [14244.603200] fi; [14244.603200] exit $rc [14245.223420] Lustre: DEBUG MARKER: dmesg [14245.862926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference \(1\) ============= 14:30:16 \(1491489016\) [14246.020993] Lustre: DEBUG MARKER: == sanity-lfsck test 14a: LFSCK can repair MDT-object with dangling LOV EA reference (1) ============= 14:30:16 (1491489016) [14246.461694] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [14246.717028] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [14247.362136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [14247.663407] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [14247.988177] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [14248.269621] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [14248.538634] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [14249.004689] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14249.298446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14249.298446] awk '/^status/ { print $2 }' [14249.553071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14249.553071] awk '/^status/ { print $2 }' [14249.808778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14250.068304] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -c [14250.399109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14250.399109] awk '/^status/ { print $2 }' [14250.661142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14250.661142] awk '/^status/ { print $2 }' [14250.974732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14251.278766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [14251.851262] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [14252.804424] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14253.059722] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14253.342191] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [14253.344434] Lustre: Skipped 47 previous similar messages [14259.396322] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14266.548695] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14266.809955] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14273.287219] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14290.857979] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14290.857979] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14290.857979] if [ $running -ne $mpts ]; then [14290.857979] echo $(hostname) env are INSANE!; [14290.857979] exit 1; [14290.857979] fi [14291.412963] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14291.412963] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14291.412963] if [ $running -ne $mpts ]; then [14291.412963] echo $(hostname) env are INSANE!; [14291.412963] exit 1; [14291.412963] fi [14291.991126] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14291.991126] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14291.991126] if [ $running -ne $mpts ]; then [14291.991126] echo $(hostname) env are INSANE!; [14291.991126] exit 1; [14291.991126] fi [14292.913412] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14293.168872] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14293.418310] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14293.666141] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14293.820689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14294.041861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14294.295457] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14294.788575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14294.791301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14294.946387] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14294.950601] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14295.090952] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14295.353775] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14295.605669] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14295.863075] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14296.117729] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14298.231758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14298.394348] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14299.806877] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14300.073438] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14300.341325] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14300.610248] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14300.762538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14300.960978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14301.218797] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14301.714615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14301.714617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14301.877431] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14301.878416] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14302.026846] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14302.281848] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14302.536132] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14302.800371] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14303.057107] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14305.197892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14305.357086] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14305.809133] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491489071/real 1491489071] req@ffff880063f4aa00 x1563924895977312/t0(0) o8->lustre-OST0003-osc-MDT0002@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491489076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [14305.817898] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [14308.643267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14308.806037] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14311.594190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14311.768275] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14314.596602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14314.755681] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14317.573968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14317.758616] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14319.875451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14320.056606] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14321.502476] Lustre: DEBUG MARKER: lctl get_param -n timeout [14321.815493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14321.976239] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14322.107753] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14327.434914] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14328.316823] Lustre: DEBUG MARKER: rc=0; [14328.316823] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14328.316823] if [[ $? -eq 0 && $val -ne 0 ]]; then [14328.316823] echo $(hostname -s): $val; [14328.316823] rc=$val; [14328.316823] fi; [14328.316823] exit $rc [14328.924836] Lustre: DEBUG MARKER: dmesg [14329.488612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference \(2\) ============= 14:31:40 \(1491489100\) [14329.644040] Lustre: DEBUG MARKER: == sanity-lfsck test 14b: LFSCK can repair MDT-object with dangling LOV EA reference (2) ============= 14:31:40 (1491489100) [14330.153500] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [14330.412452] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [14330.994802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [14331.290229] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [14331.616733] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [14331.901832] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [14332.157845] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [14332.631772] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -d [14332.964846] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [14332.964846] awk '/^layout_mdts_completed/ { print $2 }' [14363.252593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14363.508618] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c -d [14363.828680] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t layout -M lustre-MDT0000 -w | [14363.828680] awk '/^layout_mdts_completed/ { print $2 }' [14367.145026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14367.448670] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [14368.053469] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [14369.015407] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14369.286608] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14369.611402] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14376.730609] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14376.985789] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14380.911068] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004a113600 x1563924895992832/t0(0) o13->lustre-OST0003-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14380.915892] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [14383.361889] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14400.831006] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14400.831006] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14400.831006] if [ $running -ne $mpts ]; then [14400.831006] echo $(hostname) env are INSANE!; [14400.831006] exit 1; [14400.831006] fi [14401.384556] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14401.384556] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14401.384556] if [ $running -ne $mpts ]; then [14401.384556] echo $(hostname) env are INSANE!; [14401.384556] exit 1; [14401.384556] fi [14401.964548] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14401.964548] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14401.964548] if [ $running -ne $mpts ]; then [14401.964548] echo $(hostname) env are INSANE!; [14401.964548] exit 1; [14401.964548] fi [14402.911605] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14403.168927] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14403.415106] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14403.660420] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14403.804511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14404.012581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14404.271362] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14404.762602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14404.768140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14404.920685] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14404.922137] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14405.065519] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14405.321300] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14405.574759] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14405.834472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14406.088195] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14408.211562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14408.367692] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14409.805185] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14410.074031] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14410.337053] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14410.604578] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14410.767272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14410.964530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14411.222624] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14411.722210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14411.728808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14411.884109] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14411.885389] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14412.030604] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14412.287636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14412.555678] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14412.816246] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14413.074932] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14415.210928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14415.374227] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14418.642550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14418.812893] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14421.601800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14421.764663] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14424.593191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14424.766769] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14427.576555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14427.729663] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14429.819537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14430.063780] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14431.527851] Lustre: DEBUG MARKER: lctl get_param -n timeout [14431.836791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14431.991746] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14432.127811] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14437.468316] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14438.383822] Lustre: DEBUG MARKER: rc=0; [14438.383822] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14438.383822] if [[ $? -eq 0 && $val -ne 0 ]]; then [14438.383822] echo $(hostname -s): $val; [14438.383822] rc=$val; [14438.383822] fi; [14438.383822] exit $rc [14438.979731] Lustre: DEBUG MARKER: dmesg [14439.543925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object\/OST-object pairs \(1\) ================= 14:33:30 \(1491489210\) [14439.699597] Lustre: DEBUG MARKER: == sanity-lfsck test 15a: LFSCK can repair unmatched MDT-object/OST-object pairs (1) ================= 14:33:30 (1491489210) [14440.654599] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14440.920872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14440.920872] awk '/^status/ { print $2 }' [14441.180990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14441.180990] awk '/^status/ { print $2 }' [14442.439400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14442.439400] awk '/^status/ { print $2 }' [14443.696766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14443.696766] awk '/^status/ { print $2 }' [14444.951958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14444.951958] awk '/^status/ { print $2 }' [14446.212862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14446.212862] awk '/^status/ { print $2 }' [14446.474804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14446.777686] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14447.673821] Lustre: DEBUG MARKER: rc=0; [14447.673821] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14447.673821] if [[ $? -eq 0 && $val -ne 0 ]]; then [14447.673821] echo $(hostname -s): $val; [14447.673821] rc=$val; [14447.673821] fi; [14447.673821] exit $rc [14448.274691] Lustre: DEBUG MARKER: dmesg [14448.838160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object\/OST-object pairs \(2\) ================= 14:33:39 \(1491489219\) [14449.005179] Lustre: DEBUG MARKER: == sanity-lfsck test 15b: LFSCK can repair unmatched MDT-object/OST-object pairs (2) ================= 14:33:39 (1491489219) [14449.917646] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14450.181399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14450.181399] awk '/^status/ { print $2 }' [14450.437813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14450.437813] awk '/^status/ { print $2 }' [14451.693523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14451.693523] awk '/^status/ { print $2 }' [14451.952459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14452.268264] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14453.170627] Lustre: DEBUG MARKER: rc=0; [14453.170627] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14453.170627] if [[ $? -eq 0 && $val -ne 0 ]]; then [14453.170627] echo $(hostname -s): $val; [14453.170627] rc=$val; [14453.170627] fi; [14453.170627] exit $rc [14453.767366] Lustre: DEBUG MARKER: dmesg [14454.325293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object\/OST-object pairs \(3\) ================= 14:33:45 \(1491489225\) [14454.483333] Lustre: DEBUG MARKER: == sanity-lfsck test 15c: LFSCK can repair unmatched MDT-object/OST-object pairs (3) ================= 14:33:45 (1491489225) [14454.634029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14454.634029] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14454.634029] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14454.949624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-lfsck test_15c Skip the test after 2.7.55 see LU-6437 [14455.104462] Lustre: DEBUG MARKER: SKIP: sanity-lfsck test_15c Skip the test after 2.7.55 see LU-6437 [14455.293324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14456.174902] Lustre: DEBUG MARKER: rc=0; [14456.174902] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14456.174902] if [[ $? -eq 0 && $val -ne 0 ]]; then [14456.174902] echo $(hostname -s): $val; [14456.174902] rc=$val; [14456.174902] fi; [14456.174902] exit $rc [14456.769699] Lustre: DEBUG MARKER: dmesg [14457.323481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object\/OST-object owner =================== 14:33:48 \(1491489228\) [14457.473260] Lustre: DEBUG MARKER: == sanity-lfsck test 16: LFSCK can repair inconsistent MDT-object/OST-object owner =================== 14:33:48 (1491489228) [14457.682811] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1613 [14457.822189] Lustre: *** cfs_fail_loc=1613, val=0*** [14457.949819] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14458.204135] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14458.471109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14458.471109] awk '/^status/ { print $2 }' [14458.730857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14458.730857] awk '/^status/ { print $2 }' [14459.988784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14459.988784] awk '/^status/ { print $2 }' [14461.248174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14461.248174] awk '/^status/ { print $2 }' [14462.510206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14462.510206] awk '/^status/ { print $2 }' [14463.767694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14463.767694] awk '/^status/ { print $2 }' [14464.027509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14464.328540] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14465.422269] Lustre: DEBUG MARKER: rc=0; [14465.422269] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14465.422269] if [[ $? -eq 0 && $val -ne 0 ]]; then [14465.422269] echo $(hostname -s): $val; [14465.422269] rc=$val; [14465.422269] fi; [14465.422269] exit $rc [14466.546785] Lustre: DEBUG MARKER: dmesg [14467.105947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 17: LFSCK can repair multiple references ======================================== 14:33:58 \(1491489238\) [14467.255300] Lustre: DEBUG MARKER: == sanity-lfsck test 17: LFSCK can repair multiple references ======================================== 14:33:58 (1491489238) [14467.423634] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0x1614 [14467.559390] Lustre: *** cfs_fail_loc=1614, val=0*** [14467.561379] Lustre: Skipped 2 previous similar messages [14467.770332] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14468.035674] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14468.302075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14468.302075] awk '/^status/ { print $2 }' [14468.559402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14468.559402] awk '/^status/ { print $2 }' [14468.819253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14469.235080] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14470.108234] Lustre: DEBUG MARKER: rc=0; [14470.108234] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14470.108234] if [[ $? -eq 0 && $val -ne 0 ]]; then [14470.108234] echo $(hostname -s): $val; [14470.108234] rc=$val; [14470.108234] fi; [14470.108234] exit $rc [14470.707746] Lustre: DEBUG MARKER: dmesg [14471.244874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18a: Find out orphan OST-object and repair it \(1\) =============================== 14:34:02 \(1491489242\) [14471.393101] Lustre: DEBUG MARKER: == sanity-lfsck test 18a: Find out orphan OST-object and repair it (1) =============================== 14:34:02 (1491489242) [14471.838126] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1615 [14471.972280] Lustre: *** cfs_fail_loc=1615, val=0*** [14471.974245] Lustre: Skipped 1 previous similar message [14474.418537] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14474.969498] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14475.295833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14475.295833] awk '/^status/ { print $2 }' [14475.570208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14475.570208] awk '/^status/ { print $2 }' [14476.901259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14476.901259] awk '/^status/ { print $2 }' [14477.682246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14477.682246] awk '/^status/ { print $2 }' [14477.957388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14477.957388] awk '/^status/ { print $2 }' [14479.764583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14480.365699] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14481.270256] Lustre: DEBUG MARKER: rc=0; [14481.270256] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14481.270256] if [[ $? -eq 0 && $val -ne 0 ]]; then [14481.270256] echo $(hostname -s): $val; [14481.270256] rc=$val; [14481.270256] fi; [14481.270256] exit $rc [14481.867435] Lustre: DEBUG MARKER: dmesg [14482.394849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18b: Find out orphan OST-object and repair it \(2\) =============================== 14:34:13 \(1491489253\) [14482.555967] Lustre: DEBUG MARKER: == sanity-lfsck test 18b: Find out orphan OST-object and repair it (2) =============================== 14:34:13 (1491489253) [14483.122741] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [14483.258158] Lustre: *** cfs_fail_loc=1616, val=0*** [14483.260094] Lustre: Skipped 3 previous similar messages [14485.701065] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14486.261876] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14486.570865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14486.570865] awk '/^status/ { print $2 }' [14486.864118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14486.864118] awk '/^status/ { print $2 }' [14488.916085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14488.916085] awk '/^status/ { print $2 }' [14489.175942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14489.175942] awk '/^status/ { print $2 }' [14490.989514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14491.618608] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14492.502898] Lustre: DEBUG MARKER: rc=0; [14492.502898] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14492.502898] if [[ $? -eq 0 && $val -ne 0 ]]; then [14492.502898] echo $(hostname -s): $val; [14492.502898] rc=$val; [14492.502898] fi; [14492.502898] exit $rc [14493.089861] Lustre: DEBUG MARKER: dmesg [14493.643524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18c: Find out orphan OST-object and repair it \(3\) =============================== 14:34:24 \(1491489264\) [14493.803233] Lustre: DEBUG MARKER: == sanity-lfsck test 18c: Find out orphan OST-object and repair it (3) =============================== 14:34:24 (1491489264) [14494.943333] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [14495.087683] Lustre: *** cfs_fail_loc=1616, val=0*** [14495.089820] Lustre: Skipped 3 previous similar messages [14497.539879] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14498.067309] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14498.426826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14498.426826] awk '/^status/ { print $2 }' [14498.702241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14498.702241] awk '/^status/ { print $2 }' [14500.732422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14500.732422] awk '/^status/ { print $2 }' [14500.991885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14500.991885] awk '/^status/ { print $2 }' [14502.808712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14503.394841] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14504.279522] Lustre: DEBUG MARKER: rc=0; [14504.279522] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14504.279522] if [[ $? -eq 0 && $val -ne 0 ]]; then [14504.279522] echo $(hostname -s): $val; [14504.279522] rc=$val; [14504.279522] fi; [14504.279522] exit $rc [14504.876678] Lustre: DEBUG MARKER: dmesg [14505.416332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18d: Find out orphan OST-object and repair it \(4\) =============================== 14:34:36 \(1491489276\) [14505.569267] Lustre: DEBUG MARKER: == sanity-lfsck test 18d: Find out orphan OST-object and repair it (4) =============================== 14:34:36 (1491489276) [14505.822474] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1618 [14505.964254] Lustre: *** cfs_fail_loc=1618, val=0*** [14505.966278] Lustre: Skipped 3 previous similar messages [14508.176697] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14509.103614] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14509.361808] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14511.408118] LustreError: 20239:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491489282 with bad export cookie 7038122449855266897 [14511.412280] LustreError: 20239:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 4 previous similar messages [14511.704363] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14518.838973] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14519.094340] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14522.975067] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14522.979200] LustreError: Skipped 7 previous similar messages [14525.506621] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14542.968568] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14542.968568] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14542.968568] if [ $running -ne $mpts ]; then [14542.968568] echo $(hostname) env are INSANE!; [14542.968568] exit 1; [14542.968568] fi [14543.524330] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14543.524330] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14543.524330] if [ $running -ne $mpts ]; then [14543.524330] echo $(hostname) env are INSANE!; [14543.524330] exit 1; [14543.524330] fi [14544.092842] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14544.092842] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14544.092842] if [ $running -ne $mpts ]; then [14544.092842] echo $(hostname) env are INSANE!; [14544.092842] exit 1; [14544.092842] fi [14545.020041] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14545.282758] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14545.534194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14545.781524] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14545.938352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14546.160653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14546.415474] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14546.913144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14546.913152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14547.073328] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14547.075141] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14547.217207] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14547.470199] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14547.721725] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14547.979782] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14548.230908] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14550.354985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14550.510567] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14551.923389] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14552.188730] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14552.445881] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14552.700547] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14552.859165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14553.053384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14553.309625] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14553.801728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14553.802861] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14553.963783] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14553.967442] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14554.112282] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14554.373484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14554.633805] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14554.892283] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14555.148395] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14557.283688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14557.447066] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14560.699953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14560.867941] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14563.636627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14563.800667] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14566.563670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14566.730296] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14569.583355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14569.738490] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14571.822787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14571.982778] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14573.419949] Lustre: DEBUG MARKER: lctl get_param -n timeout [14573.733827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14573.886735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14574.020030] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14579.304766] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c -d [14579.623629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14579.623629] awk '/^status/ { print $2 }' [14579.898449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14579.898449] awk '/^status/ { print $2 }' [14580.675095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14580.675095] awk '/^status/ { print $2 }' [14580.935565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14580.935565] awk '/^status/ { print $2 }' [14582.194411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14582.194411] awk '/^status/ { print $2 }' [14583.455312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14583.455312] awk '/^status/ { print $2 }' [14584.715732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14584.715732] awk '/^status/ { print $2 }' [14585.974138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14585.974138] awk '/^status/ { print $2 }' [14587.233333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14587.233333] awk '/^status/ { print $2 }' [14588.494481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14588.494481] awk '/^status/ { print $2 }' [14589.760551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14589.760551] awk '/^status/ { print $2 }' [14591.023792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14591.023792] awk '/^status/ { print $2 }' [14592.283236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14592.283236] awk '/^status/ { print $2 }' [14593.543621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14593.543621] awk '/^status/ { print $2 }' [14594.802468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14594.802468] awk '/^status/ { print $2 }' [14596.062575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14596.062575] awk '/^status/ { print $2 }' [14597.324546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14597.324546] awk '/^status/ { print $2 }' [14598.584782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14598.584782] awk '/^status/ { print $2 }' [14599.841774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14599.841774] awk '/^status/ { print $2 }' [14601.102359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14601.102359] awk '/^status/ { print $2 }' [14602.360669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14602.360669] awk '/^status/ { print $2 }' [14603.620827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14603.620827] awk '/^status/ { print $2 }' [14604.886062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14604.886062] awk '/^status/ { print $2 }' [14606.146140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14606.146140] awk '/^status/ { print $2 }' [14607.404042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14607.404042] awk '/^status/ { print $2 }' [14608.671355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14608.671355] awk '/^status/ { print $2 }' [14609.936770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14609.936770] awk '/^status/ { print $2 }' [14611.743023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14612.002304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14612.349600] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14613.269797] Lustre: DEBUG MARKER: rc=0; [14613.269797] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14613.269797] if [[ $? -eq 0 && $val -ne 0 ]]; then [14613.269797] echo $(hostname -s): $val; [14613.269797] rc=$val; [14613.269797] fi; [14613.269797] exit $rc [14613.879947] Lustre: DEBUG MARKER: dmesg [14614.448247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18e: Find out orphan OST-object and repair it \(5\) =============================== 14:36:25 \(1491489385\) [14614.595994] Lustre: DEBUG MARKER: == sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) =============================== 14:36:25 (1491489385) [14614.871299] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1618 [14615.014439] Lustre: *** cfs_fail_loc=1618, val=0*** [14615.016601] Lustre: Skipped 3 previous similar messages [14617.293557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14618.214180] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14618.477324] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14618.861158] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14618.865794] Lustre: Skipped 19 previous similar messages [14624.801243] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14631.919284] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14632.171394] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14638.532072] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14655.975315] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14655.975315] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14655.975315] if [ $running -ne $mpts ]; then [14655.975315] echo $(hostname) env are INSANE!; [14655.975315] exit 1; [14655.975315] fi [14656.527300] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14656.527300] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14656.527300] if [ $running -ne $mpts ]; then [14656.527300] echo $(hostname) env are INSANE!; [14656.527300] exit 1; [14656.527300] fi [14657.100202] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14657.100202] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14657.100202] if [ $running -ne $mpts ]; then [14657.100202] echo $(hostname) env are INSANE!; [14657.100202] exit 1; [14657.100202] fi [14658.046443] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14658.311735] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14658.573781] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14658.836148] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14658.983214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14659.206268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14659.465321] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14659.955324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14659.956167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14660.110901] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14660.114118] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14660.254395] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14660.508891] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14660.763215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14661.021017] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14661.272974] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14663.386648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14663.558143] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14664.969888] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [14665.240714] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [14665.494808] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [14665.748902] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [14665.908415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14666.102227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14666.358915] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14666.853098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14666.861137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14667.014460] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14667.016856] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14667.160118] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14667.415869] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14667.672094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14667.932779] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [14668.195574] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14670.340022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14670.516128] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14673.800565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14673.949386] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14676.740714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14676.902625] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14679.729618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14679.889202] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14682.696643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14682.861295] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14684.963626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14685.130848] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14686.566267] Lustre: DEBUG MARKER: lctl get_param -n timeout [14686.881790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [14687.036725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [14687.167340] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [14692.444795] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x1602 [14692.708689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [14693.026229] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [14693.378590] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [14693.663025] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c [14693.971316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14693.971316] awk '/^status/ { print $2 }' [14694.175356] LustreError: 8550:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 10000ms [14694.178402] LustreError: 8550:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [14694.278030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14694.278030] awk '/^status/ { print $2 }' [14696.559304] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [14696.819928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14696.819928] awk '/^status/ { print $2 }' [14697.077647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14697.077647] awk '/^status/ { print $2 }' [14698.341335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14698.341335] awk '/^status/ { print $2 }' [14699.601405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14699.601405] awk '/^status/ { print $2 }' [14700.864664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14700.864664] awk '/^status/ { print $2 }' [14702.132735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14702.132735] awk '/^status/ { print $2 }' [14703.391709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14703.391709] awk '/^status/ { print $2 }' [14704.181046] LustreError: 8550:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [14704.183814] LustreError: 8550:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [14704.650214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14704.650214] awk '/^status/ { print $2 }' [14705.422648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14705.422648] awk '/^status/ { print $2 }' [14705.682105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14705.682105] awk '/^status/ { print $2 }' [14707.558573] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [14708.132955] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [14708.428820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14708.806112] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14709.730349] Lustre: DEBUG MARKER: rc=0; [14709.730349] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14709.730349] if [[ $? -eq 0 && $val -ne 0 ]]; then [14709.730349] echo $(hostname -s): $val; [14709.730349] rc=$val; [14709.730349] fi; [14709.730349] exit $rc [14710.333560] Lustre: DEBUG MARKER: dmesg [14710.886755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18f: Skip the failed OST\(s\) when handle orphan OST-objects ====================== 14:38:01 \(1491489481\) [14711.059623] Lustre: DEBUG MARKER: == sanity-lfsck test 18f: Skip the failed OST(s) when handle orphan OST-objects ====================== 14:38:01 (1491489481) [14711.819199] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [14711.964093] Lustre: *** cfs_fail_loc=1616, val=0*** [14711.967352] Lustre: Skipped 3 previous similar messages [14714.389588] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14714.920763] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161c fail_val=0 [14715.175062] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14715.312391] Lustre: *** cfs_fail_loc=161c, val=0*** [14715.471727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14715.471727] awk '/^status/ { print $2 }' [14715.751411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14715.751411] awk '/^status/ { print $2 }' [14716.522034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14716.522034] awk '/^status/ { print $2 }' [14716.777513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14716.777513] awk '/^status/ { print $2 }' [14718.577941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14718.837411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14719.353071] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14719.650025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14719.650025] awk '/^status/ { print $2 }' [14719.952267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14719.952267] awk '/^status/ { print $2 }' [14720.721458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14720.721458] awk '/^status/ { print $2 }' [14720.979564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14720.979564] awk '/^status/ { print $2 }' [14722.789571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14723.355545] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14724.268576] Lustre: DEBUG MARKER: rc=0; [14724.268576] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14724.268576] if [[ $? -eq 0 && $val -ne 0 ]]; then [14724.268576] echo $(hostname -s): $val; [14724.268576] rc=$val; [14724.268576] fi; [14724.268576] exit $rc [14724.866452] Lustre: DEBUG MARKER: dmesg [14725.453267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18g: Find out orphan OST-object and repair it \(7\) =============================== 14:38:16 \(1491489496\) [14725.629827] Lustre: DEBUG MARKER: == sanity-lfsck test 18g: Find out orphan OST-object and repair it (7) =============================== 14:38:16 (1491489496) [14725.999320] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162e [14726.131683] Lustre: *** cfs_fail_loc=162e, val=0*** [14726.133711] Lustre: Skipped 1 previous similar message [14726.260332] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14726.525445] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14726.842274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14726.842274] awk '/^status/ { print $2 }' [14727.116924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14727.116924] awk '/^status/ { print $2 }' [14728.387311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14728.387311] awk '/^status/ { print $2 }' [14729.645555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14729.645555] awk '/^status/ { print $2 }' [14730.911283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14730.911283] awk '/^status/ { print $2 }' [14732.177779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14732.177779] awk '/^status/ { print $2 }' [14732.962639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14732.962639] awk '/^status/ { print $2 }' [14733.230876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14733.230876] awk '/^status/ { print $2 }' [14735.024588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14735.347755] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14736.242852] Lustre: DEBUG MARKER: rc=0; [14736.242852] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14736.242852] if [[ $? -eq 0 && $val -ne 0 ]]; then [14736.242852] echo $(hostname -s): $val; [14736.242852] rc=$val; [14736.242852] fi; [14736.242852] exit $rc [14736.844110] Lustre: DEBUG MARKER: dmesg [14737.447543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ================================== 14:38:28 \(1491489508\) [14737.604079] Lustre: DEBUG MARKER: == sanity-lfsck test 18h: LFSCK can repair crashed PFL extent range ================================== 14:38:28 (1491489508) [14737.868156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162f [14738.210088] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14738.479220] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14738.801879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14738.801879] awk '/^status/ { print $2 }' [14739.061262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14739.061262] awk '/^status/ { print $2 }' [14740.320646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14740.320646] awk '/^status/ { print $2 }' [14741.578945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14741.578945] awk '/^status/ { print $2 }' [14742.854856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14742.854856] awk '/^status/ { print $2 }' [14743.637160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14743.637160] awk '/^status/ { print $2 }' [14743.892783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14743.892783] awk '/^status/ { print $2 }' [14745.698084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14746.120104] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14747.001832] Lustre: DEBUG MARKER: rc=0; [14747.001832] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14747.001832] if [[ $? -eq 0 && $val -ne 0 ]]; then [14747.001832] echo $(hostname -s): $val; [14747.001832] rc=$val; [14747.001832] fi; [14747.001832] exit $rc [14747.584588] Lustre: DEBUG MARKER: dmesg [14748.124300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 19a: OST-object inconsistency self detect ======================================= 14:38:39 \(1491489519\) [14748.279380] Lustre: DEBUG MARKER: == sanity-lfsck test 19a: OST-object inconsistency self detect ======================================= 14:38:39 (1491489519) [14751.151343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14752.020030] Lustre: DEBUG MARKER: rc=0; [14752.020030] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14752.020030] if [[ $? -eq 0 && $val -ne 0 ]]; then [14752.020030] echo $(hostname -s): $val; [14752.020030] rc=$val; [14752.020030] fi; [14752.020030] exit $rc [14752.607863] Lustre: DEBUG MARKER: dmesg [14753.277588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 19b: OST-object inconsistency self repair ======================================= 14:38:44 \(1491489524\) [14753.433655] Lustre: DEBUG MARKER: == sanity-lfsck test 19b: OST-object inconsistency self repair ======================================= 14:38:44 (1491489524) [14757.684240] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14758.581862] Lustre: DEBUG MARKER: rc=0; [14758.581862] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14758.581862] if [[ $? -eq 0 && $val -ne 0 ]]; then [14758.581862] echo $(hostname -s): $val; [14758.581862] rc=$val; [14758.581862] fi; [14758.581862] exit $rc [14759.177585] Lustre: DEBUG MARKER: dmesg [14759.778668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================== 14:38:50 \(1491489530\) [14759.936887] Lustre: DEBUG MARKER: == sanity-lfsck test 20a: Handle the orphan with dummy LOV EA slot properly ========================== 14:38:50 (1491489530) [14760.449533] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [14760.582880] Lustre: *** cfs_fail_loc=1616, val=0*** [14760.584795] Lustre: Skipped 7 previous similar messages [14760.710916] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161a [14760.970337] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 [14761.232338] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 [14763.648877] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14763.907629] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14764.204787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14764.204787] awk '/^status/ { print $2 }' [14764.485054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14764.485054] awk '/^status/ { print $2 }' [14765.751852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14765.751852] awk '/^status/ { print $2 }' [14766.646278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14766.646278] awk '/^status/ { print $2 }' [14766.904563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14766.904563] awk '/^status/ { print $2 }' [14768.699254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14768.855176] Lustre: MGS: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [14768.857501] Lustre: Skipped 133 previous similar messages [14769.796406] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14770.698112] Lustre: DEBUG MARKER: rc=0; [14770.698112] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14770.698112] if [[ $? -eq 0 && $val -ne 0 ]]; then [14770.698112] echo $(hostname -s): $val; [14770.698112] rc=$val; [14770.698112] fi; [14770.698112] exit $rc [14771.291252] Lustre: DEBUG MARKER: dmesg [14771.837195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case =============== 14:39:02 \(1491489542\) [14772.003324] Lustre: DEBUG MARKER: == sanity-lfsck test 20b: Handle the orphan with dummy LOV EA slot properly - PFL case =============== 14:39:02 (1491489542) [14772.704082] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 [14772.964057] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161a [14773.225670] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 [14775.500707] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14775.755005] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o [14776.051321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14776.051321] awk '/^status/ { print $2 }' [14776.366955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout | [14776.366955] awk '/^status/ { print $2 }' [14777.141394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14777.141394] awk '/^status/ { print $2 }' [14777.400484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.lfsck_layout | [14777.400484] awk '/^status/ { print $2 }' [14779.187635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14780.256905] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14781.147509] Lustre: DEBUG MARKER: rc=0; [14781.147509] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14781.147509] if [[ $? -eq 0 && $val -ne 0 ]]; then [14781.147509] echo $(hostname -s): $val; [14781.147509] rc=$val; [14781.147509] fi; [14781.147509] exit $rc [14781.739865] Lustre: DEBUG MARKER: dmesg [14782.264192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 21: run all LFSCK components by default ========================================= 14:39:13 \(1491489553\) [14782.424987] Lustre: DEBUG MARKER: == sanity-lfsck test 21: run all LFSCK components by default ========================================= 14:39:13 (1491489553) [14782.564816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14782.564816] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14782.564816] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14782.980992] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -s 1 -r [14783.257607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14783.523083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14783.778142] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [14787.904876] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14788.788081] Lustre: DEBUG MARKER: rc=0; [14788.788081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14788.788081] if [[ $? -eq 0 && $val -ne 0 ]]; then [14788.788081] echo $(hostname -s): $val; [14788.788081] rc=$val; [14788.788081] fi; [14788.788081] exit $rc [14789.369309] Lustre: DEBUG MARKER: dmesg [14789.939093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 22a: LFSCK can repair unmatched pairs \(1\) ======================================= 14:39:20 \(1491489560\) [14790.097220] Lustre: DEBUG MARKER: == sanity-lfsck test 22a: LFSCK can repair unmatched pairs (1) ======================================= 14:39:20 (1491489560) [14790.414043] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161e [14790.629788] Lustre: *** cfs_fail_loc=161e, val=0*** [14790.631649] Lustre: Skipped 3499 previous similar messages [14790.635354] Lustre: *** cfs_fail_loc=161e, val=0*** [14790.637241] Lustre: Skipped 1 previous similar message [14790.781247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14791.045196] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r [14791.310225] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14791.310225] awk '/^namespace_mdts_completed/ { print $2 }' [14791.568952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14791.885026] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14792.767920] Lustre: DEBUG MARKER: rc=0; [14792.767920] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14792.767920] if [[ $? -eq 0 && $val -ne 0 ]]; then [14792.767920] echo $(hostname -s): $val; [14792.767920] rc=$val; [14792.767920] fi; [14792.767920] exit $rc [14793.356879] Lustre: DEBUG MARKER: dmesg [14793.907332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 22b: LFSCK can repair unmatched pairs \(2\) ======================================= 14:39:24 \(1491489564\) [14794.069183] Lustre: DEBUG MARKER: == sanity-lfsck test 22b: LFSCK can repair unmatched pairs (2) ======================================= 14:39:24 (1491489564) [14794.311972] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161e [14794.592073] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14794.917141] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r [14795.182237] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14795.182237] awk '/^namespace_mdts_completed/ { print $2 }' [14795.443932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14795.759873] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14796.633410] Lustre: DEBUG MARKER: rc=0; [14796.633410] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14796.633410] if [[ $? -eq 0 && $val -ne 0 ]]; then [14796.633410] echo $(hostname -s): $val; [14796.633410] rc=$val; [14796.633410] fi; [14796.633410] exit $rc [14797.228359] Lustre: DEBUG MARKER: dmesg [14797.771080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23a: LFSCK can repair dangling name entry \(1\) =================================== 14:39:28 \(1491489568\) [14797.930696] Lustre: DEBUG MARKER: == sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) =================================== 14:39:28 (1491489568) [14798.820753] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r [14799.084110] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14799.084110] awk '/^namespace_mdts_completed/ { print $2 }' [14799.344198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14799.601395] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r -C [14799.863583] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14799.863583] awk '/^namespace_mdts_completed/ { print $2 }' [14800.121405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14800.435656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14801.323860] Lustre: DEBUG MARKER: rc=0; [14801.323860] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14801.323860] if [[ $? -eq 0 && $val -ne 0 ]]; then [14801.323860] echo $(hostname -s): $val; [14801.323860] rc=$val; [14801.323860] fi; [14801.323860] exit $rc [14801.918176] Lustre: DEBUG MARKER: dmesg [14802.780202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23b: LFSCK can repair dangling name entry \(2\) =================================== 14:39:33 \(1491489573\) [14803.200091] Lustre: DEBUG MARKER: == sanity-lfsck test 23b: LFSCK can repair dangling name entry (2) =================================== 14:39:33 (1491489573) [14803.748861] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=137 fail_loc=0x1621 [14804.109322] Lustre: *** cfs_fail_loc=1621, val=0*** [14804.111250] Lustre: Skipped 1 previous similar message [14804.239217] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [14804.513212] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -C [14804.769038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14804.769038] awk '/^status/ { print $2 }' [14805.026061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14805.026061] awk '/^status/ { print $2 }' [14806.290949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14806.290949] awk '/^status/ { print $2 }' [14807.553581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14807.553581] awk '/^status/ { print $2 }' [14808.811530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14808.811530] awk '/^status/ { print $2 }' [14809.073374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14809.330207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14809.643243] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14810.546420] Lustre: DEBUG MARKER: rc=0; [14810.546420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14810.546420] if [[ $? -eq 0 && $val -ne 0 ]]; then [14810.546420] echo $(hostname -s): $val; [14810.546420] rc=$val; [14810.546420] fi; [14810.546420] exit $rc [14811.144814] Lustre: DEBUG MARKER: dmesg [14811.713072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 23c: LFSCK can repair dangling name entry \(3\) =================================== 14:39:42 \(1491489582\) [14811.872610] Lustre: DEBUG MARKER: == sanity-lfsck test 23c: LFSCK can repair dangling name entry (3) =================================== 14:39:42 (1491489582) [14812.014384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [14812.311996] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [14812.661143] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [14813.062306] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=151 fail_loc=0x1621 [14813.320594] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [14813.610399] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x1602 [14813.866218] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -C [14814.042069] LustreError: 20966:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 sleeping for 10000ms [14814.045157] LustreError: 20966:0:(fail.c:129:__cfs_fail_timeout_set()) Skipped 1 previous similar message [14814.191852] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=0 fail_loc=0 [14814.447535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14814.447535] awk '/^status/ { print $2 }' [14814.704198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14814.704198] awk '/^status/ { print $2 }' [14815.966230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14815.966230] awk '/^status/ { print $2 }' [14817.225427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14817.225427] awk '/^status/ { print $2 }' [14818.482051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14818.482051] awk '/^status/ { print $2 }' [14819.744508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14819.744508] awk '/^status/ { print $2 }' [14821.003970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14821.003970] awk '/^status/ { print $2 }' [14822.260861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14822.260861] awk '/^status/ { print $2 }' [14823.524711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14823.524711] awk '/^status/ { print $2 }' [14824.047034] LustreError: 20966:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 1602 awake [14824.049410] LustreError: 20966:0:(fail.c:133:__cfs_fail_timeout_set()) Skipped 1 previous similar message [14824.788077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14824.788077] awk '/^status/ { print $2 }' [14825.092417] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [14825.685392] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [14825.975165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14826.292990] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14827.175649] Lustre: DEBUG MARKER: rc=0; [14827.175649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14827.175649] if [[ $? -eq 0 && $val -ne 0 ]]; then [14827.175649] echo $(hostname -s): $val; [14827.175649] rc=$val; [14827.175649] fi; [14827.175649] exit $rc [14827.772586] Lustre: DEBUG MARKER: dmesg [14828.351753] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ============================= 14:39:59 \(1491489599\) [14828.505064] Lustre: DEBUG MARKER: == sanity-lfsck test 24: LFSCK can repair multiple-referenced name entry ============================= 14:39:59 (1491489599) [14828.754511] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1622 [14828.914361] Lustre: *** cfs_fail_loc=1622, val=0*** [14828.916429] Lustre: Skipped 3492 previous similar messages [14828.955688] Lustre: *** cfs_fail_loc=1622, val=0*** [14828.957712] Lustre: Skipped 3 previous similar messages [14829.090449] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14829.389121] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r [14829.672180] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14829.672180] awk '/^namespace_mdts_completed/ { print $2 }' [14829.945989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14830.269272] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14831.145770] Lustre: DEBUG MARKER: rc=0; [14831.145770] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14831.145770] if [[ $? -eq 0 && $val -ne 0 ]]; then [14831.145770] echo $(hostname -s): $val; [14831.145770] rc=$val; [14831.145770] fi; [14831.145770] exit $rc [14831.733706] Lustre: DEBUG MARKER: dmesg [14832.274350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ============================ 14:40:03 \(1491489603\) [14832.427071] Lustre: DEBUG MARKER: == sanity-lfsck test 25: LFSCK can repair bad file type in the name entry ============================ 14:40:03 (1491489603) [14832.641191] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1623 [14832.779152] Lustre: *** cfs_fail_loc=1623, val=0*** [14832.781088] Lustre: Skipped 1 previous similar message [14832.909264] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14833.164018] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r [14833.425950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14833.425950] awk '/^status/ { print $2 }' [14833.682236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14833.682236] awk '/^status/ { print $2 }' [14833.939675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14834.255676] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14835.160512] Lustre: DEBUG MARKER: rc=0; [14835.160512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14835.160512] if [[ $? -eq 0 && $val -ne 0 ]]; then [14835.160512] echo $(hostname -s): $val; [14835.160512] rc=$val; [14835.160512] fi; [14835.160512] exit $rc [14835.753748] Lustre: DEBUG MARKER: dmesg [14836.481263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace =========== 14:40:07 \(1491489607\) [14836.637757] Lustre: DEBUG MARKER: == sanity-lfsck test 26a: LFSCK can add the missing local name entry back to the namespace =========== 14:40:07 (1491489607) [14836.840923] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14837.098126] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14837.353988] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14837.614837] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14837.614837] awk '/^namespace_mdts_completed/ { print $2 }' [14840.876527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14841.194619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14842.071521] Lustre: DEBUG MARKER: rc=0; [14842.071521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14842.071521] if [[ $? -eq 0 && $val -ne 0 ]]; then [14842.071521] echo $(hostname -s): $val; [14842.071521] rc=$val; [14842.071521] fi; [14842.071521] exit $rc [14842.657998] Lustre: DEBUG MARKER: dmesg [14843.241892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========== 14:40:14 \(1491489614\) [14843.404565] Lustre: DEBUG MARKER: == sanity-lfsck test 26b: LFSCK can add the missing remote name entry back to the namespace ========== 14:40:14 (1491489614) [14843.678216] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14843.985223] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14844.255438] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14844.514730] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14844.514730] awk '/^namespace_mdts_completed/ { print $2 }' [14844.774710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14845.105400] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14845.982232] Lustre: DEBUG MARKER: rc=0; [14845.982232] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14845.982232] if [[ $? -eq 0 && $val -ne 0 ]]; then [14845.982232] echo $(hostname -s): $val; [14845.982232] rc=$val; [14845.982232] fi; [14845.982232] exit $rc [14846.588020] Lustre: DEBUG MARKER: dmesg [14847.120728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan =============== 14:40:18 \(1491489618\) [14847.274200] Lustre: DEBUG MARKER: == sanity-lfsck test 27a: LFSCK can recreate the lost local parent directory as orphan =============== 14:40:18 (1491489618) [14847.475623] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14847.736557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14847.996880] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14848.260633] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14848.260633] awk '/^namespace_mdts_completed/ { print $2 }' [14848.520106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14848.842356] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14849.728313] Lustre: DEBUG MARKER: rc=0; [14849.728313] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14849.728313] if [[ $? -eq 0 && $val -ne 0 ]]; then [14849.728313] echo $(hostname -s): $val; [14849.728313] rc=$val; [14849.728313] fi; [14849.728313] exit $rc [14850.325276] Lustre: DEBUG MARKER: dmesg [14850.875596] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ============== 14:40:21 \(1491489621\) [14851.035425] Lustre: DEBUG MARKER: == sanity-lfsck test 27b: LFSCK can recreate the lost remote parent directory as orphan ============== 14:40:21 (1491489621) [14851.270255] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14851.590204] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14851.975611] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14852.236210] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14852.236210] awk '/^namespace_mdts_completed/ { print $2 }' [14852.498602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14852.865133] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14853.738775] Lustre: DEBUG MARKER: rc=0; [14853.738775] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14853.738775] if [[ $? -eq 0 && $val -ne 0 ]]; then [14853.738775] echo $(hostname -s): $val; [14853.738775] rc=$val; [14853.738775] fi; [14853.738775] exit $rc [14854.352534] Lustre: DEBUG MARKER: dmesg [14854.927951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 28: Skip the failed MDT\(s\) when handle orphan MDT-objects ======================= 14:40:25 \(1491489625\) [14855.099096] Lustre: DEBUG MARKER: == sanity-lfsck test 28: Skip the failed MDT(s) when handle orphan MDT-objects ======================= 14:40:25 (1491489625) [14855.395142] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14855.941462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14856.703123] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14856.968799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14856.968799] awk '/^status/ { print $2 }' [14857.223847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | [14857.223847] awk '/^status/ { print $2 }' [14858.250433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14858.766827] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14859.029973] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14859.029973] awk '/^namespace_mdts_completed/ { print $2 }' [14859.294036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14859.852710] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14860.733913] Lustre: DEBUG MARKER: rc=0; [14860.733913] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14860.733913] if [[ $? -eq 0 && $val -ne 0 ]]; then [14860.733913] echo $(hostname -s): $val; [14860.733913] rc=$val; [14860.733913] fi; [14860.733913] exit $rc [14861.326329] Lustre: DEBUG MARKER: dmesg [14861.871533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 29b: LFSCK can repair bad nlink count \(2\) ======================================= 14:40:32 \(1491489632\) [14862.035190] Lustre: DEBUG MARKER: == sanity-lfsck test 29b: LFSCK can repair bad nlink count (2) ======================================= 14:40:32 (1491489632) [14862.346321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1626 [14862.485160] Lustre: *** cfs_fail_loc=1626, val=0*** [14862.487085] Lustre: Skipped 6 previous similar messages [14862.613411] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14862.881935] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14863.147638] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14863.147638] awk '/^namespace_mdts_completed/ { print $2 }' [14863.409316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14863.731843] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14864.638771] Lustre: DEBUG MARKER: rc=0; [14864.638771] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14864.638771] if [[ $? -eq 0 && $val -ne 0 ]]; then [14864.638771] echo $(hostname -s): $val; [14864.638771] rc=$val; [14864.638771] fi; [14864.638771] exit $rc [14865.239084] Lustre: DEBUG MARKER: dmesg [14865.857983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 29c: verify linkEA size limitation ============================================== 14:40:36 \(1491489636\) [14866.020357] Lustre: DEBUG MARKER: == sanity-lfsck test 29c: verify linkEA size limitation ============================================== 14:40:36 (1491489636) [14871.617995] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14871.924767] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14871.924767] awk '/^namespace_mdts_completed/ { print $2 }' [14872.186690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14872.447229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14873.195917] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14874.134175] Lustre: DEBUG MARKER: rc=0; [14874.134175] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14874.134175] if [[ $? -eq 0 && $val -ne 0 ]]; then [14874.134175] echo $(hostname -s): $val; [14874.134175] rc=$val; [14874.134175] fi; [14874.134175] exit $rc [14874.739952] Lustre: DEBUG MARKER: dmesg [14875.281252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 30: LFSCK can recover the orphans from backend \/lost+found ====================== 14:40:46 \(1491489646\) [14875.439570] Lustre: DEBUG MARKER: == sanity-lfsck test 30: LFSCK can recover the orphans from backend /lost+found ====================== 14:40:46 (1491489646) [14875.658506] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x161d [14875.914155] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14876.176591] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1624 [14876.441615] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14876.841254] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14877.096895] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [14877.228175] Lustre: Failing over lustre-MDT0000 [14877.229915] Lustre: Skipped 11 previous similar messages [14877.330774] Lustre: server umount lustre-MDT0000 complete [14877.332552] Lustre: Skipped 9 previous similar messages [14877.465558] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14877.728920] Lustre: DEBUG MARKER: e2fsck -d -v -t -t -f -y /dev/lvm-Role_MDS/P1 [14878.063836] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14878.325320] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14878.401115] LustreError: 5306:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491489649 with bad export cookie 7038122449855284964 [14878.405015] LustreError: 5306:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [14878.408562] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14878.412581] LustreError: Skipped 92 previous similar messages [14878.586912] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14878.839591] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14879.000394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14879.017396] LustreError: 11-0: MGC10.9.5.243@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [14879.019900] LustreError: Skipped 17 previous similar messages [14879.020074] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d538333 to 0x61ac6f141d543e7a [14879.020075] Lustre: Skipped 3 previous similar messages [14879.057114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14879.059480] Lustre: Skipped 9 previous similar messages [14879.083597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [14879.236663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14879.505755] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14880.000430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14880.002299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14880.183018] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14880.183058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14880.332504] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14880.592650] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14880.847728] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14881.114644] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14881.378891] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14881.634558] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14881.911302] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14881.911302] awk '/^namespace_mdts_completed/ { print $2 }' [14884.054615] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [14884.059657] LustreError: Skipped 3 previous similar messages [14885.481222] Lustre: lustre-MDT0000: Recovery over after 0:07, of 4 clients 4 recovered and 0 were evicted. [14888.171346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14888.591278] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14889.490208] Lustre: DEBUG MARKER: rc=0; [14889.490208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14889.490208] if [[ $? -eq 0 && $val -ne 0 ]]; then [14889.490208] echo $(hostname -s): $val; [14889.490208] rc=$val; [14889.490208] fi; [14889.490208] exit $rc [14890.090947] Lustre: DEBUG MARKER: dmesg [14890.626333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31a: The LFSCK can find\/repair the name entry with bad name hash \(1\) ============ 14:41:01 \(1491489661\) [14890.798417] Lustre: DEBUG MARKER: == sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) ============ 14:41:01 (1491489661) [14890.897588] Lustre: 5319:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 256 < left 271, rollback = 9 [14890.902568] Lustre: 5319:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/0, destroy: 0/0/0 [14890.905301] Lustre: 5319:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/271/0 [14890.908447] Lustre: 5319:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14890.911083] Lustre: 5319:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14890.913910] Lustre: 5319:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14891.064628] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14891.350072] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14891.350072] awk '/^namespace_mdts_completed/ { print $2 }' [14894.620525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14895.219345] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14896.114819] Lustre: DEBUG MARKER: rc=0; [14896.114819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14896.114819] if [[ $? -eq 0 && $val -ne 0 ]]; then [14896.114819] echo $(hostname -s): $val; [14896.114819] rc=$val; [14896.114819] fi; [14896.114819] exit $rc [14896.722411] Lustre: DEBUG MARKER: dmesg [14897.284389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31b: The LFSCK can find\/repair the name entry with bad name hash \(2\) ============ 14:41:08 \(1491489668\) [14897.453044] Lustre: DEBUG MARKER: == sanity-lfsck test 31b: The LFSCK can find/repair the name entry with bad name hash (2) ============ 14:41:08 (1491489668) [14897.520616] Lustre: 25868:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 255 < left 271, rollback = 9 [14897.524626] Lustre: 25868:0:(osd_internal.h:1100:osd_trans_exec_op()) Skipped 14 previous similar messages [14897.526964] Lustre: 25868:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/1, destroy: 0/0/0 [14897.529428] Lustre: 25868:0:(osd_handler.c:1665:osd_trans_dump_creds()) Skipped 14 previous similar messages [14897.531839] Lustre: 25868:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/271/0 [14897.534225] Lustre: 25868:0:(osd_handler.c:1672:osd_trans_dump_creds()) Skipped 14 previous similar messages [14897.536531] Lustre: 25868:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14897.538860] Lustre: 25868:0:(osd_handler.c:1682:osd_trans_dump_creds()) Skipped 14 previous similar messages [14897.541130] Lustre: 25868:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14897.543347] Lustre: 25868:0:(osd_handler.c:1689:osd_trans_dump_creds()) Skipped 14 previous similar messages [14897.545631] Lustre: 25868:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14897.547845] Lustre: 25868:0:(osd_handler.c:1696:osd_trans_dump_creds()) Skipped 14 previous similar messages [14897.694032] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14897.957668] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14897.957668] awk '/^namespace_mdts_completed/ { print $2 }' [14898.798819] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14899.701779] Lustre: DEBUG MARKER: rc=0; [14899.701779] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14899.701779] if [[ $? -eq 0 && $val -ne 0 ]]; then [14899.701779] echo $(hostname -s): $val; [14899.701779] rc=$val; [14899.701779] fi; [14899.701779] exit $rc [14900.299472] Lustre: DEBUG MARKER: dmesg [14900.837270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory =================== 14:41:11 \(1491489671\) [14900.991222] Lustre: DEBUG MARKER: == sanity-lfsck test 31c: Re-generate the lost master LMV EA for striped directory =================== 14:41:11 (1491489671) [14901.214387] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1629 [14901.358989] Lustre: 5318:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 255 < left 271, rollback = 9 [14901.363462] Lustre: 5318:0:(osd_internal.h:1100:osd_trans_exec_op()) Skipped 14 previous similar messages [14901.366647] Lustre: 5318:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/1, destroy: 0/0/0 [14901.370579] Lustre: 5318:0:(osd_handler.c:1665:osd_trans_dump_creds()) Skipped 14 previous similar messages [14901.373343] Lustre: 5318:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/271/0 [14901.375824] Lustre: 5318:0:(osd_handler.c:1672:osd_trans_dump_creds()) Skipped 14 previous similar messages [14901.378089] Lustre: 5318:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14901.380660] Lustre: 5318:0:(osd_handler.c:1682:osd_trans_dump_creds()) Skipped 14 previous similar messages [14901.382913] Lustre: 5318:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14901.385337] Lustre: 5318:0:(osd_handler.c:1689:osd_trans_dump_creds()) Skipped 14 previous similar messages [14901.387615] Lustre: 5318:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14901.390056] Lustre: 5318:0:(osd_handler.c:1696:osd_trans_dump_creds()) Skipped 14 previous similar messages [14901.392755] Lustre: *** cfs_fail_loc=1629, val=0*** [14901.394654] Lustre: Skipped 20 previous similar messages [14901.530597] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [14901.786928] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14902.054452] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14902.054452] awk '/^namespace_mdts_completed/ { print $2 }' [14902.314971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14902.971598] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14903.866607] Lustre: DEBUG MARKER: rc=0; [14903.866607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14903.866607] if [[ $? -eq 0 && $val -ne 0 ]]; then [14903.866607] echo $(hostname -s): $val; [14903.866607] rc=$val; [14903.866607] fi; [14903.866607] exit $rc [14904.459701] Lustre: DEBUG MARKER: dmesg [14905.049317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31d: Set broken striped directory \(modified after broken\) as read-only ========== 14:41:16 \(1491489676\) [14905.210581] Lustre: DEBUG MARKER: == sanity-lfsck test 31d: Set broken striped directory (modified after broken) as read-only ========== 14:41:16 (1491489676) [14905.388159] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1629 [14905.523031] Lustre: 5318:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 256 < left 271, rollback = 9 [14905.526858] Lustre: 5318:0:(osd_internal.h:1100:osd_trans_exec_op()) Skipped 13 previous similar messages [14905.529129] Lustre: 5318:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/0, destroy: 0/0/0 [14905.531355] Lustre: 5318:0:(osd_handler.c:1665:osd_trans_dump_creds()) Skipped 13 previous similar messages [14905.533608] Lustre: 5318:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/271/0 [14905.536007] Lustre: 5318:0:(osd_handler.c:1672:osd_trans_dump_creds()) Skipped 13 previous similar messages [14905.538309] Lustre: 5318:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14905.540660] Lustre: 5318:0:(osd_handler.c:1682:osd_trans_dump_creds()) Skipped 13 previous similar messages [14905.542937] Lustre: 5318:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14905.545185] Lustre: 5318:0:(osd_handler.c:1689:osd_trans_dump_creds()) Skipped 13 previous similar messages [14905.547400] Lustre: 5318:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14905.549561] Lustre: 5318:0:(osd_handler.c:1696:osd_trans_dump_creds()) Skipped 13 previous similar messages [14905.679430] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 [14906.152606] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14906.494438] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14906.494438] awk '/^namespace_mdts_completed/ { print $2 }' [14906.756963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14907.155268] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14908.027691] Lustre: DEBUG MARKER: rc=0; [14908.027691] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14908.027691] if [[ $? -eq 0 && $val -ne 0 ]]; then [14908.027691] echo $(hostname -s): $val; [14908.027691] rc=$val; [14908.027691] fi; [14908.027691] exit $rc [14908.621663] Lustre: DEBUG MARKER: dmesg [14909.211464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory \(1\) ================ 14:41:20 \(1491489680\) [14909.367556] Lustre: DEBUG MARKER: == sanity-lfsck test 31e: Re-generate the lost slave LMV EA for striped directory (1) ================ 14:41:20 (1491489680) [14909.538939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162a fail_val=0 [14909.673925] Lustre: 5319:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 256 < left 271, rollback = 9 [14909.677708] Lustre: 5319:0:(osd_internal.h:1100:osd_trans_exec_op()) Skipped 13 previous similar messages [14909.679943] Lustre: 5319:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/0, destroy: 0/0/0 [14909.682153] Lustre: 5319:0:(osd_handler.c:1665:osd_trans_dump_creds()) Skipped 13 previous similar messages [14909.684381] Lustre: 5319:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/207/0 [14909.686767] Lustre: 5319:0:(osd_handler.c:1672:osd_trans_dump_creds()) Skipped 13 previous similar messages [14909.689021] Lustre: 5319:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14909.691385] Lustre: 5319:0:(osd_handler.c:1682:osd_trans_dump_creds()) Skipped 13 previous similar messages [14909.693655] Lustre: 5319:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14909.695964] Lustre: 5319:0:(osd_handler.c:1689:osd_trans_dump_creds()) Skipped 13 previous similar messages [14909.698210] Lustre: 5319:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14909.700458] Lustre: 5319:0:(osd_handler.c:1696:osd_trans_dump_creds()) Skipped 13 previous similar messages [14909.831016] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 fail_val=0 [14910.089098] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14910.356355] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14910.356355] awk '/^namespace_mdts_completed/ { print $2 }' [14910.617051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14911.001308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14911.877234] Lustre: DEBUG MARKER: rc=0; [14911.877234] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14911.877234] if [[ $? -eq 0 && $val -ne 0 ]]; then [14911.877234] echo $(hostname -s): $val; [14911.877234] rc=$val; [14911.877234] fi; [14911.877234] exit $rc [14912.476963] Lustre: DEBUG MARKER: dmesg [14913.059090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory \(2\) ================ 14:41:24 \(1491489684\) [14913.216282] Lustre: DEBUG MARKER: == sanity-lfsck test 31f: Re-generate the lost slave LMV EA for striped directory (2) ================ 14:41:24 (1491489684) [14913.387890] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162a fail_val=1 [14913.650090] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 fail_val=0 [14913.909370] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14914.176502] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14914.176502] awk '/^namespace_mdts_completed/ { print $2 }' [14914.844778] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14915.734944] Lustre: DEBUG MARKER: rc=0; [14915.734944] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14915.734944] if [[ $? -eq 0 && $val -ne 0 ]]; then [14915.734944] echo $(hostname -s): $val; [14915.734944] rc=$val; [14915.734944] fi; [14915.734944] exit $rc [14916.336805] Lustre: DEBUG MARKER: dmesg [14916.897597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================== 14:41:27 \(1491489687\) [14917.052944] Lustre: DEBUG MARKER: == sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================== 14:41:27 (1491489687) [14917.220205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162b fail_val=0 [14917.492124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 fail_val=0 [14917.749152] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14918.011354] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14918.011354] awk '/^namespace_mdts_completed/ { print $2 }' [14918.269342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14918.915070] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14919.816605] Lustre: DEBUG MARKER: rc=0; [14919.816605] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14919.816605] if [[ $? -eq 0 && $val -ne 0 ]]; then [14919.816605] echo $(hostname -s): $val; [14919.816605] rc=$val; [14919.816605] fi; [14919.816605] exit $rc [14920.405799] Lustre: DEBUG MARKER: dmesg [14920.944812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 31h: Repair the corrupted shard\'s name entry ==================================== 14:41:31 \(1491489691\) [14921.112602] Lustre: DEBUG MARKER: == sanity-lfsck test 31h: Repair the corrupted shard's name entry ==================================== 14:41:31 (1491489691) [14921.351470] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x162c fail_val=0 [14921.487128] Lustre: 5319:0:(osd_internal.h:1100:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 255 < left 271, rollback = 9 [14921.490945] Lustre: 5319:0:(osd_internal.h:1100:osd_trans_exec_op()) Skipped 40 previous similar messages [14921.493207] Lustre: 5319:0:(osd_handler.c:1665:osd_trans_dump_creds()) create: 3/24/1, destroy: 0/0/0 [14921.495543] Lustre: 5319:0:(osd_handler.c:1665:osd_trans_dump_creds()) Skipped 40 previous similar messages [14921.497903] Lustre: 5319:0:(osd_handler.c:1672:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/271/0 [14921.500182] Lustre: 5319:0:(osd_handler.c:1672:osd_trans_dump_creds()) Skipped 40 previous similar messages [14921.502487] Lustre: 5319:0:(osd_handler.c:1682:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 2/2/0 [14921.504738] Lustre: 5319:0:(osd_handler.c:1682:osd_trans_dump_creds()) Skipped 40 previous similar messages [14921.506962] Lustre: 5319:0:(osd_handler.c:1689:osd_trans_dump_creds()) insert: 13/271/0, delete: 0/0/0 [14921.509090] Lustre: 5319:0:(osd_handler.c:1689:osd_trans_dump_creds()) Skipped 40 previous similar messages [14921.511307] Lustre: 5319:0:(osd_handler.c:1696:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 0/0/0 [14921.513433] Lustre: 5319:0:(osd_handler.c:1696:osd_trans_dump_creds()) Skipped 40 previous similar messages [14921.643850] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0 fail_val=0 [14921.903976] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A [14922.169665] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w | [14922.169665] awk '/^namespace_mdts_completed/ { print $2 }' [14922.430538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace [14923.017846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14923.896451] Lustre: DEBUG MARKER: rc=0; [14923.896451] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14923.896451] if [[ $? -eq 0 && $val -ne 0 ]]; then [14923.896451] echo $(hostname -s): $val; [14923.896451] rc=$val; [14923.896451] fi; [14923.896451] exit $rc [14924.489072] Lustre: DEBUG MARKER: dmesg [14925.051576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test 32: stop LFSCK when some OST failed ============================================= 14:41:36 \(1491489696\) [14925.212490] Lustre: DEBUG MARKER: == sanity-lfsck test 32: stop LFSCK when some OST failed ============================================= 14:41:36 (1491489696) [14926.121743] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x162d [14926.381424] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r [14926.517491] Lustre: *** cfs_fail_loc=162d, val=0*** [14926.520334] LustreError: 3638:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 162d sleeping for 3000ms [14926.649637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_layout [14927.726624] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [14928.985202] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [14929.523043] LustreError: 3638:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 162d awake [14929.525477] LustreError: 3638:0:(osp_object.c:527:osp_attr_get()) lustre-OST0002-osc-MDT0000:osp_attr_get update error [0x100020000:0x2c2:0x0]: rc = -4 [14929.719469] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14930.608052] Lustre: DEBUG MARKER: rc=0; [14930.608052] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14930.608052] if [[ $? -eq 0 && $val -ne 0 ]]; then [14930.608052] echo $(hostname -s): $val; [14930.608052] rc=$val; [14930.608052] fi; [14930.608052] exit $rc [14931.217623] Lustre: DEBUG MARKER: dmesg [14932.400633] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14932.658992] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [14934.053344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14934.055471] Lustre: Skipped 41 previous similar messages [14935.583080] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005dcc8c00 x1563924896079232/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [14935.587617] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 18 previous similar messages [14939.029042] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14945.407041] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491489709/real 1491489709] req@ffff88005dccb000 x1563924896079312/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491489716 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [14945.413629] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 54 previous similar messages [14946.201668] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14946.468479] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [14952.875027] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14965.083729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [14965.083729] /usr/sbin/lctl lustre_build_version 2>/dev/null || [14965.083729] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [14965.408341] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [14966.266533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14966.280894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14966.290574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14966.309666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14966.325377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [14966.595285] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [14966.627386] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [14966.666764] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14966.692992] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [14966.705793] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [14966.995333] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [14967.258005] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14967.546850] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [14967.736360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14968.862651] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [14969.112223] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [14969.366573] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [14969.549752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14978.355550] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lfsck test complete, duration 1732 sec ===================================================== 14:42:29 \(1491489749\) [14978.511968] Lustre: DEBUG MARKER: == sanity-lfsck test complete, duration 1732 sec ===================================================== 14:42:29 (1491489749) [14978.730780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-hsm ============----- Thu Apr 6 14:42:29 UTC 2017 [14978.888174] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-hsm ============----- Thu Apr 6 14:42:29 UTC 2017 [14979.169213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration -o sec ========================================================= 14:42:30 \(1491489750\) [14979.315430] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration -o sec ========================================================= 14:42:30 (1491489750) [14981.758618] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14981.758618] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14981.758618] if [ $running -ne $mpts ]; then [14981.758618] echo $(hostname) env are INSANE!; [14981.758618] exit 1; [14981.758618] fi [14982.316590] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [14982.316590] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [14982.316590] if [ $running -ne $mpts ]; then [14982.316590] echo $(hostname) env are INSANE!; [14982.316590] exit 1; [14982.316590] fi [14982.894809] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [14982.894809] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [14982.894809] if [ $running -ne $mpts ]; then [14982.894809] echo $(hostname) env are INSANE!; [14982.894809] exit 1; [14982.894809] fi [14983.826704] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [14984.090987] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [14984.353282] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [14984.609138] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [14984.764906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14984.809105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14984.841831] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [14984.844386] Lustre: Skipped 4 previous similar messages [14984.858305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [14984.874055] Lustre: lustre-MDT0000: new disk, initializing [14984.906042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14984.908650] Lustre: Skipped 7 previous similar messages [14985.068155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [14985.323439] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [14985.812300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14985.816506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14985.972758] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14985.974576] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14986.117178] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14986.370060] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14986.624757] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [14986.877190] Lustre: DEBUG MARKER: sync; sync; sync [14992.181557] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [14992.436402] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [14993.656622] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [14993.659055] Lustre: Skipped 4 previous similar messages [14994.690959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [14994.852379] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15001.573731] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [15001.836889] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [15002.093103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [15002.343397] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [15002.501031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15002.547171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15002.561719] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [15002.564348] Lustre: Skipped 4 previous similar messages [15002.571576] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [15002.574071] Lustre: Skipped 1 previous similar message [15002.589505] Lustre: lustre-MDT0002: new disk, initializing [15002.634705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [15002.637863] Lustre: Skipped 1 previous similar message [15002.640399] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [15002.806351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15003.061957] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [15003.555063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15003.555818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15003.716391] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15003.718330] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15003.861908] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15004.117318] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15004.372234] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15004.629207] Lustre: DEBUG MARKER: sync; sync; sync [15009.920144] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [15010.185562] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15013.024247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15013.191077] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15022.999696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15023.162698] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15031.283029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15031.448215] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15039.687584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15039.850158] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15040.812073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000380000400-0x00000003c0000400]:2:ost [15040.815213] Lustre: Skipped 3 previous similar messages [15048.030809] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15048.211470] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15056.428089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15056.588078] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15064.750194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15064.924923] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15073.293246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15073.459036] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15081.733188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15081.893856] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15089.745144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15089.910135] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15092.501064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15092.744483] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15093.018751] Lustre: DEBUG MARKER: lctl get_param -n timeout [15093.336807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [15093.497527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [15093.628908] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [15099.040133] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [15099.317254] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15099.833280] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15100.357685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15100.357685] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15100.357685] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15101.062216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [15101.220969] Lustre: DEBUG MARKER: excepting tests: [15101.843736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir \/shared_test\/autotest2\/2017-04-06\/064517-70343306542720 [15101.850613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir \/shared_test\/autotest2\/2017-04-06\/064517-70343306542720 [15102.071406] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir /shared_test/autotest2/2017-04-06/064517-70343306542720 [15102.072971] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir /shared_test/autotest2/2017-04-06/064517-70343306542720 [15102.998734] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [15103.134752] Lustre: Modifying parameter general.mdt.lustre-MDT0000.hsm_control in log params [15103.137796] Lustre: Skipped 9 previous similar messages [15103.274988] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0001.hsm_control=enabled [15103.533508] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0002.hsm_control=enabled [15103.789911] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0003.hsm_control=enabled [15110.337621] LustreError: 10872:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [15124.057200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15124.317598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15125.088554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15125.351573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15126.124834] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [15126.639927] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [15127.226187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1: lfs hsm flags root\/non-root access ============================================= 14:44:58 \(1491489898\) [15127.382669] Lustre: DEBUG MARKER: == sanity-hsm test 1: lfs hsm flags root/non-root access ============================================= 14:44:58 (1491489898) [15127.622369] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15128.514194] Lustre: DEBUG MARKER: rc=0; [15128.514194] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15128.514194] if [[ $? -eq 0 && $val -ne 0 ]]; then [15128.514194] echo $(hostname -s): $val; [15128.514194] rc=$val; [15128.514194] fi; [15128.514194] exit $rc [15129.112691] Lustre: DEBUG MARKER: dmesg [15129.666144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1a: mmap & cat a HSM released file ================================================ 14:45:00 \(1491489900\) [15129.822506] Lustre: DEBUG MARKER: == sanity-hsm test 1a: mmap [15131.088021] LustreError: 10862:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent '3067b518-b7c9-485b-b6ca-498aba1e1c42': rc = -11 [15131.214890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15131.478884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15132.739303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15134.010731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15135.279251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15136.542639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15137.813298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15139.084129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15140.345109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15141.612633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15142.882579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15144.146416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15149.136996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15149.396416] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15150.169140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15150.427972] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15151.205846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15151.461456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15152.722815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15153.992699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15155.252958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15156.509049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15156.768950] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15157.792725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15158.049362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15159.309178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15160.565136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15160.821348] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15161.857087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15162.115202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15162.895455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15163.155439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15164.625451] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15165.533837] Lustre: DEBUG MARKER: rc=0; [15165.533837] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15165.533837] if [[ $? -eq 0 && $val -ne 0 ]]; then [15165.533837] echo $(hostname -s): $val; [15165.533837] rc=$val; [15165.533837] fi; [15165.533837] exit $rc [15166.135752] Lustre: DEBUG MARKER: dmesg [15166.805612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1b: Archive, Release & Restore composite file ===================================== 14:45:37 \(1491489937\) [15166.960617] Lustre: DEBUG MARKER: == sanity-hsm test 1b: Archive, Release [15167.138587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15167.401454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15168.663288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15169.923348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15171.190708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15172.466611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15173.727528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15174.986609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15176.250418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15177.516686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15178.778080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15180.036585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15181.301945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15182.563352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15183.823930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15185.089706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15186.349688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15187.614664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15188.880086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15190.141032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15191.404593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15192.670263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15193.934814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15195.200316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15196.463064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15197.728402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15198.991307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15200.258945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15201.525058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15202.791554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15204.052942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15205.316776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15206.583988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15207.854399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15209.126280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15210.388583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15211.653689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15212.916937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15214.179524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15215.442392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15216.705558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15217.965264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15219.226466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15220.487877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15221.756780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15223.021747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15224.283392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15225.551558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15226.820728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15228.083841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15229.350387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15230.612512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15231.881398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15233.144764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15234.406534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15235.668342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15236.930630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15238.196608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15239.461953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15240.727107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15241.994903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15243.261496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15244.527911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15245.798309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15247.065469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15248.331700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15249.595896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15250.859416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15252.120712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15253.383316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15254.645172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15255.906915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15257.175804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15258.437822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15259.702702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15260.965030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15262.226309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15263.489663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15264.758631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15266.020179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15267.281725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15268.542850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15269.806843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15271.073076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15272.336836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15273.596641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15274.857812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15276.123104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15277.388313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15278.650465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15279.917638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15281.180628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15282.442796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15283.708865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15284.975269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15286.240315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15287.507639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15288.773642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15290.034848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15291.299296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15292.566553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15293.836482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15295.099770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15296.363312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15297.627470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15298.894854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15300.157230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15301.419212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15302.689257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15303.951828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15305.217699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15306.480785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15307.746223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15309.010684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15310.272383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15311.543738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15312.806411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15314.073616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15315.340506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15316.609213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15317.872855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15319.137443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15320.405082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15321.672768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15322.940573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15324.206813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15325.472498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15326.741103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15328.010768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15329.276624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15330.541373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15331.800984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15333.067849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15334.339679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15335.608746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15336.878430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15338.140033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15339.400137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15340.664299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15341.924641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15343.188005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15344.450685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15345.717625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15346.985114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15348.254910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15349.521367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15350.783379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15352.048645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15353.320548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15354.584037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15355.848453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15357.116357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15358.387253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15359.650784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15360.915474] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15362.179571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15363.442263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15364.709578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15365.972732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15367.240764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15368.502625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15369.767762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15371.033246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15372.295770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15373.559345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15374.821958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15376.091960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15377.355996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15378.619893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15379.885749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15381.147045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15382.413503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15383.677419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15384.939910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15386.203489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15387.464662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15388.725921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15389.988330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15391.251125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15392.514557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15393.776366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15395.038622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15396.299156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15397.561161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15398.826135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15400.088559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15401.351249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15402.614305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15403.876763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15405.140696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15406.409094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15407.676650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15408.943643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15410.207275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15411.469713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15412.738380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15414.001672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15415.266722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15416.531841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15417.798806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15419.068648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15420.329673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15420.655509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-hsm test_1b: @@@@@@ FAIL: request on 0x200000404:0x8:0x0 is not SUCCEED on mds1 [15420.824134] Lustre: DEBUG MARKER: sanity-hsm test_1b: @@@@@@ FAIL: request on 0x200000404:0x8:0x0 is not SUCCEED on mds1 [15421.018606] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /logdir/test_logs/2017-04-06/lustre-reviews-el7-x86_64--review-dne-part-2--1_7_1__46127__-70343306542720-064500/sanity-hsm.test_1b.debug_log.$(hostname -s).1491490192.log; [15421.018606] dmesg > /logdir/test_logs/2017-04-06/lustre-reviews-el [15436.469422] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15436.881909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 14:50:07 \(1491490207\) [15437.044540] Lustre: DEBUG MARKER: == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 14:50:07 (1491490207) [15437.296087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15438.174167] Lustre: DEBUG MARKER: rc=0; [15438.174167] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15438.174167] if [[ $? -eq 0 && $val -ne 0 ]]; then [15438.174167] echo $(hostname -s): $val; [15438.174167] rc=$val; [15438.174167] fi; [15438.174167] exit $rc [15438.785915] Lustre: DEBUG MARKER: dmesg [15439.362440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 14:50:10 \(1491490210\) [15439.515511] Lustre: DEBUG MARKER: == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 14:50:10 (1491490210) [15440.045901] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15441.522670] Lustre: DEBUG MARKER: rc=0; [15441.522670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15441.522670] if [[ $? -eq 0 && $val -ne 0 ]]; then [15441.522670] echo $(hostname -s): $val; [15441.522670] rc=$val; [15441.522670] fi; [15441.522670] exit $rc [15442.147465] Lustre: DEBUG MARKER: dmesg [15442.730140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 4: Useless cancel must not be registered ========================================== 14:50:13 \(1491490213\) [15442.883134] Lustre: DEBUG MARKER: == sanity-hsm test 4: Useless cancel must not be registered ========================================== 14:50:13 (1491490213) [15443.317415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [15443.624517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15444.527059] Lustre: DEBUG MARKER: rc=0; [15444.527059] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15444.527059] if [[ $? -eq 0 && $val -ne 0 ]]; then [15444.527059] echo $(hostname -s): $val; [15444.527059] rc=$val; [15444.527059] fi; [15444.527059] exit $rc [15445.128100] Lustre: DEBUG MARKER: dmesg [15445.710875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 8: Test default archive number ==================================================== 14:50:16 \(1491490216\) [15445.869450] Lustre: DEBUG MARKER: == sanity-hsm test 8: Test default archive number ==================================================== 14:50:16 (1491490216) [15446.818511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15447.100805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15450.043700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15450.301573] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15451.078998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15451.336607] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15452.113998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15452.371331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15453.632691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15454.898551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15456.161202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15457.424034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15457.684718] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15458.718422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15458.975489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15460.236415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15461.493949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15461.754959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15462.779149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15463.037547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15463.809084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15464.067226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15465.402762] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15466.303984] Lustre: DEBUG MARKER: rc=0; [15466.303984] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15466.303984] if [[ $? -eq 0 && $val -ne 0 ]]; then [15466.303984] echo $(hostname -s): $val; [15466.303984] rc=$val; [15466.303984] fi; [15466.303984] exit $rc [15466.914431] Lustre: DEBUG MARKER: dmesg [15467.490439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 14:50:38 \(1491490238\) [15467.638357] Lustre: DEBUG MARKER: == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 14:50:38 (1491490238) [15470.312213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15470.573148] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15471.343417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15471.600205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15472.375400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15472.642125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15473.901040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15475.171113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15476.430619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15477.691948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15477.952316] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15478.980032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15479.235427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15479.491251] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15480.527105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15480.783789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15481.555022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15481.809225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15492.304385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15492.476548] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15493.113794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre [15493.274527] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre [15493.422161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 425652db-3402-9d08-e07d-5c3c3d135eb9 [15493.937437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 425652db-3402-9d08-e07d-5c3c3d135eb9 [15494.484233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15494.758644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15497.567440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15497.842430] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15498.623925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15498.894711] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15499.675637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15499.934859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15501.195909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15502.459742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15503.722121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15504.986798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15505.248699] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15506.273926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15506.530888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15506.789192] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15507.811534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15508.069255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15508.836620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15509.093212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15510.399971] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15511.317083] Lustre: DEBUG MARKER: rc=0; [15511.317083] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15511.317083] if [[ $? -eq 0 && $val -ne 0 ]]; then [15511.317083] echo $(hostname -s): $val; [15511.317083] rc=$val; [15511.317083] fi; [15511.317083] exit $rc [15511.907844] Lustre: DEBUG MARKER: dmesg [15512.456763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9a: Multiple remote agents ======================================================== 14:51:23 \(1491490283\) [15512.614105] Lustre: DEBUG MARKER: == sanity-hsm test 9a: Multiple remote agents ======================================================== 14:51:23 (1491490283) [15512.801657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [15512.964051] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [15513.149649] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15514.053747] Lustre: DEBUG MARKER: rc=0; [15514.053747] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15514.053747] if [[ $? -eq 0 && $val -ne 0 ]]; then [15514.053747] echo $(hostname -s): $val; [15514.053747] rc=$val; [15514.053747] fi; [15514.053747] exit $rc [15514.650382] Lustre: DEBUG MARKER: dmesg [15515.230476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10a: Archive a file =============================================================== 14:51:26 \(1491490286\) [15515.401459] Lustre: DEBUG MARKER: == sanity-hsm test 10a: Archive a file =============================================================== 14:51:26 (1491490286) [15516.316721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15516.577233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15520.131457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15520.388699] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15521.176803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15521.437698] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15522.209540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15522.483402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15523.741427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15524.999124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15526.261077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15526.518720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15527.576092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15527.842401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15528.116686] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15529.155414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15529.430760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15530.223130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15530.497914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15531.824751] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15532.707508] Lustre: DEBUG MARKER: rc=0; [15532.707508] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15532.707508] if [[ $? -eq 0 && $val -ne 0 ]]; then [15532.707508] echo $(hostname -s): $val; [15532.707508] rc=$val; [15532.707508] fi; [15532.707508] exit $rc [15533.307871] Lustre: DEBUG MARKER: dmesg [15533.890193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 14:51:44 \(1491490304\) [15534.046854] Lustre: DEBUG MARKER: == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 14:51:44 (1491490304) [15534.960491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15535.217669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15535.484459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {n++}; END {print n}' [15538.290241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15538.549460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15539.321842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15539.579511] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15540.350663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15540.618678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15541.874789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15543.133287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15544.399612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15545.655960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15545.914916] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15548.200722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15548.460503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15548.720461] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15549.752967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15550.019611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15550.790303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15551.050325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15552.374324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15553.255320] Lustre: DEBUG MARKER: rc=0; [15553.255320] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15553.255320] if [[ $? -eq 0 && $val -ne 0 ]]; then [15553.255320] echo $(hostname -s): $val; [15553.255320] rc=$val; [15553.255320] fi; [15553.255320] exit $rc [15553.865668] Lustre: DEBUG MARKER: dmesg [15554.432555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10c: Check forbidden archive ====================================================== 14:52:05 \(1491490325\) [15554.587811] Lustre: DEBUG MARKER: == sanity-hsm test 10c: Check forbidden archive ====================================================== 14:52:05 (1491490325) [15558.034348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15558.292738] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15559.086004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15559.344661] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15560.120042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15560.388450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15561.652553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15562.914776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15564.171738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15565.426779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15566.683660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15566.945893] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15569.224105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15569.479731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15569.735663] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15570.755188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15571.011156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15571.778533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15572.036647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15573.370419] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15574.284268] Lustre: DEBUG MARKER: rc=0; [15574.284268] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15574.284268] if [[ $? -eq 0 && $val -ne 0 ]]; then [15574.284268] echo $(hostname -s): $val; [15574.284268] rc=$val; [15574.284268] fi; [15574.284268] exit $rc [15574.891187] Lustre: DEBUG MARKER: dmesg [15575.454187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 14:52:26 \(1491490346\) [15575.622237] Lustre: DEBUG MARKER: == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 14:52:26 (1491490346) [15576.532995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15576.793374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15577.064067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [15579.868448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15580.130935] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15580.904210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15581.162766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15581.947783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15582.222691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15583.481655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15584.744945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15586.003222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15587.265127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15587.523085] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15589.803689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15590.059629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15590.316666] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15591.340645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15591.597520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15592.382206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15592.641865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15593.973104] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15594.878684] Lustre: DEBUG MARKER: rc=0; [15594.878684] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15594.878684] if [[ $? -eq 0 && $val -ne 0 ]]; then [15594.878684] echo $(hostname -s): $val; [15594.878684] rc=$val; [15594.878684] fi; [15594.878684] exit $rc [15595.472704] Lustre: DEBUG MARKER: dmesg [15596.040966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11a: Import a file ================================================================ 14:52:47 \(1491490367\) [15596.204799] Lustre: DEBUG MARKER: == sanity-hsm test 11a: Import a file ================================================================ 14:52:47 (1491490367) [15597.691606] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15598.582603] Lustre: DEBUG MARKER: rc=0; [15598.582603] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15598.582603] if [[ $? -eq 0 && $val -ne 0 ]]; then [15598.582603] echo $(hostname -s): $val; [15598.582603] rc=$val; [15598.582603] fi; [15598.582603] exit $rc [15599.187705] Lustre: DEBUG MARKER: dmesg [15599.763045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 14:52:50 \(1491490370\) [15599.946895] Lustre: DEBUG MARKER: == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 14:52:50 (1491490370) [15600.866833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15601.125455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15604.283882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15604.539030] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15605.310491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15605.571083] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15606.354720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15606.610758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15607.870849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15609.128922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15610.385155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15611.648172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15611.907915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15612.934161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15613.191615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15613.459959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15614.495264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15614.760023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15615.537614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15615.797441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15617.143708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15618.031606] Lustre: DEBUG MARKER: rc=0; [15618.031606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15618.031606] if [[ $? -eq 0 && $val -ne 0 ]]; then [15618.031606] echo $(hostname -s): $val; [15618.031606] rc=$val; [15618.031606] fi; [15618.031606] exit $rc [15618.634254] Lustre: DEBUG MARKER: dmesg [15619.317920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 14:53:10 \(1491490390\) [15619.482747] Lustre: DEBUG MARKER: == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 14:53:10 (1491490390) [15621.128201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15621.388755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15624.453421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15624.712170] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15625.485960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15625.743836] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15626.513464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15626.774227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15628.035776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15629.297361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15630.569190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15631.826684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15632.094252] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15634.397321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15634.658665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15634.923090] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15635.949345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15636.206236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15636.974275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15637.230289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15638.550231] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15639.434229] Lustre: DEBUG MARKER: rc=0; [15639.434229] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15639.434229] if [[ $? -eq 0 && $val -ne 0 ]]; then [15639.434229] echo $(hostname -s): $val; [15639.434229] rc=$val; [15639.434229] fi; [15639.434229] exit $rc [15640.053457] Lustre: DEBUG MARKER: dmesg [15640.648298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 14:53:31 \(1491490411\) [15640.798063] Lustre: DEBUG MARKER: == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 14:53:31 (1491490411) [15645.308381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15645.583048] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15646.372456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15646.629502] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15647.398521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15647.656697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15648.914863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15650.178172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15651.442073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15652.698481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15652.959177] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15655.264167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15655.525747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15655.786080] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15656.845929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15657.120896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15657.900769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15658.171085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15659.507453] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15660.414658] Lustre: DEBUG MARKER: rc=0; [15660.414658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15660.414658] if [[ $? -eq 0 && $val -ne 0 ]]; then [15660.414658] echo $(hostname -s): $val; [15660.414658] rc=$val; [15660.414658] fi; [15660.414658] exit $rc [15661.010441] Lustre: DEBUG MARKER: dmesg [15661.593608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 14:53:52 \(1491490432\) [15661.745993] Lustre: DEBUG MARKER: == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 14:53:52 (1491490432) [15663.046724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15663.311775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15664.568576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15665.830746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15667.098199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15668.360119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15676.356437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15676.614584] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15677.386649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15677.645401] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15678.421789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15678.677063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15679.940329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15681.207125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15682.472972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15683.732442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15683.988466] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15686.275397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15686.533717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15686.792962] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15687.815925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15688.073507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15688.838870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15689.100316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15690.424838] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15691.306796] Lustre: DEBUG MARKER: rc=0; [15691.306796] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15691.306796] if [[ $? -eq 0 && $val -ne 0 ]]; then [15691.306796] echo $(hostname -s): $val; [15691.306796] rc=$val; [15691.306796] fi; [15691.306796] exit $rc [15691.899912] Lustre: DEBUG MARKER: dmesg [15692.605543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 14:54:23 \(1491490463\) [15692.774323] Lustre: DEBUG MARKER: == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 14:54:23 (1491490463) [15693.711537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [15693.980822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15694.240704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15694.508654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [15697.328175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15697.589995] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15698.359177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15698.618375] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15699.388200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15699.641436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15700.898035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15702.156632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15703.423384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15704.682873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15704.946888] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15707.234087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15707.493967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15707.747647] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15708.783729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15709.041140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15709.808475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15710.068309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15711.395129] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15712.302596] Lustre: DEBUG MARKER: rc=0; [15712.302596] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15712.302596] if [[ $? -eq 0 && $val -ne 0 ]]; then [15712.302596] echo $(hostname -s): $val; [15712.302596] rc=$val; [15712.302596] fi; [15712.302596] exit $rc [15712.911511] Lustre: DEBUG MARKER: dmesg [15713.492925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12e: Check forbidden restore ====================================================== 14:54:44 \(1491490484\) [15713.641833] Lustre: DEBUG MARKER: == sanity-hsm test 12e: Check forbidden restore ====================================================== 14:54:44 (1491490484) [15714.561125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15714.824980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15717.691843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15717.950409] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15718.738985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15719.000340] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15719.781399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15720.055513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15721.328267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15722.585945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15723.847997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15725.107804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15725.364612] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15727.641629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15727.898667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15728.157063] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15729.179508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15729.441915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15730.228844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15730.496237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15731.812631] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15732.692114] Lustre: DEBUG MARKER: rc=0; [15732.692114] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15732.692114] if [[ $? -eq 0 && $val -ne 0 ]]; then [15732.692114] echo $(hostname -s): $val; [15732.692114] rc=$val; [15732.692114] fi; [15732.692114] exit $rc [15733.291742] Lustre: DEBUG MARKER: dmesg [15733.854951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12f: Restore a released file explicitly =========================================== 14:55:04 \(1491490504\) [15734.017196] Lustre: DEBUG MARKER: == sanity-hsm test 12f: Restore a released file explicitly =========================================== 14:55:04 (1491490504) [15734.921661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15735.183090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15735.456351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15735.714779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15738.555705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15738.817517] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15739.607619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15739.863767] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15740.632697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15740.890876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15742.148366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15743.411649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15744.675288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15745.941321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15746.200432] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15748.494559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15748.749803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15749.005965] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15750.051875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15750.309111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15751.073218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15751.333646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15752.646706] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15753.524920] Lustre: DEBUG MARKER: rc=0; [15753.524920] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15753.524920] if [[ $? -eq 0 && $val -ne 0 ]]; then [15753.524920] echo $(hostname -s): $val; [15753.524920] rc=$val; [15753.524920] fi; [15753.524920] exit $rc [15754.121853] Lustre: DEBUG MARKER: dmesg [15754.702449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12g: Restore a released file implicitly =========================================== 14:55:25 \(1491490525\) [15754.858513] Lustre: DEBUG MARKER: == sanity-hsm test 12g: Restore a released file implicitly =========================================== 14:55:25 (1491490525) [15755.775411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15756.053440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15756.419783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15756.684168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15759.488417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15759.750800] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15760.524344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15760.785174] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15761.561820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15761.819528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15763.079260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15764.336105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15765.597520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15766.856945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15767.114311] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15769.395307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15769.667038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15769.927953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15770.942886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15771.202248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15771.977160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15772.237119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15773.557314] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15774.453494] Lustre: DEBUG MARKER: rc=0; [15774.453494] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15774.453494] if [[ $? -eq 0 && $val -ne 0 ]]; then [15774.453494] echo $(hostname -s): $val; [15774.453494] rc=$val; [15774.453494] fi; [15774.453494] exit $rc [15775.059425] Lustre: DEBUG MARKER: dmesg [15775.641423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 14:55:46 \(1491490546\) [15775.799902] Lustre: DEBUG MARKER: == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 14:55:46 (1491490546) [15776.718291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15776.988269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15777.573951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15777.835681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15780.629320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15780.888173] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15781.661407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15781.919182] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15782.688151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15782.947270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15784.208261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15785.465768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15786.723199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15787.981796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15788.237963] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15790.514352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15790.775618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15791.034368] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15792.056415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15792.312911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15793.079659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15793.335318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15794.653588] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15795.529252] Lustre: DEBUG MARKER: rc=0; [15795.529252] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15795.529252] if [[ $? -eq 0 && $val -ne 0 ]]; then [15795.529252] echo $(hostname -s): $val; [15795.529252] rc=$val; [15795.529252] fi; [15795.529252] exit $rc [15796.154101] Lustre: DEBUG MARKER: dmesg [15796.763947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12m: Archive\/release\/implicit restore ============================================= 14:56:07 \(1491490567\) [15796.930798] Lustre: DEBUG MARKER: == sanity-hsm test 12m: Archive/release/implicit restore ============================================= 14:56:07 (1491490567) [15797.849596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15798.127751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15801.027423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15801.286920] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15802.056342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15802.312068] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15803.086184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15803.347299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15804.610606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15805.869101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15807.126977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15808.392155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15808.652044] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15810.933230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15811.191396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15811.452395] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15812.478588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15812.752715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15813.532172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15813.804530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15815.139522] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15816.042859] Lustre: DEBUG MARKER: rc=0; [15816.042859] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15816.042859] if [[ $? -eq 0 && $val -ne 0 ]]; then [15816.042859] echo $(hostname -s): $val; [15816.042859] rc=$val; [15816.042859] fi; [15816.042859] exit $rc [15816.670303] Lustre: DEBUG MARKER: dmesg [15817.233570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12n: Import\/implicit restore\/release ============================================== 14:56:28 \(1491490588\) [15817.400375] Lustre: DEBUG MARKER: == sanity-hsm test 12n: Import/implicit restore/release ============================================== 14:56:28 (1491490588) [15821.888070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15822.146293] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15822.915854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15823.173551] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15823.947902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15824.203589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15825.460986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15826.717842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15827.972532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15829.235108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15829.493519] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15831.787161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15832.042736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15832.300576] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15833.322123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15833.591091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15834.363899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15834.623398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15835.954191] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15836.835826] Lustre: DEBUG MARKER: rc=0; [15836.835826] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15836.835826] if [[ $? -eq 0 && $val -ne 0 ]]; then [15836.835826] echo $(hostname -s): $val; [15836.835826] rc=$val; [15836.835826] fi; [15836.835826] exit $rc [15837.432304] Lustre: DEBUG MARKER: dmesg [15838.013301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 14:56:48 \(1491490608\) [15838.167195] Lustre: DEBUG MARKER: == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 14:56:48 (1491490608) [15839.077708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15839.336687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15839.605224] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x152 [15839.866343] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15840.377420] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15840.834156] Lustre: *** cfs_fail_loc=152, val=0*** [15840.964108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15841.225802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15841.487989] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15842.002935] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15842.527783] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [15842.786132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [15843.308300] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [15843.836055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15844.090528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15844.865980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15845.124275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15845.886367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15856.235116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15856.498355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15859.306521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15859.564302] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15860.331107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15860.588053] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15861.357390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15861.614796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15862.875161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15864.131403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15865.392351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15866.651476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15866.908741] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15867.939759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15868.197554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15868.461455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15869.509098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15869.784093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15870.547559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15870.806459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15872.133398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15873.050152] Lustre: DEBUG MARKER: rc=0; [15873.050152] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15873.050152] if [[ $? -eq 0 && $val -ne 0 ]]; then [15873.050152] echo $(hostname -s): $val; [15873.050152] rc=$val; [15873.050152] fi; [15873.050152] exit $rc [15873.660221] Lustre: DEBUG MARKER: dmesg [15874.231162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 14:57:25 \(1491490645\) [15874.402452] Lustre: DEBUG MARKER: == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 14:57:25 (1491490645) [15875.306882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15875.567149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15879.360963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15879.622510] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15880.398005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15880.659050] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15881.426234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15881.687314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15882.948406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15884.203371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15885.468905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15886.733271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15886.997623] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15889.287118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15889.551197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15889.809930] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15890.842844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15891.099772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15891.871257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15892.126379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15893.446042] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15894.334105] Lustre: DEBUG MARKER: rc=0; [15894.334105] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15894.334105] if [[ $? -eq 0 && $val -ne 0 ]]; then [15894.334105] echo $(hostname -s): $val; [15894.334105] rc=$val; [15894.334105] fi; [15894.334105] exit $rc [15894.934100] Lustre: DEBUG MARKER: dmesg [15895.548564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 14:57:46 \(1491490666\) [15895.715335] Lustre: DEBUG MARKER: == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 14:57:46 (1491490666) [15895.855350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15895.855350] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15895.855350] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15896.517829] Lustre: lustre-MDT0000: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [15896.520078] Lustre: Skipped 82 previous similar messages [15897.144304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15897.305930] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15898.434692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15898.696247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15899.958889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15901.228388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15918.701169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15918.959786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15919.735394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15919.997746] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15920.769115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15921.043830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15922.314103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15923.571647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15924.832597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15926.088330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15926.347233] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15929.884284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15930.138318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15930.396909] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15931.422547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15931.676823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15932.441981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15932.699099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15934.923057] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15935.847103] Lustre: DEBUG MARKER: rc=0; [15935.847103] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15935.847103] if [[ $? -eq 0 && $val -ne 0 ]]; then [15935.847103] echo $(hostname -s): $val; [15935.847103] rc=$val; [15935.847103] fi; [15935.847103] exit $rc [15936.456859] Lustre: DEBUG MARKER: dmesg [15937.034730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 13: Recursively import and restore a directory ==================================== 14:58:28 \(1491490708\) [15937.194382] Lustre: DEBUG MARKER: == sanity-hsm test 13: Recursively import and restore a directory ==================================== 14:58:28 (1491490708) [15972.154697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15972.415633] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15973.191996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15973.452267] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15974.220387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15974.476933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15975.738754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15977.010400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15978.277186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15979.533211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15979.791775] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15980.847847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15981.112372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15981.372715] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15982.407504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15982.667329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15983.437618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15983.694806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15985.035510] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15985.938186] Lustre: DEBUG MARKER: rc=0; [15985.938186] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15985.938186] if [[ $? -eq 0 && $val -ne 0 ]]; then [15985.938186] echo $(hostname -s): $val; [15985.938186] rc=$val; [15985.938186] fi; [15985.938186] exit $rc [15986.544682] Lustre: DEBUG MARKER: dmesg [15987.091112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 14:59:18 \(1491490758\) [15987.238887] Lustre: DEBUG MARKER: == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 14:59:18 (1491490758) [15988.425238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15988.686352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15989.950300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15991.212685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15996.431478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15996.691715] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15997.466754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15997.727364] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15998.501968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15998.768921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16000.029881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16001.290869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16002.549194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16003.806437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16004.063353] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16005.113142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16005.369355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16005.631156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16006.659096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16006.913350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16007.680912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16007.938203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16009.273360] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16010.162348] Lustre: DEBUG MARKER: rc=0; [16010.162348] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16010.162348] if [[ $? -eq 0 && $val -ne 0 ]]; then [16010.162348] echo $(hostname -s): $val; [16010.162348] rc=$val; [16010.162348] fi; [16010.162348] exit $rc [16010.761452] Lustre: DEBUG MARKER: dmesg [16011.341924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 15: Rebind a list of files ======================================================== 14:59:42 \(1491490782\) [16011.496756] Lustre: DEBUG MARKER: == sanity-hsm test 15: Rebind a list of files ======================================================== 14:59:42 (1491490782) [16013.989465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [16014.249629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [16015.508150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [16016.765041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [16018.027229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [16032.094207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16032.352898] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16033.120440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16033.378301] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16034.144149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16034.402066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16035.658202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16036.913974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16038.169715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16039.425081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16039.683025] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16040.716355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16040.971326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16041.228679] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16042.300975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16042.558197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16043.332976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16043.590004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16044.918906] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16045.806384] Lustre: DEBUG MARKER: rc=0; [16045.806384] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16045.806384] if [[ $? -eq 0 && $val -ne 0 ]]; then [16045.806384] echo $(hostname -s): $val; [16045.806384] rc=$val; [16045.806384] fi; [16045.806384] exit $rc [16046.407171] Lustre: DEBUG MARKER: dmesg [16046.975202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 16: Test CT bandwith control option =============================================== 15:00:17 \(1491490817\) [16047.136265] Lustre: DEBUG MARKER: == sanity-hsm test 16: Test CT bandwith control option =============================================== 15:00:17 (1491490817) [16048.519227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16048.783770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16050.045144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16051.304928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16052.567273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16053.835280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16055.097348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16056.358204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16057.623950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16058.893316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16060.154885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16061.417078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16062.682564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16063.944826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16065.213284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16066.484594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16067.749824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16069.015387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16071.811960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16072.070395] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16072.841295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16073.101888] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16073.872779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16074.141918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16075.413751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16076.683109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16077.948507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16079.215268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16079.474138] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16081.760854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16082.018232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16082.277644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16083.313198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16083.571903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16084.350934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16084.607329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16085.933327] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16086.830301] Lustre: DEBUG MARKER: rc=0; [16086.830301] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16086.830301] if [[ $? -eq 0 && $val -ne 0 ]]; then [16086.830301] echo $(hostname -s): $val; [16086.830301] rc=$val; [16086.830301] fi; [16086.830301] exit $rc [16087.440109] Lustre: DEBUG MARKER: dmesg [16088.090319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 20: Release is not permitted ====================================================== 15:00:59 \(1491490859\) [16088.239126] Lustre: DEBUG MARKER: == sanity-hsm test 20: Release is not permitted ====================================================== 15:00:59 (1491490859) [16088.478135] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16089.371495] Lustre: DEBUG MARKER: rc=0; [16089.371495] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16089.371495] if [[ $? -eq 0 && $val -ne 0 ]]; then [16089.371495] echo $(hostname -s): $val; [16089.371495] rc=$val; [16089.371495] fi; [16089.371495] exit $rc [16089.975168] Lustre: DEBUG MARKER: dmesg [16090.583920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 21: Simple release tests ========================================================== 15:01:01 \(1491490861\) [16090.743316] Lustre: DEBUG MARKER: == sanity-hsm test 21: Simple release tests ========================================================== 15:01:01 (1491490861) [16091.891778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [16092.203955] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [16092.561803] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [16092.885246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16093.163604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16094.438158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16095.706123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16096.035648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x46:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16096.298357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x46:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16096.649929] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [16097.234842] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [16100.069520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16100.329117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16101.116727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16101.377315] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16102.146506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16102.403196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16103.662300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16104.920156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16106.184633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16106.446443] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16107.476295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16107.750845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16108.024746] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16109.055359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16109.324116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16110.106130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16110.371522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16111.715346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16112.603236] Lustre: DEBUG MARKER: rc=0; [16112.603236] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16112.603236] if [[ $? -eq 0 && $val -ne 0 ]]; then [16112.603236] echo $(hostname -s): $val; [16112.603236] rc=$val; [16112.603236] fi; [16112.603236] exit $rc [16113.210580] Lustre: DEBUG MARKER: dmesg [16113.799702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 22: Could not swap a release file ================================================= 15:01:24 \(1491490884\) [16113.981332] Lustre: DEBUG MARKER: == sanity-hsm test 22: Could not swap a release file ================================================= 15:01:24 (1491490884) [16115.127713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16115.404153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16116.667043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16117.929507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16120.970582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16121.245543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16122.036073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16122.304051] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16123.082112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16123.341359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16124.603430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16125.866729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16127.131713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16127.392357] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16128.412280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16128.672616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16128.934459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16129.966602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16130.239132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16131.016566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16131.275633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16132.637202] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16133.545311] Lustre: DEBUG MARKER: rc=0; [16133.545311] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16133.545311] if [[ $? -eq 0 && $val -ne 0 ]]; then [16133.545311] echo $(hostname -s): $val; [16133.545311] rc=$val; [16133.545311] fi; [16133.545311] exit $rc [16134.156986] Lustre: DEBUG MARKER: dmesg [16134.739125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 23: Release does not change a\/mtime \(utime\) ======================================= 15:01:45 \(1491490905\) [16134.898836] Lustre: DEBUG MARKER: == sanity-hsm test 23: Release does not change a/mtime (utime) ======================================= 15:01:45 (1491490905) [16136.028288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16136.290146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16137.551621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16138.811841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16141.644954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16141.921621] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16142.706194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16142.963661] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16143.743759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16144.001237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16145.261085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16146.519153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16147.786075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16149.042381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16149.302212] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16150.346758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16150.606728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16150.863973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16151.889838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16152.148420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16152.923276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16153.184877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16154.536524] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16155.431340] Lustre: DEBUG MARKER: rc=0; [16155.431340] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16155.431340] if [[ $? -eq 0 && $val -ne 0 ]]; then [16155.431340] echo $(hostname -s): $val; [16155.431340] rc=$val; [16155.431340] fi; [16155.431340] exit $rc [16156.029272] Lustre: DEBUG MARKER: dmesg [16156.608792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24a: Archive, release, and restore does not change a\/mtime \(i\/o\) ================== 15:02:07 \(1491490927\) [16156.765286] Lustre: DEBUG MARKER: == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ================== 15:02:07 (1491490927) [16158.914554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16159.177890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16160.435960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16161.701533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16161.991047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16162.252617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16163.514696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16164.777100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16167.573448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16167.830156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16168.610146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16168.866966] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16169.635910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16169.893822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16171.151956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16172.411953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16173.684288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16174.939961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16175.196639] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16176.215785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16176.474690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16176.732352] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16177.756808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16178.014139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16178.784343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16179.041673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16180.584830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16181.494286] Lustre: DEBUG MARKER: rc=0; [16181.494286] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16181.494286] if [[ $? -eq 0 && $val -ne 0 ]]; then [16181.494286] echo $(hostname -s): $val; [16181.494286] rc=$val; [16181.494286] fi; [16181.494286] exit $rc [16182.100727] Lustre: DEBUG MARKER: dmesg [16182.677065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 15:02:33 \(1491490953\) [16182.835241] Lustre: DEBUG MARKER: == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 15:02:33 (1491490953) [16184.043827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16184.307232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16185.568147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16186.830685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16187.112256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16187.370925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16188.632816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16189.896723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16194.828557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16195.085246] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16195.863813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16196.120442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16196.891764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16197.147373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16198.413403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16199.684929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16200.949720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16202.208847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16202.472139] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16203.500155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16203.755224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16204.009473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16205.043448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16205.317718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16206.093442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16206.349592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16207.671132] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16208.561640] Lustre: DEBUG MARKER: rc=0; [16208.561640] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16208.561640] if [[ $? -eq 0 && $val -ne 0 ]]; then [16208.561640] echo $(hostname -s): $val; [16208.561640] rc=$val; [16208.561640] fi; [16208.561640] exit $rc [16209.166983] Lustre: DEBUG MARKER: dmesg [16209.746609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24c: check that user,group,other request masks work =============================== 15:03:00 \(1491490980\) [16209.895562] Lustre: DEBUG MARKER: == sanity-hsm test 24c: check that user,group,other request masks work =============================== 15:03:00 (1491490980) [16210.787588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.user_request_mask [16211.046325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.group_request_mask [16211.306908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.other_request_mask [16211.800226] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=archive [16212.312603] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.user_request_mask=archive [16213.080365] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=archive [16213.604640] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.group_request_mask=archive [16214.344770] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=archive [16214.869786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.other_request_mask=archive [16217.952234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16218.210737] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16219.006363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16219.262945] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16220.049211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16220.306547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16221.562409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16222.818528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16224.082872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16225.341430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16225.598506] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16226.649143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16226.906587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16227.164943] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16228.203860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16228.461956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16229.240901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16229.500518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16230.762361] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=RESTORE [16231.278788] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.user_request_mask=RESTORE [16231.809745] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=RESTORE [16232.331712] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.group_request_mask=RESTORE [16232.866128] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=RESTORE [16233.397850] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.other_request_mask=RESTORE [16233.974683] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16234.888170] Lustre: DEBUG MARKER: rc=0; [16234.888170] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16234.888170] if [[ $? -eq 0 && $val -ne 0 ]]; then [16234.888170] echo $(hostname -s): $val; [16234.888170] rc=$val; [16234.888170] fi; [16234.888170] exit $rc [16235.495379] Lustre: DEBUG MARKER: dmesg [16236.064357] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 15:03:27 \(1491491007\) [16236.230141] Lustre: DEBUG MARKER: == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 15:03:27 (1491491007) [16237.619424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16237.779613] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16238.725226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16238.996575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16240.258650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16240.534728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16240.801380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16242.064550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16243.332094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16248.252183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16248.511627] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16249.282091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16249.540260] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16250.326406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16250.600778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16251.866829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16253.128367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16254.389169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16255.645164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16255.902640] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16256.935353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16257.192028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16257.452587] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16258.491617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16258.762485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16259.532299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16259.789475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16261.950324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16262.847824] Lustre: DEBUG MARKER: rc=0; [16262.847824] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16262.847824] if [[ $? -eq 0 && $val -ne 0 ]]; then [16262.847824] echo $(hostname -s): $val; [16262.847824] rc=$val; [16262.847824] fi; [16262.847824] exit $rc [16263.460395] Lustre: DEBUG MARKER: dmesg [16264.053619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 15:03:55 \(1491491035\) [16264.209453] Lustre: DEBUG MARKER: == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 15:03:55 (1491491035) [16265.358140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16265.619431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16266.881568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16268.143535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16273.064817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16273.321342] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16274.099277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16274.358044] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16275.130191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16275.390962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16276.649476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16277.905891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16279.162581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16280.420256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16280.676763] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16281.711684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16281.985248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16282.244995] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16283.269934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16283.528061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16284.305454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16284.562301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16285.919790] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16286.833607] Lustre: DEBUG MARKER: rc=0; [16286.833607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16286.833607] if [[ $? -eq 0 && $val -ne 0 ]]; then [16286.833607] echo $(hostname -s): $val; [16286.833607] rc=$val; [16286.833607] fi; [16286.833607] exit $rc [16287.443400] Lustre: DEBUG MARKER: dmesg [16288.028241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 15:04:19 \(1491491059\) [16288.179224] Lustre: DEBUG MARKER: == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 15:04:19 (1491491059) [16289.112528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16289.373017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16292.304431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16292.562398] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16293.336617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16293.596563] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16294.377791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16294.636942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16295.901513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16297.164611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16298.421698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16299.677975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16299.936049] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16302.224639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16302.482294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16302.742466] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16303.776948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16304.031097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16304.806744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16305.067199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16306.388709] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16307.298747] Lustre: DEBUG MARKER: rc=0; [16307.298747] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16307.298747] if [[ $? -eq 0 && $val -ne 0 ]]; then [16307.298747] echo $(hostname -s): $val; [16307.298747] rc=$val; [16307.298747] fi; [16307.298747] exit $rc [16307.909272] Lustre: DEBUG MARKER: dmesg [16308.478294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25a: Restore lost file \(HS_LOST flag\) from import ================================= 15:04:39 \(1491491079\) [16308.634292] Lustre: DEBUG MARKER: == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ================================= 15:04:39 (1491491079) [16312.809034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16313.065555] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16313.840540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16314.101303] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16314.880166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16315.137599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16316.394173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16317.653126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16318.914904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16320.177209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16320.441507] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16322.749585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16323.009650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16323.266464] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16324.300859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16324.558753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16325.357432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16325.614332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16326.938344] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16327.837657] Lustre: DEBUG MARKER: rc=0; [16327.837657] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16327.837657] if [[ $? -eq 0 && $val -ne 0 ]]; then [16327.837657] echo $(hostname -s): $val; [16327.837657] rc=$val; [16327.837657] fi; [16327.837657] exit $rc [16328.438765] Lustre: DEBUG MARKER: dmesg [16329.031684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25b: Restore lost file \(HS_LOST flag\) after release =============================== 15:05:00 \(1491491100\) [16329.202642] Lustre: DEBUG MARKER: == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release =============================== 15:05:00 (1491491100) [16330.130162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16330.387836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16333.200399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16333.458384] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16334.238236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16334.496956] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16335.265004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16335.522589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16336.784231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16338.044266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16339.309333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16340.569826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16340.831083] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16343.114317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16343.371812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16343.630182] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16344.660977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16344.918076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16345.687907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16345.943469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16347.273273] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16348.164176] Lustre: DEBUG MARKER: rc=0; [16348.164176] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16348.164176] if [[ $? -eq 0 && $val -ne 0 ]]; then [16348.164176] echo $(hostname -s): $val; [16348.164176] rc=$val; [16348.164176] fi; [16348.164176] exit $rc [16348.769375] Lustre: DEBUG MARKER: dmesg [16349.349838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 26: Remove the archive of a valid file ============================================ 15:05:20 \(1491491120\) [16349.501936] Lustre: DEBUG MARKER: == sanity-hsm test 26: Remove the archive of a valid file ============================================ 15:05:20 (1491491120) [16351.452412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16351.712550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16352.970937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16354.230846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16355.491714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16356.759761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16358.026862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16359.287615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16360.552590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16361.811418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16363.069722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16364.335819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16365.611439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16366.887923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16368.159447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16369.418638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16370.681460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16371.943310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16373.208330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16374.474366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16375.736682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16377.002658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16378.266708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16379.528626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16380.790032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16382.052288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16383.314318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16384.581208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16385.849715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16387.117751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16388.388602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16388.658397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16388.924656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16391.733988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16392.002957] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16392.785401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16393.045133] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16393.817791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16394.077254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16395.339470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16396.605399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16397.866249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16399.126809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16399.382702] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16400.417781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16400.674548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16400.933512] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16401.965225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16402.225233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16403.019457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16403.276678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16404.598902] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16405.494032] Lustre: DEBUG MARKER: rc=0; [16405.494032] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16405.494032] if [[ $? -eq 0 && $val -ne 0 ]]; then [16405.494032] echo $(hostname -s): $val; [16405.494032] rc=$val; [16405.494032] fi; [16405.494032] exit $rc [16406.109165] Lustre: DEBUG MARKER: dmesg [16406.682642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27a: Remove the archive of an imported file \(Operation not permitted\) ============= 15:06:17 \(1491491177\) [16406.847439] Lustre: DEBUG MARKER: == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ============= 15:06:17 (1491491177) [16413.368824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16413.637548] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16414.455160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16414.714319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16415.488636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16415.749381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16417.012702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16418.270789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16419.531369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16419.788690] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16420.819183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16421.073888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16421.330629] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16422.363341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16422.621092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16423.400837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16423.660659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16425.008154] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16425.898410] Lustre: DEBUG MARKER: rc=0; [16425.898410] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16425.898410] if [[ $? -eq 0 && $val -ne 0 ]]; then [16425.898410] echo $(hostname -s): $val; [16425.898410] rc=$val; [16425.898410] fi; [16425.898410] exit $rc [16426.507783] Lustre: DEBUG MARKER: dmesg [16427.218314] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27b: Remove the archive of a relased file \(Operation not permitted\) =============== 15:06:38 \(1491491198\) [16427.387475] Lustre: DEBUG MARKER: == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) =============== 15:06:38 (1491491198) [16432.392323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16432.652223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16433.913146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16435.173636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16436.448505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16437.725539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16438.994812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16440.263283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16441.525285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16442.784242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16444.044630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16445.312442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16446.578886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16447.844958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16449.108679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16450.371467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16451.634072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16452.897108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16454.161838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16455.430745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16456.692287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16457.952246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16459.212781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16460.476240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16461.743613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16463.011893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16464.283359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16465.551684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16466.814456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16468.080001] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16469.346643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16472.174879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16472.433998] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16473.211715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16473.470388] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16474.250561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16474.523849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16475.784875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16477.046081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16478.309068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16479.574736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16479.833429] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16482.128124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16482.385527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16482.644583] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16483.672282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16483.934860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16484.708234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16484.967328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16486.317392] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16487.219423] Lustre: DEBUG MARKER: rc=0; [16487.219423] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16487.219423] if [[ $? -eq 0 && $val -ne 0 ]]; then [16487.219423] echo $(hostname -s): $val; [16487.219423] rc=$val; [16487.219423] fi; [16487.219423] exit $rc [16487.829263] Lustre: DEBUG MARKER: dmesg [16488.449155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 28: Concurrent archive\/file remove ================================================ 15:07:39 \(1491491259\) [16488.613684] Lustre: DEBUG MARKER: == sanity-hsm test 28: Concurrent archive/file remove ================================================ 15:07:39 (1491491259) [16490.585665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16490.856476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16492.121927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16493.397748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16494.666663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16495.943243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16497.209922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16498.475083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16499.741098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16501.009539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16502.271829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16503.536288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16504.803414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16506.073951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16507.335393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16508.594645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16509.855813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16511.121317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16512.398686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16513.674455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16514.944940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16516.210412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16517.481988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16518.757862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16520.028134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16521.291953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16522.553644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16523.813559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16525.074620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16526.338165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16527.599774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16527.865227] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [16528.389778] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [16528.904758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16529.167061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16529.935744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16530.193446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16530.983735] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16531.510147] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16532.043598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16532.310342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16533.075911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16533.333805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16534.098341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16534.361577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16537.172239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16537.434163] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16538.205779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16538.462787] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16539.237958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16539.498859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16540.757779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16542.021658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16542.279990] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16543.303808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16543.562193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16543.819162] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16544.847172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16545.102532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16545.872777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16546.131919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16547.507831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16548.402093] Lustre: DEBUG MARKER: rc=0; [16548.402093] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16548.402093] if [[ $? -eq 0 && $val -ne 0 ]]; then [16548.402093] echo $(hostname -s): $val; [16548.402093] rc=$val; [16548.402093] fi; [16548.402093] exit $rc [16549.007139] Lustre: DEBUG MARKER: dmesg [16549.581147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 15:08:40 \(1491491320\) [16549.737202] Lustre: DEBUG MARKER: == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 15:08:40 (1491491320) [16553.163366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16553.421544] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16554.230200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16554.501795] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16555.280607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16555.536189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16556.797769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16558.063322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16559.328210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16560.586139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16561.842180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16563.104577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16563.364858] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16564.397278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16564.652738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16564.912728] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16565.935863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16566.193275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16566.971283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16567.227631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16568.546461] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16569.459110] Lustre: DEBUG MARKER: rc=0; [16569.459110] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16569.459110] if [[ $? -eq 0 && $val -ne 0 ]]; then [16569.459110] echo $(hostname -s): $val; [16569.459110] rc=$val; [16569.459110] fi; [16569.459110] exit $rc [16570.062688] Lustre: DEBUG MARKER: dmesg [16570.627771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29b: Archive\/delete\/remove by FID from the archive. =============================== 15:09:01 \(1491491341\) [16570.778093] Lustre: DEBUG MARKER: == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. =============================== 15:09:01 (1491491341) [16571.898145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16572.156720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16573.420406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16574.684977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16574.954111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16575.215700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16578.028211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16578.296521] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16579.066630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16579.325455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16580.095793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16580.351839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16581.613651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16582.872680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16584.128695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16585.388810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16585.663210] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16586.701534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16586.958959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16587.217702] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16588.246120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16588.509714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16589.290742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16589.552800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16590.873725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16591.771140] Lustre: DEBUG MARKER: rc=0; [16591.771140] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16591.771140] if [[ $? -eq 0 && $val -ne 0 ]]; then [16591.771140] echo $(hostname -s): $val; [16591.771140] rc=$val; [16591.771140] fi; [16591.771140] exit $rc [16592.400077] Lustre: DEBUG MARKER: dmesg [16592.961798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29c: Archive\/delete\/remove by FID, using a file list. ============================= 15:09:23 \(1491491363\) [16593.110947] Lustre: DEBUG MARKER: == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ============================= 15:09:23 (1491491363) [16594.641436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16594.900403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16596.161673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16597.421901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16597.687557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16597.945796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16598.208411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16598.469879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16598.748461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16599.008099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16599.270650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16599.530564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16599.794775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16600.056854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16602.892631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16603.154172] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16603.924390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16604.183411] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16604.959463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16605.228047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16606.486438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16607.744476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16609.005874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16609.265593] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16610.295513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16610.553535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16610.811474] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16611.834174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16612.094716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16612.885036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16613.142307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16614.475129] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16615.383428] Lustre: DEBUG MARKER: rc=0; [16615.383428] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16615.383428] if [[ $? -eq 0 && $val -ne 0 ]]; then [16615.383428] echo $(hostname -s): $val; [16615.383428] rc=$val; [16615.383428] fi; [16615.383428] exit $rc [16616.002152] Lustre: DEBUG MARKER: dmesg [16616.572741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 15:09:47 \(1491491387\) [16616.724378] Lustre: DEBUG MARKER: == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 15:09:47 (1491491387) [16616.930495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [16617.090465] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [16617.300966] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16618.208932] Lustre: DEBUG MARKER: rc=0; [16618.208932] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16618.208932] if [[ $? -eq 0 && $val -ne 0 ]]; then [16618.208932] echo $(hostname -s): $val; [16618.208932] rc=$val; [16618.208932] fi; [16618.208932] exit $rc [16618.819635] Lustre: DEBUG MARKER: dmesg [16619.438793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30a: Restore at exec \(import case\) ================================================ 15:09:50 \(1491491390\) [16619.612142] Lustre: DEBUG MARKER: == sanity-hsm test 30a: Restore at exec (import case) ================================================ 15:09:50 (1491491390) [16621.251560] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [16621.770418] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [16622.585659] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [16623.099414] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [16626.163795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16626.422591] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16627.201392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16627.462376] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16628.231919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16628.504711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16629.764837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16631.022974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16632.280096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16633.536992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16633.795439] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16634.820928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16635.077447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16635.335886] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16636.362622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16636.622239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16637.390840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16637.647353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16638.987812] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16639.912434] Lustre: DEBUG MARKER: rc=0; [16639.912434] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16639.912434] if [[ $? -eq 0 && $val -ne 0 ]]; then [16639.912434] echo $(hostname -s): $val; [16639.912434] rc=$val; [16639.912434] fi; [16639.912434] exit $rc [16640.540391] Lustre: DEBUG MARKER: dmesg [16641.100175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30b: Restore at exec \(release case\) =============================================== 15:10:12 \(1491491412\) [16641.268398] Lustre: DEBUG MARKER: == sanity-hsm test 30b: Restore at exec (release case) =============================================== 15:10:12 (1491491412) [16642.202788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16642.464164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16642.737726] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [16643.256523] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [16644.132771] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [16644.645082] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [16647.719530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16647.978501] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16648.755914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16649.019578] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16649.804546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16650.063286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16651.322344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16652.579900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16653.836050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16655.089918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16655.349887] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16656.380305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16656.648802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16656.906557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16657.937908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16658.194980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16658.970478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16659.227959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16660.557613] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16661.444603] Lustre: DEBUG MARKER: rc=0; [16661.444603] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16661.444603] if [[ $? -eq 0 && $val -ne 0 ]]; then [16661.444603] echo $(hostname -s): $val; [16661.444603] rc=$val; [16661.444603] fi; [16661.444603] exit $rc [16662.054921] Lustre: DEBUG MARKER: dmesg [16662.634083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30c: Update during exec of released file must fail ================================ 15:10:33 \(1491491433\) [16662.789712] Lustre: DEBUG MARKER: == sanity-hsm test 30c: Update during exec of released file must fail ================================ 15:10:33 (1491491433) [16663.709721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16663.974193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16664.249455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [16664.762987] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [16675.647078] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [16676.162587] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [16679.218625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16679.474934] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16680.247418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16680.507086] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16681.277400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16681.535639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16682.793317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16684.054402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16685.309313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16686.564126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16686.823257] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16687.862642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16688.122880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16688.381621] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16689.413650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16689.672216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16690.441176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16690.697755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16692.029263] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16692.925552] Lustre: DEBUG MARKER: rc=0; [16692.925552] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16692.925552] if [[ $? -eq 0 && $val -ne 0 ]]; then [16692.925552] echo $(hostname -s): $val; [16692.925552] rc=$val; [16692.925552] fi; [16692.925552] exit $rc [16693.530791] Lustre: DEBUG MARKER: dmesg [16694.098975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31a: Import a large file and check size during restore ============================ 15:11:05 \(1491491465\) [16694.264408] Lustre: DEBUG MARKER: == sanity-hsm test 31a: Import a large file and check size during restore ============================ 15:11:05 (1491491465) [16698.568288] Lustre: HSM agent 3067b518-b7c9-485b-b6ca-498aba1e1c42 already registered [16698.591835] LustreError: 9230:0:(mdt_coordinator.c:1405:mdt_hsm_update_request_state()) lustre-MDT0000: Progress on [0x200000406:0x82:0x0] for cookie 0x58e65a82 does not match request FID [0x200000407:0xd3:0x0] nor data FID [0x200000406:0x81:0x0] [16731.300209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16731.560147] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16732.334463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16732.601863] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16733.375641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16733.632692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16734.893385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16736.154697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16737.412375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16738.674622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16738.930124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16739.949820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16740.208656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16740.469870] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16741.499538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16741.757627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16742.563689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16742.823466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16744.150853] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16745.061670] Lustre: DEBUG MARKER: rc=0; [16745.061670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16745.061670] if [[ $? -eq 0 && $val -ne 0 ]]; then [16745.061670] echo $(hostname -s): $val; [16745.061670] rc=$val; [16745.061670] fi; [16745.061670] exit $rc [16745.663579] Lustre: DEBUG MARKER: dmesg [16746.253773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 15:11:57 \(1491491517\) [16746.413053] Lustre: DEBUG MARKER: == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 15:11:57 (1491491517) [16748.373850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16748.635521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16749.896752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16751.170766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16752.434954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16753.710774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16754.972308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16756.237384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16757.508768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16758.769917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16760.030126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16761.291223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16762.554429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16763.817476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16765.079814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16766.344405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16767.607960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16768.874881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16770.140142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16771.405924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16772.672142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16773.945817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16775.207816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16776.472397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16777.741772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16779.012509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16780.277591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16781.540688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16782.806627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16784.066789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16785.328499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16828.221425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16828.498095] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16829.315406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16829.572771] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16830.362124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16830.621423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16831.879978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16833.143088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16833.401653] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16834.432994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16834.694738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16834.955692] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16835.986919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16836.245277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16837.012521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16837.270972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16838.612789] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16839.508167] Lustre: DEBUG MARKER: rc=0; [16839.508167] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16839.508167] if [[ $? -eq 0 && $val -ne 0 ]]; then [16839.508167] echo $(hostname -s): $val; [16839.508167] rc=$val; [16839.508167] fi; [16839.508167] exit $rc [16840.117693] Lustre: DEBUG MARKER: dmesg [16840.772342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 15:13:31 \(1491491611\) [16840.928568] Lustre: DEBUG MARKER: == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 15:13:31 (1491491611) [16844.613236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16844.871359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16846.135587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16847.401143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16848.674756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16849.935639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16851.195729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16852.458322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16853.718786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16854.986878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16856.256308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16857.523542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16858.788030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16860.049027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16861.314705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16862.576406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16863.837570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16865.103122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16866.365578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16867.628143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16868.892266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16870.155273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16871.422290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16872.689341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16873.966524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16875.230364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16876.496861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16877.759748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16920.630542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16920.891209] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16921.661355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16921.921565] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16922.713830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16922.972934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16923.230321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16925.516927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16925.776556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16926.036746] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16927.069183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16927.328509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16928.096665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16928.358423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16929.732685] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16930.637165] Lustre: DEBUG MARKER: rc=0; [16930.637165] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16930.637165] if [[ $? -eq 0 && $val -ne 0 ]]; then [16930.637165] echo $(hostname -s): $val; [16930.637165] rc=$val; [16930.637165] fi; [16930.637165] exit $rc [16931.248589] Lustre: DEBUG MARKER: dmesg [16931.826929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 33: Kill a restore waiting process ================================================ 15:15:02 \(1491491702\) [16931.980440] Lustre: DEBUG MARKER: == sanity-hsm test 33: Kill a restore waiting process ================================================ 15:15:02 (1491491702) [16933.728112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16933.995458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16935.254944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16936.523380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16937.784419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16939.053813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16940.318839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16941.582097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16942.844338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16944.104284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16945.365627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16946.627109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16947.894669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16949.164721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16950.430132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16951.697109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16952.959382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16954.219285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16955.482701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16956.749464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16958.016615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16959.281468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16960.548788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16961.820895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16963.083962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16964.346927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16965.611276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16966.872338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16968.143454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16969.407132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16970.669534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16970.955184] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [16971.471067] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [16971.997706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16972.257579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16973.022696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16973.282341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16974.046468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [16984.314888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [16984.584744] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=100 [16985.102542] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=100 [16985.617671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16985.878823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16987.152397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16987.414090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16988.682220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16989.950993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16991.219247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16992.488627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16993.752119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16995.014915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16996.278524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16997.545679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [16998.810643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17000.072698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17001.336573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17002.598938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17003.865154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17005.128890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17006.391841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17007.661524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17008.924848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17010.188775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17011.460757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17012.722958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17013.985958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17015.250061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17016.514192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [17016.781226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17017.042378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [17017.306911] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [17017.830390] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [17020.883660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17021.142124] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17021.912063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17022.171807] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17022.943289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17023.204646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17024.472108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17025.730342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17025.992196] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17027.021572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17027.283467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17027.540465] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17028.562125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17028.817468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17029.588064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17029.849835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17031.182346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17032.076602] Lustre: DEBUG MARKER: rc=0; [17032.076602] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17032.076602] if [[ $? -eq 0 && $val -ne 0 ]]; then [17032.076602] echo $(hostname -s): $val; [17032.076602] rc=$val; [17032.076602] fi; [17032.076602] exit $rc [17032.695741] Lustre: DEBUG MARKER: dmesg [17033.352464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 34: Remove file during restore ==================================================== 15:16:44 \(1491491804\) [17033.535594] Lustre: DEBUG MARKER: == sanity-hsm test 34: Remove file during restore ==================================================== 15:16:44 (1491491804) [17035.978973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17036.240091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17037.510573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17038.774912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17040.036150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17041.299344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17042.564906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17043.838510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17045.098921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17046.366831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17047.630731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17048.901424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17050.168421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17051.436382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17052.712266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17053.974261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17055.238948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17056.502445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17057.769128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17059.029813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17060.294475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17061.573547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17062.843944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17064.107142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17065.370508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17066.640941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17067.909308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17069.172628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17070.439391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17071.705378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17072.970278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17073.261755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17073.523911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17073.797147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17074.077573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17074.358629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17074.638451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17075.918793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17077.198244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17078.465348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17079.734243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17081.006779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17082.269483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17083.537764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17084.802942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17086.064477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17087.326815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17088.588495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17089.857577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17091.121342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17092.386818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17093.652863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17094.921309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17096.192807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17097.466387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17098.734370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17099.996916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17101.259517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17102.524177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17103.793340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17105.061766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17106.330353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17107.597022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17108.865428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17110.129831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17111.420372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17114.238005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17114.498885] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17115.278050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17115.537437] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17116.310345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17116.571181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17117.827991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17119.092360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17120.357512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17121.614791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17121.876516] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17122.907266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17123.166208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17123.426570] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17124.489397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17124.757789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17125.543261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17125.800135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17127.132585] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17128.043649] Lustre: DEBUG MARKER: rc=0; [17128.043649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17128.043649] if [[ $? -eq 0 && $val -ne 0 ]]; then [17128.043649] echo $(hostname -s): $val; [17128.043649] rc=$val; [17128.043649] fi; [17128.043649] exit $rc [17128.674070] Lustre: DEBUG MARKER: dmesg [17129.230079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 35: Overwrite file during restore ================================================= 15:18:20 \(1491491900\) [17129.385901] Lustre: DEBUG MARKER: == sanity-hsm test 35: Overwrite file during restore ================================================= 15:18:20 (1491491900) [17131.371620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17131.631804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17132.894267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17134.162249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17135.422984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17136.691040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17137.966376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17139.226174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17140.488350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17141.757681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17143.018721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17144.279365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17145.538584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17146.801497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17148.064177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17149.326944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17150.593471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17151.853781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17153.113700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17154.377517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17155.640874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17156.905167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17158.170359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17159.437439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17160.698773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17161.960974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17163.223819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17164.491260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17165.758400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17167.022172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17168.292546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17168.588106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17168.851939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17169.122460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17169.386776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17169.651268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17169.916416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17171.183137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17172.447312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17173.709259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17174.972799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17176.241167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17177.511208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17178.788051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17180.061950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17181.326149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17182.593300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17183.855088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17185.118144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17186.383732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17187.654961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17188.918825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17190.181935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17191.443352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17192.705627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17193.968020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17195.237325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17196.515218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17197.779289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17199.042169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17200.313052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17201.576792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17202.841360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17204.106744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17205.372097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17206.652889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17209.503145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17209.765216] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17210.545475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17210.810165] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17211.582121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17211.859548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17213.122226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17214.385893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17215.645348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17216.908382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17217.167141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17218.203936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17218.462779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17218.721826] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17219.756220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17220.015511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17220.795728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17221.056774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17222.405253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17223.296663] Lustre: DEBUG MARKER: rc=0; [17223.296663] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17223.296663] if [[ $? -eq 0 && $val -ne 0 ]]; then [17223.296663] echo $(hostname -s): $val; [17223.296663] rc=$val; [17223.296663] fi; [17223.296663] exit $rc [17223.917139] Lustre: DEBUG MARKER: dmesg [17224.482923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 36: Move file during restore ====================================================== 15:19:55 \(1491491995\) [17224.634139] Lustre: DEBUG MARKER: == sanity-hsm test 36: Move file during restore ====================================================== 15:19:55 (1491491995) [17227.272768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17227.533860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17228.801765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17230.061695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17231.322286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17232.587599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17233.847965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17235.107986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17236.377432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17237.638544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17238.901314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17240.164239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17241.424797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17242.692495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17243.952149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17245.212169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17246.476521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17247.745050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17249.014630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17250.281388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17251.559854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17252.825185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17254.085949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17255.349953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17256.613997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17257.879295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17259.146748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17260.409808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17261.673517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17262.940927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17264.203797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17264.503723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17264.766223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17265.034970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17265.294499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17265.561579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17265.825222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17267.089792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17268.358739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17269.627789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17270.908742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17272.173030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17273.439270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17274.700172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17275.971160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17277.249992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17278.519231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17279.784824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17281.049143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17282.313804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17283.576406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17284.842175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17286.102165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17287.366681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17288.631260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17289.897172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17291.169185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17292.432312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17293.696766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17294.965407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17296.231985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17297.509614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17298.782863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17300.057067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17301.319383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17302.588583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17305.415511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17305.681288] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17306.461504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17306.723640] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17307.501910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17307.759176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17309.018959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17310.277226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17311.540632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17312.805652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17313.076991] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17314.110212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17314.370296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17314.629209] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17315.662729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17315.920840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17316.689523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17316.949696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17318.284938] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17319.189848] Lustre: DEBUG MARKER: rc=0; [17319.189848] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17319.189848] if [[ $? -eq 0 && $val -ne 0 ]]; then [17319.189848] echo $(hostname -s): $val; [17319.189848] rc=$val; [17319.189848] fi; [17319.189848] exit $rc [17319.797081] Lustre: DEBUG MARKER: dmesg [17320.371151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 37: re-archive a dirty file ======================================================= 15:21:31 \(1491492091\) [17320.529082] Lustre: DEBUG MARKER: == sanity-hsm test 37: re-archive a dirty file ======================================================= 15:21:31 (1491492091) [17323.199145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17323.459868] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17324.228298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17324.486592] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17325.257265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17325.516692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17326.772708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17328.029393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17329.284652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17330.540234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17331.796971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17333.055461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17334.317908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17334.575868] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17335.597373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17335.855206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17336.120103] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17337.164761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17337.421104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17338.211452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17338.467751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17340.714165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17340.993118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17342.268226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17343.545090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17343.815562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [17356.242575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17356.502694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17357.765415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17360.568634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17360.828709] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17361.621389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17361.889102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17362.666349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17362.940638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17364.212517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17365.471240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17366.734946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17367.994960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17368.262694] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17369.283609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17369.542439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17369.801196] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17370.827075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17371.088202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17371.864303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17372.121791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17373.482064] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17374.378348] Lustre: DEBUG MARKER: rc=0; [17374.378348] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17374.378348] if [[ $? -eq 0 && $val -ne 0 ]]; then [17374.378348] echo $(hostname -s): $val; [17374.378348] rc=$val; [17374.378348] fi; [17374.378348] exit $rc [17374.993466] Lustre: DEBUG MARKER: dmesg [17375.585737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 40: Parallel archive requests ===================================================== 15:22:26 \(1491492146\) [17375.740444] Lustre: DEBUG MARKER: == sanity-hsm test 40: Parallel archive requests ===================================================== 15:22:26 (1491492146) [17381.611164] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [17382.128413] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [17382.639915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17382.895771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17383.660641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17383.918761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17384.688366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [17395.931951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17396.237276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17397.564152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17400.378940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17400.645878] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17401.419478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17401.679723] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17402.460910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17402.717795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17403.984714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17405.246023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17406.501976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17407.760287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17408.035139] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17410.318344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17410.577693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17410.834630] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17411.860441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17412.134910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17412.910270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17413.169410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17414.582744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17415.510432] Lustre: DEBUG MARKER: rc=0; [17415.510432] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17415.510432] if [[ $? -eq 0 && $val -ne 0 ]]; then [17415.510432] echo $(hostname -s): $val; [17415.510432] rc=$val; [17415.510432] fi; [17415.510432] exit $rc [17416.122772] Lustre: DEBUG MARKER: dmesg [17416.708217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 15:23:07 \(1491492187\) [17416.877348] Lustre: DEBUG MARKER: == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 15:23:07 (1491492187) [17417.843518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17418.104707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17418.395799] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client a37041b4-d54f-f25f-fce3-981d01accc78 [17418.535930] Lustre: 10191:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a37041b4-d54f-f25f-fce3-981d01accc78 at adminstrative request [17419.674606] Lustre: lustre-MDT0000: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [17419.676820] Lustre: Skipped 4 previous similar messages [17423.501568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17423.765344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17424.546881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17424.806659] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17425.575421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17425.839166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17427.097919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17428.356609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17428.614743] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17430.898562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17431.158787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17431.417433] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17432.443378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17432.701925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17433.474383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17433.731769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17435.070100] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17435.973142] Lustre: DEBUG MARKER: rc=0; [17435.973142] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17435.973142] if [[ $? -eq 0 && $val -ne 0 ]]; then [17435.973142] echo $(hostname -s): $val; [17435.973142] rc=$val; [17435.973142] fi; [17435.973142] exit $rc [17436.585704] Lustre: DEBUG MARKER: dmesg [17437.173382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 15:23:28 \(1491492208\) [17437.334022] Lustre: DEBUG MARKER: == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 15:23:28 (1491492208) [17438.247449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040b:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17438.513107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040b:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17438.790053] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client a37041b4-d54f-f25f-fce3-981d01accc78 [17438.922762] Lustre: 11882:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a37041b4-d54f-f25f-fce3-981d01accc78 at adminstrative request [17443.875033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17444.137364] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17444.907371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17445.168297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17445.941354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17446.197171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17447.456058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17448.716150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17448.973255] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17451.250990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17451.512525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17451.770651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17452.799759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17453.056859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17453.832517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17454.090758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17455.415922] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17456.309126] Lustre: DEBUG MARKER: rc=0; [17456.309126] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17456.309126] if [[ $? -eq 0 && $val -ne 0 ]]; then [17456.309126] echo $(hostname -s): $val; [17456.309126] rc=$val; [17456.309126] fi; [17456.309126] exit $rc [17456.928157] Lustre: DEBUG MARKER: dmesg [17457.507736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 54: Write during an archive cancels it ============================================ 15:23:48 \(1491492228\) [17457.671574] Lustre: DEBUG MARKER: == sanity-hsm test 54: Write during an archive cancels it ============================================ 15:23:48 (1491492228) [17460.447066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17460.719043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17460.994193] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [17461.520277] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [17462.071983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17462.331616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17463.594663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17464.862818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17466.125279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17467.387225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17468.651343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17469.912103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17471.173101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17472.434236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17473.692609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17474.953209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17476.214588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17477.484468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17478.761050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17480.036720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17481.307310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17482.572842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17483.840348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17485.104960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17486.372965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17487.646113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17488.909724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17490.177848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17491.457670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17492.722543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17493.985578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17495.257567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17496.529181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17497.808301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17498.082769] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17498.599756] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17501.658463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17501.920245] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17502.701065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17502.962684] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17503.739004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17503.997968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17505.255609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17506.512022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17507.770929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17508.031327] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17511.569298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17511.826460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17512.086364] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17513.125549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17513.386057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17514.159797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17514.417895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17515.753328] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17516.679082] Lustre: DEBUG MARKER: rc=0; [17516.679082] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17516.679082] if [[ $? -eq 0 && $val -ne 0 ]]; then [17516.679082] echo $(hostname -s): $val; [17516.679082] rc=$val; [17516.679082] fi; [17516.679082] exit $rc [17517.296469] Lustre: DEBUG MARKER: dmesg [17517.886424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 15:24:48 \(1491492288\) [17518.068588] Lustre: DEBUG MARKER: == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 15:24:48 (1491492288) [17521.468537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17521.727735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17521.996685] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [17522.520278] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [17523.102541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17523.361350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17523.632654] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17524.149416] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17527.203881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17527.461679] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17528.231229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17528.491895] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17529.271456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17529.532900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17530.796517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17532.055515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17533.313745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17533.571442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17534.597544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17534.857862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17535.117218] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17536.145063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17536.403746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17537.173562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17537.434096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17538.761725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17539.685005] Lustre: DEBUG MARKER: rc=0; [17539.685005] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17539.685005] if [[ $? -eq 0 && $val -ne 0 ]]; then [17539.685005] echo $(hostname -s): $val; [17539.685005] rc=$val; [17539.685005] fi; [17539.685005] exit $rc [17540.328063] Lustre: DEBUG MARKER: dmesg [17540.911467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 56: Setattr during an archive is ok =============================================== 15:25:11 \(1491492311\) [17541.075417] Lustre: DEBUG MARKER: == sanity-hsm test 56: Setattr during an archive is ok =============================================== 15:25:11 (1491492311) [17542.983836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17543.249970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17543.534423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17543.798173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17545.064487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17546.326436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17547.593496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17548.856397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17550.127383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17551.389611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17552.652408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17553.914882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17555.178559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17556.442306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17557.704399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17558.972362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17560.239063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17561.502283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17562.765409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17564.025488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17565.286714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17566.556481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17567.825438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17569.096447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17570.359391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17571.629282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17572.892995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17574.154990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17575.415819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17576.679338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17577.942220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17579.207116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17580.471024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17581.742688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17584.570839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17584.834646] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17585.615922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17585.876197] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17586.646299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17586.915993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17588.189074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17589.447144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17590.707147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17590.964775] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17595.752444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17596.010454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17596.279111] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17597.304266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17597.561034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17598.332952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17598.590182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17599.948814] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17600.875925] Lustre: DEBUG MARKER: rc=0; [17600.875925] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17600.875925] if [[ $? -eq 0 && $val -ne 0 ]]; then [17600.875925] echo $(hostname -s): $val; [17600.875925] rc=$val; [17600.875925] fi; [17600.875925] exit $rc [17601.503828] Lustre: DEBUG MARKER: dmesg [17602.121245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 15:26:13 \(1491492373\) [17602.292817] Lustre: DEBUG MARKER: == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 15:26:13 (1491492373) [17603.889490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17604.167318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17605.439550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17606.705390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17609.541386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17609.807083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17612.612798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17612.875297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17613.645960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17613.902909] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17614.679469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17614.934598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17616.189467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17617.449936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17618.707480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17619.963431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17620.219351] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17621.254891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17621.526967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17621.785709] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17622.813974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17623.069816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17623.837803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17624.094621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17625.444077] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17626.379250] Lustre: DEBUG MARKER: rc=0; [17626.379250] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17626.379250] if [[ $? -eq 0 && $val -ne 0 ]]; then [17626.379250] echo $(hostname -s): $val; [17626.379250] rc=$val; [17626.379250] fi; [17626.379250] exit $rc [17626.994836] Lustre: DEBUG MARKER: dmesg [17627.574600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 15:26:38 \(1491492398\) [17627.729920] Lustre: DEBUG MARKER: == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 15:26:38 (1491492398) [17628.682234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17628.939109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17629.295801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17629.620620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17629.881922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17630.250306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17630.570148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17630.830486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17631.200956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17634.022567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17634.285885] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17635.073087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17635.334694] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17636.106559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17636.366112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17637.631269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17638.892175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17640.153549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17641.413191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17641.679693] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17642.746224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17643.005926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17643.264136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17644.296548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17644.552622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17645.359747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17645.636598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17646.975162] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17647.897989] Lustre: DEBUG MARKER: rc=0; [17647.897989] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17647.897989] if [[ $? -eq 0 && $val -ne 0 ]]; then [17647.897989] echo $(hostname -s): $val; [17647.897989] rc=$val; [17647.897989] fi; [17647.897989] exit $rc [17648.517357] Lustre: DEBUG MARKER: dmesg [17649.075826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 15:27:00 \(1491492420\) [17649.244021] Lustre: DEBUG MARKER: == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 15:27:00 (1491492420) [17649.381822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [17649.381822] /usr/sbin/lctl lustre_build_version 2>/dev/null || [17649.381822] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [17650.434718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17650.693943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17653.517824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17653.791790] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17654.574318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17654.832347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17655.611511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17655.866630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17657.127800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17658.387233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17659.650260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17660.916348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17661.174095] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17663.459207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17663.714929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17663.974644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17665.009387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17665.267022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17666.034996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17666.295051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17667.639588] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17668.584307] Lustre: DEBUG MARKER: rc=0; [17668.584307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17668.584307] if [[ $? -eq 0 && $val -ne 0 ]]; then [17668.584307] echo $(hostname -s): $val; [17668.584307] rc=$val; [17668.584307] fi; [17668.584307] exit $rc [17669.205840] Lustre: DEBUG MARKER: dmesg [17669.805495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 60: Changing progress update interval from default ================================ 15:27:20 \(1491492440\) [17669.976645] Lustre: DEBUG MARKER: == sanity-hsm test 60: Changing progress update interval from default ================================ 15:27:20 (1491492440) [17672.683044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17672.941963] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17673.715451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17673.976191] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17674.758996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17675.014711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17676.271397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17677.534568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17678.797495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17680.053950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17681.310017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17681.572582] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17683.849794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17684.109967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17684.365668] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17685.402166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17685.662030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17686.439787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17686.699245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17688.965582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [17689.221350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [17689.994588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [17691.257499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [17692.519150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [17693.783547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [17695.046478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [17699.541628] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17700.058443] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17703.111985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17703.371918] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17704.149761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17704.408825] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17705.200260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17705.461903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17706.718482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17707.974173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17709.236998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17709.494018] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17714.293665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17714.563159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17714.821147] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17715.859816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17716.116135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17716.900117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17717.160132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17718.506293] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17719.417649] Lustre: DEBUG MARKER: rc=0; [17719.417649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17719.417649] if [[ $? -eq 0 && $val -ne 0 ]]; then [17719.417649] echo $(hostname -s): $val; [17719.417649] rc=$val; [17719.417649] fi; [17719.417649] exit $rc [17720.037742] Lustre: DEBUG MARKER: dmesg [17720.700443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 70: Copytool logs JSON register\/unregister events to FIFO ========================= 15:28:11 \(1491492491\) [17720.850064] Lustre: DEBUG MARKER: == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================= 15:28:11 (1491492491) [17723.534497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17723.789692] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17724.564804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17724.821324] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17725.592180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17725.852745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17727.110894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17728.373219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17729.634764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17729.891732] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17734.695172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17734.954319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17735.214553] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17736.231296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17736.493308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17737.264833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17737.519431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17741.093697] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17741.627876] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17742.154871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [17742.411043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [17745.236535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17745.504587] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17746.276956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17746.535454] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17747.313552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17747.571004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17748.834266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17750.092504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17750.350074] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17755.155598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17755.417615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17755.677190] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17756.715498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17756.982679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17757.761408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17758.017595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17760.198362] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17761.125343] Lustre: DEBUG MARKER: rc=0; [17761.125343] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17761.125343] if [[ $? -eq 0 && $val -ne 0 ]]; then [17761.125343] echo $(hostname -s): $val; [17761.125343] rc=$val; [17761.125343] fi; [17761.125343] exit $rc [17761.745344] Lustre: DEBUG MARKER: dmesg [17762.318745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 15:28:53 \(1491492533\) [17762.475831] Lustre: DEBUG MARKER: == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 15:28:53 (1491492533) [17765.160555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17765.434109] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17766.246712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17766.518791] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17767.295788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17767.551476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17768.808170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17770.071327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17771.328667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17771.586644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17776.377821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17776.634805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17776.892072] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17777.925316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17778.184610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17778.962149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17779.222597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17783.695495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17783.963368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17785.221401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17786.482783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17787.747124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17789.014137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17790.274965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17791.537720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17792.800282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17794.063187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17795.322683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17796.586787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17797.850469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17799.114222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17800.376308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17801.646210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17802.913437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17804.177668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17805.441371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17806.706408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17807.972328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17809.233765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17810.502563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17811.767192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17813.036180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17814.297595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17815.565232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17816.827751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17818.094836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17819.362157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17820.624946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17821.887911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17823.155649] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17823.681406] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17826.741407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17827.002571] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17827.780899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17828.042875] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17828.817697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17829.076259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17830.340805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17831.612812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17831.885924] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17836.686441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17836.943785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17837.204045] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17838.244292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17838.505044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17839.266741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17839.525433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17841.361656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17842.294264] Lustre: DEBUG MARKER: rc=0; [17842.294264] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17842.294264] if [[ $? -eq 0 && $val -ne 0 ]]; then [17842.294264] echo $(hostname -s): $val; [17842.294264] rc=$val; [17842.294264] fi; [17842.294264] exit $rc [17842.917919] Lustre: DEBUG MARKER: dmesg [17843.533622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 15:30:14 \(1491492614\) [17843.706225] Lustre: DEBUG MARKER: == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 15:30:14 (1491492614) [17846.404766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17846.663998] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17847.430734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17847.695985] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17848.475176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17848.733229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17849.996424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17851.258668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17852.517100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17852.775264] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17857.586994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17857.842804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17858.102205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17859.134457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17859.389961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17860.162280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17860.424219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17866.901497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17867.163717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17868.422177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17869.683826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17870.942946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17872.204568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17873.467364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17874.733809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17875.994062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17877.255228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17878.515125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17879.779727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17881.041276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17882.311153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17883.269431] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [17883.790222] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [17886.842791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17887.104269] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17887.871769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17888.132072] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17888.906128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17889.162160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17890.421819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17891.679311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17892.936628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17893.194412] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17897.986872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17898.248381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17898.510992] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17899.542286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17899.802134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17900.569872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17900.828088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17902.661244] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17903.579316] Lustre: DEBUG MARKER: rc=0; [17903.579316] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17903.579316] if [[ $? -eq 0 && $val -ne 0 ]]; then [17903.579316] echo $(hostname -s): $val; [17903.579316] rc=$val; [17903.579316] fi; [17903.579316] exit $rc [17904.202707] Lustre: DEBUG MARKER: dmesg [17904.854240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 90: Archive\/restore a file list =================================================== 15:31:15 \(1491492675\) [17905.012939] Lustre: DEBUG MARKER: == sanity-hsm test 90: Archive/restore a file list =================================================== 15:31:15 (1491492675) [17906.795483] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [17907.318973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [17907.844414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17908.103154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17908.875195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17909.132633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17909.922493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [17920.349039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17920.615187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17921.277079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17921.572168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [17924.367818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17924.631091] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17925.409224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17925.683784] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17926.453166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17926.709943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17927.969324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17929.228436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17930.489132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17931.750453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17932.012597] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17933.050070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17933.305175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17933.561260] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17934.613419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17934.873307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17935.644859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17935.903086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17937.264937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17938.197675] Lustre: DEBUG MARKER: rc=0; [17938.197675] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17938.197675] if [[ $? -eq 0 && $val -ne 0 ]]; then [17938.197675] echo $(hostname -s): $val; [17938.197675] rc=$val; [17938.197675] fi; [17938.197675] exit $rc [17938.817542] Lustre: DEBUG MARKER: dmesg [17939.469928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 100: Set coordinator \/proc tunables =============================================== 15:31:50 \(1491492710\) [17939.635148] Lustre: DEBUG MARKER: == sanity-hsm test 100: Set coordinator /proc tunables =============================================== 15:31:50 (1491492710) [17939.771688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [17940.035765] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=20 [17940.555813] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=20 [17941.074328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [17941.333611] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=0 [17941.846214] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=0 [17942.365593] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [17942.885119] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=10 [17943.410584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [17943.671275] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=20 [17944.181297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=20 [17944.700253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [17944.959237] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=0 [17945.486617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=0 [17946.000832] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [17946.522108] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [17947.035235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [17947.298620] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=7200 [17947.825074] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=7200 [17948.345182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [17948.608465] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=0 [17949.123769] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=0 [17949.649924] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [17950.165524] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=3600 [17950.692653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [17950.955956] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=6 [17951.472899] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=6 [17951.992742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [17952.256224] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=0 [17952.779049] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=0 [17953.293829] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=3 [17953.808307] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=3 [17954.330701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [17954.606207] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [17955.131393] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=2 [17955.647464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [17955.912136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=0 [17956.425141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=0 [17956.937202] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=1 [17957.449377] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=1 [17958.218452] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17959.377282] Lustre: DEBUG MARKER: rc=0; [17959.377282] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17959.377282] if [[ $? -eq 0 && $val -ne 0 ]]; then [17959.377282] echo $(hostname -s): $val; [17959.377282] rc=$val; [17959.377282] fi; [17959.377282] exit $rc [17960.012584] Lustre: DEBUG MARKER: dmesg [17960.636823] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 102: Verify coordinator control =================================================== 15:32:11 \(1491492731\) [17960.813330] Lustre: DEBUG MARKER: == sanity-hsm test 102: Verify coordinator control =================================================== 15:32:11 (1491492731) [17960.960686] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [17961.480851] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [17961.998511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17962.268790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17963.043655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17963.301515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17964.096376] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17964.616796] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17965.129178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17965.388366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17966.163866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17966.424275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17967.193753] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17967.714701] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17968.225781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17968.481727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17969.740616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17971.001972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17972.259719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17973.521832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17974.785441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17975.558940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17975.819582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17976.589565] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17977.098146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17977.614634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17977.877597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17978.654989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17978.912887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17979.766928] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17980.677942] Lustre: DEBUG MARKER: rc=0; [17980.677942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17980.677942] if [[ $? -eq 0 && $val -ne 0 ]]; then [17980.677942] echo $(hostname -s): $val; [17980.677942] rc=$val; [17980.677942] fi; [17980.677942] exit $rc [17981.298547] Lustre: DEBUG MARKER: dmesg [17981.922657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 103: Purge all requests =========================================================== 15:32:32 \(1491492752\) [17982.097292] Lustre: DEBUG MARKER: == sanity-hsm test 103: Purge all requests =========================================================== 15:32:32 (1491492752) [17983.306699] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [17983.821140] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [17984.340666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17984.598462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17985.376647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17985.642068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17986.405802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep -v CANCELED | grep -v SUCCEED | grep -v FAILED [17989.211524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17989.470622] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17990.241467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17990.502842] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17991.276073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17991.542901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17992.807830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17994.070227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17994.327533] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17997.864985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17998.124469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17998.384777] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17999.419044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17999.681684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18000.459815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18000.719797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18002.074366] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18003.026714] Lustre: DEBUG MARKER: rc=0; [18003.026714] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18003.026714] if [[ $? -eq 0 && $val -ne 0 ]]; then [18003.026714] echo $(hostname -s): $val; [18003.026714] rc=$val; [18003.026714] fi; [18003.026714] exit $rc [18003.657639] Lustre: DEBUG MARKER: dmesg [18004.266631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 104: Copy tool data field ========================================================= 15:32:55 \(1491492775\) [18004.427056] Lustre: DEBUG MARKER: == sanity-hsm test 104: Copy tool data field ========================================================= 15:32:55 (1491492775) [18006.632945] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18007.160218] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18007.682665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18007.939608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18008.712316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18008.974847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18009.747816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep 0x200000405:0x218:0x0 | cut -f16 -d= [18010.007977] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18010.534127] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18011.058621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18011.318464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18012.087181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18012.347513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18015.694559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18015.956592] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18016.750472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18017.007778] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18017.797907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18018.055338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18019.316514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18020.576405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18020.840626] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18021.870207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18022.138819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18022.399195] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18023.425159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18023.680871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18024.458524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18024.718477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18026.081922] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18027.003266] Lustre: DEBUG MARKER: rc=0; [18027.003266] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18027.003266] if [[ $? -eq 0 && $val -ne 0 ]]; then [18027.003266] echo $(hostname -s): $val; [18027.003266] rc=$val; [18027.003266] fi; [18027.003266] exit $rc [18027.629088] Lustre: DEBUG MARKER: dmesg [18028.202527] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 105: Restart of coordinator ======================================================= 15:33:19 \(1491492799\) [18028.365257] Lustre: DEBUG MARKER: == sanity-hsm test 105: Restart of coordinator ======================================================= 15:33:19 (1491492799) [18028.515805] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18029.042292] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18029.564157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18029.822171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18030.626536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18030.887664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18031.742767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18032.002580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18032.512420] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18033.035427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18033.294229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18034.551172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18035.815057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18037.081213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18038.342716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18039.602099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18040.860762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18042.120220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18042.897573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18043.172261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18043.948522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18044.475762] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18044.999985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18045.260395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18046.024506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18046.292456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18047.065248] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18047.579103] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18048.091283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18048.346133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18049.118981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18049.374754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18050.146270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18050.411949] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18050.943609] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18051.474154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18051.748077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18052.524337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18052.781981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18053.559115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [18054.071315] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [18054.588873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18054.848307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18055.625907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18055.884433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18056.715905] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18057.633333] Lustre: DEBUG MARKER: rc=0; [18057.633333] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18057.633333] if [[ $? -eq 0 && $val -ne 0 ]]; then [18057.633333] echo $(hostname -s): $val; [18057.633333] rc=$val; [18057.633333] fi; [18057.633333] exit $rc [18058.257964] Lustre: DEBUG MARKER: dmesg [18058.820040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 106: Copytool register\/unregister ================================================= 15:33:49 \(1491492829\) [18058.992455] Lustre: DEBUG MARKER: == sanity-hsm test 106: Copytool register/unregister ================================================= 15:33:49 (1491492829) [18060.401362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [18060.625890] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [18061.256350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [18061.421791] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [18061.583505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18062.118688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18065.457991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18065.718735] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18066.501401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18066.761069] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18067.545709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18067.803025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18069.062247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18070.321669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18071.586704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18071.844035] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18072.872367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18073.129945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18073.387680] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18074.414475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18074.669705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18075.445594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18075.702929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18076.976202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18077.498772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18079.281628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [18079.443386] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [18080.067548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [18080.229668] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [18080.389097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18080.913776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [18083.981554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18084.242739] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18085.028328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18085.290483] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18086.074902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18086.335603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18087.594117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18088.853742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18090.117487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18091.375410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18092.635717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18092.897999] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18093.946467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18094.206776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18094.465999] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18095.503842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18095.760906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18096.548646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18096.805163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18098.156303] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18099.091076] Lustre: DEBUG MARKER: rc=0; [18099.091076] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18099.091076] if [[ $? -eq 0 && $val -ne 0 ]]; then [18099.091076] echo $(hostname -s): $val; [18099.091076] rc=$val; [18099.091076] fi; [18099.091076] exit $rc [18099.717856] Lustre: DEBUG MARKER: dmesg [18100.301719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 15:34:31 \(1491492871\) [18100.458941] Lustre: DEBUG MARKER: == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 15:34:31 (1491492871) [18101.397764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18101.661193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18101.936432] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [18102.195341] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [18102.328154] Lustre: Failing over lustre-MDT0000 [18102.399235] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880049250c00 x1563924896378752/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [18102.403883] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [18102.444535] Lustre: server umount lustre-MDT0000 complete [18102.446403] Lustre: Skipped 2 previous similar messages [18102.577027] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [18104.726262] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [18104.728608] LustreError: Skipped 6 previous similar messages [18104.730346] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18104.734125] Lustre: Skipped 8 previous similar messages [18104.736544] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18104.740566] LustreError: Skipped 18 previous similar messages [18111.726105] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491492875/real 1491492875] req@ffff88004aa6fc00 x1563924896378832/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491492882 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [18111.732770] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [18111.735155] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18111.739645] LustreError: Skipped 3 previous similar messages [18112.890659] Lustre: DEBUG MARKER: hostname [18113.207451] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [18113.467014] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [18113.722657] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [18113.885221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18114.015076] Lustre: MGS: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [18114.017746] Lustre: Skipped 1 previous similar message [18117.741330] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5485b5 to 0x61ac6f141d587163 [18117.779069] LustreError: 1753:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [18117.786429] LustreError: 1758:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [18117.789404] LustreError: 1758:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [18117.810540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18117.813287] Lustre: Skipped 2 previous similar messages [18118.002136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [18118.265358] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [18118.781026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18118.785401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18118.953564] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18118.957599] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18119.015282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [18119.104626] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [18119.367086] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [18119.629103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [18119.898114] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [18120.679392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18120.679476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18120.904313] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18120.905859] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18122.796055] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [18122.814074] Lustre: lustre-MDT0000: Recovery over after 0:04, of 7 clients 7 recovered and 0 were evicted. [18123.191106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18123.193948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18123.382665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18123.479048] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18124.791554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18125.057550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18127.870980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18128.147447] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18128.935876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18129.195842] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18129.977292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18130.238869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18131.496456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18132.756515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18134.018413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18135.280485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18135.539050] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18136.569049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18136.829909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18137.089285] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18138.130589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18138.390350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18139.196860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18139.455939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18140.831472] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18141.749436] Lustre: DEBUG MARKER: rc=0; [18141.749436] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18141.749436] if [[ $? -eq 0 && $val -ne 0 ]]; then [18141.749436] echo $(hostname -s): $val; [18141.749436] rc=$val; [18141.749436] fi; [18141.749436] exit $rc [18142.380552] Lustre: DEBUG MARKER: dmesg [18142.982938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 109: Policy display\/change ======================================================== 15:35:13 \(1491492913\) [18143.140658] Lustre: DEBUG MARKER: == sanity-hsm test 109: Policy display/change ======================================================== 15:35:13 (1491492913) [18143.280384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18143.542149] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NBR" [18143.800987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18144.060455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NRA" [18144.321977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18144.582814] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NBR" [18144.843022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18145.102369] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NRA" [18145.361747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18145.622076] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="NRA NBR" [18145.885449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18146.147732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18146.424520] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="wrong" [18146.556340] Lustre: 5611:0:(mdt_coordinator.c:1804:mdt_hsm_policy_seq_write()) lustre-MDT0000: 'wrong' is unknown, supported policies are: [18146.688417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [18146.949389] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [18147.469407] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [18147.985659] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [18148.500456] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [18149.085061] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18149.997516] Lustre: DEBUG MARKER: rc=0; [18149.997516] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18149.997516] if [[ $? -eq 0 && $val -ne 0 ]]; then [18149.997516] echo $(hostname -s): $val; [18149.997516] rc=$val; [18149.997516] fi; [18149.997516] exit $rc [18150.616315] Lustre: DEBUG MARKER: dmesg [18151.173814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110a: Non blocking restore policy \(import case\) =================================== 15:35:22 \(1491492922\) [18151.346343] Lustre: DEBUG MARKER: == sanity-hsm test 110a: Non blocking restore policy (import case) =================================== 15:35:22 (1491492922) [18153.003624] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [18153.520355] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NBR [18154.054642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18154.319783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18154.581655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [18155.104816] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [18158.171520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18158.436396] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18159.222313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18159.480726] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18160.250789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18160.509503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18161.767648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18163.025886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18164.290370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18164.548719] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18166.839275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18167.099507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18167.356694] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18168.394281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18168.660452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18169.435326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18169.708576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18171.086145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18172.001582] Lustre: DEBUG MARKER: rc=0; [18172.001582] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18172.001582] if [[ $? -eq 0 && $val -ne 0 ]]; then [18172.001582] echo $(hostname -s): $val; [18172.001582] rc=$val; [18172.001582] fi; [18172.001582] exit $rc [18172.624736] Lustre: DEBUG MARKER: dmesg [18173.207828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110b: Non blocking restore policy \(release case\) ================================== 15:35:44 \(1491492944\) [18173.367411] Lustre: DEBUG MARKER: == sanity-hsm test 110b: Non blocking restore policy (release case) ================================== 15:35:44 (1491492944) [18174.287955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18174.553285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18174.828342] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [18175.346217] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NBR [18175.866020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18176.130892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18176.395303] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [18176.912777] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [18179.968825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18180.230983] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18181.015286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18181.276595] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18182.051525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18182.308756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18183.572736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18184.830569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18186.102162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18186.375165] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18187.411297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18187.671176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18187.928580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18188.960411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18189.232828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18190.020470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18190.292151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18191.636551] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18192.563828] Lustre: DEBUG MARKER: rc=0; [18192.563828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18192.563828] if [[ $? -eq 0 && $val -ne 0 ]]; then [18192.563828] echo $(hostname -s): $val; [18192.563828] rc=$val; [18192.563828] fi; [18192.563828] exit $rc [18193.206207] Lustre: DEBUG MARKER: dmesg [18193.828852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111a: No retry policy \(import case\), restore will error =========================== 15:36:04 \(1491492964\) [18193.988837] Lustre: DEBUG MARKER: == sanity-hsm test 111a: No retry policy (import case), restore will error =========================== 15:36:04 (1491492964) [18195.625841] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [18196.146655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [18197.158394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18197.418802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18197.683791] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [18198.208057] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [18201.288035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18201.547132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18202.322329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18202.589457] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18203.359176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18203.619988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18204.881085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18206.140617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18207.400268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18207.655953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18208.689420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18208.948433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18209.209720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18210.275635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18210.538065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18211.318989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18211.576770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18212.915306] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18213.815393] Lustre: DEBUG MARKER: rc=0; [18213.815393] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18213.815393] if [[ $? -eq 0 && $val -ne 0 ]]; then [18213.815393] echo $(hostname -s): $val; [18213.815393] rc=$val; [18213.815393] fi; [18213.815393] exit $rc [18214.445818] Lustre: DEBUG MARKER: dmesg [18215.057113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111b: No retry policy \(release case\), restore will error ========================== 15:36:26 \(1491492986\) [18215.208635] Lustre: DEBUG MARKER: == sanity-hsm test 111b: No retry policy (release case), restore will error ========================== 15:36:26 (1491492986) [18216.135469] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [18216.651115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [18217.180330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18217.443540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18218.211793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18218.473507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18218.740307] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [18219.263938] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [18222.327135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18222.595585] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18223.372285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18223.638653] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18224.417240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18224.678198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18225.937042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18227.196848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18228.453568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18228.711742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18229.750547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18230.008279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18230.266162] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18231.297226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18231.554606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18232.330990] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18232.594082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18233.922213] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18234.849344] Lustre: DEBUG MARKER: rc=0; [18234.849344] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18234.849344] if [[ $? -eq 0 && $val -ne 0 ]]; then [18234.849344] echo $(hostname -s): $val; [18234.849344] rc=$val; [18234.849344] fi; [18234.849344] exit $rc [18235.468269] Lustre: DEBUG MARKER: dmesg [18236.072147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 112: State of recorded request ==================================================== 15:36:47 \(1491493007\) [18236.233031] Lustre: DEBUG MARKER: == sanity-hsm test 112: State of recorded request ==================================================== 15:36:47 (1491493007) [18237.165968] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18237.681567] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18238.196461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18238.458736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18239.245149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18239.502166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18240.287151] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18240.808278] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18241.326757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18241.584351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18242.362183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18242.628388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18243.405460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18243.667184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18246.481764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18246.739319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18247.521150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18247.779121] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18248.552138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18248.811297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18250.075416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18251.336612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18251.599130] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18252.629226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18252.888354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18253.147209] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18254.171394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18254.430585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18255.214304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18255.471816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18256.818413] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18257.722047] Lustre: DEBUG MARKER: rc=0; [18257.722047] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18257.722047] if [[ $? -eq 0 && $val -ne 0 ]]; then [18257.722047] echo $(hostname -s): $val; [18257.722047] rc=$val; [18257.722047] fi; [18257.722047] exit $rc [18258.351258] Lustre: DEBUG MARKER: dmesg [18258.953194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 200: Register\/Cancel archive ====================================================== 15:37:09 \(1491493029\) [18259.115756] Lustre: DEBUG MARKER: == sanity-hsm test 200: Register/Cancel archive ====================================================== 15:37:09 (1491493029) [18263.076320] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18263.600135] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18264.118591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18264.378169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18265.410111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18266.172371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18267.116632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18267.376203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18267.650149] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18268.176275] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18268.702585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18268.967979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18269.746745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18270.005970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18270.780542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18271.040716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18271.301255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18271.563621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18274.380734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18274.640533] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18275.428801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18275.690658] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18276.466538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18276.725435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18277.983852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18278.246629] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18279.270726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18279.530228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18279.788857] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18280.830436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18281.087576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18281.881533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18282.138868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18283.475434] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18284.378450] Lustre: DEBUG MARKER: rc=0; [18284.378450] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18284.378450] if [[ $? -eq 0 && $val -ne 0 ]]; then [18284.378450] echo $(hostname -s): $val; [18284.378450] rc=$val; [18284.378450] fi; [18284.378450] exit $rc [18285.005790] Lustre: DEBUG MARKER: dmesg [18285.572076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 201: Register\/Cancel restore ====================================================== 15:37:36 \(1491493056\) [18285.737373] Lustre: DEBUG MARKER: == sanity-hsm test 201: Register/Cancel restore ====================================================== 15:37:36 (1491493056) [18289.705810] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18290.234727] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18290.755775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18291.014077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18291.782824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18292.050766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18292.855509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18293.120581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18293.388053] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18293.907563] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18294.438217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18294.695944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18295.474179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18295.730759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18296.505942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18296.772189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18297.036268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18297.299628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18300.105229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18300.366357] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18301.183842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18301.448715] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18302.231493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18302.491772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18303.748487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18304.007265] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18305.037069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18305.297922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18305.558706] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18306.577093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18306.839754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18307.623352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18307.885420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18309.242549] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18310.173644] Lustre: DEBUG MARKER: rc=0; [18310.173644] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18310.173644] if [[ $? -eq 0 && $val -ne 0 ]]; then [18310.173644] echo $(hostname -s): $val; [18310.173644] rc=$val; [18310.173644] fi; [18310.173644] exit $rc [18310.803762] Lustre: DEBUG MARKER: dmesg [18311.394956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 202: Register\/Cancel remove ======================================================= 15:38:02 \(1491493082\) [18311.575525] Lustre: DEBUG MARKER: == sanity-hsm test 202: Register/Cancel remove ======================================================= 15:38:02 (1491493082) [18313.506739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18313.778981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18315.046441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18316.311891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18317.574160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18318.836454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18320.104343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18321.367582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18322.629505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18323.892575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18325.160080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18326.421759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18327.690388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18328.953339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18330.218984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18331.484196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18332.747384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18334.011178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18335.282760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18336.550428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18337.819111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18339.089640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18340.364554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18341.631948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18342.897390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18344.163428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18345.444795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18346.710821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18347.979061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18349.259465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18350.525876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18350.792766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18351.313988] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18351.845765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18352.121070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18352.906539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18353.183420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18353.958136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18354.220397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18354.489595] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18355.009298] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18355.539861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18355.798590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18356.568277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18356.829056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18357.603174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18357.866293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18360.677345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18360.945183] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18361.736105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18361.996513] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18362.775639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18363.053033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18364.316357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18365.580836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18365.842731] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18366.871847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18367.129977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18367.387093] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18368.414283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18368.672522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18369.447400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18369.707295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18371.040237] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18371.950658] Lustre: DEBUG MARKER: rc=0; [18371.950658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18371.950658] if [[ $? -eq 0 && $val -ne 0 ]]; then [18371.950658] echo $(hostname -s): $val; [18371.950658] rc=$val; [18371.950658] fi; [18371.950658] exit $rc [18372.578571] Lustre: DEBUG MARKER: dmesg [18373.200677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220: Changelog for archive ======================================================== 15:39:04 \(1491493144\) [18373.361754] Lustre: DEBUG MARKER: == sanity-hsm test 220: Changelog for archive ======================================================== 15:39:04 (1491493144) [18374.316574] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18374.446959] Lustre: lustre-MDD0000: changelog on [18374.577297] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18375.355919] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18375.486907] Lustre: lustre-MDD0002: changelog on [18375.617992] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18376.414250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18376.673690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18376.948808] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl1 [18377.079299] Lustre: lustre-MDD0000: changelog off [18377.474811] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl1 [18377.606907] Lustre: lustre-MDD0002: changelog off [18380.549650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18380.814881] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18381.594633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18381.851455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18382.631798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18382.894209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18384.155591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18385.417443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18386.680495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18386.940347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18387.973636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18388.234329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18388.492893] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18389.527795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18389.784931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18390.549869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18390.809759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18392.143871] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18393.072509] Lustre: DEBUG MARKER: rc=0; [18393.072509] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18393.072509] if [[ $? -eq 0 && $val -ne 0 ]]; then [18393.072509] echo $(hostname -s): $val; [18393.072509] rc=$val; [18393.072509] fi; [18393.072509] exit $rc [18393.703813] Lustre: DEBUG MARKER: dmesg [18394.345308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220a: Changelog for failed archive ================================================ 15:39:25 \(1491493165\) [18394.511742] Lustre: DEBUG MARKER: == sanity-hsm test 220a: Changelog for failed archive ================================================ 15:39:25 (1491493165) [18395.430617] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18395.561843] Lustre: lustre-MDD0000: changelog on [18395.692890] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18396.474983] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18396.735869] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18397.932048] LustreError: 14377:0:(mdt_coordinator.c:1375:mdt_hsm_update_request_state()) lustre-MDT0000: Cannot find running request for cookie 0x58e66126 on fid=[0x200000405:0x221:0x0] [18398.064274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18398.327722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18398.599771] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl2 [18398.732101] Lustre: lustre-MDD0000: changelog off [18399.128691] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl2 [18402.204339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18402.477929] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18403.251628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18403.512190] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18404.284249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18404.544700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18405.803812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18407.068391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18408.331832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18408.591752] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18409.630875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18409.889156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18410.148179] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18411.177324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18411.436579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18412.217458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18412.476389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18413.820413] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18414.733698] Lustre: DEBUG MARKER: rc=0; [18414.733698] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18414.733698] if [[ $? -eq 0 && $val -ne 0 ]]; then [18414.733698] echo $(hostname -s): $val; [18414.733698] rc=$val; [18414.733698] fi; [18414.733698] exit $rc [18415.361259] Lustre: DEBUG MARKER: dmesg [18415.944735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 221: Changelog for archive canceled =============================================== 15:39:46 \(1491493186\) [18416.111044] Lustre: DEBUG MARKER: == sanity-hsm test 221: Changelog for archive canceled =============================================== 15:39:46 (1491493186) [18422.275313] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18422.407677] Lustre: lustre-MDD0000: changelog on [18422.409543] Lustre: Skipped 1 previous similar message [18422.542867] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18423.330225] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18423.589965] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18424.375928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18424.637938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18424.902294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18425.159928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18425.421110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18425.684766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18426.946794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18428.207161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18429.470019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18430.732267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18431.996772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18433.272526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18434.536522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18435.799429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18437.063118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18438.330591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18439.592365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18440.860084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18442.136747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18443.407856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18444.673538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18445.942503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18447.205857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18448.469254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18449.731629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18451.001206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18452.264469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18453.527437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18454.789908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18457.600144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18457.863233] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18458.665247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18458.925716] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18459.702937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18459.963691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18461.222829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18462.484448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18463.748950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18464.009188] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18465.040709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18465.299620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18465.560208] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18466.592623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18466.853778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18467.640518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18467.904557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18469.201352] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl3 [18469.334052] Lustre: lustre-MDD0000: changelog off [18469.336380] Lustre: Skipped 1 previous similar message [18469.733090] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl3 [18470.332436] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18471.261813] Lustre: DEBUG MARKER: rc=0; [18471.261813] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18471.261813] if [[ $? -eq 0 && $val -ne 0 ]]; then [18471.261813] echo $(hostname -s): $val; [18471.261813] rc=$val; [18471.261813] fi; [18471.261813] exit $rc [18471.894512] Lustre: DEBUG MARKER: dmesg [18472.469499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222a: Changelog for explicit restore ============================================== 15:40:43 \(1491493243\) [18472.626253] Lustre: DEBUG MARKER: == sanity-hsm test 222a: Changelog for explicit restore ============================================== 15:40:43 (1491493243) [18474.297977] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18474.426809] Lustre: lustre-MDD0000: changelog on [18474.428721] Lustre: Skipped 1 previous similar message [18474.561389] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18475.363036] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18475.623663] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18476.419518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18476.682242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18479.513709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18479.774548] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18480.559429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18480.821438] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18481.598683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18481.857049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18483.115288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18484.379393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18485.637575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18486.894369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18487.151149] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18488.183528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18488.446471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18488.707741] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18489.738118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18489.998019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18490.780633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18491.039385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18492.322507] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl4 [18492.452894] Lustre: lustre-MDD0000: changelog off [18492.454577] Lustre: Skipped 1 previous similar message [18492.856168] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl4 [18493.438963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18494.353125] Lustre: DEBUG MARKER: rc=0; [18494.353125] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18494.353125] if [[ $? -eq 0 && $val -ne 0 ]]; then [18494.353125] echo $(hostname -s): $val; [18494.353125] rc=$val; [18494.353125] fi; [18494.353125] exit $rc [18494.975770] Lustre: DEBUG MARKER: dmesg [18495.551925] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222b: Changelog for implicit restore ============================================== 15:41:06 \(1491493266\) [18495.705607] Lustre: DEBUG MARKER: == sanity-hsm test 222b: Changelog for implicit restore ============================================== 15:41:06 (1491493266) [18496.625064] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18496.755589] Lustre: lustre-MDD0000: changelog on [18496.757285] Lustre: Skipped 1 previous similar message [18496.890067] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18497.685050] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18497.946347] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18498.745998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18499.007021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18499.321416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18499.583186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18502.400978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18502.660764] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18503.451722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18503.712441] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18504.491319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18504.748622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18506.008351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18507.271606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18508.531783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18509.796809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18510.063282] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18511.090916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18511.350661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18511.615102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18512.644194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18512.904193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18513.686853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18513.946476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18515.219283] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl5 [18515.350554] Lustre: lustre-MDD0000: changelog off [18515.352336] Lustre: Skipped 1 previous similar message [18515.754649] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl5 [18516.340744] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18517.251657] Lustre: DEBUG MARKER: rc=0; [18517.251657] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18517.251657] if [[ $? -eq 0 && $val -ne 0 ]]; then [18517.251657] echo $(hostname -s): $val; [18517.251657] rc=$val; [18517.251657] fi; [18517.251657] exit $rc [18517.870479] Lustre: DEBUG MARKER: dmesg [18518.489791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 15:41:29 \(1491493289\) [18518.650773] Lustre: DEBUG MARKER: == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 15:41:29 (1491493289) [18520.299812] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18520.440332] Lustre: lustre-MDD0000: changelog on [18520.442167] Lustre: Skipped 1 previous similar message [18520.573843] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18521.359406] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18521.621515] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18522.945631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18523.217172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18526.027685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18526.291467] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18527.074641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18527.333725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18528.109414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18528.369712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18529.628911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18530.888277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18532.147289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18533.406956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18533.671480] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18534.700563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18534.957359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18535.214870] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18536.239561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18536.511117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18537.297313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18537.570784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18538.850606] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl6 [18538.982605] Lustre: lustre-MDD0000: changelog off [18538.984290] Lustre: Skipped 1 previous similar message [18539.393221] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl6 [18539.980128] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18540.891296] Lustre: DEBUG MARKER: rc=0; [18540.891296] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18540.891296] if [[ $? -eq 0 && $val -ne 0 ]]; then [18540.891296] echo $(hostname -s): $val; [18540.891296] rc=$val; [18540.891296] fi; [18540.891296] exit $rc [18541.509547] Lustre: DEBUG MARKER: dmesg [18542.105585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 15:41:53 \(1491493313\) [18542.266994] Lustre: DEBUG MARKER: == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 15:41:53 (1491493313) [18543.176792] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18543.440451] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18544.218465] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18544.477748] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18545.266511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18545.528237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18546.310815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18546.572508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18549.400847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18549.678529] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18550.467627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18550.738382] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18551.558820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18551.828282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18553.091465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18554.354655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18555.621403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18556.881559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18557.140979] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18558.176161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18558.437440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18558.699291] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18559.743510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18560.002294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18560.778404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18561.038494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18562.326277] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl7 [18562.864089] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl7 [18563.465593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18564.419157] Lustre: DEBUG MARKER: rc=0; [18564.419157] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18564.419157] if [[ $? -eq 0 && $val -ne 0 ]]; then [18564.419157] echo $(hostname -s): $val; [18564.419157] rc=$val; [18564.419157] fi; [18564.419157] exit $rc [18565.046738] Lustre: DEBUG MARKER: dmesg [18565.624049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223a: Changelog for restore canceled \(import case\) ================================ 15:42:16 \(1491493336\) [18565.788610] Lustre: DEBUG MARKER: == sanity-hsm test 223a: Changelog for restore canceled (import case) ================================ 15:42:16 (1491493336) [18569.499385] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18569.632575] Lustre: lustre-MDD0000: changelog on [18569.634389] Lustre: Skipped 3 previous similar messages [18569.767105] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18570.558428] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18570.822414] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18571.871552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18572.135715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18572.409619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18572.672286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18572.935983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18573.197920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18574.457616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18575.723257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18576.985764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18578.252057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18579.514294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18580.784814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18582.053805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18583.325565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18584.597398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18585.862213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18587.127822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18588.394607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18589.661526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18590.924206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18592.193592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18593.460255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18594.728740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18595.992796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18597.256170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18598.520884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18599.789584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18601.060213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18602.326828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18605.150233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18605.413839] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18606.192209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18606.468903] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18607.251762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18607.511573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18608.774365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18610.037274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18611.295264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18612.555842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18612.827713] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18613.865368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18614.126529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18614.386867] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18615.444930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18615.705325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18616.485636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18616.752974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18618.037173] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl8 [18618.168831] Lustre: lustre-MDD0000: changelog off [18618.171091] Lustre: Skipped 3 previous similar messages [18618.582840] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl8 [18619.192361] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18620.118372] Lustre: DEBUG MARKER: rc=0; [18620.118372] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18620.118372] if [[ $? -eq 0 && $val -ne 0 ]]; then [18620.118372] echo $(hostname -s): $val; [18620.118372] rc=$val; [18620.118372] fi; [18620.118372] exit $rc [18620.737314] Lustre: DEBUG MARKER: dmesg [18621.345480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223b: Changelog for restore canceled \(release case\) =============================== 15:43:12 \(1491493392\) [18621.522024] Lustre: DEBUG MARKER: == sanity-hsm test 223b: Changelog for restore canceled (release case) =============================== 15:43:12 (1491493392) [18623.662264] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18623.936936] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18624.752784] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18625.011545] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18625.803810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18626.065566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18627.327688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18628.589769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18629.855111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18631.122686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18632.388620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18633.657086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18634.931755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18636.196564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18637.475922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18638.750190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18640.018693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18641.292711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18642.562769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18643.832090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18645.094282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18646.358775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18647.622487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18648.893759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18650.155726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18651.419845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18652.682434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18653.948952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18655.224167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18656.489205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18657.754192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18659.023259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18660.288950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18661.558453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18662.824553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18664.088382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18664.381529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18664.643424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18664.914825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18665.183679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [18665.449329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18665.719129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18666.992223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18668.257120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18669.526147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18670.804937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18672.069360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18673.334974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18674.600445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18675.864359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18677.136159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18678.406880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18679.673879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18680.940694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18682.206435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18683.470192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18684.739712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18686.007493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18687.276821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18688.545965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18689.814710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18691.077281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18692.342428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18693.614786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18694.878744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [18697.711523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18697.970698] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18698.748978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18699.009516] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18699.785745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18700.047371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18701.306602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18702.566522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18703.830533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18705.097167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18705.358448] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18706.391168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18706.657175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18706.920309] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18707.961029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18708.223655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18709.004663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18709.261967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18710.538279] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl9 [18710.669037] Lustre: lustre-MDD0000: changelog off [18710.671302] Lustre: Skipped 1 previous similar message [18711.070963] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl9 [18711.661274] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18712.598101] Lustre: DEBUG MARKER: rc=0; [18712.598101] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18712.598101] if [[ $? -eq 0 && $val -ne 0 ]]; then [18712.598101] echo $(hostname -s): $val; [18712.598101] rc=$val; [18712.598101] fi; [18712.598101] exit $rc [18713.235462] Lustre: DEBUG MARKER: dmesg [18713.838144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224: Changelog for remove ========================================================= 15:44:44 \(1491493484\) [18713.996788] Lustre: DEBUG MARKER: == sanity-hsm test 224: Changelog for remove ========================================================= 15:44:44 (1491493484) [18714.926123] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18715.058724] Lustre: lustre-MDD0000: changelog on [18715.060571] Lustre: Skipped 3 previous similar messages [18715.192240] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18715.985789] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18716.248425] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18717.040549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18717.299801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18717.570225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18717.833408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18720.659697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18720.920681] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18721.698480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18721.960289] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18722.742311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18723.001746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18724.261768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18725.524135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18726.787732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18728.045678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18728.304765] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18729.333770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18729.592932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18729.856956] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18730.896106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18731.153059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18731.923340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18732.181362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18733.457067] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl10 [18733.984307] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl10 [18734.564972] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18735.525945] Lustre: DEBUG MARKER: rc=0; [18735.525945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18735.525945] if [[ $? -eq 0 && $val -ne 0 ]]; then [18735.525945] echo $(hostname -s): $val; [18735.525945] rc=$val; [18735.525945] fi; [18735.525945] exit $rc [18736.147315] Lustre: DEBUG MARKER: dmesg [18736.787729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224a: Changelog for failed remove ================================================= 15:45:07 \(1491493507\) [18736.956766] Lustre: DEBUG MARKER: == sanity-hsm test 224a: Changelog for failed remove ================================================= 15:45:07 (1491493507) [18737.893443] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18738.164325] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18738.943994] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18739.204348] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18739.991164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18740.250399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18741.544671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18741.807051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [18744.629300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18744.888864] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18745.672973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18745.933423] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18746.711523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18746.970907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18748.230224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18749.493171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18750.759597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18752.024563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18752.284126] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18753.316651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18753.576033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18753.836669] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18754.871557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18755.139723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18755.912974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18756.171059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18757.444560] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl11 [18757.966757] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl11 [18758.568619] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18759.497474] Lustre: DEBUG MARKER: rc=0; [18759.497474] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18759.497474] if [[ $? -eq 0 && $val -ne 0 ]]; then [18759.497474] echo $(hostname -s): $val; [18759.497474] rc=$val; [18759.497474] fi; [18759.497474] exit $rc [18760.122831] Lustre: DEBUG MARKER: dmesg [18760.719411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 225: Changelog for remove canceled ================================================ 15:45:31 \(1491493531\) [18760.874862] Lustre: DEBUG MARKER: == sanity-hsm test 225: Changelog for remove canceled ================================================ 15:45:31 (1491493531) [18764.314838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18764.575319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18765.353669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18765.615092] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18766.407817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18766.665135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18767.923797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18769.200434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18770.460356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18771.722790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18772.983885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18773.247051] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18774.290618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18774.552452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18774.816322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18775.850991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18776.106181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18776.878191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18777.134542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18778.468279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18779.390927] Lustre: DEBUG MARKER: rc=0; [18779.390927] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18779.390927] if [[ $? -eq 0 && $val -ne 0 ]]; then [18779.390927] echo $(hostname -s): $val; [18779.390927] rc=$val; [18779.390927] fi; [18779.390927] exit $rc [18780.011277] Lustre: DEBUG MARKER: dmesg [18780.628634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 226: changelog for last rm\/mv with exiting archive ================================ 15:45:51 \(1491493551\) [18780.786706] Lustre: DEBUG MARKER: == sanity-hsm test 226: changelog for last rm/mv with exiting archive ================================ 15:45:51 (1491493551) [18781.720284] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18781.981720] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18782.773756] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18783.037922] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18783.831361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18784.093589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18784.364810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18784.637565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18787.475033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18787.734942] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18788.516434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18788.775091] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18789.566074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18789.823678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18791.086109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18792.346473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18793.606004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18794.869381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18795.130647] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18796.166603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18796.425414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18796.691613] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18797.719135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18797.979395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18798.755898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18799.016123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18800.331908] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl12 [18800.859541] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl12 [18801.467846] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18802.389150] Lustre: DEBUG MARKER: rc=0; [18802.389150] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18802.389150] if [[ $? -eq 0 && $val -ne 0 ]]; then [18802.389150] echo $(hostname -s): $val; [18802.389150] rc=$val; [18802.389150] fi; [18802.389150] exit $rc [18803.018036] Lustre: DEBUG MARKER: dmesg [18803.625747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 15:46:14 \(1491493574\) [18803.785273] Lustre: DEBUG MARKER: == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 15:46:14 (1491493574) [18804.677205] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [18804.941324] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [18805.729367] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [18805.998312] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [18806.941200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18807.204240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18810.039220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18810.307615] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18811.087652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18811.347330] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18812.140612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18812.399928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18813.659461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18814.920232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18816.178774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18817.443431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18817.701764] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18818.744382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18819.000382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18819.258950] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18820.296161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18820.570722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18821.345906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18821.605801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18822.875382] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl13 [18823.401407] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl13 [18823.994037] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18824.899518] Lustre: DEBUG MARKER: rc=0; [18824.899518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18824.899518] if [[ $? -eq 0 && $val -ne 0 ]]; then [18824.899518] echo $(hostname -s): $val; [18824.899518] rc=$val; [18824.899518] fi; [18824.899518] exit $rc [18825.520069] Lustre: DEBUG MARKER: dmesg [18826.108986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 15:46:37 \(1491493597\) [18826.275744] Lustre: DEBUG MARKER: == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 15:46:37 (1491493597) [18827.298349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18827.560975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18828.824439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18834.209051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18834.467370] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18835.243639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18835.505355] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18836.278310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18836.543125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18837.801318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18839.063403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18840.324616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18841.588700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18841.850737] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18842.876812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18843.134094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18843.392631] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18844.421531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18844.679921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18845.446830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18845.706062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18847.043437] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18847.970447] Lustre: DEBUG MARKER: rc=0; [18847.970447] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18847.970447] if [[ $? -eq 0 && $val -ne 0 ]]; then [18847.970447] echo $(hostname -s): $val; [18847.970447] rc=$val; [18847.970447] fi; [18847.970447] exit $rc [18848.590348] Lustre: DEBUG MARKER: dmesg [18849.323569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 250: Coordinator max request ====================================================== 15:47:00 \(1491493620\) [18849.484741] Lustre: DEBUG MARKER: == sanity-hsm test 250: Coordinator max request ====================================================== 15:47:00 (1491493620) [18850.404180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [18850.665357] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18851.186590] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18851.706501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18851.965113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18852.742995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18853.001772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18861.812296] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18862.348859] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18862.882040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18863.141448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18863.934837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18864.190391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18865.978685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18866.251281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18867.528515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18867.803122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18869.081333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18869.342580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18870.610888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18870.874524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18872.136533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18872.402325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18873.666557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18873.931160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18875.194070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18875.456869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18876.726525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18876.989619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18878.251313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18878.517603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18879.789273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18880.053560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18881.326480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18881.598200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18882.861400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18883.126789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18884.394729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18884.660096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18885.923253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18886.191257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18887.455208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18887.720168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18888.983454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18889.247903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18890.512429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18890.779277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18892.041858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18892.304847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18893.573139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [18893.842256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [18896.648892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18896.909619] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18897.683303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18897.944656] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18898.730159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18898.988463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18900.254316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18901.514761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18902.772847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18904.035382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18904.295140] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18905.326573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18905.583102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18905.840614] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18906.868941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18907.126775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18907.899666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18908.165784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18909.522427] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18910.433659] Lustre: DEBUG MARKER: rc=0; [18910.433659] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18910.433659] if [[ $? -eq 0 && $val -ne 0 ]]; then [18910.433659] echo $(hostname -s): $val; [18910.433659] rc=$val; [18910.433659] fi; [18910.433659] exit $rc [18911.056553] Lustre: DEBUG MARKER: dmesg [18911.669201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 251: Coordinator request timeout ================================================== 15:48:02 \(1491493682\) [18911.846929] Lustre: DEBUG MARKER: == sanity-hsm test 251: Coordinator request timeout ================================================== 15:48:02 (1491493682) [18916.955161] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [18917.475306] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [18917.996140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18918.253181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18919.032580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18919.293527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18920.064325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [18920.327213] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=4 [18920.841112] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=4 [18921.355941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [18921.623456] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=2 [18922.135478] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=2 [18922.660609] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18923.176582] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18923.691125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18923.948124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18924.735609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18924.994554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18925.848040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18926.107792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18931.376817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18931.641698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [18931.908924] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [18932.424626] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=3600 [18932.951512] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [18933.466724] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=10 [18936.543735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18936.804311] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18937.574873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18937.835758] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18938.607184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18938.864120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18940.127595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18941.386783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18942.649555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18943.905022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18944.163779] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [18946.444921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18946.705359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18946.963430] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [18948.002286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18948.260677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18949.030972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18949.289266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18950.636349] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18951.549291] Lustre: DEBUG MARKER: rc=0; [18951.549291] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18951.549291] if [[ $? -eq 0 && $val -ne 0 ]]; then [18951.549291] echo $(hostname -s): $val; [18951.549291] rc=$val; [18951.549291] fi; [18951.549291] exit $rc [18952.170323] Lustre: DEBUG MARKER: dmesg [18952.777821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 300: On disk coordinator state kept between MDT umount\/mount ====================== 15:48:43 \(1491493723\) [18952.945656] Lustre: DEBUG MARKER: == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ====================== 15:48:43 (1491493723) [18953.092098] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [18953.634206] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [18954.177781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18954.450480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18956.483437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18956.742605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18958.003294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18958.790230] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0000.hsm_control [18958.929215] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm_control in log params [18958.931081] Lustre: Skipped 3 previous similar messages [18959.070564] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0001.hsm_control [18959.329720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0002.hsm_control [18959.590643] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0003.hsm_control [18959.860197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18960.117142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18960.889208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18961.146539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18961.928513] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [18962.186998] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [18962.317423] Lustre: Failing over lustre-MDT0000 [18962.416213] Lustre: server umount lustre-MDT0000 complete [18962.446542] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18962.449900] LustreError: Skipped 36 previous similar messages [18962.556578] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [18962.805307] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [18962.807511] LustreError: Skipped 1 previous similar message [18962.809066] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18962.812801] Lustre: Skipped 1 previous similar message [18972.329980] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18972.333928] LustreError: Skipped 22 previous similar messages [18972.887154] Lustre: DEBUG MARKER: hostname [18973.207482] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [18973.465214] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [18973.720220] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [18973.883224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18973.901554] LustreError: 11-0: MGC10.9.5.243@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [18973.901595] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18973.901695] LustreError: 6498:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [18973.901774] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [18973.901775] Lustre: Skipped 27 previous similar messages [18973.901816] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d587163 to 0x61ac6f141d58b5e9 [18973.917349] LustreError: Skipped 2 previous similar messages [18973.955445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18974.110695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [18974.371052] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [18974.870401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18974.872938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18975.037149] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18975.037166] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [18975.189225] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [18975.452129] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [18975.717161] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [18975.981247] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [18976.031843] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [18976.031872] Lustre: lustre-MDT0000: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [18976.031873] Lustre: Skipped 2 previous similar messages [18976.745896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18976.750148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18976.915991] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18977.008793] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [18978.806944] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [18978.809782] Lustre: Skipped 17 previous similar messages [18978.952387] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [18978.960236] Lustre: lustre-MDT0000: Recovery over after 0:03, of 7 clients 7 recovered and 0 were evicted. [18979.163526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18979.316441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18979.329725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18979.534148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [18980.817781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18981.074822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [18981.854717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18982.111472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [18982.888675] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [18983.026113] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm_control in log params [18983.028790] Lustre: Skipped 3 previous similar messages [18983.162641] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0001.hsm_control=enabled [18983.423369] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0002.hsm_control=enabled [18983.684588] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0003.hsm_control=enabled [18986.011341] LustreError: 16663:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [19003.949503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19004.211143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19004.993840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19005.269515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19006.068287] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [19006.328765] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [19006.460104] Lustre: Failing over lustre-MDT0000 [19006.562955] Lustre: server umount lustre-MDT0000 complete [19006.693701] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19007.066987] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19007.071671] LustreError: Skipped 13 previous similar messages [19008.953133] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [19008.955665] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19008.959996] Lustre: Skipped 1 previous similar message [19015.559035] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491493779/real 1491493779] req@ffff88004e871200 x1563924896457552/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491493786 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19015.566068] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19015.568494] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19017.023821] Lustre: DEBUG MARKER: hostname [19017.339343] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [19017.597061] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [19017.853716] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [19018.015374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19021.573283] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [19021.575594] Lustre: Skipped 6 previous similar messages [19021.577588] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d58b5e9 to 0x61ac6f141d58bbd8 [19021.611809] LustreError: 17519:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [19021.618288] LustreError: 17524:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [19021.620798] LustreError: 17524:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [19021.635990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19021.832611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19022.095628] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [19022.593313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19022.598803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19022.755706] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19022.757213] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19022.903295] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19023.161896] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19023.419411] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19023.686101] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [19024.064134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [19024.458477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19024.471118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19024.664183] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19024.713440] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19026.626086] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19026.643859] Lustre: lustre-MDT0000: Recovery over after 0:03, of 7 clients 7 recovered and 0 were evicted. [19026.917406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19027.025162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19027.080517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19027.236545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19028.531586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19028.794559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19029.570405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19029.830817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19030.680832] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19031.630674] Lustre: DEBUG MARKER: rc=0; [19031.630674] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19031.630674] if [[ $? -eq 0 && $val -ne 0 ]]; then [19031.630674] echo $(hostname -s): $val; [19031.630674] rc=$val; [19031.630674] fi; [19031.630674] exit $rc [19032.281363] Lustre: DEBUG MARKER: dmesg [19032.871939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 301: HSM tunnable are persistent ================================================== 15:50:03 \(1491493803\) [19033.039076] Lustre: DEBUG MARKER: == sanity-hsm test 301: HSM tunnable are persistent ================================================== 15:50:03 (1491493803) [19033.179226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [19033.440600] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [19033.571151] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [19033.573759] Lustre: Skipped 3 previous similar messages [19033.705151] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0001.hsm.default_archive_id=2 [19033.963112] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0002.hsm.default_archive_id=2 [19034.238808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0003.hsm.default_archive_id=2 [19034.522998] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [19034.782991] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [19034.914824] Lustre: Failing over lustre-MDT0000 [19034.998943] Lustre: server umount lustre-MDT0000 complete [19035.129473] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19036.626180] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19036.630804] Lustre: Skipped 1 previous similar message [19043.183050] Lustre: 6498:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491493806/real 1491493806] req@ffff880048d78300 x1563924896461024/t0(0) o101->MGC10.9.5.243@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1491493813 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [19043.190807] Lustre: 6498:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19043.193176] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19043.198294] LustreError: 6498:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [19045.350359] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19045.355058] LustreError: Skipped 66 previous similar messages [19045.472518] Lustre: DEBUG MARKER: hostname [19045.785485] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [19046.042550] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [19046.298345] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [19046.461281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19046.843343] Lustre: MGS: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [19046.845663] Lustre: Skipped 27 previous similar messages [19049.197351] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d58bbd8 to 0x61ac6f141d58c229 [19049.250043] LustreError: 20149:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [19049.257079] LustreError: 20154:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [19049.282081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19049.472601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19049.735799] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [19050.240629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19050.252310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19050.410603] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19050.410627] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19050.559277] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19050.821844] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19051.013678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [19051.092180] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19051.365312] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [19052.151591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19052.173857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19052.310375] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19052.431648] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19054.267516] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19054.285633] Lustre: lustre-MDT0000: Recovery over after 0:03, of 7 clients 7 recovered and 0 were evicted. [19054.586886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19054.712089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19054.738451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19054.905867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19056.199492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [19056.463667] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [19056.599055] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [19056.602084] Lustre: Skipped 3 previous similar messages [19056.738097] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0001.hsm.default_archive_id [19056.999231] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0002.hsm.default_archive_id [19057.263729] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0003.hsm.default_archive_id [19057.600917] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19058.539413] Lustre: DEBUG MARKER: rc=0; [19058.539413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19058.539413] if [[ $? -eq 0 && $val -ne 0 ]]; then [19058.539413] echo $(hostname -s): $val; [19058.539413] rc=$val; [19058.539413] fi; [19058.539413] exit $rc [19059.176163] Lustre: DEBUG MARKER: dmesg [19059.853488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 15:50:30 \(1491493830\) [19060.017233] Lustre: DEBUG MARKER: == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 15:50:30 (1491493830) [19060.163699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [19060.425935] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19060.942916] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19061.461917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19061.723721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19062.983720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19064.248256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19065.517719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19066.780231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19068.043830] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19069.312241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19070.088629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19070.346743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19071.119892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=3 [19071.383102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0001.hsm.default_archive_id=3 [19071.643611] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0002.hsm.default_archive_id=3 [19071.903166] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0003.hsm.default_archive_id=3 [19072.174748] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [19072.438004] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [19072.571248] Lustre: Failing over lustre-MDT0000 [19072.655382] Lustre: server umount lustre-MDT0000 complete [19072.784650] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19074.267281] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [19074.269403] LustreError: Skipped 3 previous similar messages [19074.270895] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19074.274527] Lustre: Skipped 1 previous similar message [19083.115979] Lustre: DEBUG MARKER: hostname [19083.430628] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [19083.685676] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [19083.939873] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [19084.104965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19084.120797] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19084.125708] LustreError: 6498:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [19084.125798] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [19084.125799] Lustre: Skipped 27 previous similar messages [19084.125833] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d58c229 to 0x61ac6f141d58c754 [19084.163161] LustreError: 23747:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [19084.332891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19084.596977] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [19085.107047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19085.109444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19085.273131] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19085.274029] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19085.423309] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19085.681698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19085.941225] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19086.206211] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [19086.970116] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19086.972182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19087.185486] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19087.204793] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19089.175308] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19091.433555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [19091.506327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [19091.586312] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [19091.746320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [19105.967444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19106.142161] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19107.858079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19107.894521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19108.096938] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19108.120622] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [19110.335563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [19110.416577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [19110.517972] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [19110.640946] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [19111.958799] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [19112.233355] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [19112.369407] Lustre: Failing over lustre-MDT0002 [19112.580873] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19120.993999] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [19121.000016] LustreError: Skipped 75 previous similar messages [19122.896931] Lustre: DEBUG MARKER: hostname [19123.217694] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [19123.477462] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [19123.736240] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [19123.885929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19123.923895] Lustre: lustre-MDT0000: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [19123.928320] Lustre: Skipped 32 previous similar messages [19123.941141] LustreError: 25785:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [19123.946337] LustreError: 25789:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [19123.964532] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [19123.967359] Lustre: Skipped 1 previous similar message [19124.117996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19124.377871] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [19124.386119] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 7 clients reconnect [19124.389933] Lustre: Skipped 1 previous similar message [19124.876392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19124.887591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19125.048145] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19125.051593] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19125.197728] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19125.454499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19125.715489] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19125.978943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [19126.775468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [19126.793368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [19126.981293] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [19127.023589] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [19128.964541] Lustre: lustre-MDT0002: Recovery over after 0:04, of 7 clients 7 recovered and 0 were evicted. [19128.967565] Lustre: Skipped 1 previous similar message [19129.265181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [19129.360303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [19129.462166] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [19129.591141] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [19143.810599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19143.980276] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19145.692836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [19145.698355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [19145.912163] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [19145.964780] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [19148.190732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [19148.337698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [19148.403814] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [19148.521642] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [19149.832911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19150.103123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19150.877573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19151.155078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19151.966511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [19152.231742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [19152.363184] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [19152.365985] Lustre: Skipped 7 previous similar messages [19152.500122] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0001.hsm.default_archive_id [19152.759035] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0002.hsm.default_archive_id [19153.019297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0003.hsm.default_archive_id [19153.378946] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19154.305407] Lustre: DEBUG MARKER: rc=0; [19154.305407] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19154.305407] if [[ $? -eq 0 && $val -ne 0 ]]; then [19154.305407] echo $(hostname -s): $val; [19154.305407] rc=$val; [19154.305407] fi; [19154.305407] exit $rc [19154.951428] Lustre: DEBUG MARKER: dmesg [19155.544495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 15:52:06 \(1491493926\) [19155.708831] Lustre: DEBUG MARKER: == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 15:52:06 (1491493926) [19157.264328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19157.528145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19158.789412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19160.054301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19165.941221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19166.204939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19166.983763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19167.246072] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19168.026567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19168.302778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19169.576343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19169.836808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19173.382736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19173.641766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19174.905803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19175.166672] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19176.206778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19176.466395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19177.236548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19177.497308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19178.871627] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19179.793186] Lustre: DEBUG MARKER: rc=0; [19179.793186] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19179.793186] if [[ $? -eq 0 && $val -ne 0 ]]; then [19179.793186] echo $(hostname -s): $val; [19179.793186] rc=$val; [19179.793186] fi; [19179.793186] exit $rc [19180.431129] Lustre: DEBUG MARKER: dmesg [19181.014952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 15:52:31 \(1491493951\) [19181.192120] Lustre: DEBUG MARKER: == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 15:52:31 (1491493951) [19182.611779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19182.874991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19184.138360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19185.406021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19188.744171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19189.003586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19189.780429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19190.039632] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19190.815991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19191.073974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19192.334224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19193.591644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19194.848412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19195.107237] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19196.139004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19196.397938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19196.658807] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19197.685111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19197.946360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19198.719204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19198.981395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19200.356393] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19201.276974] Lustre: DEBUG MARKER: rc=0; [19201.276974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19201.276974] if [[ $? -eq 0 && $val -ne 0 ]]; then [19201.276974] echo $(hostname -s): $val; [19201.276974] rc=$val; [19201.276974] fi; [19201.276974] exit $rc [19201.910190] Lustre: DEBUG MARKER: dmesg [19202.554614] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 15:52:53 \(1491493973\) [19202.707435] Lustre: DEBUG MARKER: == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 15:52:53 (1491493973) [19205.399132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19205.660381] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19206.438367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19206.695915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19207.462944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19207.721385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19208.986909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19210.244869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19211.502726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19212.768376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19214.025155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19215.285314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19215.557554] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19216.592070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19216.851432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19217.110149] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19218.144605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19218.412898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19219.186702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19219.444388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19223.728253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep uuid [19224.252212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep uuid [19225.418775] Lustre: lustre-MDT0000: Client 425652db-3402-9d08-e07d-5c3c3d135eb9 (at 10.9.5.238@tcp) reconnecting [19225.420624] Lustre: Skipped 1 previous similar message [19225.422014] Lustre: lustre-MDT0000: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [19225.424218] Lustre: Skipped 22 previous similar messages [19227.396649] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19228.320095] Lustre: DEBUG MARKER: rc=0; [19228.320095] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19228.320095] if [[ $? -eq 0 && $val -ne 0 ]]; then [19228.320095] echo $(hostname -s): $val; [19228.320095] rc=$val; [19228.320095] fi; [19228.320095] exit $rc [19228.942846] Lustre: DEBUG MARKER: dmesg [19231.984769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19232.248107] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19233.037802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19233.300603] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19234.081299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19234.337768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19235.598259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19235.858761] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19236.889270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19237.149462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19237.409491] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19238.450351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19238.714213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19239.503638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19239.763318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19241.127393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 15:53:32 \(1491494012\) [19241.294505] Lustre: DEBUG MARKER: == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 15:53:32 (1491494012) [19242.343241] LustreError: 1863:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent '3067b518-b7c9-485b-b6ca-498aba1e1c42': rc = -11 [19242.470129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [19252.348455] LustreError: 1863:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent '3067b518-b7c9-485b-b6ca-498aba1e1c42': rc = -11 [19252.733350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19252.997082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19253.513948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19253.788344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19255.059289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19256.320503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19257.589205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19258.856003] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19260.122685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19261.392773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19262.657459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19265.466972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19265.729506] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19266.525646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19266.784657] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19267.558333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19267.815546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19269.079761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19270.341749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19271.604880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19272.863712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19273.124732] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19274.176193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19274.448366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19274.710749] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19275.744309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19276.005327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19276.789796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19277.048561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19278.385572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19279.305684] Lustre: DEBUG MARKER: rc=0; [19279.305684] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19279.305684] if [[ $? -eq 0 && $val -ne 0 ]]; then [19279.305684] echo $(hostname -s): $val; [19279.305684] rc=$val; [19279.305684] fi; [19279.305684] exit $rc [19279.936519] Lustre: DEBUG MARKER: dmesg [19280.541310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 15:54:11 \(1491494051\) [19280.702394] Lustre: DEBUG MARKER: == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 15:54:11 (1491494051) [19283.409793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19283.672342] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19284.447647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19284.705862] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19285.482493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19285.745503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19287.011171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19288.269197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19289.527130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19290.790257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19292.048127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19293.306427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19293.565540] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19294.592861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19294.852737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19295.124319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19296.155634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19296.411893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19297.183817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19297.438871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19300.702383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [19300.883646] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [19301.521155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [19301.678731] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [19301.843824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [19303.413995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [19303.943424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 3067b518-b7c9-485b-b6ca-498aba1e1c42 [19307.008118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19307.267166] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19308.048625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19308.309778] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19309.095472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19309.352711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19310.620584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19311.877363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19313.135484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19314.394023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19314.652319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19315.679417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19315.935148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19316.192224] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19317.218513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19317.475693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19318.249571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19318.509449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19319.873927] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19320.825573] Lustre: DEBUG MARKER: rc=0; [19320.825573] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19320.825573] if [[ $? -eq 0 && $val -ne 0 ]]; then [19320.825573] echo $(hostname -s): $val; [19320.825573] rc=$val; [19320.825573] fi; [19320.825573] exit $rc [19321.464896] Lustre: DEBUG MARKER: dmesg [19322.076979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 15:54:53 \(1491494093\) [19322.263473] Lustre: DEBUG MARKER: == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 15:54:53 (1491494093) [19324.155219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19324.416774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19325.686774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19326.951638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19330.511337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19330.777610] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19331.552156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19331.812919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19332.598295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19332.858648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19334.121692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19335.379026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19336.635880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19336.897166] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19337.927949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19338.193927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19338.456016] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19339.489138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19339.747768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19340.524719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19340.782960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19342.139320] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19343.054537] Lustre: DEBUG MARKER: rc=0; [19343.054537] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19343.054537] if [[ $? -eq 0 && $val -ne 0 ]]; then [19343.054537] echo $(hostname -s): $val; [19343.054537] rc=$val; [19343.054537] fi; [19343.054537] exit $rc [19343.677900] Lustre: DEBUG MARKER: dmesg [19344.286202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 405: archive and release under striped directory ================================== 15:55:15 \(1491494115\) [19344.458429] Lustre: DEBUG MARKER: == sanity-hsm test 405: archive and release under striped directory ================================== 15:55:15 (1491494115) [19345.872076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19346.134088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19347.394736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19348.176377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x8a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19348.444186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x8a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19356.117756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19356.394190] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19357.178563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19357.453994] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19358.232892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19358.500732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19359.758571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19361.022134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19362.283413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19362.546940] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19363.573874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19363.834095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19364.094456] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19365.131202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19365.390478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19366.169965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19366.431469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19367.782709] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19368.699459] Lustre: DEBUG MARKER: rc=0; [19368.699459] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19368.699459] if [[ $? -eq 0 && $val -ne 0 ]]; then [19368.699459] echo $(hostname -s): $val; [19368.699459] rc=$val; [19368.699459] fi; [19368.699459] exit $rc [19369.328356] Lustre: DEBUG MARKER: dmesg [19369.927277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 15:55:40 \(1491494140\) [19370.091783] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 15:55:40 (1491494140) [19370.250108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [19370.250108] /usr/sbin/lctl lustre_build_version 2>/dev/null || [19370.250108] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [19371.530053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19371.791859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19373.054273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19374.317164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [19374.642394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [19374.902865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [19376.164621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [19377.434309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [19377.704796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [19377.966721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [19387.111240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19387.369280] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19388.146737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19388.404969] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19389.180372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19389.441203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19389.703222] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19394.504635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19394.761291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19395.023499] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19396.055606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19396.313993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19397.119335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19397.377723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19398.725961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19399.651865] Lustre: DEBUG MARKER: rc=0; [19399.651865] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19399.651865] if [[ $? -eq 0 && $val -ne 0 ]]; then [19399.651865] echo $(hostname -s): $val; [19399.651865] rc=$val; [19399.651865] fi; [19399.651865] exit $rc [19400.281627] Lustre: DEBUG MARKER: dmesg [19400.889781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 15:56:11 \(1491494171\) [19401.050411] Lustre: DEBUG MARKER: == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 15:56:11 (1491494171) [19401.197856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [19401.197856] /usr/sbin/lctl lustre_build_version 2>/dev/null || [19401.197856] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [19404.032229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19404.293442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19405.076911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19405.348653] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19406.118621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19406.375658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19407.635173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19408.900716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19410.161759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19410.424601] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19415.222848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19415.480370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19415.740172] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19416.771950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19417.030267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19417.803389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19418.063287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19427.726470] Lustre: HSM agent a37041b4-d54f-f25f-fce3-981d01accc78 already registered [19427.728238] Lustre: Skipped 1 previous similar message [19429.562992] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19430.480372] Lustre: DEBUG MARKER: rc=0; [19430.480372] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19430.480372] if [[ $? -eq 0 && $val -ne 0 ]]; then [19430.480372] echo $(hostname -s): $val; [19430.480372] rc=$val; [19430.480372] fi; [19430.480372] exit $rc [19431.101230] Lustre: DEBUG MARKER: dmesg [19434.357190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19434.637101] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [19435.449646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19435.710319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [19436.510318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19436.767679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19438.025300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19439.281884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19440.543063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19440.802347] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [19441.843769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19442.105172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19442.363351] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [19443.385343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19443.644876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [19444.426068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19444.695559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [19446.074262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration 4467 sec ======================================================= 15:56:57 \(1491494217\) [19446.251109] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration 4467 sec ======================================================= 15:56:57 (1491494217) [19448.042116] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [19448.300552] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [19448.434252] Lustre: Failing over lustre-MDT0000 [19448.579324] Lustre: server umount lustre-MDT0000 complete [19448.580812] Lustre: Skipped 1 previous similar message [19448.709424] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19448.960216] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [19448.960234] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19448.960235] Lustre: Skipped 6 previous similar messages [19448.967012] LustreError: Skipped 9 previous similar messages [19448.968949] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19452.400105] LustreError: 6487:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491494223 with bad export cookie 7038122449855630484 [19455.543033] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491494219/real 1491494219] req@ffff880061680f00 x1563924896507904/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491494226 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19455.549113] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19455.551235] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19459.034725] Lustre: DEBUG MARKER: hostname [19459.357884] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [19459.617743] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [19459.874417] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [19460.049163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19461.555267] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [19461.557599] Lustre: Skipped 3 previous similar messages [19461.559642] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d58c754 to 0x61ac6f141d5a564d [19461.599980] LustreError: 17814:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [19461.611891] LustreError: 17819:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [19461.635362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19461.842962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19462.105303] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [19462.618242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19462.619731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19462.781121] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19462.782959] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19462.929345] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19463.018860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [19463.189707] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19463.449347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19463.714971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [19464.506127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19464.507232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19464.728751] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19464.747488] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [19466.617353] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [19466.642489] Lustre: lustre-MDT0000: Recovery over after 0:04, of 7 clients 7 recovered and 0 were evicted. [19467.017800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19467.023847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19467.227143] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19467.263094] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [19489.537355] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [19489.797748] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [19491.617352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19491.620303] Lustre: Skipped 11 previous similar messages [19491.711105] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880067a52a00 x1563924896523712/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [19491.717118] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [19496.247169] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19503.401142] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [19503.617034] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491494267/real 1491494267] req@ffff880067a53300 x1563924896523968/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491494274 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19503.625571] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19503.659359] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [19504.607089] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880078deea00 x1563924896524432/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [19504.613255] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [19510.006500] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [19586.328116] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: ost-pools ============----- Thu Apr 6 15:59:16 UTC 2017 [19586.486242] Lustre: DEBUG MARKER: -----============= acceptance-small: ost-pools ============----- Thu Apr 6 15:59:16 UTC 2017 [19586.738840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration -o sec ========================================================== 15:59:17 \(1491494357\) [19586.892779] Lustre: DEBUG MARKER: == ost-pools test complete, duration -o sec ========================================================== 15:59:17 (1491494357) [19589.336671] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [19589.336671] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [19589.336671] if [ $running -ne $mpts ]; then [19589.336671] echo $(hostname) env are INSANE!; [19589.336671] exit 1; [19589.336671] fi [19589.903409] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [19589.903409] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [19589.903409] if [ $running -ne $mpts ]; then [19589.903409] echo $(hostname) env are INSANE!; [19589.903409] exit 1; [19589.903409] fi [19590.485614] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [19590.485614] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [19590.485614] if [ $running -ne $mpts ]; then [19590.485614] echo $(hostname) env are INSANE!; [19590.485614] exit 1; [19590.485614] fi [19593.009410] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [19593.278458] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [19593.526413] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [19593.774716] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [19593.931904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19594.000606] LustreError: 20123:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [19594.157785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19594.415538] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [19594.909939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19594.911288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19595.069429] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19595.072247] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19595.217130] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19595.471362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19595.723469] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19595.982795] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [19596.237110] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19598.379676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19598.537993] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19599.009118] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491494364/real 1491494364] req@ffff880067d15800 x1563924896524896/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491494369 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [19599.016593] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [19599.961246] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [19600.225113] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [19600.479341] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [19600.732547] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [19600.890855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19600.970468] LustreError: 21419:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [19601.138619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [19601.401360] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [19601.900930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19601.907890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19602.063747] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19602.072691] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19602.217449] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19602.481042] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19602.750232] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [19603.014606] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [19603.271533] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19605.664303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19606.596633] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19609.905678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19610.065973] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19613.074626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19613.223585] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19616.010518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19616.172019] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19619.302713] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19619.454435] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19622.257611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19622.430854] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19625.512215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19625.672228] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19628.599259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19628.756189] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19631.782800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19631.938311] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19634.039913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19634.201522] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [19635.655744] Lustre: DEBUG MARKER: lctl get_param -n timeout [19635.974272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [19636.140975] Lustre: DEBUG MARKER: Using TIMEOUT=20 [19636.273868] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [19636.564435] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [19647.031249] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [19647.312943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19647.836922] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [19648.423177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [19648.578407] Lustre: DEBUG MARKER: excepting tests: [19648.770274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 18 23b [19648.924117] Lustre: DEBUG MARKER: skipping tests SLOW=no: 18 23b [19649.187833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 16:00:20 \(1491494420\) [19649.345687] Lustre: DEBUG MARKER: == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 16:00:20 (1491494420) [19649.479957] Lustre: DEBUG MARKER: lctl pool_new lustre.p [19651.744474] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [19652.004027] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [19660.324281] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [19660.582272] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [19661.391561] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19661.655777] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.p [19661.916981] Lustre: DEBUG MARKER: lctl pool_destroy lustre.p [19672.190282] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [19672.446154] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [19673.225165] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [19673.483080] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [19674.329671] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19675.241634] Lustre: DEBUG MARKER: rc=0; [19675.241634] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19675.241634] if [[ $? -eq 0 && $val -ne 0 ]]; then [19675.241634] echo $(hostname -s): $val; [19675.241634] rc=$val; [19675.241634] fi; [19675.241634] exit $rc [19675.834811] Lustre: DEBUG MARKER: dmesg [19676.395175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 16:00:47 \(1491494447\) [19676.552134] Lustre: DEBUG MARKER: == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 16:00:47 (1491494447) [19676.691615] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12 [19686.957686] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [19687.216914] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [19688.002200] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [19688.257056] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [19689.049188] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19689.308594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool12 [19689.567243] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool12 [19699.832135] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [19700.091282] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [19700.876458] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [19701.134199] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [19701.982081] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19702.881783] Lustre: DEBUG MARKER: rc=0; [19702.881783] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19702.881783] if [[ $? -eq 0 && $val -ne 0 ]]; then [19702.881783] echo $(hostname -s): $val; [19702.881783] rc=$val; [19702.881783] fi; [19702.881783] exit $rc [19703.484271] Lustre: DEBUG MARKER: dmesg [19704.053944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 16:01:15 \(1491494475\) [19704.209165] Lustre: DEBUG MARKER: == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 16:01:15 (1491494475) [19704.347145] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [19714.614151] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19714.873004] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19715.648824] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19715.914757] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19716.716081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19716.974603] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [19717.230208] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [19723.495937] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19723.752414] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19724.525196] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19724.784974] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19727.659933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19728.537107] Lustre: DEBUG MARKER: rc=0; [19728.537107] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19728.537107] if [[ $? -eq 0 && $val -ne 0 ]]; then [19728.537107] echo $(hostname -s): $val; [19728.537107] rc=$val; [19728.537107] fi; [19728.537107] exit $rc [19729.156829] Lustre: DEBUG MARKER: dmesg [19729.731930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1d: Create a pool with a 16 char pool name\; should fail ============================ 16:01:40 \(1491494500\) [19729.886950] Lustre: DEBUG MARKER: == ost-pools test 1d: Create a pool with a 16 char pool name; should fail ============================ 16:01:40 (1491494500) [19730.028380] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12345678 [19730.335984] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19731.223837] Lustre: DEBUG MARKER: rc=0; [19731.223837] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19731.223837] if [[ $? -eq 0 && $val -ne 0 ]]; then [19731.223837] echo $(hostname -s): $val; [19731.223837] rc=$val; [19731.223837] fi; [19731.223837] exit $rc [19731.821705] Lustre: DEBUG MARKER: dmesg [19732.426280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1e: Create a pool with a 1000 char pool name\; should fail ========================== 16:01:43 \(1491494503\) [19732.613086] Lustre: DEBUG MARKER: == ost-pools test 1e: Create a pool with a 1000 char pool name; should fail ========================== 16:01:43 (1491494503) [19732.761856] Lustre: DEBUG MARKER: lctl pool_new lustre.testpooloooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo [19733.075449] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19733.962776] Lustre: DEBUG MARKER: rc=0; [19733.962776] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19733.962776] if [[ $? -eq 0 && $val -ne 0 ]]; then [19733.962776] echo $(hostname -s): $val; [19733.962776] rc=$val; [19733.962776] fi; [19733.962776] exit $rc [19734.567648] Lustre: DEBUG MARKER: dmesg [19735.140338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 16:01:46 \(1491494506\) [19735.306450] Lustre: DEBUG MARKER: == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 16:01:46 (1491494506) [19735.439994] Lustre: DEBUG MARKER: lctl pool_new .testpool [19735.757701] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19736.640743] Lustre: DEBUG MARKER: rc=0; [19736.640743] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19736.640743] if [[ $? -eq 0 && $val -ne 0 ]]; then [19736.640743] echo $(hostname -s): $val; [19736.640743] rc=$val; [19736.640743] fi; [19736.640743] exit $rc [19737.248861] Lustre: DEBUG MARKER: dmesg [19737.823613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 16:01:48 \(1491494508\) [19737.990787] Lustre: DEBUG MARKER: == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 16:01:48 (1491494508) [19738.135194] Lustre: DEBUG MARKER: lctl pool_new testpool [19738.457823] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19739.326146] Lustre: DEBUG MARKER: rc=0; [19739.326146] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19739.326146] if [[ $? -eq 0 && $val -ne 0 ]]; then [19739.326146] echo $(hostname -s): $val; [19739.326146] rc=$val; [19739.326146] fi; [19739.326146] exit $rc [19739.920455] Lustre: DEBUG MARKER: dmesg [19740.468225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 16:01:51 \(1491494511\) [19740.627200] Lustre: DEBUG MARKER: == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 16:01:51 (1491494511) [19740.764609] Lustre: DEBUG MARKER: lctl pool_new lustre. [19741.079838] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19741.991556] Lustre: DEBUG MARKER: rc=0; [19741.991556] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19741.991556] if [[ $? -eq 0 && $val -ne 0 ]]; then [19741.991556] echo $(hostname -s): $val; [19741.991556] rc=$val; [19741.991556] fi; [19741.991556] exit $rc [19742.786652] Lustre: DEBUG MARKER: dmesg [19743.637358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 16:01:54 \(1491494514\) [19743.798240] Lustre: DEBUG MARKER: == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 16:01:54 (1491494514) [19743.943832] Lustre: DEBUG MARKER: lctl pool_new . [19744.255187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19745.135685] Lustre: DEBUG MARKER: rc=0; [19745.135685] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19745.135685] if [[ $? -eq 0 && $val -ne 0 ]]; then [19745.135685] echo $(hostname -s): $val; [19745.135685] rc=$val; [19745.135685] fi; [19745.135685] exit $rc [19745.740302] Lustre: DEBUG MARKER: dmesg [19746.336217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 16:01:57 \(1491494517\) [19746.493842] Lustre: DEBUG MARKER: == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 16:01:57 (1491494517) [19746.625926] Lustre: DEBUG MARKER: lctl pool_new lustre,testpool [19746.936154] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19747.830910] Lustre: DEBUG MARKER: rc=0; [19747.830910] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19747.830910] if [[ $? -eq 0 && $val -ne 0 ]]; then [19747.830910] echo $(hostname -s): $val; [19747.830910] rc=$val; [19747.830910] fi; [19747.830910] exit $rc [19748.433760] Lustre: DEBUG MARKER: dmesg [19749.045409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 16:02:00 \(1491494520\) [19749.197155] Lustre: DEBUG MARKER: == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 16:02:00 (1491494520) [19749.340933] Lustre: DEBUG MARKER: lctl pool_new lustre/testpool [19749.655670] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19750.543611] Lustre: DEBUG MARKER: rc=0; [19750.543611] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19750.543611] if [[ $? -eq 0 && $val -ne 0 ]]; then [19750.543611] echo $(hostname -s): $val; [19750.543611] rc=$val; [19750.543611] fi; [19750.543611] exit $rc [19751.135164] Lustre: DEBUG MARKER: dmesg [19751.834395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 16:02:02 \(1491494522\) [19751.986811] Lustre: DEBUG MARKER: == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 16:02:02 (1491494522) [19752.119889] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [19762.384661] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19762.640491] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19763.413076] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19763.671016] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19764.475724] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [19764.734416] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19764.995639] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19765.253142] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [19775.519259] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19775.779503] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19776.551033] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19776.806938] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19777.652529] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19778.538701] Lustre: DEBUG MARKER: rc=0; [19778.538701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19778.538701] if [[ $? -eq 0 && $val -ne 0 ]]; then [19778.538701] echo $(hostname -s): $val; [19778.538701] rc=$val; [19778.538701] fi; [19778.538701] exit $rc [19779.135537] Lustre: DEBUG MARKER: dmesg [19779.749411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 16:02:30 \(1491494550\) [19779.904093] Lustre: DEBUG MARKER: == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 16:02:30 (1491494550) [19780.039185] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [19790.305259] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19790.564768] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19791.349592] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19791.609026] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19792.417818] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567 OST0000 [19804.683565] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [19804.683565] sort -u | tr '\n' ' ' [19804.944327] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [19804.944327] sort -u | tr '\n' ' ' [19805.725517] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 | [19805.725517] sort -u | tr '\n' ' ' [19805.987109] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 | [19805.987109] sort -u | tr '\n' ' ' [19809.600261] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1234567 OST0000 [19819.935886] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19820.211720] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [19820.485907] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [19830.760904] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19831.028141] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19831.808084] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19832.067553] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [19832.929037] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19833.819888] Lustre: DEBUG MARKER: rc=0; [19833.819888] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19833.819888] if [[ $? -eq 0 && $val -ne 0 ]]; then [19833.819888] echo $(hostname -s): $val; [19833.819888] rc=$val; [19833.819888] fi; [19833.819888] exit $rc [19834.428966] Lustre: DEBUG MARKER: dmesg [19835.019035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 16:03:26 \(1491494606\) [19835.204134] Lustre: DEBUG MARKER: == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 16:03:26 (1491494606) [19835.342661] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19835.602057] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [19835.924777] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19836.817641] Lustre: DEBUG MARKER: rc=0; [19836.817641] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19836.817641] if [[ $? -eq 0 && $val -ne 0 ]]; then [19836.817641] echo $(hostname -s): $val; [19836.817641] rc=$val; [19836.817641] fi; [19836.817641] exit $rc [19837.416383] Lustre: DEBUG MARKER: dmesg [19838.005872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 16:03:28 \(1491494608\) [19838.155110] Lustre: DEBUG MARKER: == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 16:03:28 (1491494608) [19838.290489] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567890 lustre-OST0000 [19838.599692] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19839.488819] Lustre: DEBUG MARKER: rc=0; [19839.488819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19839.488819] if [[ $? -eq 0 && $val -ne 0 ]]; then [19839.488819] echo $(hostname -s): $val; [19839.488819] rc=$val; [19839.488819] fi; [19839.488819] exit $rc [19840.108139] Lustre: DEBUG MARKER: dmesg [19840.664664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2c: pool_add: OST index combinations =============================================== 16:03:31 \(1491494611\) [19840.843328] Lustre: DEBUG MARKER: == ost-pools test 2c: pool_add: OST index combinations =============================================== 16:03:31 (1491494611) [19840.980004] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19847.253190] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19847.510020] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19848.297862] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19848.554830] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19851.395223] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [19859.659073] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [19869.940522] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [19878.205507] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [19886.487453] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [19890.761973] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19899.050603] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000,0001,0002,0003,0004,0005,0006,0007,] [19905.319136] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000,0001,0002,0003,0004,0005,0006,0007,] [19913.611745] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [19923.950133] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0007/1] [19932.262315] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19932.519037] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19932.776144] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19937.042410] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19937.296645] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19941.834665] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19942.092650] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19942.950073] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19943.845609] Lustre: DEBUG MARKER: rc=0; [19943.845609] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19943.845609] if [[ $? -eq 0 && $val -ne 0 ]]; then [19943.845609] echo $(hostname -s): $val; [19943.845609] rc=$val; [19943.845609] fi; [19943.845609] exit $rc [19944.434104] Lustre: DEBUG MARKER: dmesg [19945.032943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2d: pool_add: OSTs that don\'t exist should be rejected ============================= 16:05:16 \(1491494716\) [19945.213508] Lustre: DEBUG MARKER: == ost-pools test 2d: pool_add: OSTs that don't exist should be rejected ============================= 16:05:16 (1491494716) [19945.363368] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19951.629954] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19951.885413] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19952.651216] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19952.907237] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19953.716563] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0008_UUID [19953.976551] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19954.236179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19954.496326] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19960.767259] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19961.025249] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19965.603459] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19965.866873] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19966.716199] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19967.583413] Lustre: DEBUG MARKER: rc=0; [19967.583413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19967.583413] if [[ $? -eq 0 && $val -ne 0 ]]; then [19967.583413] echo $(hostname -s): $val; [19967.583413] rc=$val; [19967.583413] fi; [19967.583413] exit $rc [19968.176973] Lustre: DEBUG MARKER: dmesg [19968.775255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 16:05:39 \(1491494739\) [19968.925918] Lustre: DEBUG MARKER: == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 16:05:39 (1491494739) [19969.062369] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19975.326923] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19975.585226] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19976.360467] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19976.618502] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19979.434370] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [19987.718278] Lustre: DEBUG MARKER: LOCALE=C /usr/sbin/lctl pool_add lustre.testpool lustre-OST0000_UUID 2>&1 [19987.981274] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19988.241959] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19988.502215] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19998.766215] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20007.030723] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20007.289325] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20008.079074] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20008.336249] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20009.178162] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20010.063700] Lustre: DEBUG MARKER: rc=0; [20010.063700] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20010.063700] if [[ $? -eq 0 && $val -ne 0 ]]; then [20010.063700] echo $(hostname -s): $val; [20010.063700] rc=$val; [20010.063700] fi; [20010.063700] exit $rc [20010.655060] Lustre: DEBUG MARKER: dmesg [20011.238367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3a: pool_remove: non-existant pool ================================================= 16:06:22 \(1491494782\) [20011.397076] Lustre: DEBUG MARKER: == ost-pools test 3a: pool_remove: non-existant pool ================================================= 16:06:22 (1491494782) [20011.535957] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [20011.838422] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20012.721901] Lustre: DEBUG MARKER: rc=0; [20012.721901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20012.721901] if [[ $? -eq 0 && $val -ne 0 ]]; then [20012.721901] echo $(hostname -s): $val; [20012.721901] rc=$val; [20012.721901] fi; [20012.721901] exit $rc [20013.309465] Lustre: DEBUG MARKER: dmesg [20013.864422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 16:06:24 \(1491494784\) [20014.019265] Lustre: DEBUG MARKER: == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 16:06:24 (1491494784) [20014.154907] Lustre: DEBUG MARKER: lctl pool_remove nonexistantfs.testpool OST0000 [20014.455632] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20015.348173] Lustre: DEBUG MARKER: rc=0; [20015.348173] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20015.348173] if [[ $? -eq 0 && $val -ne 0 ]]; then [20015.348173] echo $(hostname -s): $val; [20015.348173] rc=$val; [20015.348173] fi; [20015.348173] exit $rc [20015.939339] Lustre: DEBUG MARKER: dmesg [20016.541222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 16:06:27 \(1491494787\) [20016.695373] Lustre: DEBUG MARKER: == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 16:06:27 (1491494787) [20016.829020] Lustre: DEBUG MARKER: lctl pool_remove lustre.p1234567891234567890 lustre-OST0000 [20017.144569] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20018.019122] Lustre: DEBUG MARKER: rc=0; [20018.019122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20018.019122] if [[ $? -eq 0 && $val -ne 0 ]]; then [20018.019122] echo $(hostname -s): $val; [20018.019122] rc=$val; [20018.019122] fi; [20018.019122] exit $rc [20018.614941] Lustre: DEBUG MARKER: dmesg [20019.202674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3d: pool_remove: OST index combinations ============================================ 16:06:30 \(1491494790\) [20019.352083] Lustre: DEBUG MARKER: == ost-pools test 3d: pool_remove: OST index combinations ============================================ 16:06:30 (1491494790) [20019.485957] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20027.749726] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20028.007526] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20030.038493] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20030.296856] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20031.097629] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [20033.362213] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [20041.644410] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [20047.911453] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [20058.191575] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [20062.463413] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20068.760158] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [20075.035610] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20075.035610] sort -u | tr '\n' ' ' [20075.294251] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20075.294251] sort -u | tr '\n' ' ' [20076.086808] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20076.086808] sort -u | tr '\n' ' ' [20076.348145] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20076.348145] sort -u | tr '\n' ' ' [20077.162218] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0007/1] [20084.483832] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20084.746179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20085.004629] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20089.269251] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20089.527244] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20095.335624] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20095.595194] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20096.445346] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20097.333170] Lustre: DEBUG MARKER: rc=0; [20097.333170] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20097.333170] if [[ $? -eq 0 && $val -ne 0 ]]; then [20097.333170] echo $(hostname -s): $val; [20097.333170] rc=$val; [20097.333170] fi; [20097.333170] exit $rc [20097.943622] Lustre: DEBUG MARKER: dmesg [20098.571662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 16:07:49 \(1491494869\) [20098.728888] Lustre: DEBUG MARKER: == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 16:07:49 (1491494869) [20098.874994] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20099.181305] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20100.074524] Lustre: DEBUG MARKER: rc=0; [20100.074524] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20100.074524] if [[ $? -eq 0 && $val -ne 0 ]]; then [20100.074524] echo $(hostname -s): $val; [20100.074524] rc=$val; [20100.074524] fi; [20100.074524] exit $rc [20100.669587] Lustre: DEBUG MARKER: dmesg [20101.283131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 16:07:52 \(1491494872\) [20101.436409] Lustre: DEBUG MARKER: == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 16:07:52 (1491494872) [20101.575808] Lustre: DEBUG MARKER: lctl pool_destroy nonexistantfs.testpool [20101.896328] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20102.776901] Lustre: DEBUG MARKER: rc=0; [20102.776901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20102.776901] if [[ $? -eq 0 && $val -ne 0 ]]; then [20102.776901] echo $(hostname -s): $val; [20102.776901] rc=$val; [20102.776901] fi; [20102.776901] exit $rc [20103.375192] Lustre: DEBUG MARKER: dmesg [20103.967802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 16:07:54 \(1491494874\) [20104.122754] Lustre: DEBUG MARKER: == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 16:07:54 (1491494874) [20104.254200] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20112.518730] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20112.779033] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20113.551835] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20113.806950] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20114.615192] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [20118.883505] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20118.883505] sort -u | tr '\n' ' ' [20119.141700] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20119.141700] sort -u | tr '\n' ' ' [20121.185161] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20121.185161] sort -u | tr '\n' ' ' [20121.449585] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20121.449585] sort -u | tr '\n' ' ' [20122.255770] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20122.516334] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20122.776064] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20123.032186] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20127.293970] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20137.564659] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20137.824227] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20138.630668] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20138.887549] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20139.731784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20140.617284] Lustre: DEBUG MARKER: rc=0; [20140.617284] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20140.617284] if [[ $? -eq 0 && $val -ne 0 ]]; then [20140.617284] echo $(hostname -s): $val; [20140.617284] rc=$val; [20140.617284] fi; [20140.617284] exit $rc [20141.206397] Lustre: DEBUG MARKER: dmesg [20141.813024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5a: lfs pool_list from client ====================================================== 16:08:32 \(1491494912\) [20141.967513] Lustre: DEBUG MARKER: == ost-pools test 5a: lfs pool_list from client ====================================================== 16:08:32 (1491494912) [20142.105184] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20142.365203] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20142.626139] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20152.890227] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20153.147469] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20153.926427] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20154.180759] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20154.986399] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20163.250294] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20163.506773] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20164.270902] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20164.527137] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20165.335446] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [20173.621637] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20173.884908] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20174.159989] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20182.434794] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [20190.697304] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20198.962143] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [20203.222909] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20211.485134] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [20217.750440] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [20222.011515] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [20230.275495] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20238.551219] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20238.805038] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20239.579038] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20239.836159] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20240.635525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20240.895942] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20241.156017] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [20247.427395] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20247.684299] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20249.715631] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20249.976207] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20251.839164] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20252.723700] Lustre: DEBUG MARKER: rc=0; [20252.723700] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20252.723700] if [[ $? -eq 0 && $val -ne 0 ]]; then [20252.723700] echo $(hostname -s): $val; [20252.723700] rc=$val; [20252.723700] fi; [20252.723700] exit $rc [20253.331612] Lustre: DEBUG MARKER: dmesg [20253.930454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5b: lctl pool_list from MDS ======================================================== 16:10:24 \(1491495024\) [20254.082807] Lustre: DEBUG MARKER: == ost-pools test 5b: lctl pool_list from MDS ======================================================== 16:10:24 (1491495024) [20254.220419] Lustre: DEBUG MARKER: lctl pool_list [20254.478210] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20254.735900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20254.993124] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20265.259206] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20265.515557] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20266.287642] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20266.547559] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20267.340109] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20275.605376] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20275.866451] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20276.638278] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20276.895599] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20277.704283] Lustre: DEBUG MARKER: lctl pool_list lustre [20277.960460] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [20286.237074] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [20286.496456] Lustre: DEBUG MARKER: lctl pool_list nonexistantfs [20286.752283] Lustre: DEBUG MARKER: lctl pool_list lustre.nonexistantpool [20287.009851] Lustre: DEBUG MARKER: lctl pool_list /mnt/lustrenonexistant [20287.267775] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20287.528765] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20287.790533] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20292.054165] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [20298.323427] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20304.601112] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [20314.874639] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20319.137920] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [20325.402929] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [20333.670756] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [20339.932841] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20348.205917] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20348.478979] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20349.256412] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20349.515278] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20350.314724] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20350.582601] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20350.839769] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [20355.107333] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20355.365838] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20356.134231] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20356.393222] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20357.255799] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20358.153839] Lustre: DEBUG MARKER: rc=0; [20358.153839] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20358.153839] if [[ $? -eq 0 && $val -ne 0 ]]; then [20358.153839] echo $(hostname -s): $val; [20358.153839] rc=$val; [20358.153839] fi; [20358.153839] exit $rc [20358.754238] Lustre: DEBUG MARKER: dmesg [20359.318674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 6: getstripe\/setstripe ============================================================= 16:12:10 \(1491495130\) [20359.480915] Lustre: DEBUG MARKER: == ost-pools test 6: getstripe/setstripe ============================================================= 16:12:10 (1491495130) [20359.615655] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20361.889794] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20362.145348] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20362.925640] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20363.180369] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20364.002352] Lustre: DEBUG MARKER: lctl pool_list lustre [20364.264201] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [20368.533900] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20368.533900] sort -u | tr '\n' ' ' [20368.792614] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20368.792614] sort -u | tr '\n' ' ' [20369.573079] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20369.573079] sort -u | tr '\n' ' ' [20369.843143] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20369.843143] sort -u | tr '\n' ' ' [20371.678022] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20377.948857] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20378.222976] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20379.006555] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20379.268201] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20380.066728] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [20384.331181] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [20384.331181] sort -u | tr '\n' ' ' [20384.590354] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [20384.590354] sort -u | tr '\n' ' ' [20385.368269] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [20385.368269] sort -u | tr '\n' ' ' [20385.634842] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [20385.634842] sort -u | tr '\n' ' ' [20387.506759] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20388.395272] Lustre: DEBUG MARKER: rc=0; [20388.395272] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20388.395272] if [[ $? -eq 0 && $val -ne 0 ]]; then [20388.395272] echo $(hostname -s): $val; [20388.395272] rc=$val; [20388.395272] fi; [20388.395272] exit $rc [20388.999237] Lustre: DEBUG MARKER: dmesg [20389.545807] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20389.805293] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20390.063972] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20398.329117] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [20406.593254] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20416.861246] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [20427.128493] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20435.396838] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [20443.662401] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [20451.924282] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [20462.189875] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20470.464324] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20470.729894] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20471.503444] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20471.761754] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20472.571925] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20472.832843] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20473.089774] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [20483.358427] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [20491.628626] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20491.888549] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20492.665728] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20492.932878] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20493.819182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7a: create various pool name ======================================================= 16:14:24 \(1491495264\) [20493.989135] Lustre: DEBUG MARKER: == ost-pools test 7a: create various pool name ======================================================= 16:14:24 (1491495264) [20494.144562] Lustre: DEBUG MARKER: lctl pool_new lustre.1 [20504.411425] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 2>/dev/null || echo foo [20504.670243] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 2>/dev/null || echo foo [20505.441699] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 2>/dev/null || echo foo [20505.701872] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 2>/dev/null || echo foo [20506.506125] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.1 lustre-OST[0-1/1] [20510.773966] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 | [20510.773966] sort -u | tr '\n' ' ' [20511.032052] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 | [20511.032052] sort -u | tr '\n' ' ' [20514.344343] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 | [20514.344343] sort -u | tr '\n' ' ' [20514.606995] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 | [20514.606995] sort -u | tr '\n' ' ' [20516.448350] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.1 [20516.708409] Lustre: DEBUG MARKER: lctl pool_remove lustre.1 lustre-OST0000_UUID [20524.972203] Lustre: DEBUG MARKER: lctl pool_remove lustre.1 lustre-OST0001_UUID [20531.234837] Lustre: DEBUG MARKER: lctl pool_destroy lustre.1 [20539.502536] Lustre: DEBUG MARKER: lctl pool_new lustre.119221712 [20547.766520] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221712 2>/dev/null || echo foo [20548.023941] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221712 2>/dev/null || echo foo [20548.810763] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221712 2>/dev/null || echo foo [20549.069072] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221712 2>/dev/null || echo foo [20549.873161] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.119221712 lustre-OST[0-1/1] [20560.144837] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221712 | [20560.144837] sort -u | tr '\n' ' ' [20560.405069] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221712 | [20560.405069] sort -u | tr '\n' ' ' [20561.180722] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221712 | [20561.180722] sort -u | tr '\n' ' ' [20561.443252] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221712 | [20561.443252] sort -u | tr '\n' ' ' [20562.266583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.119221712 [20562.525275] Lustre: DEBUG MARKER: lctl pool_remove lustre.119221712 lustre-OST0000_UUID [20568.788530] Lustre: DEBUG MARKER: lctl pool_remove lustre.119221712 lustre-OST0001_UUID [20577.051301] Lustre: DEBUG MARKER: lctl pool_destroy lustre.119221712 [20583.323836] Lustre: DEBUG MARKER: lctl pool_new lustre.119221024127959 [20591.588433] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221024127959 2>/dev/null || echo foo [20591.845632] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221024127959 2>/dev/null || echo foo [20592.624969] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221024127959 2>/dev/null || echo foo [20592.886168] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221024127959 2>/dev/null || echo foo [20593.699031] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.119221024127959 lustre-OST[0-1/1] [20601.981759] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221024127959 | [20601.981759] sort -u | tr '\n' ' ' [20602.258234] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.119221024127959 | [20602.258234] sort -u | tr '\n' ' ' [20603.044565] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221024127959 | [20603.044565] sort -u | tr '\n' ' ' [20603.307394] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.119221024127959 | [20603.307394] sort -u | tr '\n' ' ' [20604.138926] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.119221024127959 [20604.400164] Lustre: DEBUG MARKER: lctl pool_remove lustre.119221024127959 lustre-OST0000_UUID [20614.664966] Lustre: DEBUG MARKER: lctl pool_remove lustre.119221024127959 lustre-OST0001_UUID [20620.927927] Lustre: DEBUG MARKER: lctl pool_destroy lustre.119221024127959 [20631.238317] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20632.128471] Lustre: DEBUG MARKER: rc=0; [20632.128471] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20632.128471] if [[ $? -eq 0 && $val -ne 0 ]]; then [20632.128471] echo $(hostname -s): $val; [20632.128471] rc=$val; [20632.128471] fi; [20632.128471] exit $rc [20632.732873] Lustre: DEBUG MARKER: dmesg [20633.250116] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20633.525288] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20633.800375] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20634.152938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 16:16:45 \(1491495405\) [20634.316925] Lustre: DEBUG MARKER: == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 16:16:45 (1491495405) [20634.463372] Lustre: DEBUG MARKER: lctl pool_new qwerty [20634.727970] Lustre: DEBUG MARKER: lctl pool_new lustre. [20634.988180] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789^bdef [20635.244309] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abdefg [20635.553789] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20636.440120] Lustre: DEBUG MARKER: rc=0; [20636.440120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20636.440120] if [[ $? -eq 0 && $val -ne 0 ]]; then [20636.440120] echo $(hostname -s): $val; [20636.440120] rc=$val; [20636.440120] fi; [20636.440120] exit $rc [20637.036063] Lustre: DEBUG MARKER: dmesg [20637.586757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 16:16:48 \(1491495408\) [20637.734297] Lustre: DEBUG MARKER: == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 16:16:48 (1491495408) [20637.881522] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abcde [20646.152304] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [20646.421244] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [20647.197654] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [20647.456448] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [20648.266299] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.0123456789abcde lustre-OST[0-1/1] [20654.533482] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [20654.533482] sort -u | tr '\n' ' ' [20654.800865] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [20654.800865] sort -u | tr '\n' ' ' [20655.578370] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde | [20655.578370] sort -u | tr '\n' ' ' [20655.840176] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde | [20655.840176] sort -u | tr '\n' ' ' [20658.678348] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.0123456789abcde [20658.938873] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0000_UUID [20665.206326] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0001_UUID [20673.468468] Lustre: DEBUG MARKER: lctl pool_destroy lustre.0123456789abcde [20683.792188] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20684.674804] Lustre: DEBUG MARKER: rc=0; [20684.674804] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20684.674804] if [[ $? -eq 0 && $val -ne 0 ]]; then [20684.674804] echo $(hostname -s): $val; [20684.674804] rc=$val; [20684.674804] fi; [20684.674804] exit $rc [20685.283822] Lustre: DEBUG MARKER: dmesg [20685.772543] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20686.199051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 11: OSTs in overlapping\/multiple pools ============================================= 16:17:37 \(1491495457\) [20686.356212] Lustre: DEBUG MARKER: == ost-pools test 11: OSTs in overlapping/multiple pools ============================================= 16:17:37 (1491495457) [20686.493201] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20694.760543] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20695.018602] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20697.065153] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20697.320626] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20698.123655] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20708.391298] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20708.648664] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20709.422575] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20709.676653] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20710.493317] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [20718.762686] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [20729.028936] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20729.028936] sort -u | tr '\n' ' ' [20729.290286] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20729.290286] sort -u | tr '\n' ' ' [20730.066930] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20730.066930] sort -u | tr '\n' ' ' [20730.324163] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20730.324163] sort -u | tr '\n' ' ' [20731.269070] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20731.537299] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20731.818681] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20732.086140] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20732.365024] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20732.632632] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20732.918726] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20733.194373] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20733.481577] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20733.752596] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20734.035237] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20734.305761] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20734.596464] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20734.864641] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20735.152624] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20735.420916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20735.704149] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20735.973774] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20736.269860] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20736.546918] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20736.831363] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20737.108594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20737.390341] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20737.656178] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20737.938843] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20738.205374] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20738.489435] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20738.765566] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20739.309800] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20740.019069] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20740.755163] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20741.061985] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20741.345040] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20741.614225] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20741.897208] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20742.164199] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20742.446858] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20742.717656] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20742.998880] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20743.265609] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20743.547230] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20743.812952] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20744.093350] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20744.361541] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20744.646081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20744.921055] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20745.205568] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20745.471403] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20745.750387] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20746.019350] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20746.299129] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20746.582989] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20746.873132] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20747.143954] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20747.440782] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20747.715123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20747.994833] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20748.264280] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20748.545471] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20748.812564] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20749.099044] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20749.365489] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20749.649159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20749.924155] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20750.210857] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20750.477985] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20750.762536] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20751.031728] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20751.312350] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20751.596040] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20751.879325] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20752.145757] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20752.427680] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20752.694770] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20752.978206] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20753.245377] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20753.523109] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20753.794212] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20754.084509] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20754.353212] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20754.632184] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20754.900244] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20755.182104] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20755.449782] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20755.735711] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20756.001782] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20756.282258] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20756.551477] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20756.847940] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20757.115357] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20757.401475] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20757.669679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20757.954080] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20758.223985] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20758.507251] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20758.779984] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20759.064469] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20759.331147] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20759.611149] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20759.881352] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20760.164102] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20760.429362] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20760.721802] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20761.007688] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20761.296646] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20761.563651] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20761.856204] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20762.122779] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20762.406271] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20762.673719] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20762.956515] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20763.224114] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20763.509110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20763.775643] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20764.057673] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20764.323149] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20764.603718] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20764.873266] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20765.157590] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20765.425916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20765.707503] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20765.973110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20766.257912] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20766.526615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20766.808142] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20767.089402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20767.376324] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20767.643783] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20767.928625] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20768.194713] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20768.476719] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20768.740213] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20769.024903] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20769.290814] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20769.570979] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20769.837225] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20770.117939] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20770.389200] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20770.671498] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20770.939359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20771.218693] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20771.486436] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20771.768173] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20772.034472] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20772.324918] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20772.593120] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20772.872580] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20773.137351] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20773.420344] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20773.685759] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20773.964460] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20774.231879] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20774.508972] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20774.777843] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20775.059127] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20775.322359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20775.604152] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20775.868812] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20776.150978] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20776.419588] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20776.703607] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20776.973923] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20777.255139] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20777.531768] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20777.814404] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20778.077982] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20778.357500] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20778.628583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20778.905827] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20779.169381] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20779.447283] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20779.718284] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20780.000084] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20780.269566] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20780.552886] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20780.821302] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20781.103269] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20781.376461] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20781.657369] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20781.925252] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20782.206936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20782.476388] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20782.760127] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20783.026678] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20783.307362] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20783.575683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20783.854441] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20784.119781] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20784.399118] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20784.667658] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20784.953287] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20785.221715] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20785.503450] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20785.770858] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20786.051073] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20786.315298] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20786.594780] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20786.859950] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20787.143168] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20787.410636] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20787.811913] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20788.083104] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20788.385928] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20788.662210] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20788.948818] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20789.216004] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20789.496627] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20789.765121] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20790.056021] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20790.337801] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20790.634080] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20790.901841] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20791.182160] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20791.448323] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20791.738491] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20792.007439] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20792.288195] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20792.555359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20792.855958] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20793.126174] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20793.406187] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20793.673085] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20793.953834] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20794.219875] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20794.500168] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20794.769763] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20795.052038] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20795.318454] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20795.600654] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20795.868336] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20796.150680] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20796.420764] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20796.702201] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20796.969771] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20797.256138] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20797.520275] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20797.806320] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20798.082972] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20798.366800] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20798.633407] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20798.918363] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20799.186744] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20799.470548] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20799.740649] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20800.024158] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20800.290202] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20800.570437] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20800.841271] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20801.121812] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20801.387404] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20801.674916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20801.941723] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20802.229964] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20802.498066] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20802.776594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20803.045326] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20803.337404] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20803.604191] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20803.884781] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20804.151404] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20804.432117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20804.700962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20804.983045] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20805.257255] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20805.544881] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20805.810283] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20806.088745] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20806.356973] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20806.645712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20806.910345] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20807.189619] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20807.457349] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20807.734589] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20808.005140] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20808.293235] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20808.574151] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20808.853893] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20809.126148] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20809.414764] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20809.696836] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20809.992410] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20810.265413] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20810.549994] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20810.820745] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20811.104346] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20811.374911] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20811.659208] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20811.925515] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20812.208167] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20812.475535] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20812.757982] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20813.024446] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20813.305268] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20813.586615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20813.869504] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20814.148136] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20814.440167] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20814.705679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20814.984762] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20815.255954] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20815.538168] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20815.805906] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20816.083936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20816.350913] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20816.632642] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20816.897710] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20817.177266] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20817.446247] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20817.742878] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20818.020900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20818.302638] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20818.572162] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20818.853972] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20819.124898] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20819.408886] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20819.676112] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20819.959647] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20820.229464] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20820.513217] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20820.783732] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20821.069275] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20821.338124] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20821.622936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20821.893512] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20822.177297] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20822.451384] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20822.733186] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20823.002581] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20823.286822] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20823.557595] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20823.844431] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20824.120880] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20824.404386] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20824.675483] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20824.960441] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20825.229625] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20825.516991] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20825.783618] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20826.067349] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20826.334596] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20826.617245] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20826.887367] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20827.166556] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20827.434543] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20827.715916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20827.986137] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20828.269062] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20828.539583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20828.822583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20829.090046] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20829.385154] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20829.659522] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20829.951731] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20830.220874] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20830.509430] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20830.779380] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20831.062050] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20831.329240] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20831.609695] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20831.876589] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20832.154962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20832.416673] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20832.705915] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20832.974931] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20833.255198] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20833.522465] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20833.800749] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20834.065943] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20834.346510] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20834.621409] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20834.906297] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20835.173078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20835.450853] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20835.719562] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20836.006323] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20836.270908] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20836.549610] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20836.814265] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20837.093085] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20837.359499] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20837.642841] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20837.914915] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20838.199522] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20838.471380] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20838.751601] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20839.019308] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20839.303294] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20839.572144] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20839.872863] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20840.142629] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20840.422860] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20840.703145] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20840.991294] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20841.257293] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20841.539358] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20841.821933] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20842.116907] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20842.386056] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20842.670059] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20842.938427] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20843.409059] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20844.297038] Lustre: DEBUG MARKER: rc=0; [20844.297038] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20844.297038] if [[ $? -eq 0 && $val -ne 0 ]]; then [20844.297038] echo $(hostname -s): $val; [20844.297038] rc=$val; [20844.297038] fi; [20844.297038] exit $rc [20844.883660] Lustre: DEBUG MARKER: dmesg [20845.569122] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20845.857938] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20846.134232] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20856.407770] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20864.677208] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20872.938289] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [20879.197843] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20887.468027] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20887.725097] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20888.500711] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20888.758835] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20889.555285] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20889.814761] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20890.068714] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [20898.335262] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [20904.610217] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [20912.873289] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [20923.138143] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [20933.402036] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20933.658926] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20934.429502] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20934.687084] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20935.561877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 12: OST Pool Membership ============================================================ 16:21:46 \(1491495706\) [20935.724740] Lustre: DEBUG MARKER: == ost-pools test 12: OST Pool Membership ============================================================ 16:21:46 (1491495706) [20935.871945] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20942.133366] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20942.389688] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20943.172208] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20943.430956] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20946.258130] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20954.524162] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20954.781843] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20955.558556] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20955.815359] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20956.614353] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [20962.888898] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [20969.162780] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20969.162780] sort -u | tr '\n' ' ' [20969.419698] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20969.419698] sort -u | tr '\n' ' ' [20970.200199] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20970.200199] sort -u | tr '\n' ' ' [20970.457104] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20970.457104] sort -u | tr '\n' ' ' [20973.330906] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20973.603136] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20973.890832] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20974.158641] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20974.437214] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000] [20978.698927] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [20978.963757] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0000] [20983.227670] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [20983.227670] sort -u | tr '\n' ' ' [20983.489532] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [20983.489532] sort -u | tr '\n' ' ' [20984.278449] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [20984.278449] sort -u | tr '\n' ' ' [20984.536115] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [20984.536115] sort -u | tr '\n' ' ' [20985.330102] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool2 [20985.673973] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20985.941117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20986.218325] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20986.486105] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20986.825868] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20987.725114] Lustre: DEBUG MARKER: rc=0; [20987.725114] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20987.725114] if [[ $? -eq 0 && $val -ne 0 ]]; then [20987.725114] echo $(hostname -s): $val; [20987.725114] rc=$val; [20987.725114] fi; [20987.725114] exit $rc [20988.323372] Lustre: DEBUG MARKER: dmesg [20988.860996] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20989.125846] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20989.387897] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20993.655216] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20999.923869] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21006.201947] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21012.475112] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21012.741074] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21013.533140] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21013.793433] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21014.594820] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21014.858502] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21015.117230] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [21023.381505] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [21029.647480] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [21039.926029] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [21046.189775] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [21052.452712] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [21060.718490] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21060.973666] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21061.749204] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21062.005120] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21062.878021] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 13: Striping characteristics in a pool ============================================= 16:23:53 \(1491495833\) [21063.046407] Lustre: DEBUG MARKER: == ost-pools test 13: Striping characteristics in a pool ============================================= 16:23:53 (1491495833) [21063.189137] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21069.454557] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21069.712683] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21070.506937] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21070.763586] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21071.562033] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [21075.828748] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21075.828748] sort -u | tr '\n' ' ' [21076.087790] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21076.087790] sort -u | tr '\n' ' ' [21076.868749] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21076.868749] sort -u | tr '\n' ' ' [21077.137977] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21077.137977] sort -u | tr '\n' ' ' [21077.960361] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21078.233179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21078.530419] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21078.803492] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21079.100095] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21079.375229] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21079.678410] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21079.950302] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21080.249156] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21080.527480] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21080.833750] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21081.105846] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21081.406127] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21081.678508] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21081.974210] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21082.251133] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21082.554401] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21082.826534] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21083.125657] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21083.399908] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21083.782824] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21084.054824] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21084.343152] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21084.625283] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21084.907444] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21085.179477] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21085.459181] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21085.733121] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21086.016273] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21086.288773] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21086.569626] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21086.838159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21087.120247] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21087.390454] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21087.669525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21087.953078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21088.254470] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21088.531938] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21088.818163] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21089.090130] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21089.398847] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21089.667950] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21089.948391] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21090.225814] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21090.515367] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21090.788677] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21091.069062] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21091.511754] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21092.192345] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21092.888948] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21093.394494] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21093.663978] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21093.945087] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21094.217246] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21094.498765] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21094.774838] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21095.058679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21095.328662] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21095.619078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21095.889436] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21096.194438] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21096.464885] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21096.742829] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21097.010796] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21097.290786] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21097.561943] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21097.842221] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21098.111713] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21098.391159] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21098.660748] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21098.939800] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21099.214362] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21099.492357] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21099.769354] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21100.054370] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21100.326808] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21100.602734] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21100.870762] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21101.148756] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21101.418666] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21101.719633] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21101.990262] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21102.276034] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21102.552406] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21102.830971] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21103.101837] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21103.382962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21103.657863] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21103.936908] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21104.208141] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21104.482364] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21104.754286] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21105.034261] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21105.304874] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21105.590179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21105.858574] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21106.139147] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21106.409832] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21106.684992] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21106.954276] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21107.333076] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21108.241149] Lustre: DEBUG MARKER: rc=0; [21108.241149] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21108.241149] if [[ $? -eq 0 && $val -ne 0 ]]; then [21108.241149] echo $(hostname -s): $val; [21108.241149] rc=$val; [21108.241149] fi; [21108.241149] exit $rc [21108.844155] Lustre: DEBUG MARKER: dmesg [21109.343637] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21109.604720] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21109.859976] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21118.127210] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [21128.389480] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21134.654510] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [21140.915422] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21147.179454] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [21157.442325] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21163.708930] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [21173.976399] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21178.248153] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21178.505545] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21180.553244] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21180.810510] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21184.722045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 16:25:55 \(1491495955\) [21184.885759] Lustre: DEBUG MARKER: == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 16:25:55 (1491495955) [21185.079817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_14 OST size 16504586 is larger than 2000000 [21185.238362] Lustre: DEBUG MARKER: SKIP: ost-pools test_14 OST size 16504586 is larger than 2000000 [21185.426723] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21186.325643] Lustre: DEBUG MARKER: rc=0; [21186.325643] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21186.325643] if [[ $? -eq 0 && $val -ne 0 ]]; then [21186.325643] echo $(hostname -s): $val; [21186.325643] rc=$val; [21186.325643] fi; [21186.325643] exit $rc [21186.943524] Lustre: DEBUG MARKER: dmesg [21187.528438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 15: One directory per OST\/pool ===================================================== 16:25:58 \(1491495958\) [21187.684796] Lustre: DEBUG MARKER: == ost-pools test 15: One directory per OST/pool ===================================================== 16:25:58 (1491495958) [21187.816896] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool0 [21198.080958] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [21198.336562] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [21199.106065] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [21199.366519] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [21200.199343] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool0 lustre-OST[0000] [21206.467871] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [21206.467871] sort -u | tr '\n' ' ' [21206.728374] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [21206.728374] sort -u | tr '\n' ' ' [21207.504480] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 | [21207.504480] sort -u | tr '\n' ' ' [21207.771597] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 | [21207.771597] sort -u | tr '\n' ' ' [21209.860382] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [21216.128162] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [21216.396526] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [21217.168415] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [21217.434310] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [21219.270842] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 lustre-OST[0001] [21225.548963] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [21225.548963] sort -u | tr '\n' ' ' [21225.808365] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [21225.808365] sort -u | tr '\n' ' ' [21226.595552] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 | [21226.595552] sort -u | tr '\n' ' ' [21226.856334] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 | [21226.856334] sort -u | tr '\n' ' ' [21228.936992] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [21237.205545] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21237.463605] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21238.230671] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21238.487262] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21239.294756] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0002] [21247.561608] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21247.561608] sort -u | tr '\n' ' ' [21247.822458] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21247.822458] sort -u | tr '\n' ' ' [21248.612145] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21248.612145] sort -u | tr '\n' ' ' [21248.874239] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21248.874239] sort -u | tr '\n' ' ' [21250.918405] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool3 [21259.185716] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [21259.447628] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [21260.227112] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [21260.482721] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [21261.295471] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool3 lustre-OST[0003] [21269.566361] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 | [21269.566361] sort -u | tr '\n' ' ' [21269.844680] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 | [21269.844680] sort -u | tr '\n' ' ' [21271.926485] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 | [21271.926485] sort -u | tr '\n' ' ' [21272.187228] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 | [21272.187228] sort -u | tr '\n' ' ' [21274.228235] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool4 [21280.502347] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [21280.768374] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [21281.543839] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [21281.813377] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [21283.632910] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool4 lustre-OST[0004] [21293.905367] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 | [21293.905367] sort -u | tr '\n' ' ' [21294.177686] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 | [21294.177686] sort -u | tr '\n' ' ' [21294.964175] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 | [21294.964175] sort -u | tr '\n' ' ' [21295.224201] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 | [21295.224201] sort -u | tr '\n' ' ' [21297.229324] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool5 [21303.494616] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [21303.754157] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [21308.300662] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [21308.562404] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [21309.377051] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool5 lustre-OST[0005] [21315.644170] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 | [21315.644170] sort -u | tr '\n' ' ' [21315.904963] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 | [21315.904963] sort -u | tr '\n' ' ' [21319.228239] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 | [21319.228239] sort -u | tr '\n' ' ' [21319.491511] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 | [21319.491511] sort -u | tr '\n' ' ' [21321.498624] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool6 [21325.765649] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [21326.024139] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [21326.801405] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [21327.063971] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [21330.904982] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool6 lustre-OST[0006] [21341.172169] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 | [21341.172169] sort -u | tr '\n' ' ' [21341.432981] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 | [21341.432981] sort -u | tr '\n' ' ' [21342.213520] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 | [21342.213520] sort -u | tr '\n' ' ' [21342.471797] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 | [21342.471797] sort -u | tr '\n' ' ' [21344.527094] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool7 [21354.795656] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [21355.057050] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [21355.840091] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [21356.107614] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [21356.940151] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool7 lustre-OST[0007] [21367.211200] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 | [21367.211200] sort -u | tr '\n' ' ' [21367.470397] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 | [21367.470397] sort -u | tr '\n' ' ' [21368.259622] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 | [21368.259622] sort -u | tr '\n' ' ' [21368.523360] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 | [21368.523360] sort -u | tr '\n' ' ' [21370.615075] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21371.515532] Lustre: DEBUG MARKER: rc=0; [21371.515532] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21371.515532] if [[ $? -eq 0 && $val -ne 0 ]]; then [21371.515532] echo $(hostname -s): $val; [21371.515532] rc=$val; [21371.515532] fi; [21371.515532] exit $rc [21372.148366] Lustre: DEBUG MARKER: dmesg [21372.683199] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21372.945905] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool0 [21373.202865] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool0 lustre-OST0000_UUID [21381.468562] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool0 [21391.739185] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [21392.011661] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [21392.791762] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [21393.052903] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [21393.859215] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21394.132592] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [21394.407079] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0001_UUID [21402.678835] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [21412.942944] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [21413.200750] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [21413.972705] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [21414.229615] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [21415.034512] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21415.298789] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21415.559679] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0002_UUID [21425.829454] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [21432.097351] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21432.355790] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21433.134088] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21433.397997] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21434.200359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21434.461259] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool3 [21434.720191] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool3 lustre-OST0003_UUID [21440.981295] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool3 [21449.246928] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [21449.505919] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [21450.273672] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [21450.532990] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [21451.331932] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21451.592402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool4 [21451.850601] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool4 lustre-OST0004_UUID [21460.114136] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool4 [21470.395989] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [21470.656815] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [21471.428959] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [21471.685725] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [21472.483352] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21472.748102] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool5 [21473.006437] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool5 lustre-OST0005_UUID [21481.270223] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool5 [21491.537406] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [21491.795207] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [21492.571191] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [21492.829221] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [21493.627373] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21493.889120] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool6 [21494.147875] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool6 lustre-OST0006_UUID [21502.412842] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool6 [21508.682351] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [21508.940832] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [21509.724449] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [21509.984405] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [21510.786400] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21511.046690] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool7 [21511.304880] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool7 lustre-OST0007_UUID [21515.567518] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool7 [21525.836548] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [21526.093626] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [21526.867415] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [21527.133739] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [21528.027844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 16: Inheritance of pool properties ================================================= 16:31:39 \(1491496299\) [21528.196400] Lustre: DEBUG MARKER: == ost-pools test 16: Inheritance of pool properties ================================================= 16:31:39 (1491496299) [21528.335655] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21532.600850] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21532.856084] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21538.663648] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21538.921199] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21539.740880] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [21542.012633] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21542.012633] sort -u | tr '\n' ' ' [21542.270942] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21542.270942] sort -u | tr '\n' ' ' [21545.561254] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21545.561254] sort -u | tr '\n' ' ' [21545.821938] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21545.821938] sort -u | tr '\n' ' ' [21546.673927] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21546.940832] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21547.225263] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21547.495967] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21547.779313] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21548.048936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21548.332811] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21548.601298] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21548.898479] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21549.169346] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21549.452941] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21549.720696] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21550.004227] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21550.273084] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21550.557936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21550.827836] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21551.107793] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21551.376639] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21551.679052] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21551.947431] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21552.357799] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21553.240406] Lustre: DEBUG MARKER: rc=0; [21553.240406] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21553.240406] if [[ $? -eq 0 && $val -ne 0 ]]; then [21553.240406] echo $(hostname -s): $val; [21553.240406] rc=$val; [21553.240406] fi; [21553.240406] exit $rc [21553.842162] Lustre: DEBUG MARKER: dmesg [21554.346712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21554.604881] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21554.863126] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21563.126192] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21571.390921] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21579.663902] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21587.940623] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21598.211254] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21598.485580] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21599.276433] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21599.531709] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21600.500834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 17: Referencing an empty pool ====================================================== 16:32:51 \(1491496371\) [21600.655222] Lustre: DEBUG MARKER: == ost-pools test 17: Referencing an empty pool ====================================================== 16:32:51 (1491496371) [21600.790539] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21607.056761] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21607.312907] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21609.357946] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21609.618015] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21610.441317] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [21618.712603] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21618.712603] sort -u | tr '\n' ' ' [21618.969942] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21618.969942] sort -u | tr '\n' ' ' [21619.760203] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21619.760203] sort -u | tr '\n' ' ' [21620.018432] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21620.018432] sort -u | tr '\n' ' ' [21620.846170] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21621.116256] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21621.419206] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21621.693944] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21621.995809] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21622.277185] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21622.578335] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21622.858869] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21623.157218] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21623.445311] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21623.753278] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21624.038214] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21624.345540] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21624.632236] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21624.930367] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21625.204699] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21625.506083] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21625.780185] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21626.074786] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21626.346598] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21626.644749] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21626.905085] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21627.161043] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21633.427473] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [21643.687619] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21649.951802] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [21658.212712] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21664.477624] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [21670.740777] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21681.004454] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [21689.264937] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21695.527914] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21695.785085] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21696.555923] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21696.813789] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21697.700820] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21698.604986] Lustre: DEBUG MARKER: rc=0; [21698.604986] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21698.604986] if [[ $? -eq 0 && $val -ne 0 ]]; then [21698.604986] echo $(hostname -s): $val; [21698.604986] rc=$val; [21698.604986] fi; [21698.604986] exit $rc [21699.212614] Lustre: DEBUG MARKER: dmesg [21699.762202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_18 skipping SLOW test 18 [21699.918126] Lustre: DEBUG MARKER: SKIP: ost-pools test_18 skipping SLOW test 18 [21700.123200] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 19: Pools should not come into play when not specified ============================= 16:34:31 \(1491496471\) [21700.281157] Lustre: DEBUG MARKER: == ost-pools test 19: Pools should not come into play when not specified ============================= 16:34:31 (1491496471) [21700.418075] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21704.680527] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21704.939753] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21710.750924] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21711.025774] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21711.828125] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [21720.099344] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21720.099344] sort -u | tr '\n' ' ' [21720.359255] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21720.359255] sort -u | tr '\n' ' ' [21722.413437] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21722.413437] sort -u | tr '\n' ' ' [21722.676975] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21722.676975] sort -u | tr '\n' ' ' [21723.516624] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21723.786978] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21724.073497] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21724.341860] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21724.632623] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21724.900926] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21725.181402] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21725.451718] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21725.737110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21726.004933] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21726.295144] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21726.569571] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21726.871525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21727.151042] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21727.449101] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21727.727858] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21728.022616] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21728.290806] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21728.585980] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21728.855076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21729.140956] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21729.417609] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21729.702566] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21729.975720] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21730.409431] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21731.307453] Lustre: DEBUG MARKER: rc=0; [21731.307453] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21731.307453] if [[ $? -eq 0 && $val -ne 0 ]]; then [21731.307453] echo $(hostname -s): $val; [21731.307453] rc=$val; [21731.307453] fi; [21731.307453] exit $rc [21731.908798] Lustre: DEBUG MARKER: dmesg [21732.405979] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21732.668786] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21732.926870] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21741.199350] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21747.461330] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21755.726370] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21761.993198] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21770.257240] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21770.513653] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21771.299678] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21771.574679] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21772.456562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 16:35:43 \(1491496543\) [21772.617939] Lustre: DEBUG MARKER: == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 16:35:43 (1491496543) [21772.755641] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21779.025865] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21779.283971] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21780.057153] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21780.311218] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21781.108329] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [21789.373459] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21789.629878] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21790.410734] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21790.674926] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21791.473300] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [21797.749377] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21797.749377] sort -u | tr '\n' ' ' [21798.009131] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21798.009131] sort -u | tr '\n' ' ' [21798.813233] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21798.813233] sort -u | tr '\n' ' ' [21799.072509] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21799.072509] sort -u | tr '\n' ' ' [21801.899930] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [21806.168593] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21806.168593] sort -u | tr '\n' ' ' [21806.428457] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21806.428457] sort -u | tr '\n' ' ' [21807.209756] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21807.209756] sort -u | tr '\n' ' ' [21807.472685] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21807.472685] sort -u | tr '\n' ' ' [21811.390598] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21811.663670] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21811.949170] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21812.218576] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21812.598895] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21813.492525] Lustre: DEBUG MARKER: rc=0; [21813.492525] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21813.492525] if [[ $? -eq 0 && $val -ne 0 ]]; then [21813.492525] echo $(hostname -s): $val; [21813.492525] rc=$val; [21813.492525] fi; [21813.492525] exit $rc [21814.099534] Lustre: DEBUG MARKER: dmesg [21814.614715] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21814.880385] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21815.140764] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21823.403085] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21831.666427] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21839.933949] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21846.199240] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21856.471945] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21856.739085] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21857.538635] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21857.796477] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21858.599590] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21858.859452] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [21859.116608] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [21861.379766] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [21869.648703] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [21873.917405] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [21884.182224] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [21892.450927] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21892.711401] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21893.491687] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21893.748794] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21894.616576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 16:37:45 \(1491496665\) [21894.778289] Lustre: DEBUG MARKER: == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 16:37:45 (1491496665) [21894.916543] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21899.180141] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21899.437448] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21902.725688] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21902.985319] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21903.781439] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [21914.052404] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21914.052404] sort -u | tr '\n' ' ' [21914.314594] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21914.314594] sort -u | tr '\n' ' ' [21915.100600] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21915.100600] sort -u | tr '\n' ' ' [21915.358892] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21915.358892] sort -u | tr '\n' ' ' [21916.187617] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21916.468874] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21916.814854] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21917.714719] Lustre: DEBUG MARKER: rc=0; [21917.714719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21917.714719] if [[ $? -eq 0 && $val -ne 0 ]]; then [21917.714719] echo $(hostname -s): $val; [21917.714719] rc=$val; [21917.714719] fi; [21917.714719] exit $rc [21918.309170] Lustre: DEBUG MARKER: dmesg [21918.824306] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [21919.083377] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [21919.343645] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [21925.621427] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [21931.902826] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [21940.168907] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [21946.440034] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [21956.716114] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21956.976801] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21957.759355] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21958.017173] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21958.893847] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 16:38:49 \(1491496729\) [21959.050233] Lustre: DEBUG MARKER: == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 16:38:49 (1491496729) [21959.181845] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [21965.455582] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21965.732306] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [21966.529125] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21966.788497] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [21967.603570] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [21973.873527] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21973.873527] sort -u | tr '\n' ' ' [21974.135512] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [21974.135512] sort -u | tr '\n' ' ' [21974.912844] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21974.912844] sort -u | tr '\n' ' ' [21975.176299] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [21975.176299] sort -u | tr '\n' ' ' [21975.978644] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [21986.244252] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21986.500571] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [21987.271348] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21987.529772] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [21988.326357] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [21996.590180] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21996.590180] sort -u | tr '\n' ' ' [21996.848236] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [21996.848236] sort -u | tr '\n' ' ' [21997.635425] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21997.635425] sort -u | tr '\n' ' ' [21997.894345] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [21997.894345] sort -u | tr '\n' ' ' [21998.703376] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [21998.706666] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22004.996961] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22004.997726] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22015.301766] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22015.301800] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22025.606062] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22025.608023] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22031.908227] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22031.908263] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22040.200347] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22040.201522] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22046.501142] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22046.503083] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22056.799972] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22056.802230] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22063.094745] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22063.111437] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22073.385975] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22073.388919] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22081.689354] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22081.692514] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22087.978521] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22087.981648] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22096.268571] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22096.273398] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22104.561538] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22104.563729] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22112.864418] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22112.864824] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22123.157090] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22123.158711] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22131.460350] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22131.462215] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22141.795276] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22141.798774] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22148.094868] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [22148.094868] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [22154.383222] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [22154.383621] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [22164.735998] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22165.635603] Lustre: DEBUG MARKER: rc=0; [22165.635603] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22165.635603] if [[ $? -eq 0 && $val -ne 0 ]]; then [22165.635603] echo $(hostname -s): $val; [22165.635603] rc=$val; [22165.635603] fi; [22165.635603] exit $rc [22166.269753] Lustre: DEBUG MARKER: dmesg [22166.785928] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22167.049962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22167.309712] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [22177.575095] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [22187.841364] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22188.101352] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22188.874930] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22189.134155] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22189.939074] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22190.198513] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [22190.457127] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [22196.719418] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [22202.986049] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22203.253314] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22204.039845] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22204.301410] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22205.181706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 23a: OST pools and quota =========================================================== 16:42:56 \(1491496976\) [22205.349235] Lustre: DEBUG MARKER: == ost-pools test 23a: OST pools and quota =========================================================== 16:42:56 (1491496976) [22205.614967] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [22213.883289] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22214.138368] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22214.916189] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22215.176350] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22215.977896] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/3] [22224.250120] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [22224.250120] sort -u | tr '\n' ' ' [22224.512283] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [22224.512283] sort -u | tr '\n' ' ' [22225.295170] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [22225.295170] sort -u | tr '\n' ' ' [22225.555800] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [22225.555800] sort -u | tr '\n' ' ' [22226.373868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [22226.373868] /usr/sbin/lctl lustre_build_version 2>/dev/null || [22226.373868] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [22226.648632] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [22237.825710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first dd failed with EDQUOT. [22237.983214] Lustre: DEBUG MARKER: first dd failed with EDQUOT. [22238.201206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22239.095089] Lustre: DEBUG MARKER: rc=0; [22239.095089] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22239.095089] if [[ $? -eq 0 && $val -ne 0 ]]; then [22239.095089] echo $(hostname -s): $val; [22239.095089] rc=$val; [22239.095089] fi; [22239.095089] exit $rc [22239.706442] Lustre: DEBUG MARKER: dmesg [22240.205798] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22240.481145] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22240.744853] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [22249.012526] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [22257.278636] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [22265.544953] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [22273.820914] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22274.096680] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22274.876815] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22275.152939] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22276.030241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_23b skipping SLOW test 23b [22276.198807] Lustre: DEBUG MARKER: SKIP: ost-pools test_23b skipping SLOW test 23b [22276.413673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 16:44:07 \(1491497047\) [22276.574667] Lustre: DEBUG MARKER: == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 16:44:07 (1491497047) [22276.716042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [22276.716042] /usr/sbin/lctl lustre_build_version 2>/dev/null || [22276.716042] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [22276.985577] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [22287.256440] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22287.530682] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22288.327017] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22288.585852] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22289.393081] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [22297.673699] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [22297.673699] sort -u | tr '\n' ' ' [22297.934275] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [22297.934275] sort -u | tr '\n' ' ' [22299.982155] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [22299.982155] sort -u | tr '\n' ' ' [22300.244691] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [22300.244691] sort -u | tr '\n' ' ' [22301.101400] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22301.370969] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22301.679399] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22301.950851] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22302.258081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22302.528881] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22302.836074] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22303.109630] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22303.416401] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22303.688359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22304.000699] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22304.274120] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22304.577866] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22304.848196] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22305.154791] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22305.428419] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22305.740625] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22306.013145] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22306.359487] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22306.637364] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22306.989632] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22307.262161] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22307.554934] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22307.830810] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22308.118370] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22308.391642] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22308.678815] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22308.949516] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22309.239478] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22309.513988] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22309.806241] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22310.086117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22310.377370] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22310.649977] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22310.943193] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22311.211336] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22311.503078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22311.773898] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22312.062035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22312.335939] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22313.017454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22313.920028] Lustre: DEBUG MARKER: rc=0; [22313.920028] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22313.920028] if [[ $? -eq 0 && $val -ne 0 ]]; then [22313.920028] echo $(hostname -s): $val; [22313.920028] rc=$val; [22313.920028] fi; [22313.920028] exit $rc [22314.805089] Lustre: DEBUG MARKER: dmesg [22315.816429] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22316.092321] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [22316.366990] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [22326.663125] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [22334.932685] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [22347.197617] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [22353.464411] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [22359.730169] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [22369.997860] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [22378.262442] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [22386.528367] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [22394.795480] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22395.052247] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [22395.828182] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22396.087126] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [22396.981310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 25: Create new pool and restart MDS ================================================ 16:46:07 \(1491497167\) [22397.137376] Lustre: DEBUG MARKER: == ost-pools test 25: Create new pool and restart MDS ================================================ 16:46:07 (1491497167) [22397.278159] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [22407.542964] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [22407.801618] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [22408.581303] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [22408.837974] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [22409.630340] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 OST0000; sync [22419.974652] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [22420.233431] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [22420.363704] Lustre: Failing over lustre-MDT0000 [22420.487033] Lustre: server umount lustre-MDT0000 complete [22420.488440] Lustre: Skipped 2 previous similar messages [22420.539119] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [22420.542365] LustreError: Skipped 69 previous similar messages [22420.624122] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22421.544200] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [22421.544228] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [22421.544229] Lustre: Skipped 4 previous similar messages [22421.551403] LustreError: Skipped 4 previous similar messages [22430.935978] Lustre: DEBUG MARKER: hostname [22431.254308] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [22431.509578] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [22431.765373] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [22431.927370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22431.988519] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [22431.992109] LustreError: Skipped 1 previous similar message [22431.994067] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [22431.996189] Lustre: Skipped 60 previous similar messages [22431.998048] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5a5be8 to 0x61ac6f141d5eecd3 [22432.048173] LustreError: 20961:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [22432.052427] LustreError: 20961:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [22432.057322] LustreError: 20966:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [22432.070452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [22432.072836] Lustre: Skipped 2 previous similar messages [22432.237876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22432.516861] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [22433.041830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22433.041922] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22433.201563] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22433.203549] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22433.312718] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [22433.351622] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22433.611503] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22433.872607] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22434.143894] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [22434.422806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [22434.422806] /usr/sbin/lctl lustre_build_version 2>/dev/null || [22434.422806] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [22434.708454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [22434.708454] /usr/sbin/lctl lustre_build_version 2>/dev/null || [22434.708454] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [22434.992158] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [22435.254869] Lustre: DEBUG MARKER: lctl get_param -n at_min [22435.519562] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [22436.019883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [22436.022995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [22436.187263] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [22436.189558] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [22436.404311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [22436.404349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [22436.564377] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [22436.566085] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [22437.063279] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [22437.082073] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [22438.001759] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [22438.264583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [22445.582607] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [22445.854783] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [22446.202242] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22447.125035] Lustre: DEBUG MARKER: rc=0; [22447.125035] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22447.125035] if [[ $? -eq 0 && $val -ne 0 ]]; then [22447.125035] echo $(hostname -s): $val; [22447.125035] rc=$val; [22447.125035] fi; [22447.125035] exit $rc [22447.747508] Lustre: DEBUG MARKER: dmesg [22448.234541] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22448.499123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [22448.759387] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0000_UUID [22457.047060] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [22463.319174] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [22463.580609] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [22464.356021] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [22464.616574] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [22465.522565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 16:47:16 \(1491497236\) [22465.681577] Lustre: DEBUG MARKER: == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 16:47:16 (1491497236) [22465.822381] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [22472.089660] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22472.353469] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22475.642730] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22475.899352] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22476.705304] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000; sync [22484.129728] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0002; sync [22491.548101] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x147 [22491.833530] Lustre: DEBUG MARKER: lctl set_param fail_val=0 [22491.974750] Lustre: *** cfs_fail_loc=147, val=0*** [22492.110344] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [22492.370446] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [22492.641658] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [22492.987869] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22493.877464] Lustre: DEBUG MARKER: rc=0; [22493.877464] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22493.877464] if [[ $? -eq 0 && $val -ne 0 ]]; then [22493.877464] echo $(hostname -s): $val; [22493.877464] rc=$val; [22493.877464] fi; [22493.877464] exit $rc [22494.478052] Lustre: DEBUG MARKER: dmesg [22494.969566] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [22495.233319] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [22495.493284] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [22503.758428] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0002_UUID [22512.031614] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [22518.306319] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22518.566374] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [22521.867654] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22522.125776] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [22523.085824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration 2937 sec ======================================================== 16:48:14 \(1491497294\) [22523.241502] Lustre: DEBUG MARKER: == ost-pools test complete, duration 2937 sec ======================================================== 16:48:14 (1491497294) [22524.170133] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [22524.432083] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [22524.564258] Lustre: Failing over lustre-MDT0000 [22524.732031] Lustre: server umount lustre-MDT0000 complete [22524.860163] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22525.164322] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [22525.168227] LustreError: Skipped 31 previous similar messages [22527.062268] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [22527.062316] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [22527.062317] Lustre: Skipped 1 previous similar message [22527.069896] LustreError: Skipped 3 previous similar messages [22534.247037] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491497297/real 1491497297] req@ffff8800643a0600 x1563924897012368/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491497304 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22534.252895] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [22534.255081] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [22535.190896] Lustre: DEBUG MARKER: hostname [22535.509967] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [22535.780570] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [22536.052289] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [22536.221628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22540.258276] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [22540.260594] Lustre: Skipped 25 previous similar messages [22540.262636] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5eecd3 to 0x61ac6f141d5f84f7 [22540.335540] LustreError: 26434:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [22540.346751] LustreError: 26439:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [22540.365409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [22540.560433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22540.823538] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [22541.216109] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [22541.344133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22541.346939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22541.510443] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22541.510444] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22541.658621] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22541.918082] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22542.176907] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22542.446476] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [22543.211299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22543.224850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22543.409831] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22543.466094] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22545.354534] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [22545.379570] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [22545.678308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [22545.783921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [22545.847654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [22546.008459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [22558.009667] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [22558.279108] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [22559.717073] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [22559.717073] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [22559.717073] Lustre: Skipped 21 previous similar messages [22559.725855] Lustre: Skipped 6 previous similar messages [22560.431069] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006675c900 x1563924897032528/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [22560.436977] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [22564.524778] Lustre: server umount lustre-MDT0000 complete [22564.657294] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22565.354669] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [22565.357689] LustreError: Skipped 2 previous similar messages [22565.360167] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [22565.365549] Lustre: Skipped 3 previous similar messages [22571.806452] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [22572.061362] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [22573.311060] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491497336/real 1491497336] req@ffff8800639f6100 x1563924897032784/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491497343 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22573.319192] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [22573.321953] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [22573.551065] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006675f300 x1563924897033328/t0(0) o13->lustre-OST0003-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [22573.556783] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [22576.735055] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [22576.738395] Lustre: Skipped 13 previous similar messages [22578.466642] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22654.633290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-ost-single ============----- Thu Apr 6 16:50:24 UTC 2017 [22654.800712] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-ost-single ============----- Thu Apr 6 16:50:24 UTC 2017 [22655.052964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test complete, duration -o sec ================================================== 16:50:26 \(1491497426\) [22655.202387] Lustre: DEBUG MARKER: == replay-ost-single test complete, duration -o sec ================================================== 16:50:26 (1491497426) [22655.517991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [22655.674585] Lustre: DEBUG MARKER: excepting tests: [22655.868565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 5 [22656.031808] Lustre: DEBUG MARKER: skipping tests SLOW=no: 5 [22658.392467] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [22658.392467] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [22658.392467] if [ $running -ne $mpts ]; then [22658.392467] echo $(hostname) env are INSANE!; [22658.392467] exit 1; [22658.392467] fi [22658.950763] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [22658.950763] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [22658.950763] if [ $running -ne $mpts ]; then [22658.950763] echo $(hostname) env are INSANE!; [22658.950763] exit 1; [22658.950763] fi [22659.534124] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [22659.534124] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [22659.534124] if [ $running -ne $mpts ]; then [22659.534124] echo $(hostname) env are INSANE!; [22659.534124] exit 1; [22659.534124] fi [22662.078557] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [22662.337747] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [22662.588017] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [22662.839774] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [22662.985389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22663.006161] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [22663.008590] Lustre: Skipped 25 previous similar messages [22663.043841] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [22663.048485] LustreError: Skipped 73 previous similar messages [22663.068631] LustreError: 28889:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [22663.084586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [22663.231201] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22663.492971] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22663.993351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22663.993689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22664.154080] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22664.158287] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22664.298696] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22664.552936] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22664.807506] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22665.066196] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [22665.321125] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22667.576460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22667.734130] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22668.077118] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491497433/real 1491497433] req@ffff8800639f6400 x1563924897033792/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491497438 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22669.165099] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [22669.431922] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [22669.688323] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [22669.941958] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [22670.115602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22670.190772] LustreError: 30185:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [22670.397149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22670.660073] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22671.159777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22671.159779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22671.320442] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22671.324495] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22671.469549] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22671.731281] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22671.992741] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22672.255151] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [22672.517856] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22674.787830] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22674.946343] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22678.299222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22678.485286] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22681.282427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22681.450547] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22684.302980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22684.463473] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22687.323144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22687.483571] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22690.433604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22690.599555] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22693.450882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22693.615626] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22696.561259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22696.720269] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22699.608802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22699.767501] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22701.953785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22702.142689] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22703.614286] Lustre: DEBUG MARKER: lctl get_param -n timeout [22703.954735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [22704.114735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [22704.258870] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [22709.673384] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [22709.964107] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22710.470862] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22711.070420] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 0a: target handle mismatch \(bug 5317\) ====================================== 16:51:22 \(1491497482\) [22711.235709] Lustre: DEBUG MARKER: == replay-ost-single test 0a: target handle mismatch (bug 5317) ====================================== 16:51:22 (1491497482) [22721.908588] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22722.814712] Lustre: DEBUG MARKER: rc=0; [22722.814712] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22722.814712] if [[ $? -eq 0 && $val -ne 0 ]]; then [22722.814712] echo $(hostname -s): $val; [22722.814712] rc=$val; [22722.814712] fi; [22722.814712] exit $rc [22723.423983] Lustre: DEBUG MARKER: dmesg [22724.005349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 0b: empty replay =========================================================== 16:51:34 \(1491497494\) [22724.157096] Lustre: DEBUG MARKER: == replay-ost-single test 0b: empty replay =========================================================== 16:51:34 (1491497494) [22725.103317] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [22725.106316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [22737.179115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22737.345542] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22738.778913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22738.796235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22738.989415] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22739.073190] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22761.036724] Lustre: lustre-MDT0002: haven't heard from client 19343ee7-f114-a1e9-adde-42bea36c4c02 (at 10.9.5.237@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp ffff88007821b800, cur 1491497531 expire 1491497501 last 1491497481 [22807.453627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [22807.597778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [22807.613932] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [22807.812712] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [22809.210547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22810.130696] Lustre: DEBUG MARKER: rc=0; [22810.130696] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22810.130696] if [[ $? -eq 0 && $val -ne 0 ]]; then [22810.130696] echo $(hostname -s): $val; [22810.130696] rc=$val; [22810.130696] fi; [22810.130696] exit $rc [22810.749034] Lustre: DEBUG MARKER: dmesg [22811.356671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 1: touch =================================================================== 16:53:02 \(1491497582\) [22811.515284] Lustre: DEBUG MARKER: == replay-ost-single test 1: touch =================================================================== 16:53:02 (1491497582) [22823.603437] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [22823.606106] Lustre: Skipped 41 previous similar messages [22824.518632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22824.678383] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22826.119156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22826.163617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22826.406400] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22826.540100] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22828.750321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22828.822668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22828.934684] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22829.065574] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22830.404893] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22831.304134] Lustre: DEBUG MARKER: rc=0; [22831.304134] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22831.304134] if [[ $? -eq 0 && $val -ne 0 ]]; then [22831.304134] echo $(hostname -s): $val; [22831.304134] rc=$val; [22831.304134] fi; [22831.304134] exit $rc [22831.907564] Lustre: DEBUG MARKER: dmesg [22832.481277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 2: \|x\| 10 open\(O_CREAT\)s =================================================== 16:53:23 \(1491497603\) [22832.641147] Lustre: DEBUG MARKER: == replay-ost-single test 2: |x| 10 open(O_CREAT)s =================================================== 16:53:23 (1491497603) [22845.671302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22845.847518] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22847.275958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22847.293088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22847.475923] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22847.524650] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22852.756319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec [22852.848846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec [22852.979667] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec [22853.081872] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 5 sec [22854.487592] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22855.379416] Lustre: DEBUG MARKER: rc=0; [22855.379416] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22855.379416] if [[ $? -eq 0 && $val -ne 0 ]]; then [22855.379416] echo $(hostname -s): $val; [22855.379416] rc=$val; [22855.379416] fi; [22855.379416] exit $rc [22855.980651] Lustre: DEBUG MARKER: dmesg [22856.560806] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 3: Fail OST during write, with verification ================================ 16:53:47 \(1491497627\) [22856.714393] Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification ================================ 16:53:47 (1491497627) [22869.863196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22870.036438] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22871.492536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22871.498336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22871.710917] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22871.728533] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22876.037666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [22876.078990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [22876.193636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [22876.329094] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [22877.689918] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22878.602761] Lustre: DEBUG MARKER: rc=0; [22878.602761] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22878.602761] if [[ $? -eq 0 && $val -ne 0 ]]; then [22878.602761] echo $(hostname -s): $val; [22878.602761] rc=$val; [22878.602761] fi; [22878.602761] exit $rc [22879.213345] Lustre: DEBUG MARKER: dmesg [22879.791826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 4: Fail OST during read, with verification ================================= 16:54:10 \(1491497650\) [22879.946888] Lustre: DEBUG MARKER: == replay-ost-single test 4: Fail OST during read, with verification ================================= 16:54:10 (1491497650) [22882.731744] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 10.9.5.244@tcp failed: rc = -107 [22882.734410] LustreError: Skipped 14 previous similar messages [22882.736568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [22882.741650] Lustre: Skipped 7 previous similar messages [22893.572555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22893.761176] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22895.222731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22895.222733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22895.437184] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22895.456112] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22897.719920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22897.761844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22897.918744] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22898.003412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [22899.353590] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22900.255518] Lustre: DEBUG MARKER: rc=0; [22900.255518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22900.255518] if [[ $? -eq 0 && $val -ne 0 ]]; then [22900.255518] echo $(hostname -s): $val; [22900.255518] rc=$val; [22900.255518] fi; [22900.255518] exit $rc [22900.882368] Lustre: DEBUG MARKER: dmesg [22901.508887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-ost-single test_5 skipping SLOW test 5 [22901.662827] Lustre: DEBUG MARKER: SKIP: replay-ost-single test_5 skipping SLOW test 5 [22901.870078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 16:54:32 \(1491497672\) [22902.033125] Lustre: DEBUG MARKER: == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 16:54:32 (1491497672) [22907.221647] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [22907.483476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [22913.367448] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [22913.680209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126409296 after_dd: 126404176 took 0 seconds [22913.838026] Lustre: DEBUG MARKER: before: 126409296 after_dd: 126404176 took 0 seconds [22913.846866] Lustre: *** cfs_fail_loc=119, val=2147483648*** [22913.849166] Lustre: Skipped 6 previous similar messages [22913.851460] LustreError: 28849:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005d1c3000 x1563948410222768/t47244640345(0) o36->a35713f7-9ad7-2f9b-9a7d-926325a910f9@10.9.5.237@tcp:-1/-1 lens 624/456 e 0 to 0 dl 1491497690 ref 1 fl Interpret:/0/0 rc 0/0 [22920.861239] Lustre: lustre-MDT0000: Client a35713f7-9ad7-2f9b-9a7d-926325a910f9 (at 10.9.5.237@tcp) reconnecting [22920.863990] Lustre: Skipped 3 previous similar messages [22933.925333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22934.087193] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22935.523846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22935.531583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22935.755604] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22935.757047] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22937.034176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [22937.050069] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [22937.235043] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [22937.346128] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [22939.153063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [22939.322872] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [22939.518266] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [22939.786684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [22942.349209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22943.680680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126409296 after: 126409296 [22943.835060] Lustre: DEBUG MARKER: before: 126409296 after: 126409296 [22944.046923] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22944.956917] Lustre: DEBUG MARKER: rc=0; [22944.956917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22944.956917] if [[ $? -eq 0 && $val -ne 0 ]]; then [22944.956917] echo $(hostname -s): $val; [22944.956917] rc=$val; [22944.956917] fi; [22944.956917] exit $rc [22945.564603] Lustre: DEBUG MARKER: dmesg [22946.154599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 16:55:17 \(1491497717\) [22946.314522] Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 16:55:17 (1491497717) [22951.533315] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [22951.794142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [22956.663793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126409296 after_dd: 126404176 took 1 seconds [22956.813503] Lustre: DEBUG MARKER: before: 126409296 after_dd: 126404176 took 1 seconds [22971.479870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22971.640440] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22973.074876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22973.090381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22973.291620] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22973.330551] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [22976.551301] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [22976.658904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [22976.734753] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [22976.879052] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [22978.704682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [22978.867288] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [22979.100578] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [22979.361860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [22981.929431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22982.202860] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [22982.487247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22983.775761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22985.060802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22986.346169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [22987.705276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126409296 after: 126409296 [22987.858377] Lustre: DEBUG MARKER: before: 126409296 after: 126409296 [22988.050218] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22988.945215] Lustre: DEBUG MARKER: rc=0; [22988.945215] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22988.945215] if [[ $? -eq 0 && $val -ne 0 ]]; then [22988.945215] echo $(hostname -s): $val; [22988.945215] rc=$val; [22988.945215] fi; [22988.945215] exit $rc [22989.542170] Lustre: DEBUG MARKER: dmesg [22990.132414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error ==================== 16:56:01 \(1491497761\) [22990.282879] Lustre: DEBUG MARKER: == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error ==================== 16:56:01 (1491497761) [22990.421579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [22990.421579] /usr/sbin/lctl lustre_build_version 2>/dev/null || [22990.421579] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23017.148121] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23018.047933] Lustre: DEBUG MARKER: rc=0; [23018.047933] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23018.047933] if [[ $? -eq 0 && $val -ne 0 ]]; then [23018.047933] echo $(hostname -s): $val; [23018.047933] rc=$val; [23018.047933] fi; [23018.047933] exit $rc [23018.642538] Lustre: DEBUG MARKER: dmesg [23019.224441] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8b: Verify redo io: redo io should success after recovery ================== 16:56:30 \(1491497790\) [23019.380358] Lustre: DEBUG MARKER: == replay-ost-single test 8b: Verify redo io: redo io should success after recovery ================== 16:56:30 (1491497790) [23019.519575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23019.519575] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23019.519575] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23053.635142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23053.798917] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23055.281320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23055.304589] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23055.442893] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23055.528310] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23056.727793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [23056.820106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [23056.903261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [23057.021026] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [23066.716294] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23067.619977] Lustre: DEBUG MARKER: rc=0; [23067.619977] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23067.619977] if [[ $? -eq 0 && $val -ne 0 ]]; then [23067.619977] echo $(hostname -s): $val; [23067.619977] rc=$val; [23067.619977] fi; [23067.619977] exit $rc [23068.225213] Lustre: DEBUG MARKER: dmesg [23068.815482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction ===================== 16:57:19 \(1491497839\) [23068.979482] Lustre: DEBUG MARKER: == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction ===================== 16:57:19 (1491497839) [23069.122925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23069.122925] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23069.122925] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23112.649237] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23113.550034] Lustre: DEBUG MARKER: rc=0; [23113.550034] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23113.550034] if [[ $? -eq 0 && $val -ne 0 ]]; then [23113.550034] echo $(hostname -s): $val; [23113.550034] rc=$val; [23113.550034] fi; [23113.550034] exit $rc [23114.155402] Lustre: DEBUG MARKER: dmesg [23114.765207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS =================================== 16:58:05 \(1491497885\) [23114.926614] Lustre: DEBUG MARKER: == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS =================================== 16:58:05 (1491497885) [23115.067571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23115.067571] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23115.067571] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23115.353340] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x187 [23115.494895] Lustre: *** cfs_fail_loc=187, val=0*** [23115.497567] Lustre: Skipped 5 previous similar messages [23125.499989] Lustre: *** cfs_fail_loc=187, val=0*** [23125.501956] Lustre: Skipped 3 previous similar messages [23135.626930] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [23136.632889] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x187 [23142.767032] Lustre: *** cfs_fail_loc=187, val=0*** [23142.768864] Lustre: Skipped 4 previous similar messages [23156.899384] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [23157.947285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23158.838774] Lustre: DEBUG MARKER: rc=0; [23158.838774] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23158.838774] if [[ $? -eq 0 && $val -ne 0 ]]; then [23158.838774] echo $(hostname -s): $val; [23158.838774] rc=$val; [23158.838774] fi; [23158.838774] exit $rc [23159.442020] Lustre: DEBUG MARKER: dmesg [23160.014564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS ===================== 16:58:50 \(1491497930\) [23160.174083] Lustre: DEBUG MARKER: == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS ===================== 16:58:50 (1491497930) [23160.321836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23160.321836] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23160.321836] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23181.926349] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23182.821230] Lustre: DEBUG MARKER: rc=0; [23182.821230] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23182.821230] if [[ $? -eq 0 && $val -ne 0 ]]; then [23182.821230] echo $(hostname -s): $val; [23182.821230] rc=$val; [23182.821230] fi; [23182.821230] exit $rc [23183.423701] Lustre: DEBUG MARKER: dmesg [23185.946039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 9: Verify that no req deadline happened during recovery ==================== 16:59:16 \(1491497956\) [23186.097797] Lustre: DEBUG MARKER: == replay-ost-single test 9: Verify that no req deadline happened during recovery ==================== 16:59:16 (1491497956) [23190.025299] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation obd_ping to node 10.9.5.244@tcp failed: rc = -107 [23190.025318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [23190.025320] Lustre: Skipped 7 previous similar messages [23190.033516] LustreError: Skipped 8 previous similar messages [23200.656548] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [23200.658707] Lustre: Skipped 28 previous similar messages [23201.574241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23201.741439] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23203.186621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23203.197543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23203.410863] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23203.425776] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23266.910082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 63 sec [23266.912478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 63 sec [23267.107197] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 63 sec [23267.190187] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 63 sec [23269.124897] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23270.025277] Lustre: DEBUG MARKER: rc=0; [23270.025277] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23270.025277] if [[ $? -eq 0 && $val -ne 0 ]]; then [23270.025277] echo $(hostname -s): $val; [23270.025277] rc=$val; [23270.025277] fi; [23270.025277] exit $rc [23270.622066] Lustre: DEBUG MARKER: dmesg [23271.206140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 10: conflicting PW & PR locks on a client ================================== 17:00:42 \(1491498042\) [23271.365826] Lustre: DEBUG MARKER: == replay-ost-single test 10: conflicting PW [23286.431431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23286.612133] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23331.595243] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23332.477386] Lustre: DEBUG MARKER: rc=0; [23332.477386] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23332.477386] if [[ $? -eq 0 && $val -ne 0 ]]; then [23332.477386] echo $(hostname -s): $val; [23332.477386] rc=$val; [23332.477386] fi; [23332.477386] exit $rc [23333.093651] Lustre: DEBUG MARKER: dmesg [23333.654250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test complete, duration 679 sec ================================================= 17:01:44 \(1491498104\) [23333.824086] Lustre: DEBUG MARKER: == replay-ost-single test complete, duration 679 sec ================================================= 17:01:44 (1491498104) [23344.665950] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23344.925391] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [23345.135073] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880078de8000 x1563924897096720/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23345.140372] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [23345.496116] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [23345.498648] Lustre: Skipped 7 previous similar messages [23346.287214] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880078de8900 x1563924897096960/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23346.292530] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [23348.528956] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [23348.531427] Lustre: Skipped 3 previous similar messages [23351.167791] Lustre: server umount lustre-MDT0000 complete [23351.169958] Lustre: Skipped 1 previous similar message [23351.301569] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23353.528542] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [23353.533246] LustreError: Skipped 3 previous similar messages [23358.449915] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [23358.705052] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [23360.175078] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004a23c300 x1563924897097600/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23360.485055] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [23360.487802] Lustre: Skipped 9 previous similar messages [23362.303040] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498125/real 1491498125] req@ffff880078de8300 x1563924897097216/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491498132 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23362.310609] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [23362.313274] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [23364.902912] Lustre: server umount lustre-MDT0002 complete [23365.033383] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23432.335605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: insanity ============----- Thu Apr 6 17:03:22 UTC 2017 [23432.487543] Lustre: DEBUG MARKER: -----============= acceptance-small: insanity ============----- Thu Apr 6 17:03:22 UTC 2017 [23432.740114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test complete, duration -o sec =========================================================== 17:03:23 \(1491498203\) [23432.887998] Lustre: DEBUG MARKER: == insanity test complete, duration -o sec =========================================================== 17:03:23 (1491498203) [23433.160791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 15 [23433.321861] Lustre: DEBUG MARKER: excepting tests: 15 [23435.820735] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [23435.820735] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [23435.820735] if [ $running -ne $mpts ]; then [23435.820735] echo $(hostname) env are INSANE!; [23435.820735] exit 1; [23435.820735] fi [23436.384765] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [23436.384765] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [23436.384765] if [ $running -ne $mpts ]; then [23436.384765] echo $(hostname) env are INSANE!; [23436.384765] exit 1; [23436.384765] fi [23436.960043] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [23436.960043] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [23436.960043] if [ $running -ne $mpts ]; then [23436.960043] echo $(hostname) env are INSANE!; [23436.960043] exit 1; [23436.960043] fi [23439.514870] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [23439.773400] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23440.023087] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23440.271423] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23440.428331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23440.510456] LustreError: 11350:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [23440.514918] LustreError: 11350:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [23440.528651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [23440.530971] Lustre: Skipped 1 previous similar message [23440.676915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23440.932533] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23441.441364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23441.443411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23441.601632] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23441.604773] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23441.744392] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23441.997752] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23442.254184] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23442.514378] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23442.769098] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23445.078417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23445.242906] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23445.521090] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498211/real 1491498211] req@ffff880060e7bc00 x1563924897098080/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491498216 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23446.661064] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [23446.924587] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [23447.181407] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [23447.438342] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [23447.598959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23447.696248] LustreError: 12649:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [23447.701339] LustreError: 12653:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [23447.935023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23448.197355] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23448.696304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23448.699796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23448.862114] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23448.868459] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23449.012561] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23449.271346] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23449.529537] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23449.798995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [23450.059906] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23452.376337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23452.541165] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23455.889309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23456.048519] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23458.895972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23459.052241] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23461.910404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23462.076506] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23464.958201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23465.127118] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23467.990588] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23468.153363] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23471.028248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23471.196601] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23474.099839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23474.267875] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23477.208128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23477.380975] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23479.589831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23479.751635] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23481.191994] Lustre: DEBUG MARKER: lctl get_param -n timeout [23481.520399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [23481.677556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [23481.817810] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [23487.230948] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [23487.530972] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23488.057656] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23488.753570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 0: Fail all nodes, independently ==================================================== 17:04:19 \(1491498259\) [23488.919714] Lustre: DEBUG MARKER: == insanity test 0: Fail all nodes, independently ==================================================== 17:04:19 (1491498259) [23489.070958] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23489.330102] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23489.461604] Lustre: Failing over lustre-MDT0000 [23489.541175] Lustre: server umount lustre-MDT0000 complete [23489.674613] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23499.807042] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498263/real 1491498263] req@ffff88005dc5b900 x1563924897107056/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491498270 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23499.815843] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [23499.818843] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [23499.988387] Lustre: DEBUG MARKER: hostname [23500.292880] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23500.549645] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23500.806980] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23500.969211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23505.824187] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5fa5f1 to 0x61ac6f141d5faedb [23505.879275] LustreError: 15355:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [23505.886835] LustreError: 15360:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [23505.899126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [23505.901877] Lustre: Skipped 1 previous similar message [23506.072299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23506.335203] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23506.834033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23506.841980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23506.997617] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23507.002080] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23507.147296] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23507.407754] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23507.665271] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23507.927826] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23508.690531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23508.713225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23508.902700] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23508.956336] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23509.495729] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [23510.892823] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23510.916680] Lustre: lustre-MDT0000: Recovery over after 0:01, of 5 clients 5 recovered and 0 were evicted. [23511.136990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23511.262872] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23511.299558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23511.477877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23525.779130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23525.949694] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23527.635218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23527.635219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23527.854829] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23527.868325] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23530.146156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23530.148027] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23530.336676] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23530.405252] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23531.718153] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [23531.978294] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [23532.112424] Lustre: Failing over lustre-MDT0002 [23532.278816] Lustre: server umount lustre-MDT0002 complete [23532.409701] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23542.701292] Lustre: DEBUG MARKER: hostname [23543.010940] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [23543.270777] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [23543.528887] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [23543.679867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23543.765661] LustreError: 17382:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [23543.770950] LustreError: 17386:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [23543.816473] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [23543.981376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23544.240112] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23544.737911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23544.738814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23544.868106] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 5 clients reconnect [23544.898513] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23544.904699] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23545.053253] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23545.315118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23545.574906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23545.842209] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [23546.606577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [23546.646698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [23546.819461] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [23546.867387] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [23550.316476] Lustre: lustre-MDT0002: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [23551.140294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 4 sec [23551.156122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 4 sec [23551.343995] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 4 sec [23551.435094] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 4 sec [23565.831286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23565.998362] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23567.704511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [23567.717053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [23567.887221] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [23567.981216] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [23571.186262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [23571.251772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [23571.395153] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [23571.478258] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [23585.667387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23585.835218] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23587.277793] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23587.280742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23587.492279] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23587.524764] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [23590.777635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [23590.879710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [23590.950556] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [23591.085224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [23605.257778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23605.434590] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23606.952912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [23606.955090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [23607.123230] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [23607.141475] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [23611.380959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [23611.451610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [23611.585366] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [23611.680774] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [23625.930270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23626.102207] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23627.563893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [23627.564330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [23627.797146] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [23627.804288] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [23631.075194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [23631.086899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [23631.279777] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [23631.312383] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [23645.468843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23645.638848] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23647.078477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [23647.081797] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [23647.309810] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [23647.323038] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [23649.673427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 2 sec [23649.676577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 2 sec [23649.926439] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 2 sec [23649.932395] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 2 sec [23664.110718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23664.299139] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23665.785262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [23665.809538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [23666.019129] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [23666.021471] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [23671.271383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 5 sec [23671.357716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 5 sec [23671.430380] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 5 sec [23671.562990] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 5 sec [23685.709916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23685.877528] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23687.313729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [23687.329539] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [23687.502146] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [23687.580723] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [23689.732795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [23689.875670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [23689.884567] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [23690.076186] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [23704.228505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23704.398133] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23705.842583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [23705.860188] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [23706.068715] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [23706.112028] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [23706.302792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [23706.407974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [23706.462109] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [23706.631324] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [23720.764206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23720.929492] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23722.392587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [23722.394719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [23722.588186] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [23722.656091] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [23724.877639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [23724.926803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [23725.087544] Lustre: DEBUG MARKER: osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [23725.123260] Lustre: DEBUG MARKER: osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [23726.516792] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23727.419574] Lustre: DEBUG MARKER: rc=0; [23727.419574] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23727.419574] if [[ $? -eq 0 && $val -ne 0 ]]; then [23727.419574] echo $(hostname -s): $val; [23727.419574] rc=$val; [23727.419574] fi; [23727.419574] exit $rc [23728.038266] Lustre: DEBUG MARKER: dmesg [23728.667896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 1: MDS\/MDS failure ================================================================== 17:08:19 \(1491498499\) [23728.839171] Lustre: DEBUG MARKER: == insanity test 1: MDS/MDS failure ================================================================== 17:08:19 (1491498499) [23730.105988] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23730.365906] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23730.500795] Lustre: Failing over lustre-MDT0000 [23730.527310] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004b358600 x1563924897133952/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23730.536174] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [23730.606707] Lustre: server umount lustre-MDT0000 complete [23730.736369] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23733.813329] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [23733.818555] LustreError: Skipped 94 previous similar messages [23741.511032] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498504/real 1491498504] req@ffff88004b35aa00 x1563924897133984/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491498511 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23741.518726] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [23741.521332] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [23828.226999] Lustre: lustre-MDT0002: Connection restored to 10.9.5.239@tcp (at 10.9.5.239@tcp) [23828.229728] Lustre: Skipped 125 previous similar messages [23836.526101] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498582/real 1491498582] req@ffff880067149800 x1563924897137136/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491498607 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [23836.534078] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [23894.145124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23895.159892] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23901.471827] Lustre: DEBUG MARKER: hostname [23901.748944] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [23902.014108] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23902.268942] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23902.523250] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23902.671876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23926.526220] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5faedb to 0x61ac6f141d5fbc79 [23926.573843] LustreError: 22726:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [23926.581256] LustreError: 22731:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [23926.605072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [23926.836919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23927.111408] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23927.302263] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [23927.614974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23927.615644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23928.385859] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23928.391677] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23928.600383] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23928.873294] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23929.132649] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23929.398068] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23929.668312] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23931.587353] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23931.610378] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [23933.059554] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23933.949056] Lustre: DEBUG MARKER: rc=0; [23933.949056] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23933.949056] if [[ $? -eq 0 && $val -ne 0 ]]; then [23933.949056] echo $(hostname -s): $val; [23933.949056] rc=$val; [23933.949056] fi; [23933.949056] exit $rc [23934.545659] Lustre: DEBUG MARKER: dmesg [23935.144659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 2: Second Failure Mode: MDS\/OST Thu Apr 6 17:11:46 UTC 2017 ======================== 17:11:46 \(1491498706\) [23935.317929] Lustre: DEBUG MARKER: == insanity test 2: Second Failure Mode: MDS/OST Thu Apr 6 17:11:46 UTC 2017 ======================== 17:11:46 (1491498706) [23936.608471] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23936.868482] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23937.001889] Lustre: Failing over lustre-MDT0000 [23937.172850] Lustre: server umount lustre-MDT0000 complete [23937.303243] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23941.586181] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [23941.586185] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [23941.586186] Lustre: Skipped 32 previous similar messages [23941.596320] LustreError: Skipped 33 previous similar messages [23948.586060] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [23978.489258] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [23978.743042] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [23978.870966] Lustre: Failing over lustre-MDT0002 [23979.149930] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24117.261157] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24117.430653] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24118.387923] Lustre: DEBUG MARKER: hostname [24118.654241] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [24118.916680] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [24119.166793] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [24119.416202] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [24119.565181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24119.645092] LustreError: 24889:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [24119.660543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [24119.680857] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [24119.807265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24120.063980] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24120.551476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24120.555121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24120.719570] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24120.719623] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24120.870145] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24121.127647] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24121.382311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24121.648748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [24121.904682] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24124.478025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24124.637152] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24124.653051] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491498890/real 1491498890] req@ffff880066982d00 x1563924897147840/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491498895 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [24124.661265] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [24126.133146] Lustre: DEBUG MARKER: hostname [24126.411249] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [24126.679419] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [24126.940710] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [24127.196305] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [24127.351356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24127.447323] LustreError: 26258:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [24127.657802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24127.927293] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24128.437410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24128.439381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24128.607519] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24128.620964] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24128.761482] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24129.023627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24129.283357] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24129.548121] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [24129.807234] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24132.426204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24132.636771] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24136.488102] Lustre: lustre-MDT0000: Recovery over after 0:17, of 5 clients 5 recovered and 0 were evicted. [24142.096834] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24143.007391] Lustre: DEBUG MARKER: rc=0; [24143.007391] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24143.007391] if [[ $? -eq 0 && $val -ne 0 ]]; then [24143.007391] echo $(hostname -s): $val; [24143.007391] rc=$val; [24143.007391] fi; [24143.007391] exit $rc [24143.608843] Lustre: DEBUG MARKER: dmesg [24144.240029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 3: Thirdb Failure Mode: MDS\/CLIENT Thu Apr 6 17:15:15 UTC 2017 ===================== 17:15:15 \(1491498915\) [24144.394086] Lustre: DEBUG MARKER: == insanity test 3: Thirdb Failure Mode: MDS/CLIENT Thu Apr 6 17:15:15 UTC 2017 ===================== 17:15:15 (1491498915) [24144.547738] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [24144.804779] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [24144.935297] Lustre: Failing over lustre-MDT0000 [24145.167616] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24155.431062] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [24155.484948] Lustre: DEBUG MARKER: hostname [24155.808981] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [24156.067525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [24156.326799] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [24156.491533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24161.439249] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5fc05b to 0x61ac6f141d5fc7e7 [24161.499446] LustreError: 28087:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [24161.504349] LustreError: 28087:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [24161.509399] LustreError: 28092:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [24161.684563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24161.944207] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24162.437281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24162.439646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24162.605435] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24162.607306] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24162.633963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [24162.636712] Lustre: Skipped 1 previous similar message [24162.754053] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24163.013771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24163.271155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24163.535278] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [24164.301750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24164.316765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24164.519829] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24164.554917] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24166.514723] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [24166.770173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24166.870547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24166.930942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24167.089128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24181.441574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24181.601877] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24183.291836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24183.298197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24183.503842] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24183.550875] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24185.742856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24185.859439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24185.900049] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24186.100796] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24187.388204] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [24187.643056] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [24187.773590] Lustre: Failing over lustre-MDT0002 [24187.965299] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24198.261966] Lustre: DEBUG MARKER: hostname [24198.575042] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [24198.845431] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [24199.116744] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [24199.327474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24199.416952] LustreError: 30112:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [24199.634356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24199.910200] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24200.432162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24200.433089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24200.598558] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24200.598782] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24200.745497] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24201.005263] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24201.268056] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24201.529638] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [24202.311306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24202.337505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24202.516872] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24202.566778] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24204.472519] Lustre: lustre-MDT0002: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [24204.475740] Lustre: Skipped 2 previous similar messages [24204.788771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24204.895407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24204.940534] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24205.093160] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24219.356990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24219.525135] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24221.300826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24221.307381] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24221.536341] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24221.545101] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24223.769707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24223.811074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24224.018863] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24224.067729] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24226.590644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [24226.745007] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [24274.459192] Lustre: lustre-MDT0002: haven't heard from client 1af483de-6dd5-412f-1ca5-2848e56e8183 (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff8800614a5c00, cur 1491499045 expire 1491499015 last 1491498997 [24274.465454] Lustre: Skipped 1 previous similar message [24281.353178] Lustre: lustre-MDT0000: haven't heard from client 1af483de-6dd5-412f-1ca5-2848e56e8183 (at 10.9.5.238@tcp) in 55 seconds. I think it's dead, and I am evicting it. exp ffff880049d42c00, cur 1491499052 expire 1491499022 last 1491498997 [24281.360099] LNet: Service thread pid 24849 completed after 43.72s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [24284.067824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24284.234435] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24287.705703] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24288.623868] Lustre: DEBUG MARKER: rc=0; [24288.623868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24288.623868] if [[ $? -eq 0 && $val -ne 0 ]]; then [24288.623868] echo $(hostname -s): $val; [24288.623868] rc=$val; [24288.623868] fi; [24288.623868] exit $rc [24289.240562] Lustre: DEBUG MARKER: dmesg [24289.863261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 4: Fourth Failure Mode: OST\/MDS Thu Apr 6 17:17:40 UTC 2017 ======================== 17:17:40 \(1491499060\) [24290.028852] Lustre: DEBUG MARKER: == insanity test 4: Fourth Failure Mode: OST/MDS Thu Apr 6 17:17:40 UTC 2017 ======================== 17:17:40 (1491499060) [24295.990556] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [24296.252925] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [24296.385699] Lustre: Failing over lustre-MDT0000 [24296.552718] Lustre: server umount lustre-MDT0000 complete [24296.555141] Lustre: Skipped 3 previous similar messages [24296.685564] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24306.655051] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [24335.453684] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [24335.459021] LustreError: Skipped 842 previous similar messages [24337.929376] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [24338.182879] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [24338.513875] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24456.844162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24457.010390] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24457.973300] Lustre: DEBUG MARKER: hostname [24458.239844] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [24458.503232] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [24458.760735] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [24459.008632] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [24459.154286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24459.172754] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [24459.175294] Lustre: Skipped 124 previous similar messages [24459.239471] LustreError: 551:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [24459.244065] LustreError: 551:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [24459.260877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [24459.263311] Lustre: Skipped 3 previous similar messages [24459.414800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24459.675102] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24460.167160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24460.168185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24460.330485] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24460.332925] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24460.358118] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [24460.360885] Lustre: Skipped 1 previous similar message [24460.480461] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24460.734467] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24460.993852] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24461.255857] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [24461.510726] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24464.068690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24464.220624] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24464.251122] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491499229/real 1491499229] req@ffff88006296cf00 x1563924897174768/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491499234 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [24464.259795] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [24465.663157] Lustre: DEBUG MARKER: hostname [24465.945670] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [24466.212705] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [24466.471979] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [24466.728517] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [24466.879466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24466.966359] LustreError: 1970:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [24467.178404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24467.443156] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24468.041993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24468.049671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24468.213544] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24468.217679] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24468.367930] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24468.626833] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24468.887720] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24469.153461] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [24469.415163] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24472.025865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24472.191288] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24476.116664] Lustre: lustre-MDT0002: Recovery over after 0:08, of 5 clients 5 recovered and 0 were evicted. [24480.917747] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24481.822783] Lustre: DEBUG MARKER: rc=0; [24481.822783] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24481.822783] if [[ $? -eq 0 && $val -ne 0 ]]; then [24481.822783] echo $(hostname -s): $val; [24481.822783] rc=$val; [24481.822783] fi; [24481.822783] exit $rc [24482.427283] Lustre: DEBUG MARKER: dmesg [24483.134317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 5: Fifth Failure Mode: OST\/OST Thu Apr 6 17:20:54 UTC 2017 ========================= 17:20:54 \(1491499254\) [24483.292296] Lustre: DEBUG MARKER: == insanity test 5: Fifth Failure Mode: OST/OST Thu Apr 6 17:20:54 UTC 2017 ========================= 17:20:54 (1491499254) [24518.702582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24518.867847] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24522.060251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24522.229248] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24544.491057] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24545.400718] Lustre: DEBUG MARKER: rc=0; [24545.400718] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24545.400718] if [[ $? -eq 0 && $val -ne 0 ]]; then [24545.400718] echo $(hostname -s): $val; [24545.400718] rc=$val; [24545.400718] fi; [24545.400718] exit $rc [24546.002706] Lustre: DEBUG MARKER: dmesg [24546.572940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 6: Sixth Failure Mode: OST\/CLIENT Thu Apr 6 17:21:57 UTC 2017 ====================== 17:21:57 \(1491499317\) [24546.750304] Lustre: DEBUG MARKER: == insanity test 6: Sixth Failure Mode: OST/CLIENT Thu Apr 6 17:21:57 UTC 2017 ====================== 17:21:57 (1491499317) [24552.111405] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [24552.114106] LustreError: Skipped 24 previous similar messages [24552.116301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [24552.121048] Lustre: Skipped 18 previous similar messages [24564.206601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: , to fail: 1, failed: 0 [24564.356531] Lustre: DEBUG MARKER: Request fail clients: , to fail: 1, failed: 0 [24610.369102] Lustre: lustre-MDT0002: haven't heard from client 41864a5c-c4a9-9e4d-d2c1-d2822aeb679e (at 10.9.5.238@tcp) in 51 seconds. I think it's dead, and I am evicting it. exp ffff880063968c00, cur 1491499381 expire 1491499351 last 1491499330 [24619.404057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24619.570271] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24621.841264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24622.004341] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24691.825036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24692.718859] Lustre: DEBUG MARKER: rc=0; [24692.718859] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24692.718859] if [[ $? -eq 0 && $val -ne 0 ]]; then [24692.718859] echo $(hostname -s): $val; [24692.718859] rc=$val; [24692.718859] fi; [24692.718859] exit $rc [24693.328561] Lustre: DEBUG MARKER: dmesg [24693.942161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 7: Seventh Failure Mode: CLIENT\/MDS Thu Apr 6 17:24:24 UTC 2017 ==================== 17:24:24 \(1491499464\) [24694.111201] Lustre: DEBUG MARKER: == insanity test 7: Seventh Failure Mode: CLIENT/MDS Thu Apr 6 17:24:24 UTC 2017 ==================== 17:24:24 (1491499464) [24695.693172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [24695.845313] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [24742.009211] Lustre: lustre-MDT0000: haven't heard from client 75746e55-0652-f681-3560-9b9184760af5 (at 10.9.5.238@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff88005f257400, cur 1491499512 expire 1491499482 last 1491499465 [24742.014766] Lustre: Skipped 1 previous similar message [24805.477274] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [24805.736545] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [24805.869453] Lustre: Failing over lustre-MDT0000 [24805.871681] Lustre: Skipped 1 previous similar message [24806.063590] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24813.695076] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [24816.390830] Lustre: DEBUG MARKER: hostname [24816.700059] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [24816.960600] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [24817.216356] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [24817.369352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24819.699210] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5fd2fa to 0x61ac6f141d5fe005 [24819.762035] LustreError: 4865:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [24819.766897] LustreError: 4865:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [24819.771862] LustreError: 4870:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [24819.783532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [24819.786068] Lustre: Skipped 1 previous similar message [24819.958240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24820.222768] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24820.724266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24820.736160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24820.889067] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24820.892098] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24821.037358] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24821.295840] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24821.555524] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24821.820204] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [24822.374519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [24822.377506] Lustre: Skipped 1 previous similar message [24822.581209] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24822.583163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24822.744439] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24822.850604] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [24824.777314] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [24824.802708] Lustre: lustre-MDT0000: Recovery over after 0:02, of 4 clients 4 recovered and 0 were evicted. [24824.805770] Lustre: Skipped 1 previous similar message [24824.992658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24825.140226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [24839.396632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24839.552369] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24841.227814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24841.271571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24841.532889] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24841.547555] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [24843.815505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24843.975380] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [24845.251773] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [24845.520290] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [24845.877885] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [24856.171472] Lustre: DEBUG MARKER: hostname [24856.481065] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [24856.740590] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [24856.997953] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [24857.161555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [24857.459615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [24857.718415] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [24858.215046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24858.217674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24858.378178] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24858.379975] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24858.530218] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [24858.790284] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24859.050117] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [24859.317925] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [24860.088480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24860.099604] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24860.304833] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24860.348553] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [24862.608477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24862.767125] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [24877.015335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24877.177460] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24878.863611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24878.867097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24879.074936] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24879.111744] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [24881.418892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24881.571524] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [24884.389690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24884.588096] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [24946.051742] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24946.954206] Lustre: DEBUG MARKER: rc=0; [24946.954206] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24946.954206] if [[ $? -eq 0 && $val -ne 0 ]]; then [24946.954206] echo $(hostname -s): $val; [24946.954206] rc=$val; [24946.954206] fi; [24946.954206] exit $rc [24947.563649] Lustre: DEBUG MARKER: dmesg [24948.198421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 8: Eighth Failure Mode: CLIENT\/OST Thu Apr 6 17:28:39 UTC 2017 ===================== 17:28:39 \(1491499719\) [24948.357202] Lustre: DEBUG MARKER: == insanity test 8: Eighth Failure Mode: CLIENT/OST Thu Apr 6 17:28:39 UTC 2017 ===================== 17:28:39 (1491499719) [24949.926751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [24950.076544] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [24997.285204] Lustre: lustre-MDT0002: haven't heard from client d2f061dc-8668-cda0-5aa8-526374c142de (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff88004a425400, cur 1491499767 expire 1491499737 last 1491499719 [24997.291247] Lustre: Skipped 1 previous similar message [25002.374659] LNet: Service thread pid 484 completed after 41.41s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [25081.866754] Lustre: MGS: Connection restored to a915cd29-8cc4-d397-ae54-56b596806902 (at 10.9.5.238@tcp) [25081.869774] Lustre: Skipped 109 previous similar messages [25082.549187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25082.706967] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25085.161933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25085.317628] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25149.218220] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25150.124358] Lustre: DEBUG MARKER: rc=0; [25150.124358] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25150.124358] if [[ $? -eq 0 && $val -ne 0 ]]; then [25150.124358] echo $(hostname -s): $val; [25150.124358] rc=$val; [25150.124358] fi; [25150.124358] exit $rc [25150.749103] Lustre: DEBUG MARKER: dmesg [25151.389512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 9: Ninth Failure Mode: CLIENT\/CLIENT Thu Apr 6 17:32:02 UTC 2017 =================== 17:32:02 \(1491499922\) [25151.549529] Lustre: DEBUG MARKER: == insanity test 9: Ninth Failure Mode: CLIENT/CLIENT Thu Apr 6 17:32:02 UTC 2017 =================== 17:32:02 (1491499922) [25153.253758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [25153.405034] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [25175.296042] Lustre: 3780:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491499934/real 1491499934] req@ffff88007adb1e00 x1563924897242272/t0(0) o104->lustre-MDT0000@10.9.5.238@tcp:15/16 lens 296/224 e 0 to 1 dl 1491499945 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [25175.303670] Lustre: 3780:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [25201.092525] Lustre: lustre-MDT0002: haven't heard from client 9a0d9b7b-84f9-dbf5-e90b-e5c858a3ca7c (at 10.9.5.238@tcp) in 49 seconds. I think it's dead, and I am evicting it. exp ffff88005dc46800, cur 1491499971 expire 1491499941 last 1491499922 [25201.098132] Lustre: Skipped 1 previous similar message [25207.374872] LNet: Service thread pid 3780 completed after 43.07s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [25270.299101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 1 [25270.467672] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 1 [25270.665837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark No clients failed! [25270.817728] Lustre: DEBUG MARKER: No clients failed! [25273.634939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25273.803656] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25335.265952] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25336.151969] Lustre: DEBUG MARKER: rc=0; [25336.151969] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25336.151969] if [[ $? -eq 0 && $val -ne 0 ]]; then [25336.151969] echo $(hostname -s): $val; [25336.151969] rc=$val; [25336.151969] fi; [25336.151969] exit $rc [25336.755687] Lustre: DEBUG MARKER: dmesg [25337.325556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 10: Tenth Failure Mode: MDT0\/OST\/MDT1 Thu Apr 6 17:35:08 UTC 2017 ================== 17:35:08 \(1491500108\) [25337.488201] Lustre: DEBUG MARKER: == insanity test 10: Tenth Failure Mode: MDT0/OST/MDT1 Thu Apr 6 17:35:08 UTC 2017 ================== 17:35:08 (1491500108) [25337.632428] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [25337.892112] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [25338.022947] Lustre: Failing over lustre-MDT0000 [25338.025057] Lustre: Skipped 1 previous similar message [25338.122952] Lustre: server umount lustre-MDT0000 complete [25338.125096] Lustre: Skipped 3 previous similar messages [25338.254405] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [25339.263727] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [25339.268767] LustreError: Skipped 85 previous similar messages [25342.286169] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [25342.288881] LustreError: Skipped 10 previous similar messages [25342.291133] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [25342.295861] Lustre: Skipped 10 previous similar messages [25456.595921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25456.764309] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25478.611669] Lustre: DEBUG MARKER: hostname [25478.888566] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [25479.157839] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [25479.418645] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [25479.683447] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [25479.845079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [25504.095290] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5fe005 to 0x61ac6f141d5ff090 [25504.139413] LustreError: 10306:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [25504.144029] LustreError: 10306:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [25504.148720] LustreError: 10311:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [25504.151318] LustreError: 10311:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [25504.174467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [25504.177032] Lustre: Skipped 1 previous similar message [25504.357056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [25504.390173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [25504.392684] Lustre: Skipped 1 previous similar message [25504.619467] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [25505.123167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25505.135145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25505.291069] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25505.295463] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25505.438345] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25505.698982] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25505.956194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25506.219314] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [25506.478143] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25509.069828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25509.156915] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [25509.244369] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25513.182333] Lustre: lustre-MDT0000: Recovery over after 0:08, of 5 clients 5 recovered and 0 were evicted. [25513.185799] Lustre: Skipped 1 previous similar message [25514.692232] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25515.597600] Lustre: DEBUG MARKER: rc=0; [25515.597600] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25515.597600] if [[ $? -eq 0 && $val -ne 0 ]]; then [25515.597600] echo $(hostname -s): $val; [25515.597600] rc=$val; [25515.597600] fi; [25515.597600] exit $rc [25516.204613] Lustre: DEBUG MARKER: dmesg [25516.914174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 11: Eleventh Failure Mode: MDS0\/CLIENT\/MDS1 Thu Apr 6 17:38:07 UTC 2017 ============ 17:38:07 \(1491500287\) [25517.071628] Lustre: DEBUG MARKER: == insanity test 11: Eleventh Failure Mode: MDS0/CLIENT/MDS1 Thu Apr 6 17:38:07 UTC 2017 ============ 17:38:07 (1491500287) [25517.220209] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [25517.477829] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [25517.814663] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [25526.156039] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [25526.160772] LustreError: Skipped 1 previous similar message [25528.129745] Lustre: DEBUG MARKER: hostname [25528.441183] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [25528.700167] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [25528.954668] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [25529.115037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [25532.163252] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5ff090 to 0x61ac6f141d5ff4b1 [25532.390852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [25532.652874] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [25533.150056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25533.152965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25533.314086] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25533.314088] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25533.463384] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25533.722315] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25533.982620] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25534.245408] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [25535.023248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [25535.026323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [25535.284683] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [25535.286782] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [25537.231099] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [25537.554254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25537.560918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25537.801899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25537.811381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25540.376446] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [25540.527387] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [25590.191506] Lustre: lustre-MDT0000: haven't heard from client 2b0cf76a-2f92-fcda-ae6b-6e1d7b2a8c07 (at 10.9.5.238@tcp) in 49 seconds. I think it's dead, and I am evicting it. exp ffff8800600f7000, cur 1491500360 expire 1491500330 last 1491500311 [25590.197445] Lustre: Skipped 1 previous similar message [25592.884508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25593.046673] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25606.160421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25606.326885] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25608.021557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25608.034031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25608.238154] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25608.265823] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25610.542670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [25610.547375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [25610.714211] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [25610.774807] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [25615.287942] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25616.230415] Lustre: DEBUG MARKER: rc=0; [25616.230415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25616.230415] if [[ $? -eq 0 && $val -ne 0 ]]; then [25616.230415] echo $(hostname -s): $val; [25616.230415] rc=$val; [25616.230415] fi; [25616.230415] exit $rc [25616.844759] Lustre: DEBUG MARKER: dmesg [25617.443179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 12: Twelve Failure Mode: MDS0,MDS1\/OST0, OST1\/CLIENTS Thu Apr 6 17:39:48 UTC 2017 == 17:39:48 \(1491500388\) [25617.605199] Lustre: DEBUG MARKER: == insanity test 12: Twelve Failure Mode: MDS0,MDS1/OST0, OST1/CLIENTS Thu Apr 6 17:39:48 UTC 2017 == 17:39:48 (1491500388) [25617.753873] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [25618.019228] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [25618.383729] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [25629.582802] Lustre: DEBUG MARKER: hostname [25629.888180] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [25630.143347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [25630.401613] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [25630.563687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [25635.543201] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5ff4b1 to 0x61ac6f141d5ffc05 [25635.775991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [25636.039977] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [25636.531854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25636.535523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25636.695252] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25636.696574] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25636.844713] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25637.100615] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25637.361646] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25637.628280] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [25640.610611] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [25649.996321] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25650.167847] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25651.895256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25651.911849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25652.097299] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25652.165441] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25654.409105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25654.445571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25654.629772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25654.657193] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25654.864197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25654.939766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25655.065452] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25655.167351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25671.315362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25671.486559] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25683.544917] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [25683.547879] Lustre: Skipped 108 previous similar messages [25684.466568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25684.621940] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25686.080956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25686.104606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25686.265051] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25686.359685] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25686.505531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25686.643392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25686.681481] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25686.873570] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25688.080990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [25688.324876] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [25688.923744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25689.073890] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25691.526747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [25691.688415] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [25739.079413] Lustre: lustre-MDT0000: haven't heard from client 3f8b3a10-15bd-136c-8a3b-165c45d8db22 (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff88005f257400, cur 1491500509 expire 1491500479 last 1491500461 [25739.085488] Lustre: Skipped 1 previous similar message [25741.776611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25741.935051] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25745.398823] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25746.307906] Lustre: DEBUG MARKER: rc=0; [25746.307906] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25746.307906] if [[ $? -eq 0 && $val -ne 0 ]]; then [25746.307906] echo $(hostname -s): $val; [25746.307906] rc=$val; [25746.307906] fi; [25746.307906] exit $rc [25746.920255] Lustre: DEBUG MARKER: dmesg [25747.546213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 13: Thirteen Failure Mode: MDS0,MDS1\/CLIENTS\/OST0,OST1 Thu Apr 6 17:41:58 UTC 2017 ====================================================================================================== 17:41:58 \(1491500518\) [25747.707187] Lustre: DEBUG MARKER: == insanity test 13: Thirteen Failure Mode: MDS0,MDS1/CLIENTS/OST0,OST1 Thu Apr 6 17:41:58 UTC 2017 ====================================================================================================== 17:41:58 (1491500518) [25747.859851] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [25748.119252] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [25748.476545] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [25759.634060] Lustre: DEBUG MARKER: hostname [25759.943896] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [25760.202219] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [25760.457630] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [25760.619603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [25763.610285] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d5ffc05 to 0x61ac6f141d60042b [25763.848897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [25764.110989] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [25764.616112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25764.616794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25764.775684] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25764.777798] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25764.925787] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25765.181715] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25765.441279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25765.706291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [25768.670287] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [25778.074542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25778.247650] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25779.984941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25779.986267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25780.202523] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25780.239075] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25782.498289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25782.502702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25782.715881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25782.730697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [25782.946905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25783.014632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25783.153008] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25783.256172] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [25785.736213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [25785.890656] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [25807.782039] Lustre: 484:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491500567/real 1491500567] req@ffff880077bf9e00 x1563924897298544/t0(0) o104->lustre-MDT0000@10.9.5.238@tcp:15/16 lens 296/224 e 0 to 1 dl 1491500578 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [25807.790688] Lustre: 484:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [25833.670164] Lustre: lustre-MDT0000: haven't heard from client 93aa09a8-8207-02c8-0e96-348c9a4a3158 (at 10.9.5.238@tcp) in 49 seconds. I think it's dead, and I am evicting it. exp ffff880078c8ac00, cur 1491500604 expire 1491500574 last 1491500555 [25833.676353] Lustre: Skipped 1 previous similar message [25836.347352] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25836.516161] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25853.629911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25853.807864] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25866.802627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25866.958826] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25868.406704] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25868.420627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25868.620511] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25868.654930] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25868.884214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25868.955101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25869.050442] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25869.196607] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25871.279456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25871.423849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25871.436640] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25871.645401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25874.114813] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25875.032283] Lustre: DEBUG MARKER: rc=0; [25875.032283] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25875.032283] if [[ $? -eq 0 && $val -ne 0 ]]; then [25875.032283] echo $(hostname -s): $val; [25875.032283] rc=$val; [25875.032283] fi; [25875.032283] exit $rc [25875.647974] Lustre: DEBUG MARKER: dmesg [25876.229934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 14: Fourteen Failure Mode: OST0,OST1\/CLIENTS\/MDS0,MDS1 Thu Apr 6 17:44:07 UTC 2017 ====================================================================================================== 17:44:07 \(1491500647\) [25876.395786] Lustre: DEBUG MARKER: == insanity test 14: Fourteen Failure Mode: OST0,OST1/CLIENTS/MDS0,MDS1 Thu Apr 6 17:44:07 UTC 2017 ====================================================================================================== 17:44:07 (1491500647) [25890.196964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25890.365758] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25903.305662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25903.472954] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25904.919545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25904.930480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25905.141304] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25905.159394] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [25905.404155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25905.445937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25905.595004] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25905.668195] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [25907.861577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25907.936662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25908.068434] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25908.133545] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [25910.647370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [25910.798682] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [25961.339845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25961.502137] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25962.662352] Lustre: lustre-MDT0002: haven't heard from client 773d1cf2-69de-7ffb-ab70-9d09b38d7f49 (at 10.9.5.238@tcp) in 52 seconds. I think it's dead, and I am evicting it. exp ffff88005d1e5800, cur 1491500733 expire 1491500703 last 1491500681 [25962.668228] Lustre: Skipped 2 previous similar messages [25964.943178] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [25965.202472] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [25965.334753] Lustre: Failing over lustre-MDT0000 [25965.337152] Lustre: Skipped 3 previous similar messages [25965.445475] Lustre: server umount lustre-MDT0000 complete [25965.447992] Lustre: Skipped 3 previous similar messages [25965.579320] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [25967.164335] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [25967.169886] LustreError: Skipped 602 previous similar messages [25968.670172] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [25968.670172] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [25968.670174] Lustre: Skipped 25 previous similar messages [25968.681293] LustreError: Skipped 29 previous similar messages [25976.752234] Lustre: DEBUG MARKER: hostname [25977.072848] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [25977.331562] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [25977.586507] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [25977.749595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [25982.095309] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d60042b to 0x61ac6f141d600e8f [25982.331946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [25982.593910] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [25983.104259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25983.104678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25983.269802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25983.275580] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25983.430554] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [25983.690846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25983.948672] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [25984.211951] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [25987.159232] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [25996.587990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25996.750469] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [25998.478401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25998.478445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25998.702316] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [25998.704070] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [26001.005949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [26001.009554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [26001.222293] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [26001.249660] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [26001.478453] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [26001.503955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [26001.670868] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [26001.752079] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [26004.248365] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26005.143931] Lustre: DEBUG MARKER: rc=0; [26005.143931] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26005.143931] if [[ $? -eq 0 && $val -ne 0 ]]; then [26005.143931] echo $(hostname -s): $val; [26005.143931] rc=$val; [26005.143931] fi; [26005.143931] exit $rc [26005.740573] Lustre: DEBUG MARKER: dmesg [26006.332581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: insanity test_15 skipping ALWAYS excluded test 15 [26006.488280] Lustre: DEBUG MARKER: SKIP: insanity test_15 skipping ALWAYS excluded test 15 [26006.680099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test complete, duration 2574 sec ========================================================= 17:46:17 \(1491500777\) [26006.833212] Lustre: DEBUG MARKER: == insanity test complete, duration 2574 sec ========================================================= 17:46:17 (1491500777) [26006.993991] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [26007.249920] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [26007.597292] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [26017.909613] Lustre: DEBUG MARKER: hostname [26018.224650] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [26018.489186] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [26018.750867] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [26018.912593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [26018.931057] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d600e8f to 0x61ac6f141d6014a1 [26018.979634] LustreError: 23402:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [26018.984230] LustreError: 23402:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 4 previous similar messages [26018.989141] LustreError: 23407:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [26018.991739] LustreError: 23407:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 4 previous similar messages [26019.159479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [26019.421029] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [26020.007162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26020.012951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26020.256305] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26020.258109] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26020.417033] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26020.684724] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26020.948780] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26021.211634] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [26021.979752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [26022.023102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [26022.186075] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [26022.291732] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [26023.996340] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [26025.423772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [26025.648287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [26025.651076] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [26025.859546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [26037.820776] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [26038.083663] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [26038.923223] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [26038.923226] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [26038.928802] Lustre: Skipped 2 previous similar messages [26039.807078] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f23d800 x1563924897325248/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [26041.416992] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [26041.420040] Lustre: Skipped 9 previous similar messages [26043.922666] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [26043.925466] Lustre: Skipped 3 previous similar messages [26044.448490] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [26051.617525] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [26051.878787] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [26052.319083] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004bb06700 x1563924897325936/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [26052.324885] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [26053.923138] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [26053.925890] Lustre: Skipped 2 previous similar messages [26053.951067] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004bb06400 x1563924897325952/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [26058.264290] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [26126.053416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-sec ============----- Thu Apr 6 17:48:16 UTC 2017 [26126.224597] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-sec ============----- Thu Apr 6 17:48:16 UTC 2017 [26126.477947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test complete, duration -o sec ========================================================= 17:48:17 \(1491500897\) [26126.627174] Lustre: DEBUG MARKER: == sanity-sec test complete, duration -o sec ========================================================= 17:48:17 (1491500897) [26126.878450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26126.878450] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26126.878450] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26129.768771] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [26129.768771] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [26129.768771] if [ $running -ne $mpts ]; then [26129.768771] echo $(hostname) env are INSANE!; [26129.768771] exit 1; [26129.768771] fi [26130.409977] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [26130.409977] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [26130.409977] if [ $running -ne $mpts ]; then [26130.409977] echo $(hostname) env are INSANE!; [26130.409977] exit 1; [26130.409977] fi [26130.978213] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [26130.978213] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [26130.978213] if [ $running -ne $mpts ]; then [26130.978213] echo $(hostname) env are INSANE!; [26130.978213] exit 1; [26130.978213] fi [26133.493166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [26133.751141] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [26134.000334] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [26134.246783] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [26134.402959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [26134.496089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [26134.498489] Lustre: Skipped 5 previous similar messages [26134.648414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [26134.906635] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [26135.400955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26135.402855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26135.563157] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26135.563251] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26135.706203] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26135.958968] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26136.212922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26136.475235] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [26136.733704] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26139.041962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26139.192305] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26140.620278] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [26140.886808] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [26141.143454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [26141.397146] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [26141.558187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [26141.855035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [26142.116276] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [26142.612233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26142.618855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26142.775795] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26142.782689] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26142.926083] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26143.187138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26143.445359] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [26143.709871] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [26143.971265] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26146.251892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26146.410401] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26149.959189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26150.107909] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26152.944548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26153.107437] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26156.050843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26156.206499] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26159.079149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26159.248098] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26162.181462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26162.329365] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26165.207641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26165.405360] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26168.449246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26168.599803] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26171.526269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26171.696395] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26174.050663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26174.205347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [26175.639838] Lustre: DEBUG MARKER: lctl get_param -n timeout [26175.958545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [26176.118303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [26176.252393] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [26181.671041] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [26181.956125] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26182.471178] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [26182.994912] Lustre: DEBUG MARKER: lctl get_param -N mdt.\*MDT0000 [26183.257430] Lustre: DEBUG MARKER: mkdir -p /etc/lustre [26183.585190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 2 5 6 [26183.734250] Lustre: DEBUG MARKER: excepting tests: 2 5 6 [26183.926478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 26 [26184.079943] Lustre: DEBUG MARKER: skipping tests SLOW=no: 26 [26184.219303] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [26184.479003] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [26184.737260] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall /usr/sbin/l_getidentity [26185.002560] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [26186.295963] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*2 [26186.559636] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0002.identity_upcall [26186.818852] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0002.identity_upcall /usr/sbin/l_getidentity [26187.078092] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0002/identity_flush=-1 [26188.452254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 0: uid permission ================================================================= 17:49:19 \(1491500959\) [26188.607367] Lustre: DEBUG MARKER: == sanity-sec test 0: uid permission ================================================================= 17:49:19 (1491500959) [26188.957540] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26189.842544] Lustre: DEBUG MARKER: rc=0; [26189.842544] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26189.842544] if [[ $? -eq 0 && $val -ne 0 ]]; then [26189.842544] echo $(hostname -s): $val; [26189.842544] rc=$val; [26189.842544] fi; [26189.842544] exit $rc [26190.441040] Lustre: DEBUG MARKER: dmesg [26191.082306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 1: setuid\/gid ===================================================================== 17:49:22 \(1491500962\) [26191.249021] Lustre: DEBUG MARKER: == sanity-sec test 1: setuid/gid ===================================================================== 17:49:22 (1491500962) [26191.434154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-sec test_1 without GSS support. [26191.586328] Lustre: DEBUG MARKER: SKIP: sanity-sec test_1 without GSS support. [26191.788802] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26192.686294] Lustre: DEBUG MARKER: rc=0; [26192.686294] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26192.686294] if [[ $? -eq 0 && $val -ne 0 ]]; then [26192.686294] echo $(hostname -s): $val; [26192.686294] rc=$val; [26192.686294] fi; [26192.686294] exit $rc [26193.279651] Lustre: DEBUG MARKER: dmesg [26193.888478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 4: set supplementary group ======================================================== 17:49:24 \(1491500964\) [26194.067942] Lustre: DEBUG MARKER: == sanity-sec test 4: set supplementary group ======================================================== 17:49:24 (1491500964) [26194.242023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26194.242023] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26194.242023] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26194.544186] Lustre: DEBUG MARKER: echo '* 501 setgrp' > /etc/lustre/perm.conf [26194.798626] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_flush=-1 [26195.067360] Lustre: DEBUG MARKER: rm -f /etc/lustre/perm.conf [26195.320024] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_flush=-1 [26195.624387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26196.520777] Lustre: DEBUG MARKER: rc=0; [26196.520777] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26196.520777] if [[ $? -eq 0 && $val -ne 0 ]]; then [26196.520777] echo $(hostname -s): $val; [26196.520777] rc=$val; [26196.520777] fi; [26196.520777] exit $rc [26197.118309] Lustre: DEBUG MARKER: dmesg [26197.646233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26197.646233] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26197.646233] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26197.920994] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26198.180415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26198.180415] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26198.180415] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26198.454602] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26198.795821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 7: nodemap create and delete ====================================================== 17:49:29 \(1491500969\) [26198.952688] Lustre: DEBUG MARKER: == sanity-sec test 7: nodemap create and delete ====================================================== 17:49:29 (1491500969) [26199.116815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26199.116815] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26199.116815] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26199.392275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26199.392275] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26199.392275] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26199.662560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26199.922504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26199.922504] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26199.922504] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26200.190589] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26200.449211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26200.707210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26200.965111] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26201.221243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26201.478455] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26201.734532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26201.996643] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26202.254514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26202.511246] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26202.767763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26203.027981] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26203.288173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26203.551531] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26203.811158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26204.073444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26204.331073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26204.592080] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26204.851573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26205.107514] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26205.367327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26205.629296] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26205.889801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26206.158232] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26206.415529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26206.674654] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26206.933142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26207.190832] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26207.450561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26207.708024] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26207.967565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26208.230209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26208.492776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26208.752611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26209.009686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26209.269084] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26209.529531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26209.788688] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26210.046738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26210.309759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26210.578176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26210.836835] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26211.095045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26211.354162] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26211.611724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26211.872105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26212.129989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26212.386558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26212.645886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26212.903394] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26213.161508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26213.417579] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26213.674065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26213.929461] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26214.186979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26214.445823] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26214.700683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26214.960083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26215.216092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26215.472130] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26215.725800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26215.982690] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26216.238429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26216.497939] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26216.756095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26217.068139] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26217.942279] Lustre: DEBUG MARKER: rc=0; [26217.942279] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26217.942279] if [[ $? -eq 0 && $val -ne 0 ]]; then [26217.942279] echo $(hostname -s): $val; [26217.942279] rc=$val; [26217.942279] fi; [26217.942279] exit $rc [26218.552651] Lustre: DEBUG MARKER: dmesg [26219.329756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 8: nodemap reject duplicates ====================================================== 17:49:50 \(1491500990\) [26219.500968] Lustre: DEBUG MARKER: == sanity-sec test 8: nodemap reject duplicates ====================================================== 17:49:50 (1491500990) [26219.669404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26219.669404] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26219.669404] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26219.955689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26219.955689] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26219.955689] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26220.239276] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26220.518425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26220.518425] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26220.518425] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26220.802781] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26221.076622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26221.371754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26221.646384] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26221.920928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26222.195417] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26222.469631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26222.731973] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26222.990196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26223.249687] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26223.509482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26223.768849] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26224.025846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26224.286256] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26224.545714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26224.803222] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26225.062967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26225.323403] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26225.579948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26225.844099] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26226.101654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26226.362154] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26226.621468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26226.880770] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26227.138491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26227.396359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26227.651540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26227.908208] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26228.164771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26228.422198] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26228.694737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26228.957346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26229.215019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26229.482941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26229.482941] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26229.482941] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26229.757077] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26230.015250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26230.015250] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26230.015250] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26230.284156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26230.545822] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26230.804203] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26231.061341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26231.317703] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26231.586522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26231.844509] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26232.103051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26232.361162] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26232.619519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26232.878580] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26233.137282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26233.396779] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26233.653205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26233.922728] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26234.181111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26234.438538] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26234.696121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26234.954945] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26235.213123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26235.472480] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26235.729358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26235.986280] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26236.244542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26236.501616] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26236.757645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26237.015680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26237.273193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26237.533974] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26237.794954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26238.056314] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26238.313077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26238.572123] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26238.831994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26239.156282] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26240.061179] Lustre: DEBUG MARKER: rc=0; [26240.061179] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26240.061179] if [[ $? -eq 0 && $val -ne 0 ]]; then [26240.061179] echo $(hostname -s): $val; [26240.061179] rc=$val; [26240.061179] fi; [26240.061179] exit $rc [26240.655759] Lustre: DEBUG MARKER: dmesg [26241.260396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 9: nodemap range add ============================================================== 17:50:12 \(1491501012\) [26241.426843] Lustre: DEBUG MARKER: == sanity-sec test 9: nodemap range add ============================================================== 17:50:12 (1491501012) [26241.583535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26241.583535] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26241.583535] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26241.855474] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26241.855474] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26241.855474] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26242.122525] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26242.379863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26242.379863] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26242.379863] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26242.647743] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26242.900967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26243.157616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26243.417374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26243.673688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26243.928312] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26244.182743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26244.439549] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26244.696779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26244.954218] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26245.213724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26245.471508] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26245.730506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26245.985454] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26246.239731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26246.496465] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26246.755955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26247.016444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26247.273157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26247.529245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26247.785282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26248.040966] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26248.298072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26248.557841] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26248.815515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26249.076222] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26249.335339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26249.592113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26249.851765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26250.109861] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26250.366901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26250.624723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26250.883522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26251.144059] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26251.403390] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26251.660974] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26251.919268] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26252.180190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26252.438895] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26252.697106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26252.956784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26253.213582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26253.483590] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26253.756433] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26254.014624] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26254.272413] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26254.532623] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26254.799598] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26255.056607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26255.314189] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26255.570771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26255.832558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26256.097311] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26256.354394] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26256.611966] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26256.868678] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26257.131767] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26257.387662] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26257.644225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26257.902103] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26258.163377] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26258.419524] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26258.678554] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26258.935001] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26259.192927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26259.450971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26259.708088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26259.962376] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26260.219721] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26260.477902] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26260.732638] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26260.990867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26261.247997] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26261.504824] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26261.763639] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26262.026850] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26262.287177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26262.545795] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26262.801391] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26263.056765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26263.315774] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26263.576052] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26263.831252] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26264.087898] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26264.345624] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26264.602353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26264.857474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26265.119601] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26265.376667] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26265.634684] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26265.891117] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26266.149388] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26266.407752] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26266.663217] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26266.920941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26267.179056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26267.436445] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26267.692349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26267.950420] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26268.207967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26268.465073] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26268.722715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26268.977656] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26269.234956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26269.494334] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26269.752806] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26270.014464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26270.272346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26270.531213] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26270.783031] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26271.038052] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26271.291769] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26271.546727] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26271.801174] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26272.055808] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26272.312107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26272.566120] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26272.826762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26273.083600] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26273.337966] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26273.595522] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26273.850547] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26274.108600] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26274.365759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26274.619285] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26274.871357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26275.126738] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26275.383241] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26275.636594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26275.889747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26276.143285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26276.398755] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26276.655835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26276.913718] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26277.172340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26277.428307] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26277.686427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26277.945390] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26278.203852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26278.459255] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26278.716097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26278.973958] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26279.231194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26279.486727] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26279.741427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26280.009218] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26280.278157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26280.536337] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26280.793166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26281.053316] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26281.313192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26281.570236] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26281.828932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26282.088083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26282.345149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26282.604353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26282.862932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26283.120626] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26283.388755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26283.651865] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26283.911510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26284.230386] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26285.113377] Lustre: DEBUG MARKER: rc=0; [26285.113377] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26285.113377] if [[ $? -eq 0 && $val -ne 0 ]]; then [26285.113377] echo $(hostname -s): $val; [26285.113377] rc=$val; [26285.113377] fi; [26285.113377] exit $rc [26285.729870] Lustre: DEBUG MARKER: dmesg [26286.369849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 10: nodemap reject duplicate ranges =============================================== 17:50:57 \(1491501057\) [26286.530743] Lustre: DEBUG MARKER: == sanity-sec test 10: nodemap reject duplicate ranges =============================================== 17:50:57 (1491501057) [26286.687716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26286.687716] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26286.687716] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26286.960975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26286.960975] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26286.960975] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26287.233743] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26287.493114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26287.493114] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26287.493114] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26287.761728] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26288.019189] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26288.276768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26288.547828] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26288.805333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26289.060900] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26289.317315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26289.577929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26289.834368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26290.090816] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26290.350558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26290.610966] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26290.868765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26291.132267] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26291.393097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26291.651107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26291.909047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26292.179006] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26292.438336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26292.694665] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26292.951324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26293.210360] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26293.467367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26293.723550] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26293.983220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26294.240829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26294.496383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26294.752656] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26295.010345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26295.265869] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26295.523709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26295.785531] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26296.043588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26296.311239] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26296.568354] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26296.826762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26297.084646] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26297.348076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26297.606307] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26297.862156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26298.117037] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26298.373600] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26298.629915] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26298.895940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26299.153633] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26299.407582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26299.663069] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26299.919759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26300.176985] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26300.443753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26300.703272] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26300.961851] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26301.217802] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26301.471496] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26301.728424] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26301.985653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26302.242693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26302.497777] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26302.755585] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26303.011061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26303.266955] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26303.522652] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26303.784108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26304.041184] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26304.297695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26304.555793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26304.811759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26305.068872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26305.325666] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26305.592079] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26305.850594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26306.105063] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26306.366771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26306.624067] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26306.883081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26307.141159] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26307.403793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26307.663785] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26307.918691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26308.177823] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26308.432604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26308.691839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26308.821078] LustreError: 29130:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_0': rc = -17 [26308.823023] LustreError: 29130:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_0: rc = -12 [26308.955043] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26309.214858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26309.343227] LustreError: 29276:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_0': rc = -17 [26309.345620] LustreError: 29276:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 1 previous similar message [26309.347188] LustreError: 29276:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_0: rc = -12 [26309.349568] LustreError: 29276:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 1 previous similar message [26309.479237] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26309.734782] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26309.992161] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26310.249827] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26310.381192] LustreError: 29568:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_2': rc = -17 [26310.384124] LustreError: 29568:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 3 previous similar messages [26310.386022] LustreError: 29568:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_2: rc = -12 [26310.389038] LustreError: 29568:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 3 previous similar messages [26310.518853] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26310.775388] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26311.032661] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26311.287390] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26311.544972] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26311.802889] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26312.058753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26312.314548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26312.443427] LustreError: 30154:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_4': rc = -17 [26312.446943] LustreError: 30154:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 7 previous similar messages [26312.449121] LustreError: 30154:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_4: rc = -12 [26312.452705] LustreError: 30154:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 7 previous similar messages [26312.582142] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26312.840070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26313.096854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26313.353907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26313.612843] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26313.873170] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26314.133223] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26314.403504] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26314.673982] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26314.942138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26315.200898] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26315.460791] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26315.720687] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26315.979426] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26316.241467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26316.513943] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26316.645821] LustreError: 31322:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_10': rc = -17 [26316.649416] LustreError: 31322:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 15 previous similar messages [26316.651578] LustreError: 31322:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_10: rc = -12 [26316.655029] LustreError: 31322:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 15 previous similar messages [26316.784641] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26317.040945] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26317.298251] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26317.553492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26317.810258] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26318.067175] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26318.325374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26318.583925] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26318.840363] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26319.098358] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26319.357088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26319.613161] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26319.870065] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26320.128854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26320.387139] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26320.644527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26320.902413] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26321.162868] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26321.424895] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26321.683116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26321.941921] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26322.200270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26322.457296] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26322.714608] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26322.974569] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26323.230682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26323.488063] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26323.745887] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26324.002657] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26324.262468] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26324.520805] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26324.778670] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26325.037291] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26325.292711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26325.548312] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26325.804898] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26326.063398] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26326.328793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26326.593982] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26326.854766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26327.113975] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26327.373485] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26327.632122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26327.890553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26328.143172] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26328.403216] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26328.663113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26328.919921] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26329.176467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26329.441928] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26329.697959] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26329.953171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26330.206576] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26330.460838] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26330.717501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26330.973531] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26331.233559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26331.490396] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26331.748148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26332.005773] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26332.260326] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26332.519666] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26332.778313] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26333.032209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26333.284764] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26333.539469] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26333.795230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26334.053733] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26334.310873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26334.568179] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26334.823140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26335.085345] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26335.350122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26335.613802] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26335.866634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26336.123158] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26336.375358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26336.631526] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26336.886569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26337.143259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26337.400026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26337.657636] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26337.914357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26338.170966] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26338.424798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26338.685186] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26338.944140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26339.202936] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26339.461640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26339.723559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26339.992312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26340.250829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26340.506319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26340.763776] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26341.023398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26341.279829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26341.537971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26341.853173] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26342.752926] Lustre: DEBUG MARKER: rc=0; [26342.752926] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26342.752926] if [[ $? -eq 0 && $val -ne 0 ]]; then [26342.752926] echo $(hostname -s): $val; [26342.752926] rc=$val; [26342.752926] fi; [26342.752926] exit $rc [26343.349808] Lustre: DEBUG MARKER: dmesg [26343.952180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 11: nodemap modify ================================================================ 17:51:54 \(1491501114\) [26344.106137] Lustre: DEBUG MARKER: == sanity-sec test 11: nodemap modify ================================================================ 17:51:54 (1491501114) [26344.262798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26344.262798] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26344.262798] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26344.534216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26344.534216] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26344.534216] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26344.803615] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26345.062301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26345.062301] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26345.062301] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26345.331906] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26345.590054] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26345.851217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26346.120498] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26346.393648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26346.652976] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26346.910290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26347.170327] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26347.427957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26347.688751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26347.946017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26348.204257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26348.462763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26348.719377] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26348.979183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26349.241722] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26349.499988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26349.760093] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26350.020508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26350.279047] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26350.545276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26350.819379] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26351.084061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26351.350331] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26351.623546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26351.898399] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26352.171789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26352.445479] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26352.704506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26352.966056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26353.229190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26353.488757] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26353.748327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26354.008275] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 1 [26354.270909] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 0 [26354.526990] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 1 [26354.786724] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 0 [26355.046044] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 1 [26355.308170] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 0 [26355.566801] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 1 [26355.843537] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 0 [26356.111380] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 1 [26356.368017] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 0 [26356.624145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 1 [26356.884252] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 0 [26357.142407] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 1 [26357.400687] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 0 [26357.658194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 1 [26357.919519] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 0 [26358.179707] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 1 [26358.436101] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 0 [26358.694417] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 1 [26358.952209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 0 [26359.214511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 1 [26359.470867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 0 [26359.724701] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 1 [26359.981336] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 0 [26360.236315] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 1 [26360.493735] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 0 [26360.752530] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 1 [26361.008548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 0 [26361.264603] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 1 [26361.519693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 0 [26361.773245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 1 [26362.029192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 0 [26362.286939] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 1 [26362.544444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 0 [26362.801941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 1 [26363.058183] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 0 [26363.315387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 1 [26363.568624] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 0 [26363.825356] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 1 [26364.079108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 0 [26364.333906] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 1 [26364.590185] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 0 [26364.846744] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 1 [26365.104606] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 0 [26365.361711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 1 [26365.617634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 0 [26365.871332] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 1 [26366.132771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 0 [26366.395434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 1 [26366.653020] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 0 [26366.911969] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 1 [26367.170793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 0 [26367.430474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 1 [26367.688131] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 0 [26367.944083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 1 [26368.200165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 0 [26368.457141] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 1 [26368.715768] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 0 [26368.974341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 1 [26369.228478] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 0 [26369.484451] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 1 [26369.741175] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 0 [26369.997120] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 1 [26370.254201] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 0 [26370.511188] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26370.779973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26371.053502] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26371.310159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26371.567623] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26371.821871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26372.079567] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26372.335951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26372.592350] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26372.850865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26373.112259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26373.368989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26373.627790] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26373.886550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26374.144995] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26374.405229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26374.664992] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26374.922341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26375.182193] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26375.437888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26375.704289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26375.962566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26376.223130] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26376.483811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26376.741695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26376.997101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26377.258782] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26377.513380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26377.772121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26378.028658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26378.288099] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26378.542770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26378.851901] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26379.741930] Lustre: DEBUG MARKER: rc=0; [26379.741930] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26379.741930] if [[ $? -eq 0 && $val -ne 0 ]]; then [26379.741930] echo $(hostname -s): $val; [26379.741930] rc=$val; [26379.741930] fi; [26379.741930] exit $rc [26380.344592] Lustre: DEBUG MARKER: dmesg [26380.968788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 12: nodemap set squash ids ======================================================== 17:52:31 \(1491501151\) [26381.130396] Lustre: DEBUG MARKER: == sanity-sec test 12: nodemap set squash ids ======================================================== 17:52:31 (1491501151) [26381.278697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26381.278697] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26381.278697] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26381.550284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26381.550284] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26381.550284] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26381.818807] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26382.077523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26382.077523] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26382.077523] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26382.350137] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26382.606357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26382.864355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26383.124766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26383.381542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26383.638637] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26383.894962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26384.155372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26384.414609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26384.678262] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26384.953942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26385.229864] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26385.505165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26385.781104] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26386.057539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26386.332160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26386.602248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26386.862153] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26387.119181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26387.378267] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26387.637771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26387.895381] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26388.153032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26388.413081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26388.671636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26388.930510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26389.186433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26389.443133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26389.699863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26389.962752] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26390.225101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26390.484735] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26390.741351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26390.998771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26390.998771] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26390.998771] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26391.272994] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_0 --value 88 [26391.531669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26391.531669] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26391.531669] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26391.799266] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_1 --value 88 [26392.057959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26392.057959] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26392.057959] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26392.328474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_2 --value 88 [26392.587311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26392.587311] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26392.587311] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26392.857289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_3 --value 88 [26393.114486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26393.114486] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26393.114486] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26393.380218] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_4 --value 88 [26393.639309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26393.639309] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26393.639309] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26393.915867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_5 --value 88 [26394.172704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26394.172704] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26394.172704] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26394.442193] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_6 --value 88 [26394.698836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26394.698836] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26394.698836] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26394.969864] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_7 --value 88 [26395.229723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26395.229723] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26395.229723] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26395.511913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_8 --value 88 [26395.771199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26395.771199] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26395.771199] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26396.044032] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_9 --value 88 [26396.303349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26396.303349] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26396.303349] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26396.573196] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_10 --value 88 [26396.832695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26396.832695] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26396.832695] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26397.102781] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_11 --value 88 [26397.362214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26397.362214] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26397.362214] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26397.636829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_12 --value 88 [26397.899701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26397.899701] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26397.899701] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26398.176983] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_13 --value 88 [26398.437973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26398.437973] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26398.437973] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26398.708140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_14 --value 88 [26398.969740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26398.969740] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26398.969740] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26399.242929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_15 --value 88 [26399.500723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26399.500723] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26399.500723] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26399.774925] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_0 --value 88 [26400.031966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26400.031966] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26400.031966] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26400.302114] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_1 --value 88 [26400.557881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26400.557881] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26400.557881] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26400.832173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_2 --value 88 [26401.102361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26401.102361] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26401.102361] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26401.385739] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_3 --value 88 [26401.666383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26401.666383] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26401.666383] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26401.942473] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_4 --value 88 [26402.200982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26402.200982] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26402.200982] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26402.471328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_5 --value 88 [26402.733586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26402.733586] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26402.733586] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26403.003831] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_6 --value 88 [26403.263157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26403.263157] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26403.263157] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26403.535754] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_7 --value 88 [26403.794646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26403.794646] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26403.794646] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26404.069820] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_8 --value 88 [26404.335763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26404.335763] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26404.335763] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26404.608117] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_9 --value 88 [26404.865949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26404.865949] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26404.865949] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26405.138893] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_10 --value 88 [26405.402036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26405.402036] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26405.402036] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26405.675897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_11 --value 88 [26405.935434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26405.935434] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26405.935434] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26406.214199] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_12 --value 88 [26406.477842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26406.477842] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26406.477842] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26406.753675] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_13 --value 88 [26407.016420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26407.016420] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26407.016420] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26407.306414] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_14 --value 88 [26407.581398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26407.581398] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26407.581398] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26407.867764] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_15 --value 88 [26408.142522] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26408.416769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26408.677452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26408.937436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26409.198027] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26409.458277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26409.719167] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26409.979828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26410.240892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26410.499348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26410.759861] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26411.016785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26411.276165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26411.533839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26411.793691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26412.054590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26412.316542] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26412.577145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26412.838762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26413.096394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26413.356549] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26413.614644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26413.876203] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26414.141564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26414.402823] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26414.660771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26414.924339] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26415.184879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26415.443123] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26415.700931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26415.958583] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26416.220099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26416.532060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26417.418796] Lustre: DEBUG MARKER: rc=0; [26417.418796] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26417.418796] if [[ $? -eq 0 && $val -ne 0 ]]; then [26417.418796] echo $(hostname -s): $val; [26417.418796] rc=$val; [26417.418796] fi; [26417.418796] exit $rc [26418.016466] Lustre: DEBUG MARKER: dmesg [26418.677972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 13: test nids ===================================================================== 17:53:09 \(1491501189\) [26418.824562] Lustre: DEBUG MARKER: == sanity-sec test 13: test nids ===================================================================== 17:53:09 (1491501189) [26418.978530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26418.978530] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26418.978530] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26419.250229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26419.250229] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26419.250229] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26419.523554] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26419.781881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26419.781881] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26419.781881] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26420.049950] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26420.307747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26420.565937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26420.823917] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26421.080525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26421.338241] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26421.595076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26421.850658] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26422.106565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26422.363907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26422.620531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26422.876766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26423.131606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26423.384634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26423.640382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26423.895211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26424.151998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26424.410359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26424.666686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26424.922447] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26425.184536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26425.440648] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26425.696718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26425.954138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26426.211638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26426.468112] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26426.725939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26426.987135] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26427.243227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26427.502156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26427.759868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26428.020283] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26428.275250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26428.534218] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26428.793784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26429.051810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26429.311979] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26429.577534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26429.844899] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26430.103372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26430.361876] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26430.620166] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26430.877633] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26431.136798] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26431.394347] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26431.650095] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26431.909042] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26432.170176] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26432.428146] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26432.684766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26432.939210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26433.197088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26433.454063] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26433.711060] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26433.965566] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26434.224536] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26434.484948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26434.742236] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26434.996981] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26435.255084] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26435.511344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26435.769294] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26436.025184] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26436.283244] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26436.542270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26436.798876] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26437.055788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26437.312998] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26437.581569] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26437.840296] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26438.096492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26438.354649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26438.612127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26438.868208] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26439.124612] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26439.379889] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26439.638582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26439.893695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26440.148482] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26440.404128] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26440.659985] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26440.916700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.1 [26441.175768] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.10 [26441.432363] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.64 [26441.689676] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.128 [26441.946085] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.200 [26442.204922] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.250 [26442.461157] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.1 [26442.718727] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.10 [26442.974909] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.64 [26443.232421] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.128 [26443.488971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.200 [26443.743641] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.250 [26444.003078] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.1 [26444.257655] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.10 [26444.517699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.64 [26444.777429] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.128 [26445.050637] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.200 [26445.323955] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.250 [26445.596083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.1 [26445.867551] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.10 [26446.139704] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.64 [26446.405434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.128 [26446.662255] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.200 [26446.920473] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.250 [26447.176556] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.1 [26447.434783] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.10 [26447.690682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.64 [26447.945973] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.128 [26448.201935] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.200 [26448.456342] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.250 [26448.712630] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.1 [26448.968152] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.10 [26449.222287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.64 [26449.477533] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.128 [26449.733601] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.200 [26449.990847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.250 [26450.245357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.1 [26450.500232] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.10 [26450.757843] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.64 [26451.012797] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.128 [26451.269187] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.200 [26451.525631] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.250 [26451.779396] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.1 [26452.038410] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.10 [26452.296647] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.64 [26452.554664] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.128 [26452.809625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.200 [26453.063735] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.250 [26453.320180] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.1 [26453.576749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.10 [26453.831688] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.64 [26454.084400] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.128 [26454.340098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.200 [26454.594885] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.250 [26454.848927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.1 [26455.102702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.10 [26455.356603] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.64 [26455.612057] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.128 [26455.866428] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.200 [26456.124088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.250 [26456.381365] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.1 [26456.638381] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.10 [26456.895057] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.64 [26457.151621] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.128 [26457.415031] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.200 [26457.672614] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.250 [26457.927690] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.1 [26458.184370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.10 [26458.445177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.64 [26458.718302] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.128 [26458.990629] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.200 [26459.262666] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.250 [26459.534736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.1 [26459.805751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.10 [26460.075405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.64 [26460.333700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.128 [26460.589543] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.200 [26460.844326] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.250 [26461.102574] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.1 [26461.368654] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.10 [26461.626756] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.64 [26461.885763] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.128 [26462.142974] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.200 [26462.399135] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.250 [26462.653650] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.1 [26462.910477] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.10 [26463.164913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.64 [26463.421945] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.128 [26463.677649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.200 [26463.934245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.250 [26464.191368] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.1 [26464.451372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.10 [26464.709107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.64 [26464.967151] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.128 [26465.223943] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.200 [26465.480710] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.250 [26465.740634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.1 [26465.997373] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.10 [26466.254146] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.64 [26466.512207] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.128 [26466.772069] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.200 [26467.028548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.250 [26467.287845] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.1 [26467.543535] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.10 [26467.797228] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.64 [26468.055115] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.128 [26468.311194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.200 [26468.565932] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.250 [26468.821427] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.1 [26469.079518] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.10 [26469.337343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.64 [26469.591988] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.128 [26469.846198] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.200 [26470.102711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.250 [26470.358468] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.1 [26470.620766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.10 [26470.874359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.64 [26471.127499] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.128 [26471.382949] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.200 [26471.639350] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.250 [26471.895114] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.1 [26472.151271] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.10 [26472.410702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.64 [26472.665599] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.128 [26472.921426] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.200 [26473.175908] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.250 [26473.436848] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.1 [26473.697694] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.10 [26473.951751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.64 [26474.207418] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.128 [26474.464721] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.200 [26474.722252] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.250 [26474.981005] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.1 [26475.237855] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.10 [26475.495960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.64 [26475.762365] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.128 [26476.018142] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.200 [26476.274697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.250 [26476.531278] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.1 [26476.786125] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.10 [26477.042441] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.64 [26477.301633] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.128 [26477.558094] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.200 [26477.814698] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.250 [26478.068090] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.1 [26478.325065] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.10 [26478.580181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.64 [26478.837793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.128 [26479.090770] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.200 [26479.351977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.250 [26479.970967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.1 [26480.661596] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.10 [26481.356946] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.64 [26481.617018] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.128 [26481.870717] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.200 [26482.127306] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.250 [26482.391813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.1 [26482.656161] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.10 [26482.911693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.64 [26483.168181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.128 [26483.426109] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.200 [26483.679702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.250 [26483.933337] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.1 [26484.187850] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.10 [26484.442800] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.64 [26484.697487] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.128 [26484.953093] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.200 [26485.211442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.250 [26485.465855] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.1 [26485.719540] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.10 [26485.975061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.64 [26486.229359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.128 [26486.494697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.200 [26486.750298] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.250 [26487.007630] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.1 [26487.264849] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.10 [26487.522815] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.64 [26487.781389] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.128 [26488.037581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.200 [26488.295118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.250 [26488.550053] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.1 [26488.807907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.10 [26489.064729] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.64 [26489.321940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.128 [26489.578118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.200 [26489.835670] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.250 [26490.093615] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.1 [26490.353398] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.10 [26490.609226] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.64 [26490.864464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.128 [26491.122533] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.200 [26491.380847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.250 [26491.637086] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.1 [26491.895711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.10 [26492.154343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.64 [26492.410725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.128 [26492.667121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.200 [26492.924024] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.250 [26493.179653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.1 [26493.436025] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.10 [26493.699680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.64 [26493.958196] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.128 [26494.214604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.200 [26494.473603] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.250 [26494.730026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.1 [26494.983630] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.10 [26495.243717] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.64 [26495.505177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.128 [26495.763497] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.200 [26496.020722] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.250 [26496.278220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.1 [26496.534294] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.10 [26496.794579] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.64 [26497.051751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.128 [26497.307993] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.200 [26497.564325] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.250 [26497.820198] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.1 [26498.079098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.10 [26498.336070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.64 [26498.594220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.128 [26498.849846] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.200 [26499.106262] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.250 [26499.363334] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.1 [26499.619145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.10 [26499.874675] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.64 [26500.139114] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.128 [26500.396312] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.200 [26500.654216] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.250 [26500.911784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.1 [26501.166221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.10 [26501.422257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.64 [26501.680032] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.128 [26501.938402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.200 [26502.193623] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.250 [26502.448259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.1 [26502.705402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.10 [26502.960959] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.64 [26503.217983] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.128 [26503.473824] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.200 [26503.729461] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.250 [26503.987380] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.1 [26504.241813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.10 [26504.509430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.64 [26504.764084] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.128 [26505.021088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.200 [26505.279867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.250 [26505.542787] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.1 [26505.809619] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.10 [26506.065322] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.64 [26506.322730] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.128 [26506.578444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.200 [26506.836965] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.250 [26507.095403] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.1 [26507.352659] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.10 [26507.612439] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.64 [26507.871259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.128 [26508.128177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.200 [26508.382003] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.250 [26508.634883] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.1 [26508.890288] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.10 [26509.146711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.64 [26509.404211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.128 [26509.660592] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.200 [26509.913783] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.250 [26510.171793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.1 [26510.426598] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.10 [26510.680848] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.64 [26510.935941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.128 [26511.191177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.200 [26511.449525] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.250 [26511.706265] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.1 [26511.963470] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.10 [26512.227259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.64 [26512.499597] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.128 [26512.757488] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.200 [26513.013336] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.250 [26513.269779] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.1 [26513.534338] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.10 [26513.804026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.64 [26514.058922] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.128 [26514.314707] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.200 [26514.569113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.250 [26514.825746] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.1 [26515.077691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.10 [26515.338548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.64 [26515.593387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.128 [26515.846603] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.200 [26516.100611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.250 [26516.357531] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26516.611089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26516.868140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26517.129065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26517.385789] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26517.641966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26517.901217] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26518.157038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26518.415507] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26518.671902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26518.927976] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26519.187820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26519.445374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26519.703520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26519.961926] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26520.218123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26520.476194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26520.735635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26520.992137] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26521.248126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26521.509685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26521.772128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26522.029949] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26522.286875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26522.544788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26522.801822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26523.061167] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26523.316135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26523.573288] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26523.829100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26524.089848] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26524.346656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26524.654596] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26525.558105] Lustre: DEBUG MARKER: rc=0; [26525.558105] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26525.558105] if [[ $? -eq 0 && $val -ne 0 ]]; then [26525.558105] echo $(hostname -s): $val; [26525.558105] rc=$val; [26525.558105] fi; [26525.558105] exit $rc [26526.152715] Lustre: DEBUG MARKER: dmesg [26526.751107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 14: test default nodemap nid lookup =============================================== 17:54:57 \(1491501297\) [26526.908554] Lustre: DEBUG MARKER: == sanity-sec test 14: test default nodemap nid lookup =============================================== 17:54:57 (1491501297) [26527.060427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26527.060427] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26527.060427] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26527.336108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26527.336108] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26527.336108] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26527.606419] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26527.866770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26527.866770] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26527.866770] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26528.145391] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26528.415970] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26528.675514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26528.936955] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26529.193916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26529.451073] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26529.703755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26529.960017] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26530.227259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26530.492396] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26530.753244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26531.010177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26531.270631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26531.529125] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26531.787452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26532.048406] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26532.304932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26532.565021] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26532.823092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26533.083282] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26533.339660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26533.596377] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26533.853136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26534.113644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26534.370188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26534.627786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26534.886209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26535.143620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26535.401752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26535.658812] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26535.915147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26536.172883] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26536.429824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26536.689627] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.1 [26536.945339] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.10 [26537.205542] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.64 [26537.463102] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.128 [26537.717217] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.200 [26537.973125] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.250 [26538.231689] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.1 [26538.486166] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.10 [26538.744538] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.64 [26539.004914] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.128 [26539.260589] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.200 [26539.515701] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.250 [26539.773094] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.1 [26540.029145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.10 [26540.287418] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.64 [26540.544483] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.128 [26540.799888] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.200 [26541.053672] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.250 [26541.312747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.1 [26541.565987] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.10 [26541.820181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.64 [26542.077364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.128 [26542.336686] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.200 [26542.594219] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.250 [26542.849444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.1 [26543.103205] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.10 [26543.359140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.64 [26543.615821] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.128 [26543.870052] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.200 [26544.125816] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.250 [26544.381680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.1 [26544.635874] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.10 [26544.889834] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.64 [26545.145124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.128 [26545.399273] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.200 [26545.654313] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.250 [26545.912304] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.1 [26546.165648] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.10 [26546.419064] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.64 [26546.671874] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.128 [26546.926224] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.200 [26547.182813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.250 [26547.437959] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.1 [26547.695191] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.10 [26547.949975] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.64 [26548.209446] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.128 [26548.467464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.200 [26548.723785] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.250 [26548.982774] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.1 [26549.240359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.10 [26549.495915] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.64 [26549.753958] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.128 [26550.009411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.200 [26550.269262] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.250 [26550.529580] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.1 [26550.786239] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.10 [26551.044813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.64 [26551.299872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.128 [26551.559761] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.200 [26551.816792] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.250 [26552.074416] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.1 [26552.331194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.10 [26552.590714] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.64 [26552.850374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.128 [26553.104950] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.200 [26553.362073] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.250 [26553.626917] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.1 [26553.882220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.10 [26554.139164] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.64 [26554.395560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.128 [26554.653267] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.200 [26554.910532] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.250 [26555.170588] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.1 [26555.427413] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.10 [26555.685742] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.64 [26555.949902] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.128 [26556.219848] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.200 [26556.477721] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.250 [26556.732222] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.1 [26556.986379] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.10 [26557.242192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.64 [26557.497950] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.128 [26557.752988] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.200 [26558.010740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.250 [26558.266483] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.1 [26558.527805] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.10 [26558.785518] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.64 [26559.043149] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.128 [26559.301335] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.200 [26559.558916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.250 [26559.814589] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.1 [26560.071310] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.10 [26560.326561] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.64 [26560.580334] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.128 [26560.838440] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.200 [26561.092924] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.250 [26561.351256] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.1 [26561.607185] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.10 [26561.863517] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.64 [26562.118821] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.128 [26562.376878] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.200 [26562.634897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.250 [26562.889707] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.1 [26563.146928] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.10 [26563.403877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.64 [26563.664757] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.128 [26563.920207] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.200 [26564.178540] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.250 [26564.433880] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.1 [26564.690177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.10 [26564.946535] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.64 [26565.206853] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.128 [26565.468878] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.200 [26565.726402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.250 [26565.984520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.1 [26566.240611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.10 [26566.498960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.64 [26566.755229] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.128 [26567.010786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.200 [26567.264805] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.250 [26567.522249] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.1 [26567.792872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.10 [26568.061605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.64 [26568.330075] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.128 [26568.599803] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.200 [26568.854154] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.250 [26569.107767] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.1 [26569.363327] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.10 [26569.618481] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.64 [26569.871109] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.128 [26570.127140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.200 [26570.383835] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.250 [26570.642038] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.1 [26570.899395] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.10 [26571.155894] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.64 [26571.411432] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.128 [26571.667301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.200 [26571.923740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.250 [26572.179334] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.1 [26572.432945] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.10 [26572.688793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.64 [26572.948501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.128 [26573.212798] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.200 [26573.468960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.250 [26573.724137] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.1 [26573.988231] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.10 [26574.245991] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.64 [26574.500250] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.128 [26574.754513] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.200 [26575.009438] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.250 [26575.265802] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.1 [26575.519822] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.10 [26575.776213] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.64 [26576.032411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.128 [26576.289791] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.200 [26576.545992] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.250 [26576.800985] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.1 [26577.058543] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.10 [26577.316220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.64 [26577.577359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.128 [26577.834668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.200 [26578.094919] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.250 [26578.355825] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.1 [26578.623658] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.10 [26578.895953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.64 [26579.168099] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.128 [26579.438894] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.200 [26579.709261] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.250 [26579.980712] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.1 [26580.252004] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.10 [26580.521270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.64 [26580.779141] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.128 [26581.034581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.200 [26581.291148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.250 [26581.548521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.1 [26581.805292] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.10 [26582.061376] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.64 [26582.317623] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.128 [26582.573130] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.200 [26582.828194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.250 [26583.082983] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.1 [26583.339725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.10 [26583.597929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.64 [26583.860447] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.128 [26584.117118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.200 [26584.374070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.250 [26584.632469] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.1 [26584.889839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.10 [26585.148604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.64 [26585.407794] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.128 [26585.665004] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.200 [26585.916892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.250 [26586.172858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.1 [26586.427866] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.10 [26586.685527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.64 [26586.941192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.128 [26587.196151] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.200 [26587.455143] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.250 [26587.711046] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.1 [26587.965758] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.10 [26588.227127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.64 [26588.482528] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.128 [26588.741684] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.200 [26588.996134] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.250 [26589.253694] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.1 [26589.508836] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.10 [26589.773541] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.64 [26590.036404] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.128 [26590.300719] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.200 [26590.560521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.250 [26590.820943] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.1 [26591.076948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.10 [26591.336666] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.64 [26591.590686] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.128 [26591.849546] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.200 [26592.105786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.250 [26592.361630] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.1 [26592.618248] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.10 [26592.874999] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.64 [26593.130628] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.128 [26593.388261] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.200 [26593.643505] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.250 [26593.899768] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.1 [26594.156737] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.10 [26594.412960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.64 [26594.669501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.128 [26594.926105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.200 [26595.186917] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.250 [26595.443005] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.1 [26595.703213] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.10 [26595.960111] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.64 [26596.217143] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.128 [26596.474221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.200 [26596.730930] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.250 [26596.988538] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.1 [26597.245108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.10 [26597.504455] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.64 [26597.763198] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.128 [26598.018322] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.200 [26598.272699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.250 [26598.528017] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.1 [26598.782397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.10 [26599.038160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.64 [26599.291540] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.128 [26599.544281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.200 [26599.799225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.250 [26600.065872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.1 [26600.327810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.10 [26600.582644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.64 [26600.837066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.128 [26601.093618] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.200 [26601.349398] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.250 [26601.603360] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.1 [26601.860121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.10 [26602.114307] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.64 [26602.369442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.128 [26602.626424] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.200 [26602.880399] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.250 [26603.138056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.1 [26603.393270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.10 [26603.648135] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.64 [26603.903264] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.128 [26604.159355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.200 [26604.411927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.250 [26604.664796] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.1 [26604.919810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.10 [26605.173651] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.64 [26605.431907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.128 [26605.685385] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.200 [26605.943151] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.250 [26606.202942] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.1 [26606.461452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.10 [26606.718674] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.64 [26606.973493] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.128 [26607.230588] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.200 [26607.485654] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.250 [26607.740460] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.1 [26607.994349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.10 [26608.250620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.64 [26608.505066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.128 [26608.760333] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.200 [26609.015163] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.250 [26609.272093] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.1 [26609.530511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.10 [26609.788290] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.64 [26610.047606] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.128 [26610.302399] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.200 [26610.559107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.250 [26610.816548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26611.071341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26611.329515] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26611.586154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26611.843356] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26612.098953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26612.358333] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26612.614870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26612.875094] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26613.132870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26613.393082] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26613.648354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26613.906462] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26614.161642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26614.421197] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26614.682105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26614.944080] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26615.198683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26615.455124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26615.713905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26615.971361] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26616.228509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26616.488613] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26616.747084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26617.005270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26617.261405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26617.519722] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26617.778665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26618.033527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26618.288286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26618.544462] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26618.801451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26619.111754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26619.995996] Lustre: DEBUG MARKER: rc=0; [26619.995996] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26619.995996] if [[ $? -eq 0 && $val -ne 0 ]]; then [26619.995996] echo $(hostname -s): $val; [26619.995996] rc=$val; [26619.995996] fi; [26619.995996] exit $rc [26620.583801] Lustre: DEBUG MARKER: dmesg [26621.209237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 15: test id mapping =============================================================== 17:56:32 \(1491501392\) [26621.363398] Lustre: DEBUG MARKER: == sanity-sec test 15: test id mapping =============================================================== 17:56:32 (1491501392) [26621.511496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26621.511496] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26621.511496] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26621.783996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26621.783996] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26621.783996] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26622.052712] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [26622.312630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26622.312630] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26622.312630] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26622.585272] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [26622.842770] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [26623.102527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26623.363200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [26623.620361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26623.879312] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [26624.137054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26624.395357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [26624.652158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26624.907467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [26625.163981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26625.424668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [26625.684017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26625.942749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [26626.199468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26626.460976] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [26626.721166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26626.979746] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [26627.233679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26627.490397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [26627.748761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26628.007571] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [26628.261069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26628.518629] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [26628.776892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26629.034205] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [26629.292876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26629.555403] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [26629.811124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26630.067661] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [26630.322378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26630.578105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [26630.835539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26631.094923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [26631.351870] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [26631.609723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [26631.866863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [26632.123223] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [26632.379070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [26632.634141] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [26632.896974] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [26633.151186] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [26633.410557] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [26633.665430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [26633.921511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [26634.178119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [26634.434466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [26634.695435] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [26634.953790] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [26635.211634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [26635.465157] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [26635.723171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [26635.978378] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [26636.235656] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [26636.491309] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [26636.749933] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [26637.005459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [26637.260636] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [26637.516402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [26637.771580] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [26638.026907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [26638.283372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [26638.540369] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [26638.799755] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [26639.056439] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [26639.312645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [26639.571343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [26639.826497] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [26640.084784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [26640.342351] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [26640.599179] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [26640.855190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [26641.111938] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [26641.368167] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [26641.625245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [26641.883200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [26642.139632] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [26642.394872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [26642.651033] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [26642.907387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [26643.160749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [26643.415559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26643.671030] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [26643.926558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 1 [26644.181667] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 1 [26644.438606] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 1 [26644.692986] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 1 [26644.947810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 1 [26645.203530] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 1 [26645.461926] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 1 [26645.718761] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 1 [26645.972279] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 1 [26646.227248] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 1 [26646.481892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 1 [26646.740088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 1 [26646.997444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 1 [26647.251120] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 1 [26647.509129] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 1 [26647.765953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 1 [26648.020891] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 1 [26648.279187] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 1 [26648.537627] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 1 [26648.794936] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 1 [26649.050017] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 1 [26649.307355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 1 [26649.566116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 1 [26649.821994] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 1 [26650.077434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 1 [26650.333976] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 1 [26650.592209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 1 [26650.847202] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 1 [26651.102172] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 1 [26651.360242] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 1 [26651.614695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 1 [26651.871140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 1 [26652.126239] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.0.100@tcp --idtype uid --id 0 [26652.384029] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.1.100@tcp --idtype uid --id 0 [26652.639779] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.2.100@tcp --idtype uid --id 0 [26652.895902] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 0 [26653.152622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 0 [26653.409328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 0 [26653.664515] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 0 [26653.921412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 0 [26654.177020] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 0 [26654.432739] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 0 [26654.687134] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 0 [26654.945913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 0 [26655.202286] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 0 [26655.469822] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 0 [26655.741254] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 0 [26656.007529] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 0 [26656.278524] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 0 [26656.535474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 0 [26656.791638] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 0 [26657.046045] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.0.100@tcp --idtype uid --id 0 [26657.300090] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.1.100@tcp --idtype uid --id 0 [26657.556628] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.2.100@tcp --idtype uid --id 0 [26657.815809] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 0 [26658.071208] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 0 [26658.326572] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 0 [26658.581056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 0 [26658.834216] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 0 [26659.089342] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 0 [26659.344858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 0 [26659.601754] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 0 [26659.857173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 0 [26660.113161] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 0 [26660.365627] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 0 [26660.617870] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 0 [26660.873403] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 0 [26661.127267] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 0 [26661.384454] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 0 [26661.642747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 0 [26661.894760] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [26662.147918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [26662.403616] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [26662.663170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [26662.920328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [26663.175759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [26663.432344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [26663.686893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [26663.945897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [26664.209122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [26664.464642] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [26664.723100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [26664.984066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [26665.240190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [26665.498510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [26665.754499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [26666.012435] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [26666.270837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [26666.526371] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [26666.789225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [26667.045858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [26667.312082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [26667.573096] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [26667.830200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [26668.089114] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [26668.344617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [26668.602735] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [26668.860049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [26669.119434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [26669.377026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [26669.634150] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [26669.891750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [26670.204088] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26671.111818] Lustre: DEBUG MARKER: rc=0; [26671.111818] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26671.111818] if [[ $? -eq 0 && $val -ne 0 ]]; then [26671.111818] echo $(hostname -s): $val; [26671.111818] rc=$val; [26671.111818] fi; [26671.111818] exit $rc [26671.726978] Lustre: DEBUG MARKER: dmesg [26672.347543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 16: test nodemap all_off fileops ================================================== 17:57:23 \(1491501443\) [26672.505150] Lustre: DEBUG MARKER: == sanity-sec test 16: test nodemap all_off fileops ================================================== 17:57:23 (1491501443) [26672.655054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26672.655054] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26672.655054] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26672.936250] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [26673.091170] Lustre: 32427:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [26673.242046] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [26673.507026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [26673.764453] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [26674.026191] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [26674.283913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [26674.542453] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [26674.802123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26675.061186] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26675.351063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.idmap [26677.693742] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26680.048721] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26682.399958] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26683.490309] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [26683.762541] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [26684.021093] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [26684.278765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [26684.536877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [26684.792173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [26685.048916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26685.311080] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26685.586501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.idmap [26694.089427] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26694.930251] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26695.187047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26695.445658] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26695.721403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26702.675897] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26705.043338] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26705.867653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [26706.127609] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [26706.387895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26706.644166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26706.902018] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26707.159854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26707.417406] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26707.674287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26707.930765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26708.243812] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26708.499389] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26708.777250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26709.036833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26709.296514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26709.551878] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26709.809511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26711.670317] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26712.618722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26712.882967] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26713.178924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26714.469293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26715.766191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26717.053369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26718.342385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26719.624391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26720.904342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26722.812287] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26723.810787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26724.078218] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26724.370629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26725.654740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26726.936406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26728.216563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26729.496421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26730.782429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26732.059560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26733.348395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26734.627728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26736.387492] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26737.398902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26737.679621] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26737.966023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26739.250796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26740.527026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26741.808114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26743.086876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26744.367456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26745.654766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26746.940191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26748.219801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26748.536367] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26748.800187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26749.058347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26749.314243] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26749.569952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26752.489302] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26753.906421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26754.177958] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26754.521634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26755.807684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26757.092133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26758.370328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26759.658915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26760.937630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26762.219329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26763.500529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26764.785981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26767.525336] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26768.962491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26769.228624] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26769.513349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26770.790110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26772.083848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26773.363760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26774.640327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26775.924297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26777.208991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26778.494686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26781.172398] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26782.592415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26782.859622] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26783.149503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26784.430418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26785.718146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26787.185396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26788.708244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26789.999262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26791.288879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26792.575188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26793.130520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26793.389083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26793.647164] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26793.905418] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26794.164048] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26794.423215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26794.763387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26795.022715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26795.303729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26795.571967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26795.833182] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26796.090727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26796.348897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26796.611435] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26796.868109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26798.632327] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26799.547921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26808.316382] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26809.280162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26818.042866] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26818.971608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26826.234050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26826.505219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26826.763324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26827.025844] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26827.286386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26830.265424] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26831.764578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26843.959613] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26845.472125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26858.988476] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26860.462172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26871.824470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26872.086771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26872.346834] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26872.607923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26872.867383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26873.136195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26873.479867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26873.737149] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26874.004442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26874.261861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26874.521677] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26874.778843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26875.036370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26875.295268] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26875.554353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26877.316155] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26878.276644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26878.548257] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26878.838665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26880.124694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26881.408783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26882.705388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26883.994117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26885.274908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26886.559785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26887.852606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26889.653389] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26890.629598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26890.907563] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26891.205982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26892.493088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26893.773641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26895.052517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26896.332374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26897.620092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26898.906375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26900.676353] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26901.612635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26901.883808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26902.222146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26903.514176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26904.800740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26906.092254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26907.384512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26908.669470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26909.947326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26911.231214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26912.524572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26913.812435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26914.118711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26914.381139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26914.638458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26914.898023] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26915.152841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26918.046367] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26919.464401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26919.733698] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26920.026141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26921.309017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26922.593310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26923.876807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26925.155892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26926.432973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26927.714340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26928.997772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26931.687784] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26933.188600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26933.459350] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26933.749132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26935.031113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26936.316295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26937.602823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26938.887071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26940.171394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26941.461642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26942.745733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26944.026358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26946.742021] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26948.153881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26948.423207] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26948.713920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26950.003273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26951.280951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26952.568414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26953.858913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26955.140984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26956.438196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26957.722735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26959.012207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26959.580191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26959.836377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26960.096374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26960.357850] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26960.617186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26960.877270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26961.223324] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26961.481270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26961.750754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26962.008707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26962.267362] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26962.525289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26962.784086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26963.044782] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26963.300866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26965.042708] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26965.990921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26978.557621] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26979.542655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26988.271601] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26989.307130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26996.563342] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26996.824752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26997.081200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26997.340955] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26997.597953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27000.509855] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27001.938642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27014.108874] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27015.541920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27029.033208] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27030.490481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27036.746424] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27037.009724] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27037.270257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27037.529164] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27037.793418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27038.057867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27038.320551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27038.581402] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27038.865544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27039.661508] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27040.556883] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27041.436195] Lustre: DEBUG MARKER: rc=0; [27041.436195] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27041.436195] if [[ $? -eq 0 && $val -ne 0 ]]; then [27041.436195] echo $(hostname -s): $val; [27041.436195] rc=$val; [27041.436195] fi; [27041.436195] exit $rc [27042.026247] Lustre: DEBUG MARKER: dmesg [27042.668895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 17: test nodemap trusted_noadmin fileops ========================================== 18:03:33 \(1491501813\) [27042.829082] Lustre: DEBUG MARKER: == sanity-sec test 17: test nodemap trusted_noadmin fileops ========================================== 18:03:33 (1491501813) [27042.987694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [27042.987694] /usr/sbin/lctl lustre_build_version 2>/dev/null || [27042.987694] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [27043.272960] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [27043.417801] Lustre: 21047:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [27043.421503] Lustre: 21047:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [27043.563804] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [27043.821350] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [27044.081140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [27044.341631] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [27044.601682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [27044.860289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [27045.121960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27045.655776] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [27045.914364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [27046.171120] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [27046.431535] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [27046.693488] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [27046.950736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [27047.210200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27047.467345] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [27047.728952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27047.992220] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27048.267464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27055.244135] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27056.053110] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [27056.314070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [27056.572442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27056.831711] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27057.119042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27062.562936] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27063.378279] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27063.636453] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27063.899384] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27064.157033] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27064.417158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27064.677476] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27064.950602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27065.761837] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27066.577928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27066.839573] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27067.115047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [27067.927265] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27068.760376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27069.025319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27069.287126] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27069.547784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27069.805282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27070.064216] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27070.342078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27072.695386] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27073.533304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27073.793902] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27074.067999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27074.873071] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27075.751977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27076.011181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27076.278299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27076.536073] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27076.812629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27082.249810] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27084.611859] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27085.430624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27085.705549] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27085.995510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27086.814095] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27087.631251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [27087.891181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [27088.150768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27088.412407] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27088.687306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27098.816687] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27101.120198] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27102.097549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27102.363366] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27102.646707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27103.927119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27105.213773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27106.504285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27107.800594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27109.086343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27110.846645] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27111.829339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27112.097555] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27112.391564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27113.673026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27114.955677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27116.243382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27117.541562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27118.824020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27120.584209] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27121.514244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27121.788318] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27122.076437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27123.364670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27124.648671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27125.931575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27127.213789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27128.497677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27128.809797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27129.067114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27129.324374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27129.583295] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27129.839307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27130.097958] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27130.371845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27135.793536] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27138.195052] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27139.012489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27139.282166] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27139.567339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27140.382499] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27141.273765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27141.533308] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27141.805650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27142.068069] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27142.346074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27150.851694] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27151.667595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27151.927213] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27152.201108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27152.999046] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27155.276530] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27156.243373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27168.812985] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27169.829293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27178.560309] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27179.510694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27188.065623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27188.323994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27188.581729] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27188.839209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27189.097165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27189.356231] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27189.630891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27199.713342] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27200.532719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27200.798284] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27201.070324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27201.868364] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27202.797328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27203.058386] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27203.330202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27203.603375] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27203.889276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27210.873241] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27211.685530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27211.945349] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27212.217826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27213.027740] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27215.337513] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27216.288185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27216.571480] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27216.861769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27218.141708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27219.430521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27220.712184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27221.992931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27223.273697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27225.069430] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27226.043566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27226.311657] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27226.599293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27227.884882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27229.169878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27230.453385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27231.732694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27233.013450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27234.296744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27235.579288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27237.343201] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27238.276525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27238.540364] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27238.830265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27240.107655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27241.388347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27242.670232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27243.949719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27245.232458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27245.540353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27245.800372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27246.059548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27246.319156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27246.579315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27246.838232] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27247.122737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27255.641172] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27256.453491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27256.717602] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27256.992687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27257.800886] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27258.680644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27258.940202] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27259.207623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27259.466757] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27259.744365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27266.741286] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27267.562546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27267.821485] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27268.096441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27268.901323] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27271.237508] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27272.213578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27283.504222] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27284.437673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27293.165216] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27294.255327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27302.768412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27303.029108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27303.286753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27303.546460] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27303.806443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27304.067972] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27304.362362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27308.251657] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27309.072488] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27309.334542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27309.594806] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27309.872079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27316.857039] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27319.226631] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27320.125917] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27321.055708] Lustre: DEBUG MARKER: rc=0; [27321.055708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27321.055708] if [[ $? -eq 0 && $val -ne 0 ]]; then [27321.055708] echo $(hostname -s): $val; [27321.055708] rc=$val; [27321.055708] fi; [27321.055708] exit $rc [27321.652808] Lustre: DEBUG MARKER: dmesg [27322.386103] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 18: test nodemap mapped_noadmin fileops =========================================== 18:08:13 \(1491502093\) [27322.543899] Lustre: DEBUG MARKER: == sanity-sec test 18: test nodemap mapped_noadmin fileops =========================================== 18:08:13 (1491502093) [27322.695973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [27322.695973] /usr/sbin/lctl lustre_build_version 2>/dev/null || [27322.695973] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [27322.983355] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [27323.131786] Lustre: 5061:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [27323.135406] Lustre: 5061:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [27323.277615] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [27323.538032] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [27323.796173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [27324.054207] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [27324.310652] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [27324.573502] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [27324.830039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27325.353546] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [27325.611553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [27325.869941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [27326.129368] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [27326.388176] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [27326.646640] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [27326.905929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27327.163484] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [27327.420822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27327.683697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27327.959802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27330.293077] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27331.104093] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [27331.364065] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [27331.624706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27331.885314] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27332.159170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27342.277087] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27343.092172] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27343.348867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27343.608397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27343.865227] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27344.120052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27344.381758] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27344.656125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27345.474130] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27346.298324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27346.556377] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27346.830332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [27347.640249] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27348.453710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27348.711437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27348.972702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27349.228627] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27349.488856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27349.752703] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27350.029786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27355.462637] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27356.283020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27356.543026] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27356.818750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27357.637112] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27358.502307] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27358.764234] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27359.031078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27359.287950] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27359.562604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27369.625409] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27370.440316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27370.703913] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27370.979779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27371.782198] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27372.603427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [27372.863354] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [27373.123031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27373.383777] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27373.657454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27380.624784] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27382.991649] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27385.324766] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27386.287915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27386.563543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27386.856289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27388.142714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27389.427282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27390.724616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27392.011433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27393.295339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27394.583572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27396.426776] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27397.365376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27397.639403] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27397.928771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27399.217105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27400.500137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27401.785004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27403.072630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27404.354425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27405.639106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27406.925229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27408.730601] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27409.708091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27409.975584] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27410.266506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27411.546671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27412.823485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27414.106390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27415.402917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27416.678668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27417.963426] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27419.243506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27420.530034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27420.838137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27421.095171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27421.352142] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27421.610759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27421.868393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27422.128750] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27422.403255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27432.494729] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27433.307085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27433.565845] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27433.838475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27434.642819] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27435.547884] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27435.810954] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27436.080591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27436.342864] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27436.628419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27443.588979] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27444.424566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27444.685727] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27444.968492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27445.782515] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27448.042374] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27449.034760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27459.110135] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27460.072341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27470.171992] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27471.114035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27478.367618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27478.628137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27478.889087] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27479.147856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27479.406365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27479.665190] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27479.950494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27488.491428] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27489.316645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27489.576951] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27489.861841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27490.663447] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27491.558044] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27491.813628] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27492.100513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27492.376244] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27492.649603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27499.610727] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27500.435515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27500.695192] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27500.971572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27501.793004] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27504.072397] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27505.061979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27505.332484] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27505.622407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27506.901919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27508.181174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27509.461525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27510.738451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27512.024890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27513.302469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27514.582198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27516.419122] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27517.370984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27517.649169] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27517.938286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27519.229086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27520.510360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27521.793760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27523.074392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27524.357516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27526.199041] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27527.127659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27527.395066] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27527.687176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27528.972912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27530.264101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27531.555547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27532.844870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27534.138686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27534.448210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27534.707548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27534.966972] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27535.228227] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27535.491387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27535.755124] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27536.032104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27543.036462] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27543.857194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27544.119224] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27544.393699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27545.196820] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27546.104723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27546.374122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27546.641730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27546.905449] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27547.183584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27554.171878] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27554.995213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27555.254445] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27555.528040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27556.342665] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27558.610323] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27559.543960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27569.588693] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27570.573976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27579.326542] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27580.363159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27587.641534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27587.910964] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27588.168830] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27588.426444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27588.683400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27588.941958] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27589.220928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27596.212082] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27598.571998] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27599.420915] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27599.682743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27599.946322] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27600.225251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27605.680107] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27608.041393] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27608.923643] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27609.801405] Lustre: DEBUG MARKER: rc=0; [27609.801405] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27609.801405] if [[ $? -eq 0 && $val -ne 0 ]]; then [27609.801405] echo $(hostname -s): $val; [27609.801405] rc=$val; [27609.801405] fi; [27609.801405] exit $rc [27610.416931] Lustre: DEBUG MARKER: dmesg [27611.005702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 19: test nodemap trusted_admin fileops ============================================ 18:13:01 \(1491502381\) [27611.193110] Lustre: DEBUG MARKER: == sanity-sec test 19: test nodemap trusted_admin fileops ============================================ 18:13:01 (1491502381) [27611.355702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [27611.355702] /usr/sbin/lctl lustre_build_version 2>/dev/null || [27611.355702] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [27611.637604] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [27611.784157] Lustre: 21821:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [27611.787800] Lustre: 21821:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [27611.934037] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [27612.195235] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [27612.454206] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [27612.712582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [27612.972356] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [27613.233159] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [27613.500940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27614.030906] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [27614.290138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [27614.546969] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [27614.806136] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [27615.064763] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [27615.321468] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [27615.579618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27615.841872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [27616.099368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27616.357169] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27616.630609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27622.055868] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27622.875339] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [27623.134705] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [27623.393323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27623.652171] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27623.924896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27630.962163] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27631.795319] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27632.053283] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27632.310119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27632.568238] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27632.826709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27633.087695] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27633.364535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27634.165056] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27634.973327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27635.233400] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27635.507115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [27636.314527] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27637.122907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27637.382976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27637.640364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27637.897939] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27638.155592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27638.416490] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27638.693519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27641.036421] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27641.862205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27642.123470] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27642.400495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27643.197420] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27644.057355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27644.314631] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27644.582872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27644.838894] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27645.112560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27645.927021] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27646.742820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27647.003444] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27647.276782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27648.074425] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27648.904755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [27649.164234] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [27649.429126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27649.698507] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27649.973209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27655.426058] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27657.729157] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27658.693495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27658.960469] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27659.247412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27660.534240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27661.816077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27663.099297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27664.405240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27665.692335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27666.986652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27668.776934] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27669.753193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27670.016939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27670.306068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27671.591896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27672.874364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27674.153914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27675.439165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27676.721211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27678.003766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27679.288236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27680.568172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27681.849933] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27683.637742] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27684.634432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27684.905363] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27685.251957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27686.530191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27687.810777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27689.092660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27690.419513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27691.697422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27692.981892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27694.273940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27694.582610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27694.842138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27695.101845] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27695.359865] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27695.617660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27695.881203] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27696.157281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27696.965515] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27697.779180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27698.041452] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27698.318147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27699.124437] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27700.020101] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27700.294568] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27700.578525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27700.836485] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27701.112687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27701.941690] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27702.756862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27703.019150] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27703.293837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27704.125294] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27706.451775] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27707.413089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27719.998209] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27721.005215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27729.709651] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27730.706748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27737.993539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27738.261816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27738.523118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27738.780710] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27739.039200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27739.298395] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27739.573457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27740.376602] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27741.192437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27741.454227] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27741.729907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27742.537666] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27743.428076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27743.685121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27743.953866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27744.215747] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27744.496629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27745.311208] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27746.134777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27746.401952] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27746.685458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27747.487938] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27749.774777] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27750.732425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27751.001265] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27751.292799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27752.575137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27753.858833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27755.145902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27756.426744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27757.705529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27758.983244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27760.268484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27762.048301] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27763.037052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27763.307352] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27763.599056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27764.884801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27766.170162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27767.459726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27768.744874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27770.033934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27771.822572] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27772.839234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27773.105965] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27773.407303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27774.688359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27775.970775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27777.248960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27778.535219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27779.811149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27780.124879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27780.381618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27780.640508] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27780.898549] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27781.157159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27781.417056] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27781.692771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27782.507950] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27783.327037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27783.594326] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27783.869966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27784.677423] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27785.563061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27785.838314] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27786.117920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27786.391723] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27786.682254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27787.501568] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27788.327097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27788.585427] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27788.860755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27789.674725] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27791.984791] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27792.970698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27805.537359] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27806.524290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27815.256346] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27816.231405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27827.340960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27827.615276] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27827.871160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27828.138407] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27828.412813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27828.686901] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27828.964599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27835.975977] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27836.796533] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27837.054742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27837.316334] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27837.591957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27847.675382] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27848.566756] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27849.440171] Lustre: DEBUG MARKER: rc=0; [27849.440171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27849.440171] if [[ $? -eq 0 && $val -ne 0 ]]; then [27849.440171] echo $(hostname -s): $val; [27849.440171] rc=$val; [27849.440171] fi; [27849.440171] exit $rc [27850.034269] Lustre: DEBUG MARKER: dmesg [27850.656235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 20: test nodemap mapped_admin fileops ============================================= 18:17:01 \(1491502621\) [27850.813639] Lustre: DEBUG MARKER: == sanity-sec test 20: test nodemap mapped_admin fileops ============================================= 18:17:01 (1491502621) [27850.960724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [27850.960724] /usr/sbin/lctl lustre_build_version 2>/dev/null || [27850.960724] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [27851.248088] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [27851.393691] Lustre: 6130:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [27851.397345] Lustre: 6130:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [27851.538634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [27851.796212] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [27852.053140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [27852.310759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [27852.569985] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [27852.828693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [27853.086071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27853.610735] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [27853.866881] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [27854.124228] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [27854.381770] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [27854.638879] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [27854.896221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [27855.152798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27855.409993] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [27855.671463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27855.932548] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27856.213541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27860.122169] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27860.938715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [27861.196617] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [27861.455513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27861.714961] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27861.992459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27870.542221] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27871.361127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27871.616958] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27871.874644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27872.131087] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27872.389190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27872.648296] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27872.923422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27873.736864] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27874.562559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27874.837395] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27875.111369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [27875.934170] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27876.750526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27877.010642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27877.266982] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27877.523828] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27877.783188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27878.040329] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27878.327920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27879.132778] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27879.945864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27880.204820] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27880.477509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27881.276418] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27882.132284] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27882.393362] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27882.663679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27882.925506] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27883.199077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27884.006164] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27884.818418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27885.078217] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27885.355825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27889.264876] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27891.622365] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27892.451959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [27892.707122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [27892.964833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27893.223884] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27893.505777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [27902.058639] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27904.366198] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27905.361351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27905.633988] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27905.919552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27907.207128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27908.495596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27909.789520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27911.076211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27912.845735] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27913.804964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27914.074466] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27914.370337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27915.656400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27916.942870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27918.221578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27919.501349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27920.783386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27922.064423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27923.347261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27924.638087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27926.458274] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27927.414168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27927.692504] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [27927.991093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27929.278325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27930.568349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27932.020128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27933.550312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27934.843897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27935.157949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27935.419769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27935.680287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27935.939152] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27936.193961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27936.452117] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27936.729690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27937.532766] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27938.360702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27938.623153] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27938.899566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27945.879932] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27946.770619] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27947.045163] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27947.325402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27947.585253] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27947.861785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27948.659901] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27949.494540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27949.755229] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27950.031215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27957.041263] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27959.359950] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27960.295519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27970.375444] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27971.331795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27980.097953] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [27981.069611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [27992.194468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27992.453578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27992.710861] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27992.967685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27993.228109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27993.483088] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27993.762731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27994.559296] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27995.380721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27995.638132] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27995.942502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28001.441770] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28002.305978] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28002.567578] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [28002.836268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28003.099074] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28003.378661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28004.190883] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28005.027883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28005.292154] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28005.567562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28012.575054] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28014.891079] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28015.889607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28016.160658] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28016.446136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28017.728440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28019.016128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28020.307395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28021.591596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28022.885940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28024.164323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28025.442366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28027.216690] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28028.168354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28028.437423] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28028.725741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28030.010316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28031.299140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28032.585293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28033.870930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28035.155572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28036.972630] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28037.986692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28038.271708] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28038.575143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28039.859596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28041.141603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28042.419411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28043.704931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28044.989255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28045.321279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28045.588523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28045.862920] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28046.136887] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28046.408070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28046.672280] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28046.947689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28047.760447] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28048.602018] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28048.861050] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28049.136746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28056.111841] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28056.998907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28057.259277] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [28057.526726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28057.784821] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28058.059593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28058.862234] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28059.671304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28059.935102] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28060.207053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28067.198037] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28069.554439] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28070.476133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28080.485773] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28081.443982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28088.908766] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28089.933314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28098.504611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [28098.765466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [28099.037647] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [28099.311158] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [28099.582739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28099.850775] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28100.126521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [28110.199717] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28111.016412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [28111.277533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28111.537119] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28111.811674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [28120.346866] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28121.242873] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [28122.130061] Lustre: DEBUG MARKER: rc=0; [28122.130061] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [28122.130061] if [[ $? -eq 0 && $val -ne 0 ]]; then [28122.130061] echo $(hostname -s): $val; [28122.130061] rc=$val; [28122.130061] fi; [28122.130061] exit $rc [28122.732934] Lustre: DEBUG MARKER: dmesg [28123.366702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 21: test nodemap mapped_trusted_noadmin fileops =================================== 18:21:34 \(1491502894\) [28123.519315] Lustre: DEBUG MARKER: == sanity-sec test 21: test nodemap mapped_trusted_noadmin fileops =================================== 18:21:34 (1491502894) [28123.675188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [28123.675188] /usr/sbin/lctl lustre_build_version 2>/dev/null || [28123.675188] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [28123.957634] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [28124.106803] Lustre: 22428:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [28124.110469] Lustre: 22428:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [28124.255508] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [28124.513961] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [28124.769793] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [28125.029204] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [28125.289199] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [28125.550620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [28125.810328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28126.339408] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [28126.596913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [28126.855096] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [28127.113747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [28127.370665] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [28127.629907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [28127.887754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28128.148962] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [28128.407818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28128.669914] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28128.943790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [28131.318726] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28132.142059] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [28132.413880] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [28132.673232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28132.932272] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28133.215364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [28138.686088] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28139.492763] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [28139.752217] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28140.010363] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [28140.270653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [28140.531087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28140.791161] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28141.068610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [28141.890335] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28142.709761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28142.973971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28143.229889] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28143.487155] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28143.747593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28144.007348] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28144.279898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28149.727224] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28150.555602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28150.816314] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28151.093332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28151.893248] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28152.764524] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [28153.026038] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28153.294957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28153.554682] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28153.829106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28160.806361] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28161.616029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28161.875089] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28162.152189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28162.958195] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28163.776449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28164.033736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [28164.294174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28164.555653] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28164.830048] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28174.960324] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28177.221188] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28178.174568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28178.439554] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28178.726998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28180.008301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28181.290915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28182.573690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28183.850946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28185.133619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28186.875515] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28187.895295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28188.171678] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28188.462221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28189.745174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28191.025800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28192.312315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28193.589141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28194.880442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28196.644960] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28197.617109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28197.883329] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28198.173074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28199.462750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28200.749583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28202.036774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28203.334776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28204.611449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28205.893846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28206.203614] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [28206.463626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28206.725915] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28207.016646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28213.994649] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28214.822248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28215.082364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28215.344188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28215.604540] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28215.880025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28224.394617] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28227.333827] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28228.773792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28229.038520] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28229.323795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28230.602884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28231.888514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28233.167657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28234.448796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28235.732377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28238.542799] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28240.219795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28240.488656] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28240.778855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28242.063456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28243.347265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28244.639598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28245.935450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28247.217604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28249.950340] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28251.356573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28251.624175] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28251.916777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28253.210928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28254.494313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28255.785812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28257.062171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28258.342681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28259.633442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28260.917450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28262.199292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28262.757530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28263.027635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28263.283830] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28263.541157] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28263.804374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28264.074849] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28264.348386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28265.158430] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28265.985540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28266.244277] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28266.524660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28267.335812] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28268.302411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28268.563611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28268.829680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28269.088638] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28269.362327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28270.163934] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28270.978718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28271.238108] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28271.510744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28272.316471] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28273.136178] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [28273.398989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28273.656934] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28273.944257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28279.389633] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28281.739942] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28282.564178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28282.832351] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [28283.096181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28283.356254] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28283.641641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28293.711423] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28296.076309] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28297.043093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28305.749182] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28306.755778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28315.468969] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28316.469004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28325.000564] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [28325.261289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28325.522961] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28325.800896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28331.255689] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28332.068381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28332.327667] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28332.588091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28332.849363] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28333.125165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28343.201446] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28346.154840] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28347.625349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28362.415980] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28363.873271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28377.400094] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28378.819994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28390.149623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28390.409452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28390.668758] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28390.927764] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28391.184127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28391.443706] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28391.719662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28392.523355] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28393.337183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28393.598882] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28393.875461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28394.679762] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28395.593702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28395.848371] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28396.118963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28396.376521] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28396.650297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28397.452334] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28398.261259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28398.522318] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28398.797848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28399.599649] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28400.425201] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [28400.687157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28400.949557] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28401.224697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28406.664795] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28409.026943] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28411.396329] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28412.211752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28412.468961] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [28412.725681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28412.984200] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28413.258640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28418.718209] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28421.044369] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28422.035627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28422.312651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28422.665436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28423.953197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28425.238436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28426.520600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28427.808479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28429.087590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28430.375299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28432.239364] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28433.256612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28433.529655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28433.806190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28435.090787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28436.379973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28437.668800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28438.955657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28440.242039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28441.529796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28442.809164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28444.590925] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28445.598116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28445.863208] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28446.202794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28447.490606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28448.775005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28450.056926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28451.337607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28452.623488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28453.912605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28455.194623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28456.477929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28457.759699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28458.070286] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [28458.338045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28458.601408] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28458.878914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28467.401428] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28468.225158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28468.483759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28468.746587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28469.010583] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28469.283874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28476.242417] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28479.311585] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28480.783903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28481.059797] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28481.335772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28482.619803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28483.905604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28485.187449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28486.466072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28487.758943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28489.043086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28490.325901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28493.047514] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28494.614780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28494.880877] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28495.172093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28496.459113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28497.741655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28499.023167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28500.304348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28501.595783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28502.887578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28504.172246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28505.459317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28508.190418] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28509.614495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28509.879526] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28510.167959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28511.456288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28512.743376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28514.032101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28515.316402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28516.595049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28517.875774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28519.158086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28520.449988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28521.001229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28521.260251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28521.519199] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28521.777918] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28522.038974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28522.297162] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28522.576065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28523.373502] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28524.226132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28524.502103] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28524.789866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28525.629707] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28526.499281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28526.757556] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28527.023820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28527.286644] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28527.561748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28528.371538] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28529.204834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28529.469694] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28529.744269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28530.549157] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28531.379971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [28531.638210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28531.896529] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28532.172947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28539.182953] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28539.993942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28540.251753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [28540.512870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28540.774901] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28541.050957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28546.496386] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28548.778909] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28549.725482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28559.775331] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28560.737112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28570.809433] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28571.735306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28580.301614] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [28580.560627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28580.823428] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28581.101291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [28589.610599] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28590.417405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28590.676110] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28590.934796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28591.194092] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28591.471430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28599.977296] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28602.973521] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28604.387165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28617.906397] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28619.398639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28633.034016] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28634.497116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28644.586138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [28644.845339] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [28645.105043] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [28645.362946] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [28645.626980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28645.892190] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28646.168891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [28653.200598] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28654.020647] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [28654.281024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28654.542197] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28654.828160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [28660.287279] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28662.691493] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28663.569974] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [28664.469044] Lustre: DEBUG MARKER: rc=0; [28664.469044] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [28664.469044] if [[ $? -eq 0 && $val -ne 0 ]]; then [28664.469044] echo $(hostname -s): $val; [28664.469044] rc=$val; [28664.469044] fi; [28664.469044] exit $rc [28665.063915] Lustre: DEBUG MARKER: dmesg [28665.699860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 22: test nodemap mapped_trusted_admin fileops ===================================== 18:30:36 \(1491503436\) [28665.856662] Lustre: DEBUG MARKER: == sanity-sec test 22: test nodemap mapped_trusted_admin fileops ===================================== 18:30:36 (1491503436) [28666.012121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [28666.012121] /usr/sbin/lctl lustre_build_version 2>/dev/null || [28666.012121] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [28666.298337] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [28666.444676] Lustre: 18018:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [28666.448317] Lustre: 18018:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [28666.590704] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [28666.848952] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [28667.103604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [28667.360938] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [28667.617736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [28667.875969] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [28668.134767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28668.669358] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [28668.927547] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [28669.186128] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [28669.443924] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [28669.703131] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [28669.962207] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [28670.219226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28670.476746] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [28670.731481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28670.990780] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28671.265764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [28673.599593] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28674.414192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [28674.676007] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [28674.950786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28675.210382] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28675.488435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [28680.986807] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28681.801611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28682.069510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28682.342450] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [28682.602547] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [28682.863801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28683.121278] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28683.398760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [28684.200291] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28685.036610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28685.292491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28685.551058] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28685.810931] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28686.069292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28686.329842] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28686.607110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28690.488619] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28691.299462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28691.559499] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28691.834128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28692.639125] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28693.507471] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28693.766202] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28694.032438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28694.294881] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28694.574269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28695.377988] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28696.210444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28696.469881] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28696.749600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28697.574190] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28698.386376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28700.170292] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28701.153310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28701.429062] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28701.717853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28702.999864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28704.280136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28705.557706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28706.851353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28708.140171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28709.954675] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28710.931363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28711.196759] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28711.482716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28712.761864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28714.050315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28715.332291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28716.614617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28717.904276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28719.680696] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28720.646081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28720.919724] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28721.197569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28722.486569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28723.773651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28725.055655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28726.350128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28727.634394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28728.920302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28730.201780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28731.504477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28731.817638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28734.686677] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28736.107849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28736.379406] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28736.674935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28737.955436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28739.236151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28740.526635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28741.812810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28743.107747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28745.836557] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28747.445924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28747.711324] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28747.997238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28749.281849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28750.570801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28751.855666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28753.150269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28754.435409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28755.726118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28757.003777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28759.701151] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28761.159415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28761.422941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28761.710711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28762.993110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28764.269832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28765.549546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28766.831772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28768.115039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28769.394983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28770.680191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28771.960827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28772.505300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28772.762216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28773.019391] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28773.278180] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28773.535177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28773.798155] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28774.073509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28774.873827] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28775.687443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28775.948115] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28776.222232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28777.041281] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28777.918475] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28778.174498] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28778.446308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28778.720446] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28778.995991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28779.807432] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28780.622936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28780.892184] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28781.183564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28781.998849] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28782.827155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28784.603567] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28785.587227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28796.892325] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28797.828246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28806.614748] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28807.654906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28820.044256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28823.016309] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28824.486194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28838.023085] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28839.485840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28851.740360] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28853.206567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28864.563984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28864.822800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28865.081119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28865.342916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28865.599257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28865.855743] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28866.131381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28866.935943] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28867.758347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28868.018956] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28868.295141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28869.101311] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28869.943478] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28870.201652] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28870.469786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28870.730152] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28871.004342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28871.822452] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28872.634560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28872.895291] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28873.173593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28873.983554] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28874.795531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28876.566651] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28877.493526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28877.761201] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28878.051115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28879.331709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28880.612565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28881.900525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28883.199124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28884.491784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28885.772265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28887.052097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28888.840956] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28889.794766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28890.060973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28890.406654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28891.691113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28892.972956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28894.258284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28895.564233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28896.853491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28898.136526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28899.417367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28900.715326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28901.994783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28903.748487] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28904.728284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28904.997179] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28905.289459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28906.568109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28907.849813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28909.133605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28910.444758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28911.728209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28912.039136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28914.954350] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28916.409894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28916.674135] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28916.957702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28918.248218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28919.537761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28920.825232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28922.103619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28923.390901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28924.678927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28925.970442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28928.713268] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28930.155047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28930.421705] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28930.713893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28931.993043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28933.276989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28934.558751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28935.836901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28937.124121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28939.861076] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28941.306797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28941.570585] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [28941.861122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28943.143261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28944.423637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28945.702857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28946.989774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28948.285386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28949.569425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28950.857503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28952.138181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28952.701453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [28952.959997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [28953.223885] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28953.492186] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28953.749915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28954.009313] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28954.285733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28955.100914] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28955.912845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28956.172879] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28956.447742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28957.268812] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28958.206715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [28958.481562] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [28958.762934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28959.036785] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28959.326706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [28960.172384] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28960.994126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [28961.251217] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28961.529614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [28962.325148] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [28963.175207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [28964.930828] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28965.915346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28977.210856] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28978.149646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28986.890268] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [28987.872924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [28998.980526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29001.926136] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [29003.356791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [29013.021986] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [29014.518768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [29026.763910] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [29028.241092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [29035.758453] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [29036.018238] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [29036.275501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [29036.533562] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [29036.793634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29037.054164] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29037.329657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29045.870080] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29046.685817] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29046.949808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29047.211207] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29047.488560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29054.501587] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29055.378421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29056.267309] Lustre: DEBUG MARKER: rc=0; [29056.267309] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29056.267309] if [[ $? -eq 0 && $val -ne 0 ]]; then [29056.267309] echo $(hostname -s): $val; [29056.267309] rc=$val; [29056.267309] fi; [29056.267309] exit $rc [29056.873321] Lustre: DEBUG MARKER: dmesg [29057.469533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 23: test mapped ACLs ============================================================== 18:37:08 \(1491503828\) [29057.626179] Lustre: DEBUG MARKER: == sanity-sec test 23: test mapped ACLs ============================================================== 18:37:08 (1491503828) [29057.778528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [29057.778528] /usr/sbin/lctl lustre_build_version 2>/dev/null || [29057.778528] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [29058.059322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [29058.211397] Lustre: 7729:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [29058.215085] Lustre: 7729:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [29058.359617] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [29058.618910] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [29058.875713] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [29059.131742] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [29059.387372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [29059.644192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [29059.902705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29060.441694] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [29060.698651] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [29060.955963] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [29061.213642] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [29061.474084] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [29061.730293] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [29061.989591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29062.246036] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [29062.504160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29062.764598] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29063.042665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29066.928920] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29069.296325] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29070.121521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [29070.380707] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [29070.643369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29070.902849] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29071.179312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29078.126158] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29080.499817] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29081.309951] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29081.571659] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29081.828732] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [29082.088560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [29082.349142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29082.609600] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29082.886068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [29083.684926] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29084.504882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29084.767979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29085.027958] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29085.286171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29085.544504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29085.804976] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29086.081212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29089.967800] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29090.776135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29091.036882] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29091.328308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29092.147521] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29092.986353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29093.245638] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29093.506343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29093.765543] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29094.038359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29094.842333] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29101.256180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29101.514829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29101.774049] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29102.033148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29102.294092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29102.554046] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29102.834420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29103.646152] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29104.454386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29104.729046] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29105.014414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29105.843896] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29106.724820] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29106.994364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29107.267401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29107.530943] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29107.807360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29108.609515] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29114.950389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29115.207728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29115.466719] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29115.726316] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29115.986721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29116.248161] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29116.524203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29117.330957] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29118.152754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29118.412108] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29118.688984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29119.486552] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29120.371968] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29120.629976] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29120.888177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29121.148535] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29121.425536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29122.233776] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29128.663618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29128.923574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29129.185153] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29129.452059] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29129.711675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29129.972685] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29130.261288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29131.062229] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29131.880566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29132.140146] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29132.416765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29133.225073] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29134.120836] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29134.378675] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29134.636185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29134.894064] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29135.173315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29135.977416] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29142.312376] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [29142.569094] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [29142.828230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29143.088899] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29143.367122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29148.802496] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29151.168728] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29151.997674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29152.258386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29152.516382] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29152.775717] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29153.037818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29153.297552] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29153.584756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29160.572213] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29162.921255] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29163.731783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29163.990647] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29164.277555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29165.097144] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29165.961325] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [29166.219536] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [29166.482405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29166.742484] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29167.013431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29174.013373] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29180.473600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29180.733630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29180.989512] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29181.245527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29181.504339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29181.766713] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29182.039518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29189.016638] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29189.841805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29190.120086] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29190.412308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29191.218370] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29192.100639] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [29192.359269] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [29192.618577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29192.882511] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29193.159141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29203.238619] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29209.632106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [29209.897890] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [29210.155762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [29210.412198] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [29210.670204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29210.928694] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29211.204170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29218.181604] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29218.995374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29219.253497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29219.516351] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29219.793721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29228.334928] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29229.202042] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29230.095466] Lustre: DEBUG MARKER: rc=0; [29230.095466] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29230.095466] if [[ $? -eq 0 && $val -ne 0 ]]; then [29230.095466] echo $(hostname -s): $val; [29230.095466] rc=$val; [29230.095466] fi; [29230.095466] exit $rc [29230.690366] Lustre: DEBUG MARKER: dmesg [29231.291733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 24: check nodemap proc files for LBUGs and Oopses ================================= 18:40:02 \(1491504002\) [29231.448680] Lustre: DEBUG MARKER: == sanity-sec test 24: check nodemap proc files for LBUGs and Oopses ================================= 18:40:02 (1491504002) [29231.599801] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [29231.746306] Lustre: 20236:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [29231.750233] Lustre: 20236:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [29231.895058] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [29232.154149] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [29232.413209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [29232.671213] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [29232.931402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [29233.189247] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [29233.463568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29234.000056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [29234.260314] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [29234.521053] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [29234.788115] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [29235.044862] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [29235.303403] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [29235.560750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29235.818058] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [29236.074235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29236.335608] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29236.612527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29240.504345] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29241.317556] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [29241.708153] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [29242.349939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29243.046415] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29243.568099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29248.997526] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29249.818202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -R nodemap [29250.097343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [29250.357981] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [29250.619833] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [29250.883797] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [29251.140022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29251.400781] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29251.681860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29257.109845] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29259.466193] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29260.302168] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29260.570353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29260.830211] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29261.107729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29268.071829] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29268.946986] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29269.818741] Lustre: DEBUG MARKER: rc=0; [29269.818741] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29269.818741] if [[ $? -eq 0 && $val -ne 0 ]]; then [29269.818741] echo $(hostname -s): $val; [29269.818741] rc=$val; [29269.818741] fi; [29269.818741] exit $rc [29270.422021] Lustre: DEBUG MARKER: dmesg [29271.028085] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 25: test save and reload nodemap config =========================================== 18:40:42 \(1491504042\) [29271.196284] Lustre: DEBUG MARKER: == sanity-sec test 25: test save and reload nodemap config =========================================== 18:40:42 (1491504042) [29271.348959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [29271.348959] /usr/sbin/lctl lustre_build_version 2>/dev/null || [29271.348959] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [29272.029144] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [29272.316491] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [29272.574607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [29272.832689] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [29273.089315] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [29273.350047] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [29273.608100] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [29273.867244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29274.431923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [29274.689967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [29274.948287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [29275.208960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [29275.470434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [29275.730330] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [29275.985630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29276.242877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [29276.499698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29276.758574] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29277.036236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29282.456063] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29283.279212] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [29283.539757] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [29283.796351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29284.057658] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29284.329723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29292.907061] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29293.739411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29293.997782] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29294.256145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [29294.512213] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 1 [29294.770227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29295.034456] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29295.307594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [29300.751794] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29303.112601] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29303.925521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add test26 [29304.184808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.test26.fileset=/c0dir [29304.315124] Lustre: Setting parameter general.nodemap.test26.fileset in log params [29304.316907] Lustre: Skipped 3 previous similar messages [29304.448836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29304.709207] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29304.985895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test26.id [29310.461279] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29312.818025] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29315.194977] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29316.022691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [29316.281232] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [29318.853264] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29318.853264] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29318.853264] if [ $running -ne $mpts ]; then [29318.853264] echo $(hostname) env are INSANE!; [29318.853264] exit 1; [29318.853264] fi [29319.430950] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [29319.430950] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [29319.430950] if [ $running -ne $mpts ]; then [29319.430950] echo $(hostname) env are INSANE!; [29319.430950] exit 1; [29319.430950] fi [29320.021455] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29320.021455] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29320.021455] if [ $running -ne $mpts ]; then [29320.021455] echo $(hostname) env are INSANE!; [29320.021455] exit 1; [29320.021455] fi [29320.958835] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [29321.230188] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29321.488380] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29321.747520] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29322.011397] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29322.271952] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [29322.530852] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [29322.789155] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall NONE [29323.046238] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [29325.630757] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [29325.903749] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [29326.166900] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [29326.425805] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [29326.689618] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29326.953136] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*2 [29327.213043] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0002.identity_upcall [29327.472592] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0002.identity_upcall NONE [29327.730909] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0002/identity_flush=-1 [29338.630117] Lustre: MGS: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [29338.632217] Lustre: Skipped 146 previous similar messages [29339.938208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [29340.116853] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [29341.561267] Lustre: DEBUG MARKER: lctl get_param -n timeout [29341.896893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [29342.051979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [29342.182306] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [29347.604306] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [29347.885943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29348.402513] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29349.320846] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [29349.597857] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [29349.854762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del test26 [29350.114781] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [29350.377269] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [29350.634931] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [29350.893108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [29351.149106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29351.407563] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29351.685953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29360.238140] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29361.077064] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29361.335296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29361.594138] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29361.871326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29370.440448] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29372.385649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [29372.553791] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [29372.765825] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29373.662951] Lustre: DEBUG MARKER: rc=0; [29373.662951] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29373.662951] if [[ $? -eq 0 && $val -ne 0 ]]; then [29373.662951] echo $(hostname -s): $val; [29373.662951] rc=$val; [29373.662951] fi; [29373.662951] exit $rc [29374.265512] Lustre: DEBUG MARKER: dmesg [29374.876064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-sec test_26 skipping SLOW test 26 [29375.033378] Lustre: DEBUG MARKER: SKIP: sanity-sec test_26 skipping SLOW test 26 [29375.239034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 27: test fileset in nodemap ======================================================= 18:42:26 \(1491504146\) [29375.396940] Lustre: DEBUG MARKER: == sanity-sec test 27: test fileset in nodemap ======================================================= 18:42:26 (1491504146) [29375.540212] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [29375.689206] Lustre: 30734:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [29375.693154] Lustre: 30734:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 5 previous similar messages [29375.836582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [29376.097740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [29376.358900] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [29376.617732] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [29376.874561] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [29377.132953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [29377.391233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29377.919772] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [29378.181626] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [29378.439337] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [29378.697566] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [29378.955957] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [29379.211124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [29379.468622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29379.723026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [29379.979453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29380.236276] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29380.528028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29385.969206] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29386.787643] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [29387.046979] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [29387.306067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29387.567650] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29387.842142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29396.368391] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29397.185220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29397.445805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29397.705858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29397.965923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29398.229246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29398.487358] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29398.764887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29407.288227] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29408.115086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29408.373730] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29408.648507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29409.461140] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29410.314386] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [29410.575991] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [29410.843223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29411.107181] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29411.384956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29421.471475] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29422.282047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29422.543864] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29422.824707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29423.629294] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29424.439085] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.c0.fileset=/c0dir [29424.569442] Lustre: Setting parameter general.nodemap.c0.fileset in log params [29424.700862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29424.958434] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29433.500741] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29433.776596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [29434.605781] Lustre: MGS: Connection restored to efea90a9-b969-b9f5-8b46-4d471013c4fc (at 10.9.5.237@tcp) [29434.607645] Lustre: Skipped 11 previous similar messages [29435.170341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name c0 --fileset '' [29435.430685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [29435.689789] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.c0.fileset='' [29435.949346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29436.208440] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29436.489648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [29443.472919] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29445.825318] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29446.900773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [29447.160057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [29447.421591] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [29447.687602] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [29447.945870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29448.209836] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29448.499934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29449.310627] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29451.670877] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29452.481737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29452.744742] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29453.018683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29453.825929] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29454.668054] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [29454.931816] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [29455.200655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29455.460653] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29455.738034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [29458.076518] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29458.898980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29459.158212] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29459.437126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [29460.258729] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29461.078437] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [29461.353099] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [29461.619426] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [29461.877206] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [29462.136686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29462.397816] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29462.673857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [29466.572332] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29467.401991] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29467.675072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [29467.935407] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29468.211340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [29475.196463] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [29476.086182] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29477.024327] Lustre: DEBUG MARKER: rc=0; [29477.024327] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29477.024327] if [[ $? -eq 0 && $val -ne 0 ]]; then [29477.024327] echo $(hostname -s): $val; [29477.024327] rc=$val; [29477.024327] fi; [29477.024327] exit $rc [29477.642843] Lustre: DEBUG MARKER: dmesg [29478.244728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cleanup: ====================================================== [29478.396286] Lustre: DEBUG MARKER: cleanup: ====================================================== [29478.536700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [29489.484063] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29489.743175] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [29490.171155] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [29490.172956] Lustre: Skipped 1 previous similar message [29491.735164] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [29491.735209] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29491.735210] Lustre: Skipped 7 previous similar messages [29491.735275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [29491.735276] Lustre: Skipped 10 previous similar messages [29491.745246] LustreError: Skipped 7 previous similar messages [29491.999065] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006691fc00 x1563924897624720/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29492.003174] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [29493.711056] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006691de00 x1563924897624912/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29493.715398] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [29495.168525] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [29495.170610] Lustre: Skipped 2 previous similar messages [29495.960627] Lustre: server umount lustre-MDT0000 complete [29495.962470] Lustre: Skipped 3 previous similar messages [29496.092978] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29496.735266] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [29496.739540] LustreError: Skipped 94 previous similar messages [29503.245122] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [29503.506878] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [29503.727073] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880061ce5200 x1563924897625504/t0(0) o13->lustre-OST0006-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29503.731855] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [29505.172507] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [29505.239043] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491504267/real 1491504267] req@ffff880061ce5e00 x1563924897624976/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491504274 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29505.245887] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 34 previous similar messages [29505.248349] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [29505.252722] LustreError: Skipped 5 previous similar messages [29507.039081] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880061ce4300 x1563924897625552/t0(0) o13->lustre-OST0007-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29507.044366] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [29509.926899] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29577.037721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test complete, duration 3450 sec ======================================================= 18:45:47 \(1491504347\) [29577.188232] Lustre: DEBUG MARKER: == sanity-sec test complete, duration 3450 sec ======================================================= 18:45:47 (1491504347) [29577.397288] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: large-scale ============----- Thu Apr 6 18:45:48 UTC 2017 [29577.548506] Lustre: DEBUG MARKER: -----============= acceptance-small: large-scale ============----- Thu Apr 6 18:45:48 UTC 2017 [29577.802177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test complete, duration -o sec ======================================================== 18:45:48 \(1491504348\) [29577.945124] Lustre: DEBUG MARKER: == large-scale test complete, duration -o sec ======================================================== 18:45:48 (1491504348) [29578.259096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [29578.402137] Lustre: DEBUG MARKER: excepting tests: [29580.747494] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29580.747494] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29580.747494] if [ $running -ne $mpts ]; then [29580.747494] echo $(hostname) env are INSANE!; [29580.747494] exit 1; [29580.747494] fi [29581.364209] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [29581.364209] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [29581.364209] if [ $running -ne $mpts ]; then [29581.364209] echo $(hostname) env are INSANE!; [29581.364209] exit 1; [29581.364209] fi [29581.943062] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29581.943062] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29581.943062] if [ $running -ne $mpts ]; then [29581.943062] echo $(hostname) env are INSANE!; [29581.943062] exit 1; [29581.943062] fi [29584.461152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [29584.724553] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29584.978188] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29585.227857] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29585.373336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29585.394140] Lustre: MGS: Connection restored to 38f9f288-69a6-521d-9000-002b78c6286b (at 0@lo) [29585.396638] Lustre: Skipped 11 previous similar messages [29585.434820] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [29585.441513] LustreError: Skipped 22 previous similar messages [29585.456751] LustreError: 7202:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [29585.461156] LustreError: 7202:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 2 previous similar messages [29585.475911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [29585.478294] Lustre: Skipped 1 previous similar message [29585.624749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29585.880777] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29586.388593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29586.395871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29586.555620] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29586.556641] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29586.697439] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29586.963606] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29587.216930] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29587.474127] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29587.729603] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29590.007650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29590.170152] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29590.469182] Lustre: 5278:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491504356/real 1491504356] req@ffff880065173600 x1563924897625824/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491504361 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29591.595882] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [29591.861776] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [29592.117839] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [29592.372252] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [29592.533754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29592.765768] LustreError: 8500:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [29592.768616] LustreError: 8500:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [29593.144786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29593.402705] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29593.923491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29593.930486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29594.312981] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29594.314783] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29594.692366] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29595.537075] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29596.096502] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29596.369409] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [29596.632480] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29599.233730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29599.398574] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29602.757787] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29602.930509] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29605.868692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29606.033215] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29608.999725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29609.162480] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29612.149081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29612.320323] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29615.332662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29615.499603] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29618.640090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29618.788544] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29621.809089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29621.987352] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29625.014574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29625.162552] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29627.363887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29627.530228] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29629.246288] Lustre: DEBUG MARKER: lctl get_param -n timeout [29629.976079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [29630.125930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [29630.260107] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [29635.670463] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [29635.957646] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29636.486737] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29637.226804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test 3a: recovery time, 2 clients ===================================================== 18:46:48 \(1491504408\) [29637.381200] Lustre: DEBUG MARKER: == large-scale test 3a: recovery time, 2 clients ===================================================== 18:46:48 (1491504408) [29705.014605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 : Starting failover on mds1 [29705.182264] Lustre: DEBUG MARKER: 1 : Starting failover on mds1 [29705.342071] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29705.638512] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29705.801495] Lustre: Failing over lustre-MDT0000 [29705.804022] Lustre: Skipped 1 previous similar message [29706.393269] Lustre: server umount lustre-MDT0000 complete [29706.395768] Lustre: Skipped 1 previous similar message [29706.527721] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29707.898549] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [29707.901459] LustreError: Skipped 1 previous similar message [29707.903797] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29707.908814] Lustre: Skipped 2 previous similar messages [29715.055067] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [29716.844674] Lustre: DEBUG MARKER: hostname [29717.151575] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29717.407245] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29717.663032] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29717.822536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29721.059457] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d61a783 to 0x61ac6f141d787d63 [29721.163147] LustreError: 11293:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [29721.167926] LustreError: 11293:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [29721.173080] LustreError: 11298:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [29721.203200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [29721.205679] Lustre: Skipped 1 previous similar message [29721.387812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29721.651683] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29721.672246] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [29721.674819] Lustre: Skipped 5 previous similar messages [29722.149037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29722.153340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29722.309593] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29722.309878] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29722.457295] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29722.721967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29722.983178] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29723.250103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29723.528930] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29724.065627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29724.070680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29724.230193] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29724.233634] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29726.183266] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29726.211906] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [29726.215172] Lustre: Skipped 5 previous similar messages [29729.414413] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [29933.819176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 : Starting failover on mds1 [29934.021073] Lustre: DEBUG MARKER: 2 : Starting failover on mds1 [29934.182528] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29934.470724] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29934.629797] Lustre: Failing over lustre-MDT0000 [29935.254343] Lustre: server umount lustre-MDT0000 complete [29935.384297] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29936.100179] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [29936.105903] LustreError: Skipped 52 previous similar messages [29936.181606] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [29936.181633] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29936.181635] Lustre: Skipped 1 previous similar message [29936.192068] LustreError: Skipped 2 previous similar messages [29943.160066] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491504706/real 1491504706] req@ffff8800685ca100 x1563924898492928/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491504713 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29943.168881] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [29943.171716] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [29945.713929] Lustre: DEBUG MARKER: hostname [29946.026096] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29946.285908] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29946.544637] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29946.711067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29949.177461] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [29949.180026] Lustre: Skipped 60 previous similar messages [29949.182606] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141d787d63 to 0x61ac6f141db89ecb [29949.312778] LustreError: 13276:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [29949.320500] LustreError: 13281:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [29949.355080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [29949.524122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29949.788125] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29950.296326] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29950.297046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29950.341146] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [29950.459444] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29950.463517] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29950.605720] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29950.868520] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29951.149132] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29951.415767] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29951.682195] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29952.182482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29952.183891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29952.347004] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29952.347056] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [29954.330349] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29954.360030] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [29957.528088] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [30162.130156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 : Starting failover on mds1 [30162.348172] Lustre: DEBUG MARKER: 3 : Starting failover on mds1 [30162.525276] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30162.858300] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30163.006774] Lustre: Failing over lustre-MDT0000 [30163.120333] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [30163.123241] Lustre: Skipped 8 previous similar messages [30163.711601] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30174.030726] Lustre: DEBUG MARKER: hostname [30174.343573] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30174.600434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30174.854939] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30175.004610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30176.831445] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141db89ecb to 0x61ac6f141df89bd5 [30177.126822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30177.359332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [30177.387675] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30177.922857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30177.928331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30178.087096] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30178.087099] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30178.230907] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30178.490869] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30178.750068] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30179.013680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30179.275472] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30179.773791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30179.773795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30179.944184] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30179.946022] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30181.943585] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30181.963817] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [30185.165606] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [30363.620879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 : Starting failover on mds1 [30363.816989] Lustre: DEBUG MARKER: 1 : Starting failover on mds1 [30364.007378] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30364.316250] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30365.116883] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [30365.622992] Lustre: server umount lustre-MDT0000 complete [30365.625516] Lustre: Skipped 1 previous similar message [30365.756785] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30366.263694] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [30366.269190] LustreError: Skipped 66 previous similar messages [30366.942412] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [30366.945391] LustreError: Skipped 2 previous similar messages [30366.945468] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [30366.945469] Lustre: Skipped 3 previous similar messages [30373.975035] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491505137/real 1491505137] req@ffff88004ebb3600 x1563924900486288/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491505144 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [30373.983221] Lustre: 5279:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [30373.986063] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [30373.991145] LustreError: Skipped 1 previous similar message [30376.070555] Lustre: DEBUG MARKER: hostname [30376.386561] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30376.648683] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30376.903350] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30377.066053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30379.993448] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141df89bd5 to 0x61ac6f141e3c6c37 [30380.099311] LustreError: 17241:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [30380.104549] LustreError: 17241:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [30380.109595] LustreError: 17246:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [30380.112499] LustreError: 17246:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [30380.142227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [30380.144899] Lustre: Skipped 1 previous similar message [30380.322555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30380.585966] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30381.095060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30381.096948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30381.261627] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30381.261628] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30381.407229] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30381.666703] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30381.934675] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30382.203344] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30382.467567] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30382.970049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30382.972028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30383.134081] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30383.139790] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30385.120806] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30388.347068] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [30513.608115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 : Starting failover on mds1 [30513.792740] Lustre: DEBUG MARKER: 2 : Starting failover on mds1 [30513.961878] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30514.316006] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30514.483754] Lustre: Failing over lustre-MDT0000 [30514.486354] Lustre: Skipped 1 previous similar message [30515.124350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [30515.127075] Lustre: Skipped 7 previous similar messages [30515.406624] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30525.722634] Lustre: DEBUG MARKER: hostname [30526.034126] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30526.294924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30526.551226] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30526.711159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30532.367451] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141e3c6c37 to 0x61ac6f141e7ced06 [30532.651848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30532.912042] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30533.408415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30533.410198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30533.573045] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30533.578427] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30533.724827] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30533.986326] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30534.246927] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30534.513323] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30534.773037] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30535.269264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30535.272158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30535.434149] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30535.435802] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30535.995177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [30535.998012] Lustre: Skipped 1 previous similar message [30537.466391] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30537.497347] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [30537.500794] Lustre: Skipped 1 previous similar message [30540.623458] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [30665.829355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 : Starting failover on mds1 [30666.233854] Lustre: DEBUG MARKER: 3 : Starting failover on mds1 [30666.653237] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30666.979065] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30667.470442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [30667.473118] Lustre: Skipped 1 previous similar message [30668.123774] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30678.441404] Lustre: DEBUG MARKER: hostname [30678.754235] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30679.017421] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30679.272669] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30679.440227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30679.465449] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [30679.467963] Lustre: Skipped 103 previous similar messages [30679.470427] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x61ac6f141e7ced06 to 0x61ac6f141ebd22c6 [30679.766548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30680.027921] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30680.532079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30680.532545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30680.701761] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30680.704057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30680.849238] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30681.109284] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30681.370434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30681.635531] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30681.900740] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30682.403333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30682.403337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30682.570665] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30682.574150] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [30684.572413] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30687.783450] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [30819.887728] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30820.840565] Lustre: DEBUG MARKER: rc=0; [30820.840565] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30820.840565] if [[ $? -eq 0 && $val -ne 0 ]]; then [30820.840565] echo $(hostname -s): $val; [30820.840565] rc=$val; [30820.840565] fi; [30820.840565] exit $rc [30821.476728] Lustre: DEBUG MARKER: dmesg [30822.123606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test complete, duration 1245 sec ====================================================== 19:06:33 \(1491505593\) [30822.284030] Lustre: DEBUG MARKER: == large-scale test complete, duration 1245 sec ====================================================== 19:06:33 (1491505593) [30854.756651] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30855.015549] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [30855.160190] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006291a700 x1563924903235728/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [30855.165904] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [30856.125684] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [30856.128526] Lustre: Skipped 8 previous similar messages [30861.404859] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30868.636274] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [30868.893503] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [30869.119143] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006291a700 x1563924903236544/t0(0) o13->lustre-OST0007-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [30869.124707] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [30870.207073] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006291a700 x1563924903236624/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [30870.212552] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [30871.126297] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [30871.128904] Lustre: Skipped 13 previous similar messages [30873.919083] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800684bdb00 x1563924903236656/t0(0) o13->lustre-OST0002-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [30873.924797] LustreError: 5280:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [30875.317544] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30900.848979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mds-survey ============----- Thu Apr 6 19:07:51 UTC 2017 [30901.003140] Lustre: DEBUG MARKER: -----============= acceptance-small: mds-survey ============----- Thu Apr 6 19:07:51 UTC 2017 [30901.262126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test complete, duration -o sec ========================================================= 19:07:52 \(1491505672\) [30901.414129] Lustre: DEBUG MARKER: == mds-survey test complete, duration -o sec ========================================================= 19:07:52 (1491505672) [30901.640090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [30901.640090] /usr/sbin/lctl lustre_build_version 2>/dev/null || [30901.640090] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [30901.962856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [30902.115350] Lustre: DEBUG MARKER: excepting tests: [30904.512646] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [30904.512646] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [30904.512646] if [ $running -ne $mpts ]; then [30904.512646] echo $(hostname) env are INSANE!; [30904.512646] exit 1; [30904.512646] fi [30905.169127] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [30905.169127] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [30905.169127] if [ $running -ne $mpts ]; then [30905.169127] echo $(hostname) env are INSANE!; [30905.169127] exit 1; [30905.169127] fi [30905.754732] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [30905.754732] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [30905.754732] if [ $running -ne $mpts ]; then [30905.754732] echo $(hostname) env are INSANE!; [30905.754732] exit 1; [30905.754732] fi [30908.280137] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [30908.539500] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30908.791701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30909.041555] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30909.201419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30909.541950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30909.799961] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30910.402106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30910.402108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30910.566052] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30910.571029] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30910.722554] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30910.977469] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30911.232945] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30911.497631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30911.759849] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30914.300535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30914.466075] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30915.895465] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [30916.162499] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [30916.418472] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [30916.673393] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [30916.834109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30917.169087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30917.429440] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30917.926924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30917.928066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30918.096861] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30918.097851] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30918.247124] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30918.512218] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30918.772369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30919.032135] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [30919.290062] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30921.593854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30921.758667] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30925.243771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30925.404199] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30928.242450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30928.406912] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30931.242212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30931.406102] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30934.269252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30934.434442] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30937.328526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30937.490293] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30940.376364] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30940.556431] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30943.630264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30943.807606] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30946.740272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30946.915544] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30949.123910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30949.290260] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [30950.747688] Lustre: DEBUG MARKER: lctl get_param -n timeout [30951.059807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [30951.218726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [30951.359618] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [30956.785831] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [30957.067289] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30957.588321] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30958.901448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 1: Metadata survey with zero-stripe =============================================== 19:08:49 \(1491505729\) [30959.054912] Lustre: DEBUG MARKER: == mds-survey test 1: Metadata survey with zero-stripe =============================================== 19:08:49 (1491505729) [30959.475355] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [30960.640262] Lustre: Echo OBD driver; http://www.lustre.org/ [30965.027802] LustreError: 28149:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [30965.030943] LustreError: 28149:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [30965.349652] LustreError: 28215:0:(lod_object.c:1474:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [30965.674350] LustreError: 28281:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0002-tests: rc = -2 [30965.677464] LustreError: 28281:0:(echo_client.c:1662:echo_md_lookup()) Skipped 1 previous similar message [30965.680534] LustreError: 28281:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0002-tests: rc = -2 [30965.683729] LustreError: 28281:0:(echo_client.c:1861:echo_md_destroy_internal()) Skipped 1 previous similar message [30966.013813] LustreError: 28347:0:(lod_object.c:1474:lod_verify_md_striping()) lustre-MDT0002-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [30966.019608] LustreError: 28347:0:(lod_object.c:1474:lod_verify_md_striping()) Skipped 1 previous similar message [31087.286899] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [31088.191508] Lustre: DEBUG MARKER: rc=0; [31088.191508] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [31088.191508] if [[ $? -eq 0 && $val -ne 0 ]]; then [31088.191508] echo $(hostname -s): $val; [31088.191508] rc=$val; [31088.191508] fi; [31088.191508] exit $rc [31088.794346] Lustre: DEBUG MARKER: dmesg [31089.398788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================== 19:11:00 \(1491505860\) [31089.555410] Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================== 19:11:00 (1491505860) [31089.974416] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [31091.592519] Lustre: Echo OBD driver; http://www.lustre.org/ [31096.409773] LustreError: 30707:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [31096.412369] LustreError: 30707:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [31096.732911] LustreError: 30774:0:(lod_object.c:1474:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [31096.737711] LustreError: 30774:0:(lod_object.c:1474:lod_verify_md_striping()) Skipped 1 previous similar message [31097.060099] LustreError: 30840:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0002-tests: rc = -2 [31097.062673] LustreError: 30840:0:(echo_client.c:1662:echo_md_lookup()) Skipped 1 previous similar message [31097.065137] LustreError: 30840:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0002-tests: rc = -2 [31097.067764] LustreError: 30840:0:(echo_client.c:1861:echo_md_destroy_internal()) Skipped 1 previous similar message [31245.841199] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [31248.701135] Lustre: DEBUG MARKER: rc=0; [31248.701135] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [31248.701135] if [[ $? -eq 0 && $val -ne 0 ]]; then [31248.701135] echo $(hostname -s): $val; [31248.701135] rc=$val; [31248.701135] fi; [31248.701135] exit $rc [31250.544851] Lustre: DEBUG MARKER: dmesg [31252.537759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [31252.689731] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [31252.912174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test complete, duration 351 sec ======================================================== 19:13:43 \(1491506023\) [31253.072749] Lustre: DEBUG MARKER: == mds-survey test complete, duration 351 sec ======================================================== 19:13:43 (1491506023) [31263.916180] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [31264.177058] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [31265.833059] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [31265.835762] Lustre: Skipped 8 previous similar messages [31267.016220] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [31267.019067] LustreError: Skipped 8 previous similar messages [31267.021293] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [31267.025739] Lustre: Skipped 8 previous similar messages [31267.231063] LustreError: 5279:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006793ea00 x1563924909699344/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [31268.294128] Lustre: server umount lustre-MDT0000 complete [31268.296418] Lustre: Skipped 4 previous similar messages [31268.431597] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [31271.602475] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [31271.860796] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [31275.820242] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [31275.826137] LustreError: Skipped 143 previous similar messages [31278.959041] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491506042/real 1491506042] req@ffff8800349cf900 x1563924909699760/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491506049 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31278.966427] Lustre: 5280:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 30 previous similar messages [31280.302099] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [31356.465581] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [31356.730081] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [31357.503529] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [31357.767083] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' Console [trevis-41vm7] departed by on pts/53 at 04-06 19:19. Console [trevis-41vm7] joined by on pts/9 at 04-07 06:21. Console [trevis-41vm7] disconnected from at 04-07 06:49. Console [trevis-41vm7] connected to at 04-07 06:50. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007010] SELinux: Initializing. [ 0.008194] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.012075] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014534] Mount-cache hash table entries: 4096 [ 0.016129] Initializing cgroup subsys memory [ 0.017011] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.019005] Initializing cgroup subsys net_cls [ 0.021005] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.023006] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.026004] Initializing cgroup subsys net_prio [ 0.027115] Disabled fast string operations [ 0.028020] mce: CPU supports 10 MCE banks [ 0.030056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] tlb_flushall_shift: 6 [ 0.034176] Freeing SMP alternatives: 28k freed [ 0.039617] ACPI: Core revision 20130517 [ 0.041707] ACPI: All ACPI Tables successfully acquired [ 0.043028] ftrace: allocating 25811 entries in 101 pages [ 0.051270] smpboot: Max logical packages: 2 [ 0.052005] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.056058] Enabling x2apic [ 0.057000] Enabled x2apic [ 0.057005] Switched APIC routing to physical x2apic. [ 0.059873] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.061002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.063882] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.065604] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.319025] Brought up 2 CPUs [ 0.319017] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.321005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.322339] devtmpfs: initialized [ 0.325941] EVM: security.selinux [ 0.326005] EVM: security.ima [ 0.327004] EVM: security.capability [ 0.328726] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.329009] pinctrl core: initialized pinctrl subsystem [ 0.330157] NET: Registered protocol family 16 [ 0.331290] ACPI: bus type PCI registered [ 0.332006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.333095] PCI: Using configuration type 1 for base access [ 0.337085] ACPI: Added _OSI(Module Device) [ 0.338013] ACPI: Added _OSI(Processor Device) [ 0.339006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.342680] ACPI: Interpreter enabled [ 0.343013] ACPI: (supports S0 S5) [ 0.344005] ACPI: Using IOAPIC for interrupt routing [ 0.345029] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.348636] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.349010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.350010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.351018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352246] acpiphp: Slot [1] registered [ 0.353031] acpiphp: Slot [2] registered [ 0.354022] acpiphp: Slot [3] registered [ 0.355022] acpiphp: Slot [4] registered [ 0.356022] acpiphp: Slot [5] registered [ 0.357022] acpiphp: Slot [6] registered [ 0.358025] acpiphp: Slot [7] registered [ 0.359022] acpiphp: Slot [8] registered [ 0.360033] acpiphp: Slot [9] registered [ 0.361024] acpiphp: Slot [10] registered [ 0.362023] acpiphp: Slot [11] registered [ 0.363022] acpiphp: Slot [12] registered [ 0.364025] acpiphp: Slot [13] registered [ 0.365021] acpiphp: Slot [14] registered [ 0.366023] acpiphp: Slot [15] registered [ 0.367030] acpiphp: Slot [16] registered [ 0.368030] acpiphp: Slot [17] registered [ 0.369026] acpiphp: Slot [18] registered [ 0.370022] acpiphp: Slot [19] registered [ 0.371022] acpiphp: Slot [20] registered [ 0.372022] acpiphp: Slot [21] registered [ 0.373022] acpiphp: Slot [22] registered [ 0.374035] acpiphp: Slot [23] registered [ 0.375022] acpiphp: Slot [24] registered [ 0.376022] acpiphp: Slot [25] registered [ 0.377021] acpiphp: Slot [26] registered [ 0.378029] acpiphp: Slot [27] registered [ 0.379022] acpiphp: Slot [28] registered [ 0.380022] acpiphp: Slot [29] registered [ 0.381031] acpiphp: Slot [30] registered [ 0.382022] acpiphp: Slot [31] registered [ 0.383258] PCI host bridge to bus 0000:00 [ 0.384007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386005] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.388005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.390011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394835] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.395012] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.412677] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.414075] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.415842] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.417068] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.418732] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.420223] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.421258] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.422014] vgaarb: loaded [ 0.423005] vgaarb: bridge control possible 0000:00:02.0 [ 0.424205] SCSI subsystem initialized [ 0.425051] ACPI: bus type USB registered [ 0.426037] usbcore: registered new interface driver usbfs [ 0.427016] usbcore: registered new interface driver hub [ 0.428050] usbcore: registered new device driver usb [ 0.429206] PCI: Using ACPI for IRQ routing [ 0.430338] NetLabel: Initializing [ 0.431007] NetLabel: domain hash size = 128 [ 0.432004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.433052] NetLabel: unlabeled traffic allowed by default [ 0.434149] Switched to clocksource kvm-clock [ 0.444242] pnp: PnP ACPI init [ 0.445352] ACPI: bus type PNP registered [ 0.447033] pnp: PnP ACPI: found 5 devices [ 0.448227] ACPI: bus type PNP unregistered [ 0.458029] NET: Registered protocol family 2 [ 0.459611] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.461924] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.463652] TCP: Hash tables configured (established 16384 bind 16384) [ 0.465259] TCP: reno registered [ 0.466351] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.467815] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.469440] NET: Registered protocol family 1 [ 0.470825] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.472368] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.473879] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.475701] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.477522] Unpacking initramfs... [ 4.617627] Freeing initrd memory: 42360k freed [ 4.637372] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.639207] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.641116] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.642706] Initialise system trusted keyring [ 4.644070] audit: initializing netlink socket (disabled) [ 4.645551] type=2000 audit(1491547826.232:1): initialized [ 4.666627] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.668304] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.670907] zpool: loaded [ 4.672035] zbud: loaded [ 4.673355] VFS: Disk quotas dquot_6.5.2 [ 4.674667] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.676476] msgmni has been set to 3357 [ 4.677804] Key type big_key registered [ 4.680278] NET: Registered protocol family 38 [ 4.681640] Key type asymmetric registered [ 4.682934] Asymmetric key parser 'x509' registered [ 4.684359] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.686820] io scheduler noop registered [ 4.688096] io scheduler deadline registered (default) [ 4.689540] io scheduler cfq registered [ 4.690906] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.692399] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.694179] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.696630] ACPI: Power Button [PWRF] [ 4.698032] GHES: HEST is not enabled! [ 4.699348] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.722246] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.724156] Non-volatile memory driver v1.3 [ 4.725471] Linux agpgart interface v0.103 [ 4.726878] crash memory driver: version 1.1 [ 4.728276] rdac: device handler registered [ 4.729636] hp_sw: device handler registered [ 4.730962] emc: device handler registered [ 4.732259] alua: device handler registered [ 4.733593] libphy: Fixed MDIO Bus: probed [ 4.734937] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.736566] ehci-pci: EHCI PCI platform driver [ 4.737927] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.739505] ohci-pci: OHCI PCI platform driver [ 4.740853] uhci_hcd: USB Universal Host Controller Interface driver [ 4.742647] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.744137] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.746551] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.748047] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.749652] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.751328] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.753781] usb usb1: Product: UHCI Host Controller [ 4.755220] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.756906] usb usb1: SerialNumber: 0000:00:01.2 [ 4.758382] hub 1-0:1.0: USB hub found [ 4.759698] hub 1-0:1.0: 2 ports detected [ 4.761160] usbcore: registered new interface driver usbserial [ 4.762719] usbcore: registered new interface driver usbserial_generic [ 4.764375] usbserial: USB Serial support registered for generic [ 4.766062] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.769288] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.770647] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.772174] mousedev: PS/2 mouse device common for all mice [ 4.773933] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.776657] rtc_cmos 00:00: RTC can wake from S4 [ 4.777486] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.777908] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.784458] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.786152] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.787771] cpuidle: using governor menu [ 4.789172] hidraw: raw HID events driver (C) Jiri Kosina [ 4.790769] usbcore: registered new interface driver usbhid [ 4.792304] usbhid: USB HID core driver [ 4.793612] drop_monitor: Initializing network drop monitor service [ 4.795308] TCP: cubic registered [ 4.796564] Initializing XFRM netlink socket [ 4.798039] NET: Registered protocol family 10 [ 4.799713] NET: Registered protocol family 17 [ 4.801334] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.803241] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.804868] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.807773] Loading compiled-in X.509 certificates [ 4.809256] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.812068] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.815393] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.818239] registered taskstats version 1 [ 4.822266] Key type trusted registered [ 4.825804] Key type encrypted registered [ 4.829260] IMA: No TPM chip found, activating TPM-bypass! [ 4.831139] rtc_cmos 00:00: setting system clock to 2017-04-07 06:50:24 UTC (1491547824) [ 4.834652] Freeing unused kernel memory: 1680k freed [ 4.840865] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.860667] systemd[1]: Detected virtualization kvm. [ 4.862145] systemd[1]: Detected architecture x86-64. [ 4.863470] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.865673] systemd[1]: No hostname configured. [ 4.866941] systemd[1]: Set hostname to . [ 4.868336] systemd[1]: Initializing machine ID from KVM UUID. [ 4.894471] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Reached target Local File Systems. [ 4.897991] systemd[1]: Reached target Local File Systems. [ 4.899438] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 4.901422] systemd[1]: Reached target Timers. [ 4.902699] systemd[1]: Starting Timers. [ OK ] Reached target Swap. [ 4.904491] systemd[1]: Reached target Swap. [ 4.905711] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 4.907589] systemd[1]: Created slice Root Slice. [ 4.908868] systemd[1]: Starting Root Slice. [ OK ] Created slice System Slice. [ 4.910766] systemd[1]: Created slice System Slice. [ 4.912088] systemd[1]: Starting System Slice. [ OK ] Listening on Journal Socket. [ 4.914020] systemd[1]: Listening on Journal Socket. [ 4.915352] systemd[1]: Starting Journal Socket. [ 4.917207] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 4.922251] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.926136] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Slices. [ 4.930151] systemd[1]: Reached target Slices. [ 4.932233] systemd[1]: Starting Slices. [ OK ] Listening on udev Kernel Socket. [ 4.936161] systemd[1]: Listening on udev Kernel Socket. [ 4.938464] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ 4.941771] systemd[1]: Listening on udev Control Socket. [ 4.944046] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Encrypted Volumes. [ 4.947316] systemd[1]: Reached target Encrypted Volumes. [ 4.949574] systemd[1]: Starting Encrypted Volumes. [ 4.952644] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 4.956832] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Journal Service. [ 4.965067] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 5.028176] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.104359] dracut-cmdline[88]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.241849] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.246819] Loading iSCSI transport class v2.0-870. [ 5.251592] iscsi: registered transport (tcp) [ 5.286322] FDC 0 is a S82078B [ 5.291734] iBFT detected. [ 5.292881] iBFT has incorrect checksum (0x54)! [ 5.303287] md: raid0 personality registered for level 0 [ 5.307166] md: raid1 personality registered for level 1 [ 5.310660] async_tx: api initialized (async) [ 5.312896] xor: automatically using best checksumming function: [ 5.324012] avx : 22320.000 MB/sec [ 5.345018] raid6: sse2x1 gen() 7851 MB/s [ 5.363010] raid6: sse2x2 gen() 9890 MB/s [ 5.381014] raid6: sse2x4 gen() 11449 MB/s [ 5.382345] raid6: using algorithm sse2x4 gen() (11449 MB/s) [ 5.383868] raid6: using ssse3x2 recovery algorithm [ 5.389914] md: raid6 personality registered for level 6 [ 5.391393] md: raid5 personality registered for level 5 [ 5.392875] md: raid4 personality registered for level 4 [ 5.399357] md: raid10 personality registered for level 10 [ 5.402916] md: linear personality registered for level -1 [ 5.408385] device-mapper: uevent: version 1.0.3 [ 5.409863] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.424723] device-mapper: multipath: version 1.11.0 loaded [ 5.428364] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.460263] RPC: Registered named UNIX socket transport module. [ 5.461834] RPC: Registered udp transport module. [ 5.463252] RPC: Registered tcp transport module. [ 5.464680] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 5.639048] tsc: Refined TSC clocksource calibration: 2593.643 MHz Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... [ OK ] [ 5.682322] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.684717] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 Reached target System Initialization. [ 5.685721] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Reached target Sockets. Starting Device-Mapper Multipath Device Controller... [ 5.690355] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.697906] [drm] Initialized drm 1.1.0 20060810 [ 5.700333] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 5.704966] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 5.710160] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 5.720618] scsi host0: ata_piix [ 5.721942] scsi host1: ata_piix [ 5.723080] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.724604] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 5.729368] AES CTR mode by8 optimization enabled [ OK ] Started Device-Mapper Multipath Device Controller. Starting Open-iSCSI... [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 5.745581] 8021q: 802.1Q VLAN Support v1.8 [ OK ] Started Open-iSCSI. Starting dracut initqueue hook... %G[ 5.774600] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.776291] [TTM] Initializing pool allocator [ 5.777661] [TTM] Initializing DMA pool allocator [ 5.779420] [drm] fb mappable at 0xF0000000 [ 5.780734] [drm] vram aper at 0xF0000000 [ 5.782007] [drm] size 33554432 [ 5.783170] [drm] fb depth is 24 [ 5.784336] [drm] pitch is 3072 %G[ 5.785681] fbcon: cirrusdrmfb (fb0) is primary device [ 5.819280] Console: switching to colour frame buffer device 128x48 [ 5.825936] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 5.834619] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 5.838033] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ 5.847891] alg: No test for crc32 (crc32-pclmul) [ 5.892161] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 6.092094] vda: vda1 vda2 vda4 [ 8.610356] dracut-initqueue[567]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.611675] dracut-initqueue[567]: Dload Upload Total Spent Left Speed [ 8.825484] dracut-initqueue[567]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14606 100 14606 0 0 67656 0 --:--:-- --:--:-- --:--:-- 67934 [ 9.348208] dracut-initqueue[567]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.348922] dracut-initqueue[567]: Dload Upload Total Spent Left Speed [ 12.595367] dracut-initqueue[567]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 19 317M 19 61.1M 0 0 93.8M 0 0:00:03 --:--:-- 0:00:03 93.7M 49 317M 49 158M 0 0 95.8M 0 0:00:03 0:00:01 0:00:02 95.8M 80 317M 80 256M 0 0 96.8M 0 0:00:03 0:00:02 0:00:01 96.7M 100 317M 100 317M 0 0 97.7M 0 0:00:03 0:00:03 --:--:-- 97.7M [ 12.694805] loop: module loaded [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting dracut pre-mount hook... [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. Starting dracut mount hook... [ 12.861328] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 14.857290] systemd-journald[87]: Received SIGTERM from PID 1 (systemd). [ 15.407670] type=1403 audit(1491547835.075:2): policy loaded auid=4294967295 ses=4294967295 [ 15.413510] systemd[1]: Successfully loaded SELinux policy in 267.710ms. [ 15.514807] systemd[1]: Relabelled /dev and /run in 19.233ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Mounting Debug File System... Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Journal Service. Starting Journal Service... Starting Remount Root and Kernel File Systems... Mounting Temporary Directory... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Created slice system-anaconda\x2dtmux.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Mounted Temporary Directory. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... Starting Load/Save Random Seed... Starting Rebuild Hardware Database... Starting Create Static Device Nodes in /dev... [ OK ] Started Load/Save Random Seed. [ OK ] Started Apply Kernel Variables. [ OK ] Started Configure read-only root support. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting udev Wait for Complete Device Initialization... Starting Device-Mapper Multipath Device Controller... [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... %G[ OK ] Started Device-Mapper Multipath Device Controller. %G[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... [ OK ] Reached target Encrypted Volumes. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Dump dmesg to /var/log/dmesg... [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... Starting Service enabling compressing RAM with zRam... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... Starting firewalld - dynamic firewall daemon... Starting pre-anaconda logging service... %G (B                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (B[anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (BStarting installer, one moment...  anaconda 21.48.22.93-1 for CentOS Linux 7 started. * installation log files are stored in /tmp during the installation * shell is available on TTY2 * when reporting a bug add logs from /tmp as separate text/plain attachments06:50:41 Running pre-installation scripts06:50:43 Not asking for VNC because of an automated install06:50:43 Not asking for VNC because text mode was explicitly asked for in kickstartStarting automated install......................Checking software selectionGenerating updated storage configurationChecking storage configuration... ================================================================================ ================================================================================ Installation  1) [x] Language settings 2) [x] Time settings (English (United States)) (Etc/UTC timezone) 3) [x] Installation source 4) [x] Software selection (http://10.9.0.10/cblr/links/Ce (Custom software selected) ntOS-7.3-x86_64) 6) [x] Kdump 5) [x] Installation Destination (Kdump is enabled) (Custom partitioning selected) 8) [ ] User creation 7) [x] Network configuration (No user will be created) (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Anaconda. [ OK ] Stopped target Anaconda System Services. Stopping System Logging Service... Stopping Service enabling compressing RAM with zRam... Stopping Login Service... [ OK ] Stopped Login Service. [ OK ] Stopped System Logging Service. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Stopped target Paths. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped target Swap. Stopping Load/Save Random Seed... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Update UTMP about System Boot/Shutdown. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/run... Unmounting Configuration File System... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... Unmounting /mnt/sysimage/sys/fs/selinux... Unmounting Temporary Directory... Unmounting /mnt/sysimage/dev/shm... Unmounting /mnt/sysimage/proc... [ OK ] Unmounted /mnt/sysimage/dev/pts. [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. [ OK ] Unmounted Temporary Directory. [ OK ] Unmounted /mnt/sysimage/dev/shm. [ OK ] Unmounted Configuration File System. Unmounting /mnt/sysimage/sys... Unmounting /mnt/sysimage/dev... [ OK ] Unmounted /mnt/sysimage/run. [ OK ] Unmounted /mnt/sysimage/proc. [ OK ] Unmounted /mnt/sysimage/sys. [ OK ] Unmounted /mnt/sysimage/dev. Unmounting /mnt/sysimage... [ OK ] Deactivated swap /dev/zram0. [ OK ] Deactivated swap /dev/zram1. [ OK ] Unmounted /mnt/sysimage. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. Stopping LVM2 metadata daemon... [ OK ] Stopped LVM2 metadata daemon. [ OK ] Started Restore /run/initramfs. [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 809.745868] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fce000-0x367defff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694540k/2097140k available (6765k kernel code, 400k absent, 402200k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008190] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011287] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013406] Mount-cache hash table entries: 4096 [ 0.014256] Initializing cgroup subsys memory [ 0.016013] Initializing cgroup subsys devices [ 0.017008] Initializing cgroup subsys freezer [ 0.018005] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.023005] Initializing cgroup subsys pids [ 0.025004] Initializing cgroup subsys net_prio [ 0.026111] Disabled fast string operations [ 0.027020] mce: CPU supports 10 MCE banks [ 0.029035] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] tlb_flushall_shift: 6 [ 0.033048] Freeing SMP alternatives: 28k freed [ 0.038452] ACPI: Core revision 20130517 [ 0.040509] ACPI: All ACPI Tables successfully acquired [ 0.042015] ftrace: allocating 25813 entries in 101 pages [ 0.050195] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.052004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.054003] smpboot: Max logical packages: 2 [ 0.055245] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056005] Switched APIC routing to physical x2apic. [ 0.059274] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.060003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.063025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.065465] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.069032] Brought up 2 CPUs [ 0.069025] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.071005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.072521] devtmpfs: initialized [ 0.074887] EVM: security.selinux [ 0.075005] EVM: security.ima [ 0.076004] EVM: security.capability [ 0.077766] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.078015] pinctrl core: initialized pinctrl subsystem [ 0.079147] NET: Registered protocol family 16 [ 0.080276] ACPI: bus type PCI registered [ 0.081006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.082087] PCI: Using configuration type 1 for base access [ 0.086091] ACPI: Added _OSI(Module Device) [ 0.087013] ACPI: Added _OSI(Processor Device) [ 0.088006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.089005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.091678] ACPI: Interpreter enabled [ 0.092013] ACPI: (supports S0 S5) [ 0.093005] ACPI: Using IOAPIC for interrupt routing [ 0.094017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.097588] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.098010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.099010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.100018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.101243] acpiphp: Slot [1] registered [ 0.102031] acpiphp: Slot [2] registered [ 0.103023] acpiphp: Slot [3] registered [ 0.104022] acpiphp: Slot [4] registered [ 0.105022] acpiphp: Slot [5] registered [ 0.106022] acpiphp: Slot [6] registered [ 0.107022] acpiphp: Slot [7] registered [ 0.108022] acpiphp: Slot [8] registered [ 0.109030] acpiphp: Slot [9] registered [ 0.110023] acpiphp: Slot [10] registered [ 0.111022] acpiphp: Slot [11] registered [ 0.112022] acpiphp: Slot [12] registered [ 0.113022] acpiphp: Slot [13] registered [ 0.114022] acpiphp: Slot [14] registered [ 0.115022] acpiphp: Slot [15] registered [ 0.116029] acpiphp: Slot [16] registered [ 0.117031] acpiphp: Slot [17] registered [ 0.118022] acpiphp: Slot [18] registered [ 0.119022] acpiphp: Slot [19] registered [ 0.120022] acpiphp: Slot [20] registered [ 0.121022] acpiphp: Slot [21] registered [ 0.122022] acpiphp: Slot [22] registered [ 0.123029] acpiphp: Slot [23] registered [ 0.124022] acpiphp: Slot [24] registered [ 0.125022] acpiphp: Slot [25] registered [ 0.126022] acpiphp: Slot [26] registered [ 0.127022] acpiphp: Slot [27] registered [ 0.128029] acpiphp: Slot [28] registered [ 0.129022] acpiphp: Slot [29] registered [ 0.130026] acpiphp: Slot [30] registered [ 0.131029] acpiphp: Slot [31] registered [ 0.132257] PCI host bridge to bus 0000:00 [ 0.133006] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.134006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.135006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.136005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.137006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.139011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.140006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.141005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.142005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.143838] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.144013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.161690] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.163077] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.164830] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.166071] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.167735] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.169226] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.170273] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.171014] vgaarb: loaded [ 0.172006] vgaarb: bridge control possible 0000:00:02.0 [ 0.173221] SCSI subsystem initialized [ 0.174052] ACPI: bus type USB registered [ 0.175037] usbcore: registered new interface driver usbfs [ 0.176017] usbcore: registered new interface driver hub [ 0.177055] usbcore: registered new device driver usb [ 0.178194] PCI: Using ACPI for IRQ routing [ 0.179343] NetLabel: Initializing [ 0.180007] NetLabel: domain hash size = 128 [ 0.181004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.182041] NetLabel: unlabeled traffic allowed by default [ 0.183147] Switched to clocksource kvm-clock [ 0.192757] pnp: PnP ACPI init [ 0.193850] ACPI: bus type PNP registered [ 0.195559] pnp: PnP ACPI: found 5 devices [ 0.196769] ACPI: bus type PNP unregistered [ 0.207869] NET: Registered protocol family 2 [ 0.209493] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.211799] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.213447] TCP: Hash tables configured (established 16384 bind 16384) [ 0.215025] TCP: reno registered [ 0.216111] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.217587] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.219194] NET: Registered protocol family 1 [ 0.220567] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.222094] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.223604] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.225427] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.227270] Unpacking initramfs... [ 0.632546] Freeing initrd memory: 24644k freed [ 0.648216] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.649977] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.651975] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.653601] Initialise system trusted keyring [ 0.654997] audit: initializing netlink socket (disabled) [ 0.656513] type=2000 audit(1491548659.824:1): initialized [ 0.677744] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.679392] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.682153] zpool: loaded [ 0.683298] zbud: loaded [ 0.684630] VFS: Disk quotas dquot_6.5.2 [ 0.685981] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.687837] msgmni has been set to 3357 [ 0.689227] Key type big_key registered [ 0.692122] NET: Registered protocol family 38 [ 0.693487] Key type asymmetric registered [ 0.694774] Asymmetric key parser 'x509' registered [ 0.696244] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.698757] io scheduler noop registered [ 0.700090] io scheduler deadline registered (default) [ 0.701560] io scheduler cfq registered [ 0.702946] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.704473] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.706281] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.708721] ACPI: Power Button [PWRF] [ 0.710177] GHES: HEST is not enabled! [ 0.711544] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.734607] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.736580] Non-volatile memory driver v1.3 [ 0.737932] Linux agpgart interface v0.103 [ 0.739399] crash memory driver: version 1.1 [ 0.740823] rdac: device handler registered [ 0.742244] hp_sw: device handler registered [ 0.759968] emc: device handler registered [ 0.761276] alua: device handler registered [ 0.762612] libphy: Fixed MDIO Bus: probed [ 0.763953] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.765566] ehci-pci: EHCI PCI platform driver [ 0.766907] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.768465] ohci-pci: OHCI PCI platform driver [ 0.769809] uhci_hcd: USB Universal Host Controller Interface driver [ 0.771623] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.773100] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.775513] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.776986] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.778587] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.780348] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.782795] usb usb1: Product: UHCI Host Controller [ 0.784224] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.786727] usb usb1: SerialNumber: 0000:00:01.2 [ 0.788241] hub 1-0:1.0: USB hub found [ 0.789586] hub 1-0:1.0: 2 ports detected [ 0.791088] usbcore: registered new interface driver usbserial [ 0.792693] usbcore: registered new interface driver usbserial_generic [ 0.794341] usbserial: USB Serial support registered for generic [ 0.796021] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.799199] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.800544] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.802113] mousedev: PS/2 mouse device common for all mice [ 0.803891] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.806574] rtc_cmos 00:00: RTC can wake from S4 [ 0.807370] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.811463] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.811831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.815949] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.817665] cpuidle: using governor menu [ 0.819125] hidraw: raw HID events driver (C) Jiri Kosina [ 0.820733] usbcore: registered new interface driver usbhid [ 0.822257] usbhid: USB HID core driver [ 0.823584] drop_monitor: Initializing network drop monitor service [ 0.825267] TCP: cubic registered [ 0.826485] Initializing XFRM netlink socket [ 0.827947] NET: Registered protocol family 10 [ 0.829655] NET: Registered protocol family 17 [ 0.831233] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.833140] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.834734] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.837487] Loading compiled-in X.509 certificates [ 0.838979] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.841748] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.845059] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.847849] registered taskstats version 1 [ 0.851866] Key type trusted registered [ 0.855360] Key type encrypted registered [ 0.858836] IMA: No TPM chip found, activating TPM-bypass! [ 0.860732] rtc_cmos 00:00: setting system clock to 2017-04-07 07:04:18 UTC (1491548658) [ 0.864232] Freeing unused kernel memory: 1680k freed [ 0.870783] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.875191] systemd[1]: Detected virtualization kvm. [ 0.876569] systemd[1]: Detected architecture x86-64. [ 0.877931] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.880240] systemd[1]: Set hostname to . [ OK ] Reached target Swap. [ 0.914733] systemd[1]: Reached target Swap. [ 0.916033] systemd[1]: Starting Swap. [ OK ] Created slice Root Slice. [ 0.917905] systemd[1]: Created slice Root Slice. [ 0.919234] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 0.921240] systemd[1]: Listening on udev Kernel Socket. [ 0.922603] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on Journal Socket. [ 0.924598] systemd[1]: Listening on Journal Socket. [ 0.925897] systemd[1]: Starting Journal Socket. [ OK ] Reached target Local File Systems. [ 0.927835] systemd[1]: Reached target Local File Systems. [ 0.929262] systemd[1]: Starting Local File Systems. [ OK ] Listening on udev Control Socket. [ 0.931265] systemd[1]: Listening on udev Control Socket. [ 0.932649] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Timers. [ 0.934559] systemd[1]: Reached target Timers. [ 0.935832] systemd[1]: Starting Timers. [ OK ] Created slice System Slice. [ 0.937700] systemd[1]: Created slice System Slice. [ 0.939038] systemd[1]: Starting System Slice. [ 0.940840] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Slices. [ 0.945760] systemd[1]: Reached target Slices. [ 0.947073] systemd[1]: Starting Slices. [ 0.948899] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 0.952733] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.956557] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 0.960280] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ 0.971079] systemd[1]: Reached target Sockets. [ 0.973172] systemd[1]: Starting Sockets. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 0.977380] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ 1.108815] systemd[1]: Started Apply Kernel Variables. [ OK ] Started dracut cmdline hook. [ 1.110957] systemd[1]: Started dracut cmdline hook. [ OK ] Started Journal Service. [ 1.113084] systemd[1]: Started Journal Service. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ 1.191345] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.196207] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.198084] FDC 0 is a S82078B [ 1.200195] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.201753] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.207526] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.207987] [drm] Initialized drm 1.1.0 20060810 [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. Starting dracut initqueue hook... [ 1.215274] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.223027] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.233094] scsi host0: ata_piix [ 1.252927] scsi host1: ata_piix [ 1.254161] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.255810] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.261190] vda: vda1 vda2 [ 1.263650] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.265176] [TTM] Initializing pool allocator [ 1.266665] [TTM] Initializing DMA pool allocator %G[ 1.284268] [drm] fb mappable at 0xF0000000 [ 1.284268] [drm] vram aper at 0xF0000000 [ 1.284269] [drm] size 33554432 [ 1.284269] [drm] fb depth is 24 [ 1.284269] [drm] pitch is 3072 [ 1.345466] fbcon: cirrusdrmfb (fb0) is primary device [ 1.375723] Console: switching to colour frame buffer device 128x48 [ 1.387875] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Found device /dev/disk/by-uuid/7af240cf-d87a-4e40-aa0e-730b1006c39f. Starting File System Check on /dev/...f-d87a-4e40-aa0e-730b1006c39f... %G[ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.394039] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d...0cf-d87a-4e40-aa0e-730b1006c39f. Mounting /sysroot... [ 1.496377] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.497941] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 1.649046] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.877443] systemd-journald[97]: Received SIGTERM from PID 1 (systemd). [ 2.130809] SELinux: Disabled at runtime. [ 2.169025] type=1404 audit(1491548659.808:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.175798] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.177299] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Create list of required st... nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Listening on Device-mapper event daemon FIFOs. Mounting POSIX Message Queue File System... [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Stopped target Initrd Root File System. Mounting Debug File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Remote File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Mounted Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ 3.118914] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting Load/Save Random Seed... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Rebuild Hardware Database... [ 3.142700] systemd-journald[345]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Load/Save Random Seed. [ OK ] Started Configure read-only root support. [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ 4.271977] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 4.277237] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. Activating swap /dev/disk/by-uuid/d...6-54b7-4dec-ad18-326a879eb2d7... %G%G[ 4.315141] ppdev: user-space parallel port driver [ 4.325407] AES CTR mode by8 optimization enabled [ 4.330823] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 4.339604] alg: No test for crc32 (crc32-pclmul) [ 4.358955] intel_rapl: no valid rapl domains found in package 0 [ 4.360182] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Activated swap /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Rebuild Journal Catalog... [ 4.390433] intel_rapl: no valid rapl domains found in package 0 Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Mark the need to relabel after reboot... [ 4.399085] intel_powerclamp: No package C-state available[ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ 4.412714] intel_powerclamp: No package C-state available[ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ OK ] Started Update is Completed. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. Starting Permit User Sessions... Starting Authorization Manager... Starting OpenSSH Server Key Generation... Starting Dump dmesg to /var/log/dmesg... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ 4.623734] type=1305 audit(1491548662.262:3): audit_pid=451 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting NTP client/server... [ OK ] Started Permit User Sessions. [ OK ] Started Dump dmesg to /var/log/dmesg. Starting Terminate Plymouth Boot Screen... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ OK ] Started Login Service. [ 6.098511] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 6.100567] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-07T07:04:36+00:00] WARN: ***************************************** [2017-04-07T07:04:36+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-07T07:04:36+00:00] WARN: ***************************************** [2017-04-07T07:04:42+00:00] INFO: Forking chef instance to converge... [2017-04-07T07:04:43+00:00] INFO: *** Chef 12.4.3 *** [2017-04-07T07:04:43+00:00] INFO: Chef-client pid: 7285 [2017-04-07T07:04:44+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-07T07:04:44+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-07T07:04:44+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-07T07:04:44+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-07T07:04:44+00:00] INFO: Running start handlers [2017-04-07T07:04:44+00:00] INFO: Start handlers complete. [2017-04-07T07:04:45+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-07T07:04:45+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-07T07:04:45+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-07T07:04:45+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-07T07:04:45+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170407070445.180187 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170407070445.183184 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170407070445.186875 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170407070445.190689 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170407070445.194652 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170407070445.198562 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-07T07:04:45+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170407070445.202454 [2017-04-07T07:04:45+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-07T07:04:45+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-07T07:04:45+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170407070445.211641 [2017-04-07T07:04:45+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-07T07:04:45+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-07T07:04:52+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-07T07:04:52+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170407070452.409731 [2017-04-07T07:04:52+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-07T07:04:52+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-07T07:04:52+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-07T07:05:40+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-07T07:05:40+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-07T07:05:40+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-07T07:05:56+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-07T07:05:56+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-07T07:05:56+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-07T07:05:56+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-07T07:05:56+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-07T07:05:56+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-07T07:05:56+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-07T07:05:56+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-07T07:05:56+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-07T07:05:56+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170407070556.918981 [2017-04-07T07:05:56+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-07T07:05:56+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-07T07:05:56+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-07T07:05:57+00:00] INFO: service[sssd] restarted [2017-04-07T07:05:57+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-07T07:05:57+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:05:57+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:05:57+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-07T07:05:57+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-07T07:05:57+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-07T07:05:57+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-07T07:05:58+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-07T07:05:58+00:00] INFO: service[ntpd] enabled [2017-04-07T07:05:58+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-07T07:05:58+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170407070558.403320 [2017-04-07T07:05:58+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-07T07:05:58+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-07T07:05:58+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-07T07:05:58+00:00] INFO: service[ntpd] restarted [2017-04-07T07:05:58+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-07T07:05:58+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170407070558.706452 [2017-04-07T07:05:58+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-07T07:05:58+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-07T07:05:58+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-07T07:05:58+00:00] INFO: service[ntpd] restarted [2017-04-07T07:05:58+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-07T07:05:58+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-07T07:05:58+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-07T07:05:59+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-07T07:05:59+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-07T07:05:59+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170407070559.513280 [2017-04-07T07:05:59+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-07T07:05:59+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-07T07:05:59+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-07T07:05:59+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170407070559.520469 [2017-04-07T07:05:59+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-07T07:05:59+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-07T07:05:59+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T07:05:59+00:00] INFO: service[autofs] restarted [2017-04-07T07:05:59+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-07T07:05:59+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170407070559.823715 [2017-04-07T07:05:59+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-07T07:05:59+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-07T07:05:59+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T07:06:00+00:00] INFO: service[autofs] restarted [2017-04-07T07:06:00+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-07T07:06:00+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-07T07:06:00+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-07T07:06:00+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T07:06:01+00:00] INFO: service[autofs] restarted [2017-04-07T07:06:01+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-07T07:06:01+00:00] INFO: service[autofs] enabled [2017-04-07T07:06:01+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-07T07:06:01+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-07T07:06:01+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-07T07:06:01+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-07T07:06:02+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-07T07:06:02+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170407070602.159744 [2017-04-07T07:06:02+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-07T07:06:02+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-07T07:06:02+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170407070602.166041 [2017-04-07T07:06:02+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-07T07:06:02+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-07T07:06:02+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-07T07:06:02+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-07T07:06:02+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-07T07:06:02+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170407070602.174865 [2017-04-07T07:06:02+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-07T07:06:02+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-07T07:06:02+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-07T07:06:04+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-07T07:06:04+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-07T07:06:04+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-07T07:06:04+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-07T07:06:04+00:00] INFO: service[munge] restarted [2017-04-07T07:06:04+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-07T07:06:04+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-07T07:06:04+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-07T07:06:05+00:00] INFO: service[munge] restarted [2017-04-07T07:06:05+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-07T07:06:05+00:00] INFO: service[munge] enabled [2017-04-07T07:06:05+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-07T07:06:05+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-07T07:06:05+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T07:06:09+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-07T07:06:09+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-07T07:06:09+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T07:06:11+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-07T07:06:11+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170407070611.713708 [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-07T07:06:11+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170407070611.717745 [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-07T07:06:11+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-07T07:06:11+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-07T07:06:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:11+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-07T07:06:11+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-07T07:06:11+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:11+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-07T07:06:11+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-07T07:06:12+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-07T07:06:12+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T07:06:12+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:12+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-07T07:06:12+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-07T07:06:12+00:00] INFO: service[xinetd] started [2017-04-07T07:06:12+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-07T07:06:12+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-07T07:06:13+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-07T07:06:13+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-07T07:06:13+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-07T07:06:13+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-07T07:06:13+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170407070613.813004 [2017-04-07T07:06:13+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-07T07:06:13+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-07T07:06:13+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-07T07:06:13+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-07T07:06:13+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-07T07:06:13+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-07T07:06:13+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-07T07:06:13+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-07T07:06:13+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-07T07:06:13+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-07T07:06:13+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-07T07:06:13+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-07T07:06:13+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-07T07:06:13+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-07T07:06:13+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-07T07:06:13+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-07T07:06:13+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:13+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-07T07:06:13+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-07T07:06:13+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-07T07:06:15+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-07T07:06:15+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-07T07:06:16+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-07T07:06:18+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-07T07:06:18+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-07T07:06:18+00:00] INFO: service[rsh.socket] enabled [2017-04-07T07:06:18+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-07T07:06:18+00:00] INFO: service[rsh.socket] started [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-07T07:06:18+00:00] INFO: service[rlogin.socket] enabled [2017-04-07T07:06:18+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-07T07:06:18+00:00] INFO: service[rlogin.socket] started [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T07:06:18+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T07:06:18+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T07:06:18+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-07T07:06:18+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-07T07:06:18+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-07T07:06:18+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-07T07:06:18+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-07T07:06:19+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-07T07:06:19+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-07T07:06:19+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-07T07:06:19+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-07T07:06:19+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-07T07:06:19+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-07T07:06:21+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-07T07:06:21+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-07T07:06:21+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-07T07:06:21+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-07T07:06:21+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-07T07:06:21+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-07T07:06:23+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-07T07:06:23+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-07T07:06:23+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.01.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T07:06:27+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.01.wc1 [2017-04-07T07:06:27+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-07T07:06:27+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-07T07:06:27+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-07T07:06:27+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-07T07:06:28+00:00] INFO: service[rdma] enabled [2017-04-07T07:06:28+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 129.159457] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 129.168574] Loading iSCSI transport class v2.0-870. [ 129.172120] iscsi: registered transport (iser) [ 129.194806] RPC: Registered named UNIX socket transport module. [ 129.195389] RPC: Registered udp transport module. [ 129.195811] RPC: Registered tcp transport module. [ 129.196241] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 129.198811] RPC: Registered rdma transport module. [ 129.199279] RPC: Registered rdma backchannel transport module. [2017-04-07T07:06:28+00:00] INFO: service[rdma] started [2017-04-07T07:06:28+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-07T07:06:28+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-07T07:06:28+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 129.282345] FS-Cache: Loaded [ 129.291519] FS-Cache: Netfs 'nfs' registered for caching [ 129.294598] Key type dns_resolver registered [ 129.303101] NFS: Registering the id_resolver key type [ 129.303575] Key type id_resolver registered [ 129.303944] Key type id_legacy registered [2017-04-07T07:06:28+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-07T07:06:28+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-07T07:06:28+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170407070628.874324 [2017-04-07T07:06:28+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-07T07:06:28+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-07T07:06:28+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-07T07:07:05+00:00] INFO: service[kdump] restarted [2017-04-07T07:07:05+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-07T07:07:05+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-07T07:07:05+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-07T07:07:05+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-07T07:07:05+00:00] INFO: group[quota_usr] created [2017-04-07T07:07:05+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-07T07:07:05+00:00] INFO: user[quota_usr] created [2017-04-07T07:07:05+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-07T07:07:05+00:00] INFO: group[quota_2usr] created [2017-04-07T07:07:05+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-07T07:07:05+00:00] INFO: user[quota_2usr] created [2017-04-07T07:07:05+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-07T07:07:05+00:00] INFO: group[sanityusr] created [2017-04-07T07:07:05+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-07T07:07:05+00:00] INFO: user[sanityusr] created [2017-04-07T07:07:05+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-07T07:07:05+00:00] INFO: group[sanityusr1] created [2017-04-07T07:07:05+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-07T07:07:05+00:00] INFO: user[sanityusr1] created [2017-04-07T07:07:05+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_1] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_2] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_3] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_4] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_5] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_6] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_7] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_8] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_9] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_10] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_11] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_12] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_13] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_14] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_15] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_16] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_17] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_18] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_19] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_20] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_21] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_22] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_23] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_24] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_25] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-07T07:07:06+00:00] INFO: user[quota15_26] created [2017-04-07T07:07:06+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-07T07:07:07+00:00] INFO: user[quota15_27] created [2017-04-07T07:07:07+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-07T07:07:07+00:00] INFO: user[quota15_28] created [2017-04-07T07:07:07+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-07T07:07:07+00:00] INFO: user[quota15_29] created [2017-04-07T07:07:07+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-07T07:07:07+00:00] INFO: user[quota15_30] created [2017-04-07T07:07:07+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-07T07:07:07+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-07T07:07:07+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-07T07:07:07+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-07T07:07:07+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-07T07:07:07+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-07T07:07:07+00:00] INFO: bash[set_hostid] ran successfully [2017-04-07T07:07:07+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-07T07:07:07+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-07T07:07:07+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-07T07:07:07+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-07T07:07:07+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-07T07:07:07+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-07T07:07:07+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-07T07:07:07+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-07T07:07:07+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-07T07:07:07+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-07T07:07:07+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-07T07:07:07+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-07T07:07:07+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-07T07:07:07+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-07T07:07:07+00:00] INFO: user[mpiuser] created [2017-04-07T07:07:07+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-07T07:07:07+00:00] INFO: group[mpiuser] created [2017-04-07T07:07:07+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-07T07:07:07+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-07T07:07:07+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-07T07:07:07+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-07T07:07:07+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-07T07:07:07+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-07T07:07:07+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-07T07:07:07+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-07T07:07:08+00:00] INFO: service[postfix] restarted [2017-04-07T07:07:08+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-07T07:07:08+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-07T07:07:08+00:00] INFO: execute[newaliases] ran successfully [2017-04-07T07:07:08+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-07T07:07:08+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-07T07:07:08+00:00] INFO: service[sshd] restarted [2017-04-07T07:07:08+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-07T07:07:08+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-07T07:07:08+00:00] INFO: service[rsyslog] restarted [2017-04-07T07:07:08+00:00] INFO: Chef Run complete in 144.193201908 seconds [2017-04-07T07:07:08+00:00] INFO: Skipping removal of unused files from the cache [2017-04-07T07:07:08+00:00] INFO: Running report handlers [2017-04-07T07:07:08+00:00] INFO: Creating JSON run report [2017-04-07T07:07:08+00:00] INFO: Report handlers complete [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Unmounted /scratch. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Stopped Automounts filesystems on demand. [ OK ] Stopped target Remote File Systems. Stopping System Security Services Daemon... [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped System Security Services Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... Stopping Load/Save Random Seed... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped target Swap. Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ 2038.117331] type=1305 audit(1491550697.637:1063): audit_pid=0 old=451 auid=4294967295 ses=4294967295 res=1 [ 2038.118270] type=1130 audit(1491550697.638:1064): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2038.119850] type=1131 audit(1491550697.639:1065): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Security Auditing Service. [ OK ] Stopped Create V[ 2038.125245] type=1130 audit(1491550697.645:1066): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' olatile Files and Directories. [ 2038.128934] type=1131 audit(1491550697.649:1067): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Create Volatile Files and Directories... [ 2038.134551] type=1130 audit(1491550697.654:1068): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2038.134561] type=1131 audit(1491550697.654:1069): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs...[ 2038.146445] type=1130 audit(1491550697.666:1070): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped target Local File Systems. Unmounting /run/user/0... [ OK ] Stopped Config[ 2038.152953] type=1131 audit(1491550697.672:1071): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ure read-only root support. [ 2038.165278] type=1130 audit(1491550697.685:1072): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Configure read-only root support... [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. [ 2038.183827] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 2038.188354] systemd-journald[345]: Received SIGTERM from PID 1 (systemd-shutdow). Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Reached target Shutdown. [ 2038.195327] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 2038.198193] systemd-shutdown[1]: Unmounting file systems. [ 2038.243218] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2038.260750] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2038.262980] systemd-shutdown[1]: All filesystems unmounted. [ 2038.265288] systemd-shutdown[1]: Deactivating swaps. [ 2038.267576] systemd-shutdown[1]: All swaps deactivated. [ 2038.269787] systemd-shutdown[1]: Detaching loop devices. [ 2038.272549] systemd-shutdown[1]: All loop devices detached. [ 2038.274742] systemd-shutdown[1]: Detaching DM devices. [ 2038.276911] systemd-shutdown[1]: All DM devices detached. [ 2038.279956] systemd-shutdown[1]: Rebooting. [ 2038.282797] Restarting system. [ 2038.284633] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fce000-0x367defff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694540k/2097140k available (6765k kernel code, 400k absent, 402200k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008191] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011631] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013531] Mount-cache hash table entries: 4096 [ 0.015253] Initializing cgroup subsys memory [ 0.016013] Initializing cgroup subsys devices [ 0.017005] Initializing cgroup subsys freezer [ 0.019004] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.022006] Initializing cgroup subsys hugetlb [ 0.024004] Initializing cgroup subsys pids [ 0.025005] Initializing cgroup subsys net_prio [ 0.026106] Disabled fast string operations [ 0.028018] mce: CPU supports 10 MCE banks [ 0.029060] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] tlb_flushall_shift: 6 [ 0.033180] Freeing SMP alternatives: 28k freed [ 0.038826] ACPI: Core revision 20130517 [ 0.040853] ACPI: All ACPI Tables successfully acquired [ 0.042016] ftrace: allocating 25813 entries in 101 pages [ 0.050202] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.052004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.053004] smpboot: Max logical packages: 2 [ 0.055104] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056004] Switched APIC routing to physical x2apic. [ 0.058903] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.060002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.063025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.065467] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.069029] Brought up 2 CPUs [ 0.069024] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.071005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.072523] devtmpfs: initialized [ 0.075384] EVM: security.selinux [ 0.076005] EVM: security.ima [ 0.077004] EVM: security.capability [ 0.078761] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.079010] pinctrl core: initialized pinctrl subsystem [ 0.080148] NET: Registered protocol family 16 [ 0.081277] ACPI: bus type PCI registered [ 0.082006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.083088] PCI: Using configuration type 1 for base access [ 0.087087] ACPI: Added _OSI(Module Device) [ 0.088013] ACPI: Added _OSI(Processor Device) [ 0.089006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.090009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.092676] ACPI: Interpreter enabled [ 0.093013] ACPI: (supports S0 S5) [ 0.094005] ACPI: Using IOAPIC for interrupt routing [ 0.095016] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.098597] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.099010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.100009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.101017] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.102244] acpiphp: Slot [1] registered [ 0.103031] acpiphp: Slot [2] registered [ 0.104023] acpiphp: Slot [3] registered [ 0.105022] acpiphp: Slot [4] registered [ 0.106022] acpiphp: Slot [5] registered [ 0.107022] acpiphp: Slot [6] registered [ 0.108022] acpiphp: Slot [7] registered [ 0.109022] acpiphp: Slot [8] registered [ 0.110029] acpiphp: Slot [9] registered [ 0.111023] acpiphp: Slot [10] registered [ 0.112022] acpiphp: Slot [11] registered [ 0.113022] acpiphp: Slot [12] registered [ 0.114021] acpiphp: Slot [13] registered [ 0.115022] acpiphp: Slot [14] registered [ 0.116022] acpiphp: Slot [15] registered [ 0.117029] acpiphp: Slot [16] registered [ 0.118029] acpiphp: Slot [17] registered [ 0.119022] acpiphp: Slot [18] registered [ 0.120022] acpiphp: Slot [19] registered [ 0.121022] acpiphp: Slot [20] registered [ 0.122029] acpiphp: Slot [21] registered [ 0.123022] acpiphp: Slot [22] registered [ 0.124029] acpiphp: Slot [23] registered [ 0.125022] acpiphp: Slot [24] registered [ 0.126022] acpiphp: Slot [25] registered [ 0.127022] acpiphp: Slot [26] registered [ 0.128022] acpiphp: Slot [27] registered [ 0.129029] acpiphp: Slot [28] registered [ 0.130022] acpiphp: Slot [29] registered [ 0.131022] acpiphp: Slot [30] registered [ 0.132029] acpiphp: Slot [31] registered [ 0.133256] PCI host bridge to bus 0000:00 [ 0.134007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.135006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.136006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.137005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.138005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.140005] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.141006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.142005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.143005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.144834] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.145013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.164690] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.166077] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.167849] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.169071] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.170818] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.172244] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.173279] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.174013] vgaarb: loaded [ 0.175005] vgaarb: bridge control possible 0000:00:02.0 [ 0.176204] SCSI subsystem initialized [ 0.177052] ACPI: bus type USB registered [ 0.178037] usbcore: registered new interface driver usbfs [ 0.179017] usbcore: registered new interface driver hub [ 0.180049] usbcore: registered new device driver usb [ 0.181181] PCI: Using ACPI for IRQ routing [ 0.182345] NetLabel: Initializing [ 0.183007] NetLabel: domain hash size = 128 [ 0.184004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.185041] NetLabel: unlabeled traffic allowed by default [ 0.186145] Switched to clocksource kvm-clock [ 0.195747] pnp: PnP ACPI init [ 0.196849] ACPI: bus type PNP registered [ 0.198523] pnp: PnP ACPI: found 5 devices [ 0.199741] ACPI: bus type PNP unregistered [ 0.209556] NET: Registered protocol family 2 [ 0.211153] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.213468] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.215128] TCP: Hash tables configured (established 16384 bind 16384) [ 0.216696] TCP: reno registered [ 0.217797] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.219281] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.220925] NET: Registered protocol family 1 [ 0.222294] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.223822] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.225329] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.227151] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.228985] Unpacking initramfs... [ 0.632767] Freeing initrd memory: 24644k freed [ 0.640405] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.642158] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.644113] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.645713] Initialise system trusted keyring [ 0.647072] audit: initializing netlink socket (disabled) [ 0.648537] type=2000 audit(1491550725.459:1): initialized [ 0.669534] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.671188] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.674233] zpool: loaded [ 0.675381] zbud: loaded [ 0.676771] VFS: Disk quotas dquot_6.5.2 [ 0.678165] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.680051] msgmni has been set to 3357 [ 0.681384] Key type big_key registered [ 0.684361] NET: Registered protocol family 38 [ 0.685713] Key type asymmetric registered [ 0.687013] Asymmetric key parser 'x509' registered [ 0.688470] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.690997] io scheduler noop registered [ 0.692317] io scheduler deadline registered (default) [ 0.693784] io scheduler cfq registered [ 0.695197] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.696699] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.698517] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.700941] ACPI: Power Button [PWRF] [ 0.702402] GHES: HEST is not enabled! [ 0.703760] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.726860] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.728898] Non-volatile memory driver v1.3 [ 0.730251] Linux agpgart interface v0.103 [ 0.731715] crash memory driver: version 1.1 [ 0.733154] rdac: device handler registered [ 0.734579] hp_sw: device handler registered [ 0.752431] emc: device handler registered [ 0.753745] alua: device handler registered [ 0.755106] libphy: Fixed MDIO Bus: probed [ 0.756453] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.758147] ehci-pci: EHCI PCI platform driver [ 0.759510] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.761071] ohci-pci: OHCI PCI platform driver [ 0.762420] uhci_hcd: USB Universal Host Controller Interface driver [ 0.764242] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.765718] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.768151] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.769649] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.771295] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.772967] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.775407] usb usb1: Product: UHCI Host Controller [ 0.776837] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.779336] usb usb1: SerialNumber: 0000:00:01.2 [ 0.780852] hub 1-0:1.0: USB hub found [ 0.782148] hub 1-0:1.0: 2 ports detected [ 0.783662] usbcore: registered new interface driver usbserial [ 0.785249] usbcore: registered new interface driver usbserial_generic [ 0.786933] usbserial: USB Serial support registered for generic [ 0.788618] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.791808] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.793165] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.794701] mousedev: PS/2 mouse device common for all mice [ 0.796487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.799239] rtc_cmos 00:00: RTC can wake from S4 [ 0.800062] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.804138] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.804486] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.808632] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.810332] cpuidle: using governor menu [ 0.811801] hidraw: raw HID events driver (C) Jiri Kosina [ 0.813434] usbcore: registered new interface driver usbhid [ 0.814973] usbhid: USB HID core driver [ 0.816298] drop_monitor: Initializing network drop monitor service [ 0.817973] TCP: cubic registered [ 0.819199] Initializing XFRM netlink socket [ 0.820673] NET: Registered protocol family 10 [ 0.822394] NET: Registered protocol family 17 [ 0.823999] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.825923] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.827517] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.830267] Loading compiled-in X.509 certificates [ 0.831738] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.834519] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.837792] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.840586] registered taskstats version 1 [ 0.844712] Key type trusted registered [ 0.848213] Key type encrypted registered [ 0.851640] IMA: No TPM chip found, activating TPM-bypass! [ 0.853425] rtc_cmos 00:00: setting system clock to 2017-04-07 07:38:46 UTC (1491550726) [ 0.856743] Freeing unused kernel memory: 1680k freed [ 0.862793] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.867457] systemd[1]: Detected virtualization kvm. [ 0.868922] systemd[1]: Detected architecture x86-64. [ 0.870411] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.872733] systemd[1]: Set hostname to . [ OK ] Reached target Swap. [ 0.907413] systemd[1]: Reached target Swap. [ 0.908717] systemd[1]: Starting Swap. [ OK ] Reached target Local File Systems. [ 0.910573] systemd[1]: Reached target Local File Systems. [ 0.911993] systemd[1]: Starting Local File Systems. [ OK ] Reached target Timers. [ 0.913905] systemd[1]: Reached target Timers. [ 0.915203] systemd[1]: Starting Timers. [ OK ] Created slice Root Slice. [ 0.917120] systemd[1]: Created slice Root Slice. [ 0.918444] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Control Socket. [ 0.920386] systemd[1]: Listening on udev Control Socket. [ 0.921782] systemd[1]: Starting udev Control Socket. [ OK ] Created slice System Slice. [ 0.923786] systemd[1]: Created slice System Slice. [ 0.925157] systemd[1]: Starting System Slice. [ OK ] Reached target Slices. [ 0.927012] systemd[1]: Reached target Slices. [ 0.928311] systemd[1]: Starting Slices. [ OK ] Listening on Journal Socket. [ 0.930213] systemd[1]: Listening on Journal Socket. [ 0.931567] systemd[1]: Starting Journal Socket. [ 0.933347] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.937390] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 1.027666] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 1.031360] systemd[1]: Starting Journal Service... Starting Journal Service... [ 1.033670] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ 1.036304] systemd[1]: Listening on udev Kernel Socket. [ 1.042172] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 1.044186] systemd[1]: Reached target Sockets. [ 1.045455] systemd[1]: Starting Sockets. [ OK ] Started dracut cmdline hook. [ 1.052054] systemd[1]: Started dracut cmdline hook. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 1.056105] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ 1.061781] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Journal Service. [ 1.063853] systemd[1]: Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut pre-udev hook... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting Show Plymouth Boot Screen... Starting dracut initqueue hook... [ 1.142357] FDC 0 is a S82078B [ 1.151485] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.153363] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.155651] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.157195] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.161042] [drm] Initialized drm 1.1.0 20060810 [ 1.167103] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.171226] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.177831] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.185034] scsi host0: ata_piix [ 1.194065] scsi host1: ata_piix [ 1.195286] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.196829] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 %G[ 1.212660] vda: vda1 vda2 vda4 [ 1.220617] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.220617] [TTM] Initializing pool allocator [ 1.220620] [TTM] Initializing DMA pool allocator [ 1.224020] [drm] fb mappable at 0xF0000000 [ 1.224020] [drm] vram aper at 0xF0000000 [ 1.224021] [drm] size 33554432 [ 1.224021] [drm] fb depth is 24 [ 1.224022] [drm] pitch is 3072 [ 1.282004] fbcon: cirrusdrmfb (fb0) is primary device [ 1.310776] Console: switching to colour frame buffer device 128x48 [ 1.322697] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Found device /dev/disk/by-uuid/7af240cf-d87a-4e40-aa0e-730b1006c39f. %G[ 1.326300] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. Starting File System Check on /dev/...f-d87a-4e40-aa0e-730b1006c39f... [ OK ] Started File System Check on /dev/d...0cf-d87a-4e40-aa0e-730b1006c39f. [ 1.641049] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 1.885287] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.886863] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Plymouth switch root service. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. [ 2.112945] systemd-journald[173]: Received SIGTERM from PID 1 (systemd). Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 2.139928] SELinux: Disabled at runtime. [ 2.178029] type=1404 audit(1491550727.824:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.187304] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.188995] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Created slice User and Session Slice. Mounting Debug File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st... nodes for the current kernel... Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on LVM2 metadata daemon socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ 2.273091] RPC: Registered named UNIX socket transport module. [ 2.273091] RPC: Registered udp transport module. [ 2.273092] RPC: Registered tcp transport module. [ 2.273092] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.311971] EXT4-fs (vda1): re-mounted. Opts: (null) Starting Read and set NIS domainname from /etc/sysconfig/network... Mounting Huge Pages File System... [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Mounting POSIX Message Queue File System... Mounting NFSD configuration filesystem... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on LVM2 poll daemon socket. [ OK ] Mounted Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Apply Kernel Variables. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting Create Static Device Nodes in /dev... [ OK ] Started Remount Root and Kernel File Systems. Starting Load/Save Random Seed... Starting Configure read-only root support... [ 2.340174] systemd-journald[358]: Received request to flush runtime journal from PID 1 Starting udev Coldplug all Devices... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started Load/Save Random Seed. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Configure read-only root support. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 2.412094] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 2.417038] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. Activating swap /dev/disk/by-uuid/d...6-54b7-4dec-ad18-326a879eb2d7... [ 2.456502] AES CTR mode by8 optimization enabled %G%G[ 2.465239] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ 2.466729] ppdev: user-space parallel port driver [ OK ] Activated swap /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. [ OK ] Reached target Swap. [ 2.473091] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 2.492512] alg: No test for crc32 (crc32-pclmul) [ 2.515879] intel_rapl: no valid rapl domains found in package 0 [ 2.542204] intel_rapl: no valid rapl domains found in package 0 [ 2.553145] intel_powerclamp: No package C-state available[ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Import network configuration from initramfs... Starting Tell Plymouth To Write Out Runtime Data... Starting Preprocess NFS configuration... [ OK ] Started Preprocess NFS configuration. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Create Volatile Files and Directories. Mounting RPC Pipe File System... Starting Security Auditing Service... [ OK ] Mounted RPC Pipe File System. [ 2.609541] type=1305 audit(1491550728.255:3): audit_pid=462 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting Network Manager... Starting System Security Services Daemon... Starting NTP client/server... Starting Dump dmesg to /var/log/dmesg... Starting Authorization Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting GSSAPI Proxy Daemon... [ OK ] Reached target Timers. [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ 2.820125] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 2.831958] Loading iSCSI transport class v2.0-870. [ 2.837637] iscsi: registered transport (iser) [ 2.860621] RPC: Registered rdma transport module. [ 2.862100] RPC: Registered rdma backchannel transport module. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Authorization Manager. [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started NTP client/server. [ 3.795962] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ OK ] Mounted NFSD configuration filesystem. Starting Hostname Service... [ OK ] Started Hostname Service. [ 3.836171] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.838325] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Postfix Mail Transport Agent... Starting Dynamic System Tuning Daemon... Starting MUNGE authentication service... Starting Xinetd A Powerful Replacement For Inetd... Starting OpenSSH server daemon... Starting PowerMan... [ OK ] Reached target Network is Online. Mounting /scratch... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ OK ] Started Xinetd A Powerful Replacement For Inetd. [FAILED] Failed to start PowerMan. See 'systemctl status powerman.service' for details. [ OK ] Started OpenSSH server daemon. [ 9.802891] FS-Cache: Loaded [ OK ] Started MUNGE authentication service. [ 9.813530] FS-Cache: Netfs 'nfs' registered for caching [ 9.820779] Key type dns_resolver registered [ 9.836787] NFS: Registering the id_resolver key type [ 9.840167] Key type id_resolver registered [ 9.841528] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Postfix Mail Transport Agent. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Crash recovery kernel arming... Starting Automounts filesystems on demand... [ OK ] Started Permit User Sessions. Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Terminate Plymouth Boot Screen... [ OK ] Started Command Scheduler. Starting Command Scheduler... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [ 297.100514] device-mapper: uevent: version 1.0.3 [ 297.101148] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 527.551653] libcfs: loading out-of-tree module taints kernel. [ 527.552452] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 527.554910] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 527.557328] alg: No test for adler32 (adler32-zlib) [ 527.557877] alg: No test for crc32 (crc32-table) [ 528.416059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 528.593489] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 528.616028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 528.652760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 528.805811] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 528.806528] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 529.752749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 529.894039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 529.909727] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 530.104275] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 532.287928] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 532.527917] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 532.789886] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [ 533.073612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 534.295314] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 534.536566] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 534.780128] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [ 535.034724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 551.404643] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 551.404643] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 551.404643] if [ $running -ne $mpts ]; then [ 551.404643] echo $(hostname) env are INSANE!; [ 551.404643] exit 1; [ 551.404643] fi [ 551.940153] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 551.940153] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 551.940153] if [ $running -ne $mpts ]; then [ 551.940153] echo $(hostname) env are INSANE!; [ 551.940153] exit 1; [ 551.940153] fi [ 552.497691] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 552.497691] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 552.497691] if [ $running -ne $mpts ]; then [ 552.497691] echo $(hostname) env are INSANE!; [ 552.497691] exit 1; [ 552.497691] fi [ 553.383409] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 553.633910] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 553.876858] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 554.200805] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 554.342644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 554.363290] Lustre: Lustre: Build Version: 2.9.54_77_g2fc9ae7 [ 554.383960] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 554.385884] LNet: Accept all, port 7988 [ 554.397202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.421359] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 554.547937] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 554.569935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 554.600169] Lustre: lustre-MDT0000: new disk, initializing [ 554.641103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.650202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 554.819725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 555.071900] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 555.562730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.572880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.726554] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.728079] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 555.870048] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 556.118049] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 556.368655] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 556.619024] Lustre: DEBUG MARKER: sync; sync; sync [ 559.632252] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 562.187752] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 562.443825] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 563.808102] Lustre: MGS: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 563.862988] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [ 563.865482] Lustre: Skipped 4 previous similar messages [ 563.970947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 564.864430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 565.021528] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 571.957740] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 572.217397] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 572.468473] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 572.716509] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 572.872153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 572.915337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.935352] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [ 572.938096] Lustre: Skipped 4 previous similar messages [ 572.945960] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [ 572.948562] Lustre: Skipped 1 previous similar message [ 573.021793] Lustre: lustre-MDT0002: new disk, initializing [ 573.034741] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 573.037482] Lustre: Skipped 2 previous similar messages [ 573.044786] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 573.054624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [ 573.057718] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [ 573.218097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 573.476469] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 573.962046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 573.964658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 574.126558] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 574.129741] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 574.270868] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 574.521047] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 574.773348] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 575.025756] Lustre: DEBUG MARKER: sync; sync; sync [ 578.334125] Lustre: lustre-MDT0002: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 578.337040] Lustre: Skipped 2 previous similar messages [ 580.352102] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 580.602431] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 581.877699] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [ 581.880478] Lustre: Skipped 4 previous similar messages [ 582.217375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [ 583.128199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 583.289387] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 591.238660] Lustre: MGS: Connection restored to f0cf3274-14e6-ca46-29f0-6da151222f4c (at 10.9.5.244@tcp) [ 591.241650] Lustre: Skipped 3 previous similar messages [ 592.404976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 592.655185] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 594.956966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [ 601.136057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 601.297305] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 604.202765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:1:ost [ 608.715112] Lustre: lustre-MDT0000: Connection restored to f0cf3274-14e6-ca46-29f0-6da151222f4c (at 10.9.5.244@tcp) [ 608.718344] Lustre: Skipped 5 previous similar messages [ 609.647985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 609.809091] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 618.137719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 618.303965] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 626.880871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 627.039793] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 629.035595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000400000400-0x0000000440000400]:4:ost [ 629.038964] Lustre: Skipped 2 previous similar messages [ 636.065901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 636.242740] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 645.174689] Lustre: lustre-MDT0000: Connection restored to f0cf3274-14e6-ca46-29f0-6da151222f4c (at 10.9.5.244@tcp) [ 645.178044] Lustre: Skipped 6 previous similar messages [ 646.241185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 646.396862] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 655.040520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 655.207937] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 662.926054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 663.087352] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 665.069719] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 665.384471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 665.534529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 665.666790] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 665.952577] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 677.574139] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 678.262941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.300641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.379038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.381302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.390030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.422440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir \/logdir\/test_logs\/2017-04-07\/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.611843] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.777240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.814892] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.826332] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.851572] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 678.851587] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing check_logdir /logdir/test_logs/2017-04-07/lustre-reviews-el7-x86_64--custom--1_101_1__46174__-70344545792740-062057 [ 679.968089] Lustre: DEBUG MARKER: uname -n [ 680.525849] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 681.137223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 681.202171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 681.284447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 681.285074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 681.314845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 681.329966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing yml_node [ 681.631896] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing yml_node [ 681.632704] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing yml_node [ 681.649275] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 681.662945] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing yml_node [ 681.733848] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing yml_node [ 681.765514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || /usr/sbin/lctl lustre_build_version 2>/dev/null || /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 681.765733] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing yml_node [ 682.260243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.54_77_g2fc9ae7 [ 682.409183] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.54_77_g2fc9ae7 [ 682.548969] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 682.858513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.54_77_g2fc9ae7 [ 683.009282] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.54_77_g2fc9ae7 [ 683.466717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.54_77_g2fc9ae7 [ 683.623936] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.54_77_g2fc9ae7 [ 683.835404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-single ============----- Fri Apr 7 07:50:08 UTC 2017 [ 683.991522] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Apr 7 07:50:08 UTC 2017 [ 684.359363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test complete, duration -o sec ====================================================== 07:50:09 \(1491551409\) [ 684.514224] Lustre: DEBUG MARKER: == replay-single test complete, duration -o sec ====================================================== 07:50:09 (1491551409) [ 684.897886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 61d [ 685.055388] Lustre: DEBUG MARKER: excepting tests: 61d [ 685.234943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 44b [ 685.392756] Lustre: DEBUG MARKER: skipping tests SLOW=no: 44b [ 686.038525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 686.045560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 686.254147] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 686.254754] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 688.794047] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 688.794047] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 688.794047] if [ $running -ne $mpts ]; then [ 688.794047] echo $(hostname) env are INSANE!; [ 688.794047] exit 1; [ 688.794047] fi [ 689.378312] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 689.378312] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 689.378312] if [ $running -ne $mpts ]; then [ 689.378312] echo $(hostname) env are INSANE!; [ 689.378312] exit 1; [ 689.378312] fi [ 689.971354] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 689.971354] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 689.971354] if [ $running -ne $mpts ]; then [ 689.971354] echo $(hostname) env are INSANE!; [ 689.971354] exit 1; [ 689.971354] fi [ 690.841953] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 691.099819] Lustre: DEBUG MARKER: cat /proc/mounts [ 691.893861] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 692.150990] Lustre: DEBUG MARKER: cat /proc/mounts [ 697.270716] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 697.596434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 697.749429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 697.888529] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 698.232330] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 698.805093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 698.858575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 698.906145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 698.941794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 698.986530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.155648] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.271866] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.286172] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.288429] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.348278] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 699.603453] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 699.878971] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 700.390453] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 700.985133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0a: empty replay =============================================================== 07:50:25 \(1491551425\) [ 701.167246] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =============================================================== 07:50:25 (1491551425) [ 701.305579] Lustre: DEBUG MARKER: sync; sync; sync [ 702.305962] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 702.562125] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 702.704286] LustreError: 13151:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 702.748236] Turning device dm-0 (0xfc00000) read-only [ 702.880062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 703.011479] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 703.152900] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 703.406219] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 703.539042] Lustre: Failing over lustre-MDT0000 [ 703.601223] Removing read-only on unknown block (0xfc00000) [ 703.608475] Lustre: server umount lustre-MDT0000 complete [ 703.737424] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 703.826344] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.898221] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 703.901222] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.046831] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.052667] LustreError: Skipped 9 previous similar messages [ 709.034721] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.040623] LustreError: Skipped 1 previous similar message [ 710.906050] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551429/real 1491551429] req@ffff88005f598600 x1564004874995632/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551436 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 710.914243] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.669270] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.674762] LustreError: Skipped 10 previous similar messages [ 714.109049] Lustre: DEBUG MARKER: hostname [ 714.417078] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 714.669643] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 714.921509] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 715.078853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.511359] Lustre: MGS: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 715.514159] Lustre: Skipped 9 previous similar messages [ 715.837065] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.842518] LustreError: Skipped 7 previous similar messages [ 716.919043] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551436/real 1491551436] req@ffff88005f598f00 x1564004874995984/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491551442 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 716.926993] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b5f1a to 0x52ac6ca1757b7007 [ 716.989172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.140618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 717.399255] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 717.893654] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 717.901876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 718.062470] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 718.062477] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 718.212553] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 718.467658] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 718.670932] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 718.725368] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 718.985452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 719.737057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.750944] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.971347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.976911] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 721.985298] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 722.004635] Lustre: lustre-MDT0000: Recovery over after 0:03, of 5 clients 5 recovered and 0 were evicted. [ 722.263125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 722.267605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 722.466109] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 722.500570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 723.875511] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 724.779063] Lustre: DEBUG MARKER: rc=0; [ 724.779063] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 724.779063] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 724.779063] echo $(hostname -s): $val; [ 724.779063] rc=$val; [ 724.779063] fi; [ 724.779063] exit $rc [ 725.394067] Lustre: DEBUG MARKER: dmesg [ 725.801849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0b: ensure object created after recover exists. \(3284\) ========================= 07:50:50 \(1491551450\) [ 725.965722] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================= 07:50:50 (1491551450) [ 727.039900] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 727.042842] LustreError: Skipped 1 previous similar message [ 727.045267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 727.050490] Lustre: Skipped 1 previous similar message [ 730.191503] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 730.194396] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 739.005325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 739.178432] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 740.665293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 740.665294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 740.876346] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 740.876347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 744.136351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 744.163513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 744.299235] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 744.419064] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 745.813516] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 746.696243] Lustre: DEBUG MARKER: rc=0; [ 746.696243] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 746.696243] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 746.696243] echo $(hostname -s): $val; [ 746.696243] rc=$val; [ 746.696243] fi; [ 746.696243] exit $rc [ 747.298475] Lustre: DEBUG MARKER: dmesg [ 747.672760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0c: check replay-barrier ======================================================= 07:51:12 \(1491551472\) [ 747.840696] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier ======================================================= 07:51:12 (1491551472) [ 747.975066] Lustre: DEBUG MARKER: sync; sync; sync [ 748.804730] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 749.062358] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 749.204263] LustreError: 16055:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 749.238926] Turning device dm-0 (0xfc00000) read-only [ 749.368824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 749.498992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 749.668922] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 749.924794] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 750.056405] Lustre: Failing over lustre-MDT0000 [ 750.105918] Removing read-only on unknown block (0xfc00000) [ 750.113087] Lustre: server umount lustre-MDT0000 complete [ 750.240975] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 750.513647] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.518695] LustreError: Skipped 8 previous similar messages [ 751.985180] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 751.987984] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.456060] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551477/real 1491551477] req@ffff88005f1c3600 x1564004875001152/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 760.466460] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.558710] Lustre: DEBUG MARKER: hostname [ 760.864806] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 761.119215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 761.371526] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 761.520571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.471047] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551486/real 1491551486] req@ffff88005f1c3000 x1564004875001536/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491551492 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 766.478296] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b7007 to 0x52ac6ca1757b7d12 [ 766.532188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.705587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 766.962197] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 767.468273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 767.470710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 767.633691] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 767.633693] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 767.782329] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 767.829641] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 768.042096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 768.300056] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 768.559665] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 768.725054] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 771.517155] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 773.724840] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 778.724609] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 783.724430] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 788.724836] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:45 [ 798.724770] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:35 [ 798.730908] Lustre: Skipped 1 previous similar message [ 818.724788] Lustre: lustre-MDT0000: Denying connection for new client 7ad1e149-6cb6-9a42-fb8b-ccdfa5edde25(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:15 [ 818.730926] Lustre: Skipped 3 previous similar messages [ 834.015046] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 834.017999] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 834.437358] Lustre: lustre-MDT0000: Recovery over after 1:07, of 5 clients 4 recovered and 1 was evicted. [ 840.173874] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 841.075131] Lustre: DEBUG MARKER: rc=0; [ 841.075131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 841.075131] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 841.075131] echo $(hostname -s): $val; [ 841.075131] rc=$val; [ 841.075131] fi; [ 841.075131] exit $rc [ 841.674790] Lustre: DEBUG MARKER: dmesg [ 842.072976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 0d: expired recovery with no clients =========================================== 07:52:46 \(1491551566\) [ 842.229783] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients =========================================== 07:52:46 (1491551566) [ 842.367862] Lustre: DEBUG MARKER: sync; sync; sync [ 843.189601] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 843.445551] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 843.578898] LustreError: 18028:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 843.588437] Turning device dm-0 (0xfc00000) read-only [ 843.717297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 843.846252] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 844.018232] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 844.277133] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 844.407379] Lustre: Failing over lustre-MDT0000 [ 844.458551] Removing read-only on unknown block (0xfc00000) [ 844.465913] Lustre: server umount lustre-MDT0000 complete [ 844.599083] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 845.115077] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.120464] LustreError: Skipped 50 previous similar messages [ 846.516208] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 846.519012] LustreError: Skipped 1 previous similar message [ 846.521187] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.525905] Lustre: Skipped 1 previous similar message [ 853.528054] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551572/real 1491551572] req@ffff88005ff3a400 x1564004875009616/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551579 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 853.535162] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.911168] Lustre: DEBUG MARKER: hostname [ 855.210868] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 855.463134] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 855.716220] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 855.876852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.539194] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 859.541590] Lustre: Skipped 56 previous similar messages [ 859.543809] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b7d12 to 0x52ac6ca1757b82fa [ 859.599067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.749878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 860.007963] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 860.508423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 860.509220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 860.671456] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 860.671462] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 860.735464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 860.820969] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 861.078217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 861.332846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 861.592997] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 861.758672] Lustre: lustre-MDT0000: Denying connection for new client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0(at 10.9.5.237@tcp), waiting for 5 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 861.764295] Lustre: Skipped 3 previous similar messages [ 864.595372] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 926.757435] Lustre: lustre-MDT0000: Denying connection for new client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 926.763505] Lustre: Skipped 12 previous similar messages [ 926.943039] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 926.945727] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 926.952072] Lustre: lustre-MDT0000: Recovery over after 1:06, of 5 clients 4 recovered and 1 was evicted. [ 933.212937] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 934.104058] Lustre: DEBUG MARKER: rc=0; [ 934.104058] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 934.104058] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 934.104058] echo $(hostname -s): $val; [ 934.104058] rc=$val; [ 934.104058] fi; [ 934.104058] exit $rc [ 934.700666] Lustre: DEBUG MARKER: dmesg [ 935.080706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 1: simple create =============================================================== 07:54:19 \(1491551659\) [ 935.240577] Lustre: DEBUG MARKER: == replay-single test 1: simple create =============================================================== 07:54:19 (1491551659) [ 935.379977] Lustre: DEBUG MARKER: sync; sync; sync [ 936.215351] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 936.471466] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 936.611456] LustreError: 20005:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 936.651314] Turning device dm-0 (0xfc00000) read-only [ 936.780366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 936.909112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 937.051132] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 937.304696] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 937.434436] Lustre: Failing over lustre-MDT0000 [ 937.483756] Removing read-only on unknown block (0xfc00000) [ 937.490918] Lustre: server umount lustre-MDT0000 complete [ 937.619874] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 939.594197] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 939.596890] LustreError: Skipped 1 previous similar message [ 939.598975] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.603463] Lustre: Skipped 1 previous similar message [ 939.605592] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 939.605594] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 939.605595] LustreError: Skipped 32 previous similar messages [ 946.594046] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551665/real 1491551665] req@ffff88005edf3600 x1564004875017824/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551672 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 946.601199] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 946.603665] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.927738] Lustre: DEBUG MARKER: hostname [ 948.238712] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 948.492721] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 948.744059] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 948.914325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.608261] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b82fa to 0x52ac6ca1757b88d4 [ 952.650686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.805701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 953.030673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 953.072277] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 953.586549] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 953.592344] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 953.752327] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 953.755223] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 953.899069] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 954.157304] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 954.413354] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 954.671723] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 955.415359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.432617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.628823] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.667916] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 957.645341] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 957.664951] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 957.848514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 957.968118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 958.005035] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 958.161633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 959.570931] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 960.465972] Lustre: DEBUG MARKER: rc=0; [ 960.465972] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 960.465972] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 960.465972] echo $(hostname -s): $val; [ 960.465972] rc=$val; [ 960.465972] fi; [ 960.465972] exit $rc [ 961.064368] Lustre: DEBUG MARKER: dmesg [ 961.450859] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 2a: touch ====================================================================== 07:54:46 \(1491551686\) [ 961.616114] Lustre: DEBUG MARKER: == replay-single test 2a: touch ====================================================================== 07:54:46 (1491551686) [ 961.751854] Lustre: DEBUG MARKER: sync; sync; sync [ 962.581902] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 962.838108] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 962.973385] LustreError: 22258:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 962.984931] Turning device dm-0 (0xfc00000) read-only [ 963.118147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 963.247995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 963.393435] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 963.649316] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 963.778019] Lustre: Failing over lustre-MDT0000 [ 963.827715] Removing read-only on unknown block (0xfc00000) [ 963.834879] Lustre: server umount lustre-MDT0000 complete [ 963.963222] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 967.645232] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 967.647960] LustreError: Skipped 1 previous similar message [ 967.650051] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.654603] Lustre: Skipped 1 previous similar message [ 974.282915] Lustre: DEBUG MARKER: hostname [ 974.585080] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 974.836522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 975.088709] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 975.247874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.263952] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.268201] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b88d4 to 0x52ac6ca1757b8ed1 [ 975.307714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.459521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 975.716939] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 976.151795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 976.213126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 976.218121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 976.378036] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 976.379585] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 976.526182] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 976.783431] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 977.039227] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 977.304878] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 978.059147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.061968] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.283937] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.286962] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.304370] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 980.333979] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 980.545657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 980.546060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 980.761059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 980.763527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 982.150932] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 983.046225] Lustre: DEBUG MARKER: rc=0; [ 983.046225] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 983.046225] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 983.046225] echo $(hostname -s): $val; [ 983.046225] rc=$val; [ 983.046225] fi; [ 983.046225] exit $rc [ 983.642862] Lustre: DEBUG MARKER: dmesg [ 984.022477] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 2b: touch ====================================================================== 07:55:08 \(1491551708\) [ 984.187479] Lustre: DEBUG MARKER: == replay-single test 2b: touch ====================================================================== 07:55:08 (1491551708) [ 984.325899] Lustre: DEBUG MARKER: sync; sync; sync [ 985.168071] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 985.425034] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 985.555809] LustreError: 24511:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 985.565840] Turning device dm-0 (0xfc00000) read-only [ 985.693790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 985.822557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 985.967553] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 986.225282] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 986.356301] Lustre: Failing over lustre-MDT0000 [ 986.405631] Removing read-only on unknown block (0xfc00000) [ 986.412818] Lustre: server umount lustre-MDT0000 complete [ 986.541938] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 996.855622] Lustre: DEBUG MARKER: hostname [ 997.155501] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 997.303044] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551715/real 1491551715] req@ffff88005f701b00 x1564004875024032/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551722 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 997.309954] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 997.312384] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.411636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 997.663490] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 997.811221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.316291] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b8ed1 to 0x52ac6ca1757b9545 [ 1003.537152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1003.794790] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1004.148322] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1004.302791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1004.312966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1004.473076] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1004.479875] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1004.622232] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1004.878733] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1005.139485] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1005.403093] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1006.165667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1006.165669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1006.361163] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1006.396267] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1008.358741] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1008.380788] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 1008.613338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1008.718379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1008.764767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1008.919719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1010.251911] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1011.142226] Lustre: DEBUG MARKER: rc=0; [ 1011.142226] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1011.142226] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1011.142226] echo $(hostname -s): $val; [ 1011.142226] rc=$val; [ 1011.142226] fi; [ 1011.142226] exit $rc [ 1011.741320] Lustre: DEBUG MARKER: dmesg [ 1012.221636] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3a: replay failed open\(O_DIRECTORY\) ============================================ 07:55:37 \(1491551737\) [ 1012.385735] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ============================================ 07:55:37 (1491551737) [ 1012.521593] Lustre: DEBUG MARKER: sync; sync; sync [ 1013.413387] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1013.669143] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1013.811527] LustreError: 26770:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1013.845509] Turning device dm-0 (0xfc00000) read-only [ 1013.974451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1014.103546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1014.275949] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1014.531115] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1014.660998] Lustre: Failing over lustre-MDT0000 [ 1014.711113] Removing read-only on unknown block (0xfc00000) [ 1014.718219] Lustre: server umount lustre-MDT0000 complete [ 1014.847314] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1018.358164] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1018.360885] LustreError: Skipped 4 previous similar messages [ 1018.362963] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.367468] Lustre: Skipped 3 previous similar messages [ 1025.161791] Lustre: DEBUG MARKER: hostname [ 1025.468205] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1025.511047] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.722075] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1025.971145] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1026.127700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.515053] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551751/real 1491551751] req@ffff880064f54c00 x1564004875027616/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491551757 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1031.521879] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1031.524366] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757b9545 to 0x52ac6ca1757b9a69 [ 1031.581818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.584317] Lustre: Skipped 1 previous similar message [ 1031.732507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1031.988023] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1032.176542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1032.489878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1032.490664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1032.657148] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1032.658682] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1032.805355] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1033.061216] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1033.317548] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1033.576318] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1034.334833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.352712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.503630] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.601808] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.572685] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1036.592249] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 1036.840417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1036.990635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1037.725908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1037.887522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1039.212285] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1040.107381] Lustre: DEBUG MARKER: rc=0; [ 1040.107381] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1040.107381] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1040.107381] echo $(hostname -s): $val; [ 1040.107381] rc=$val; [ 1040.107381] fi; [ 1040.107381] exit $rc [ 1040.713787] Lustre: DEBUG MARKER: dmesg [ 1041.129653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3b: replay failed open -ENOMEM ================================================= 07:56:05 \(1491551765\) [ 1041.303986] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ================================================= 07:56:05 (1491551765) [ 1041.464562] Lustre: DEBUG MARKER: sync; sync; sync [ 1042.369530] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1042.624200] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1042.770625] Turning device dm-0 (0xfc00000) read-only [ 1042.898702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1043.027939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1043.159625] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000114 [ 1043.291638] Lustre: *** cfs_fail_loc=114, val=0*** [ 1043.418558] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 1043.680668] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1043.934157] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1044.108915] Removing read-only on unknown block (0xfc00000) [ 1044.241701] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1054.557115] Lustre: DEBUG MARKER: hostname [ 1054.855956] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1055.109151] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1055.360673] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1055.509118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.708758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1055.964670] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1056.461467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1056.463131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1056.631157] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1056.639194] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1056.778117] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1057.033949] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1057.287620] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1057.546478] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1058.301369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.303408] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.470321] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.549084] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1061.741726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1061.840826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1061.902199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1062.059903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1063.393106] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1064.274176] Lustre: DEBUG MARKER: rc=0; [ 1064.274176] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1064.274176] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1064.274176] echo $(hostname -s): $val; [ 1064.274176] rc=$val; [ 1064.274176] fi; [ 1064.274176] exit $rc [ 1064.869665] Lustre: DEBUG MARKER: dmesg [ 1065.277710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 3c: replay failed open -ENOMEM ================================================= 07:56:30 \(1491551790\) [ 1065.446694] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ================================================= 07:56:30 (1491551790) [ 1065.590374] Lustre: DEBUG MARKER: sync; sync; sync [ 1066.465724] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1066.738703] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1066.874144] LustreError: 31425:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1066.876856] LustreError: 31425:0:(osd_handler.c:2171:osd_ro()) Skipped 1 previous similar message [ 1066.887233] Turning device dm-0 (0xfc00000) read-only [ 1067.018460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1067.157213] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1067.294390] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000128 [ 1067.435852] Lustre: *** cfs_fail_loc=128, val=0*** [ 1067.569360] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 1067.834279] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1068.184331] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1068.596270] Lustre: Failing over lustre-MDT0000 [ 1068.598349] Lustre: Skipped 1 previous similar message [ 1068.662728] Removing read-only on unknown block (0xfc00000) [ 1068.669727] Lustre: server umount lustre-MDT0000 complete [ 1068.671664] Lustre: Skipped 1 previous similar message [ 1068.803325] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1070.556444] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.560799] LustreError: Skipped 161 previous similar messages [ 1077.943052] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.947118] LustreError: Skipped 1 previous similar message [ 1079.113953] Lustre: DEBUG MARKER: hostname [ 1079.414393] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1079.670279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1079.922308] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1080.070042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.949354] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757ba051 to 0x52ac6ca1757ba601 [ 1083.953197] Lustre: Skipped 1 previous similar message [ 1084.155695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1084.411771] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1084.715155] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1084.717589] Lustre: Skipped 1 previous similar message [ 1084.921929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1084.923230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1085.089004] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1085.091884] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1085.235292] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1085.489871] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1085.746441] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1086.004164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1086.801576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1086.801586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1086.981821] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1086.995134] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1088.990506] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1088.995806] LustreError: Skipped 1 previous similar message [ 1089.000402] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1089.003485] Lustre: Skipped 1 previous similar message [ 1089.245466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1089.285635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1089.495788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1089.549100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1090.939255] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1091.821361] Lustre: DEBUG MARKER: rc=0; [ 1091.821361] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1091.821361] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1091.821361] echo $(hostname -s): $val; [ 1091.821361] rc=$val; [ 1091.821361] fi; [ 1091.821361] exit $rc [ 1092.416505] Lustre: DEBUG MARKER: dmesg [ 1092.815901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 4a: \|x\| 10 open\(O_CREAT\)s ====================================================== 07:56:57 \(1491551817\) [ 1092.977782] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ====================================================== 07:56:57 (1491551817) [ 1093.124940] Lustre: DEBUG MARKER: sync; sync; sync [ 1093.998672] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1094.256443] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1094.404040] Turning device dm-0 (0xfc00000) read-only [ 1094.532248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1094.662852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1094.824380] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1095.083969] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1095.263279] Removing read-only on unknown block (0xfc00000) [ 1095.397426] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1098.990204] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1098.993046] LustreError: Skipped 6 previous similar messages [ 1098.995189] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.999898] Lustre: Skipped 5 previous similar messages [ 1105.708511] Lustre: DEBUG MARKER: hostname [ 1105.863034] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551824/real 1491551824] req@ffff8800786bd800 x1564004875036688/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551831 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1105.870160] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1106.009603] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1106.262361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1106.514174] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1106.670388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.906601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.908941] Lustre: Skipped 2 previous similar messages [ 1112.062248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1112.320447] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1112.817247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1112.817818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1112.982462] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1112.982463] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1113.128903] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1113.385916] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1113.642613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1113.902155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1114.654306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.654689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.851827] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.897499] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1116.902352] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 1116.902355] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 1116.902356] Lustre: Skipped 203 previous similar messages [ 1116.912412] Lustre: Skipped 1 previous similar message [ 1117.186705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1117.236181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1117.391380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1117.461122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1118.799187] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1119.717049] Lustre: DEBUG MARKER: rc=0; [ 1119.717049] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1119.717049] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1119.717049] echo $(hostname -s): $val; [ 1119.717049] rc=$val; [ 1119.717049] fi; [ 1119.717049] exit $rc [ 1120.314349] Lustre: DEBUG MARKER: dmesg [ 1120.695954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 4b: \|x\| rm 10 files ============================================================ 07:57:25 \(1491551845\) [ 1120.849311] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ============================================================ 07:57:25 (1491551845) [ 1121.002985] Lustre: DEBUG MARKER: sync; sync; sync [ 1121.877523] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1122.152347] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1122.304340] Turning device dm-0 (0xfc00000) read-only [ 1122.432697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1122.560958] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1122.715707] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1122.968187] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1123.148875] Removing read-only on unknown block (0xfc00000) [ 1123.282811] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1133.594220] Lustre: DEBUG MARKER: hostname [ 1133.892584] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1134.143243] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1134.391635] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1134.549303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.910301] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (not set up) [ 1140.087898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1140.344214] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1140.840278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1140.842729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1141.005664] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1141.005666] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1141.150153] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1141.406240] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1141.661432] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1141.919293] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1142.669322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.669324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.836618] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.929329] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1145.157941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1145.309379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1146.061802] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1146.211508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1147.537235] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1148.436446] Lustre: DEBUG MARKER: rc=0; [ 1148.436446] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1148.436446] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1148.436446] echo $(hostname -s): $val; [ 1148.436446] rc=$val; [ 1148.436446] fi; [ 1148.436446] exit $rc [ 1149.036068] Lustre: DEBUG MARKER: dmesg [ 1149.419435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 5: \|x\| 220 open\(O_CREAT\) ======================================================= 07:57:54 \(1491551874\) [ 1149.578497] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) ======================================================= 07:57:54 (1491551874) [ 1149.727665] Lustre: DEBUG MARKER: sync; sync; sync [ 1150.612939] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1150.867101] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1151.009397] LustreError: 6009:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1151.012178] LustreError: 6009:0:(osd_handler.c:2171:osd_ro()) Skipped 2 previous similar messages [ 1151.051202] Turning device dm-0 (0xfc00000) read-only [ 1151.180351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1151.307856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1151.798343] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1152.066107] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1152.197760] Lustre: Failing over lustre-MDT0000 [ 1152.199966] Lustre: Skipped 2 previous similar messages [ 1152.250543] Removing read-only on unknown block (0xfc00000) [ 1152.257654] Lustre: server umount lustre-MDT0000 complete [ 1152.259776] Lustre: Skipped 2 previous similar messages [ 1152.388159] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1162.095047] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.099326] LustreError: Skipped 2 previous similar messages [ 1162.697634] Lustre: DEBUG MARKER: hostname [ 1163.008633] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1163.260958] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1163.511027] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1163.669288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.101203] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757bb93a to 0x52ac6ca1757bddbb [ 1168.105269] Lustre: Skipped 2 previous similar messages [ 1168.307196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1168.564554] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1169.056954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1169.060990] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1169.221995] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1169.222093] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1169.368172] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1169.622742] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1169.879198] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1170.138893] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1170.900885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1170.900890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.103414] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.140220] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.750048] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1171.752960] Lustre: Skipped 2 previous similar messages [ 1173.150026] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1173.155436] LustreError: Skipped 2 previous similar messages [ 1173.389190] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [ 1173.392095] Lustre: Skipped 2 previous similar messages [ 1174.397316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1174.431548] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1174.554391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1174.671224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1179.751291] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1180.648629] Lustre: DEBUG MARKER: rc=0; [ 1180.648629] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1180.648629] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1180.648629] echo $(hostname -s): $val; [ 1180.648629] rc=$val; [ 1180.648629] fi; [ 1180.648629] exit $rc [ 1181.262554] Lustre: DEBUG MARKER: dmesg [ 1181.691746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 6a: mkdir + contained create =================================================== 07:58:26 \(1491551906\) [ 1181.847723] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create =================================================== 07:58:26 (1491551906) [ 1182.029586] Lustre: DEBUG MARKER: sync; sync; sync [ 1182.923664] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1183.178227] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1183.315943] Turning device dm-0 (0xfc00000) read-only [ 1183.443389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1183.571122] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1183.714339] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1183.969373] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1184.147763] Removing read-only on unknown block (0xfc00000) [ 1184.279110] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1194.584305] Lustre: DEBUG MARKER: hostname [ 1194.889201] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1195.142164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1195.392737] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1195.549813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.763767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1196.021371] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1196.519845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1196.520807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1196.682958] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1196.683764] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1196.826492] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1197.080573] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1197.334132] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1197.590344] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1198.345057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1198.345096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1198.642223] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1198.643546] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.862865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1201.868905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1202.072778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1202.072779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1205.472673] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1206.360788] Lustre: DEBUG MARKER: rc=0; [ 1206.360788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1206.360788] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1206.360788] echo $(hostname -s): $val; [ 1206.360788] rc=$val; [ 1206.360788] fi; [ 1206.360788] exit $rc [ 1206.965943] Lustre: DEBUG MARKER: dmesg [ 1207.381937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 6b: \|X\| rmdir ================================================================== 07:58:52 \(1491551932\) [ 1207.542826] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ================================================================== 07:58:52 (1491551932) [ 1207.683266] Lustre: DEBUG MARKER: sync; sync; sync [ 1208.528944] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1208.784880] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1208.934588] Turning device dm-0 (0xfc00000) read-only [ 1209.065091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1209.200612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1209.346633] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1209.599462] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1209.775998] Removing read-only on unknown block (0xfc00000) [ 1209.908930] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1220.225245] Lustre: DEBUG MARKER: hostname [ 1220.523759] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1220.775216] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1221.030472] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1221.189998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.810412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1224.072081] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1224.572362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1224.584326] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1224.755387] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1224.756778] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1224.903888] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1225.176463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1225.450744] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1225.718316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1226.559969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.579749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.872675] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.892411] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1229.179340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1229.183684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1229.357276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1229.419652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1230.783830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1231.675701] Lustre: DEBUG MARKER: rc=0; [ 1231.675701] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1231.675701] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1231.675701] echo $(hostname -s): $val; [ 1231.675701] rc=$val; [ 1231.675701] fi; [ 1231.675701] exit $rc [ 1232.286694] Lustre: DEBUG MARKER: dmesg [ 1232.694358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 7: mkdir \|X\| contained create ================================================== 07:59:17 \(1491551957\) [ 1232.850215] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ================================================== 07:59:17 (1491551957) [ 1232.988076] Lustre: DEBUG MARKER: sync; sync; sync [ 1233.865169] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1234.122736] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1234.271212] Turning device dm-0 (0xfc00000) read-only [ 1234.398324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1234.527141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1234.670106] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1234.922851] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1235.098977] Removing read-only on unknown block (0xfc00000) [ 1235.233358] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1238.643213] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1238.646036] LustreError: Skipped 11 previous similar messages [ 1238.648301] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.653175] Lustre: Skipped 9 previous similar messages [ 1245.542689] Lustre: DEBUG MARKER: hostname [ 1245.846184] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1245.879031] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491551964/real 1491551964] req@ffff88005ff38600 x1564004875060400/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491551971 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1245.885893] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 1246.104967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1246.355483] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1246.514774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.926898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.929468] Lustre: Skipped 4 previous similar messages [ 1252.080873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1252.337704] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1252.829874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1252.830139] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1252.995032] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1252.995760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1253.146888] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1253.404772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1253.660340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1253.919983] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1254.771857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1254.787727] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1254.989508] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1254.993584] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.236810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1257.352272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1257.413630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1257.530681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1258.887208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1259.773288] Lustre: DEBUG MARKER: rc=0; [ 1259.773288] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1259.773288] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1259.773288] echo $(hostname -s): $val; [ 1259.773288] rc=$val; [ 1259.773288] fi; [ 1259.773288] exit $rc [ 1260.395651] Lustre: DEBUG MARKER: dmesg [ 1260.784788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 8: creat open \|X\| close ======================================================== 07:59:45 \(1491551985\) [ 1260.940444] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ======================================================== 07:59:45 (1491551985) [ 1261.083173] Lustre: DEBUG MARKER: sync; sync; sync [ 1262.124827] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1262.378117] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1262.524667] Turning device dm-0 (0xfc00000) read-only [ 1262.655065] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1262.785364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1263.091546] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1263.345467] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1263.522939] Removing read-only on unknown block (0xfc00000) [ 1263.655742] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1273.975900] Lustre: DEBUG MARKER: hostname [ 1274.285915] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1274.540186] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1274.790385] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1274.949453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.157655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1275.414444] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1275.910709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1275.913963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1276.081502] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1276.087465] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1276.229684] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1276.486175] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1276.740583] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1277.000177] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1277.759520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.765629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.975202] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.975221] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1280.238150] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1280.260841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1280.456843] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1280.490738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1281.871381] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1282.758878] Lustre: DEBUG MARKER: rc=0; [ 1282.758878] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1282.758878] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1282.758878] echo $(hostname -s): $val; [ 1282.758878] rc=$val; [ 1282.758878] fi; [ 1282.758878] exit $rc [ 1283.376223] Lustre: DEBUG MARKER: dmesg [ 1283.775525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 9: \|X\| create \(same inum\/gen\) ================================================== 08:00:08 \(1491552008\) [ 1283.928491] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ================================================== 08:00:08 (1491552008) [ 1284.067162] Lustre: DEBUG MARKER: sync; sync; sync [ 1284.945173] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1285.200292] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1285.339539] LustreError: 17298:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1285.342257] LustreError: 17298:0:(osd_handler.c:2171:osd_ro()) Skipped 4 previous similar messages [ 1285.377226] Turning device dm-0 (0xfc00000) read-only [ 1285.506224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1285.634767] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1285.778028] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1286.033152] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1286.162143] Lustre: Failing over lustre-MDT0000 [ 1286.164252] Lustre: Skipped 4 previous similar messages [ 1286.213680] Removing read-only on unknown block (0xfc00000) [ 1286.220787] Lustre: server umount lustre-MDT0000 complete [ 1286.222836] Lustre: Skipped 4 previous similar messages [ 1286.351290] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1296.654157] Lustre: DEBUG MARKER: hostname [ 1296.954655] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1297.207147] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1297.311038] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.315040] LustreError: Skipped 4 previous similar messages [ 1297.457817] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1297.621510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.317151] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757c4a57 to 0x52ac6ca1757c507e [ 1303.321108] Lustre: Skipped 4 previous similar messages [ 1303.559240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1303.818573] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1304.314846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1304.315444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1304.480153] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1304.487056] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1304.627624] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1304.777319] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1304.779997] Lustre: Skipped 4 previous similar messages [ 1304.888841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1305.144651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1305.402845] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1306.154305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.156168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.375376] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.375377] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1308.352741] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1308.357911] LustreError: Skipped 4 previous similar messages [ 1308.372979] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1308.375977] Lustre: Skipped 4 previous similar messages [ 1308.643956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1308.655811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1308.845790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1308.873317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1310.292448] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1311.180719] Lustre: DEBUG MARKER: rc=0; [ 1311.180719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1311.180719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1311.180719] echo $(hostname -s): $val; [ 1311.180719] rc=$val; [ 1311.180719] fi; [ 1311.180719] exit $rc [ 1311.801261] Lustre: DEBUG MARKER: dmesg [ 1312.193064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 10: create \|X\| rename unlink =================================================== 08:00:37 \(1491552037\) [ 1312.353629] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink =================================================== 08:00:37 (1491552037) [ 1312.497085] Lustre: DEBUG MARKER: sync; sync; sync [ 1313.377393] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1313.634655] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1313.775339] Turning device dm-0 (0xfc00000) read-only [ 1313.905347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1314.037794] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1314.192614] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1314.452697] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1314.629251] Removing read-only on unknown block (0xfc00000) [ 1314.762143] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1325.072057] Lustre: DEBUG MARKER: hostname [ 1325.380663] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1325.633837] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1325.884691] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1326.041416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1326.245032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1326.499357] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1326.993530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1326.993937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1327.157915] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1327.159116] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1327.303401] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1327.559082] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1327.814366] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1328.073043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1328.829757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1328.846333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.060820] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.066896] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1333.347762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1333.351860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1333.534313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1333.585761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1334.974746] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1335.861649] Lustre: DEBUG MARKER: rc=0; [ 1335.861649] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1335.861649] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1335.861649] echo $(hostname -s): $val; [ 1335.861649] rc=$val; [ 1335.861649] fi; [ 1335.861649] exit $rc [ 1336.477249] Lustre: DEBUG MARKER: dmesg [ 1336.876657] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 11: create open write rename \|X\| create-old-name read ========================== 08:01:01 \(1491552061\) [ 1337.042352] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================== 08:01:01 (1491552061) [ 1337.192297] Lustre: DEBUG MARKER: sync; sync; sync [ 1338.058940] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1338.305883] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1338.449854] Turning device dm-0 (0xfc00000) read-only [ 1338.576411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1338.701427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1338.847206] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1339.093182] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1339.269090] Removing read-only on unknown block (0xfc00000) [ 1339.404900] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1339.917957] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.922934] LustreError: Skipped 390 previous similar messages [ 1349.711368] Lustre: DEBUG MARKER: hostname [ 1350.017632] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1350.263457] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1350.506868] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1350.661744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.567337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1354.815169] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1355.309146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1355.312786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1355.478814] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1355.484519] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1355.620477] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1355.869434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1356.115793] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1356.369832] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1357.114877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.126689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.342591] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.342758] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.607127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1360.639525] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1360.793557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1360.881517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1362.209451] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1363.089372] Lustre: DEBUG MARKER: rc=0; [ 1363.089372] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1363.089372] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1363.089372] echo $(hostname -s): $val; [ 1363.089372] rc=$val; [ 1363.089372] fi; [ 1363.089372] exit $rc [ 1363.690218] Lustre: DEBUG MARKER: dmesg [ 1364.076815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 12: open, unlink \|X\| close ===================================================== 08:01:28 \(1491552088\) [ 1364.230070] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ===================================================== 08:01:28 (1491552088) [ 1364.375306] Lustre: DEBUG MARKER: sync; sync; sync [ 1365.247798] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1365.496252] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1365.645922] Turning device dm-0 (0xfc00000) read-only [ 1365.770605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1365.895470] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1366.035731] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1366.281531] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1366.454390] Removing read-only on unknown block (0xfc00000) [ 1366.584664] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1376.889334] Lustre: DEBUG MARKER: hostname [ 1377.193206] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1377.439257] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1377.682058] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1377.835227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.502236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1383.752375] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1384.247211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1384.252930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1384.410808] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1384.410810] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1384.558994] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1384.806815] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1385.054295] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1385.307914] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1386.064276] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.064277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.272055] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.275031] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1388.564201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1388.718520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1389.524336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1389.680814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1390.996765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1391.906259] Lustre: DEBUG MARKER: rc=0; [ 1391.906259] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1391.906259] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1391.906259] echo $(hostname -s): $val; [ 1391.906259] rc=$val; [ 1391.906259] fi; [ 1391.906259] exit $rc [ 1392.487338] Lustre: DEBUG MARKER: dmesg [ 1392.878962] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 13: open chmod 0 \|x\| write close =============================================== 08:01:57 \(1491552117\) [ 1393.032511] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close =============================================== 08:01:57 (1491552117) [ 1393.241594] Lustre: DEBUG MARKER: sync; sync; sync [ 1394.085758] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1394.345394] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1394.489138] Turning device dm-0 (0xfc00000) read-only [ 1394.613544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1394.737515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1394.876459] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1395.124918] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1395.298385] Removing read-only on unknown block (0xfc00000) [ 1395.427910] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1405.732619] Lustre: DEBUG MARKER: hostname [ 1406.025954] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1406.270213] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1406.511622] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1406.655281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.730555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1412.979389] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1413.469686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1413.470125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1413.630716] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1413.633712] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1413.775762] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1414.026053] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1414.278863] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1414.538288] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1415.292838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1415.297519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1415.493335] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1415.536517] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.808461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1417.956481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1418.723265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1418.878364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1420.215753] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1421.102945] Lustre: DEBUG MARKER: rc=0; [ 1421.102945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1421.102945] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1421.102945] echo $(hostname -s): $val; [ 1421.102945] rc=$val; [ 1421.102945] fi; [ 1421.102945] exit $rc [ 1421.690148] Lustre: DEBUG MARKER: dmesg [ 1422.083780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 14: open\(O_CREAT\), unlink \|X\| close ============================================ 08:02:26 \(1491552146\) [ 1422.243523] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ============================================ 08:02:26 (1491552146) [ 1422.386241] Lustre: DEBUG MARKER: sync; sync; sync [ 1423.258304] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1423.505261] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1423.647746] Turning device dm-0 (0xfc00000) read-only [ 1423.772277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1423.897412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1424.039981] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1424.289725] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1424.463518] Removing read-only on unknown block (0xfc00000) [ 1424.592971] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1434.891226] Lustre: DEBUG MARKER: hostname [ 1435.185774] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1435.430572] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1435.672963] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1435.826662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.034783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1436.282143] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1436.786258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1436.786260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1436.955390] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1436.959394] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1437.098443] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1437.347167] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1437.593841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1437.845311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1438.588532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1438.588534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1438.809296] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1438.809297] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1443.096975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1443.096976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1443.303604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1443.308720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1444.720497] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1445.618189] Lustre: DEBUG MARKER: rc=0; [ 1445.618189] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1445.618189] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1445.618189] echo $(hostname -s): $val; [ 1445.618189] rc=$val; [ 1445.618189] fi; [ 1445.618189] exit $rc [ 1446.215182] Lustre: DEBUG MARKER: dmesg [ 1446.630212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 15: open\(O_CREAT\), unlink \|X\| touch new, close ================================ 08:02:51 \(1491552171\) [ 1446.782488] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ================================ 08:02:51 (1491552171) [ 1446.928270] Lustre: DEBUG MARKER: sync; sync; sync [ 1447.839165] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1448.086303] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1448.229605] Turning device dm-0 (0xfc00000) read-only [ 1448.353447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1448.477789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1448.621353] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1448.869365] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1449.043549] Removing read-only on unknown block (0xfc00000) [ 1449.173869] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1459.483324] Lustre: DEBUG MARKER: hostname [ 1459.782470] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1460.027078] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1460.270621] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1460.424694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.070145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1464.327953] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1464.908760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1464.916017] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1465.074299] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1465.076061] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1465.217098] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1465.465977] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1465.713231] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1465.965853] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1466.717965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.733450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.877439] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1467.000732] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1469.221556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1469.378531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1470.096577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1470.256669] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1471.585320] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1472.456446] Lustre: DEBUG MARKER: rc=0; [ 1472.456446] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1472.456446] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1472.456446] echo $(hostname -s): $val; [ 1472.456446] rc=$val; [ 1472.456446] fi; [ 1472.456446] exit $rc [ 1473.045557] Lustre: DEBUG MARKER: dmesg [ 1473.467824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 16: \|X\| open\(O_CREAT\), unlink, touch new, unlink new ========================== 08:03:18 \(1491552198\) [ 1473.638434] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================== 08:03:18 (1491552198) [ 1473.801392] Lustre: DEBUG MARKER: sync; sync; sync [ 1474.666289] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1474.924505] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1475.074383] Turning device dm-0 (0xfc00000) read-only [ 1475.203979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1475.329534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1475.477076] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1475.729873] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1475.901565] Removing read-only on unknown block (0xfc00000) [ 1476.031694] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1486.335114] Lustre: DEBUG MARKER: hostname [ 1486.637071] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1486.883868] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1487.126517] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1487.281483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.823414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1493.072236] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1493.579041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1493.591555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1493.756896] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1493.762776] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1493.902498] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1494.151317] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1494.397387] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1494.650578] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1495.399460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1495.441526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1495.593800] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1495.702937] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1497.931807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1498.086247] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1498.823080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1498.972515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1500.302100] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1501.171156] Lustre: DEBUG MARKER: rc=0; [ 1501.171156] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1501.171156] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1501.171156] echo $(hostname -s): $val; [ 1501.171156] rc=$val; [ 1501.171156] fi; [ 1501.171156] exit $rc [ 1501.765202] Lustre: DEBUG MARKER: dmesg [ 1502.156641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 17: \|X\| open\(O_CREAT\), \|replay\| close ========================================== 08:03:46 \(1491552226\) [ 1502.311786] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================== 08:03:46 (1491552226) [ 1502.450774] Lustre: DEBUG MARKER: sync; sync; sync [ 1503.317255] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1503.563446] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1503.695993] Turning device dm-0 (0xfc00000) read-only [ 1503.823320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1503.954696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1504.102035] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1504.355709] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1504.529495] Removing read-only on unknown block (0xfc00000) [ 1504.659336] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1507.672178] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 1507.675209] LustreError: Skipped 24 previous similar messages [ 1507.677433] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.682439] Lustre: Skipped 19 previous similar messages [ 1514.965740] Lustre: DEBUG MARKER: hostname [ 1515.260514] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1515.504638] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1515.747923] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1515.902446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.956658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.959226] Lustre: Skipped 9 previous similar messages [ 1516.108753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1516.357766] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1516.853918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1516.854329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1517.017856] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1517.019477] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1517.161278] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1517.409071] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1517.657120] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1517.909191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1518.660141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1518.705735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1518.972079] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1518.972106] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1521.260988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1521.261748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1521.489290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1521.495210] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1522.841867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1523.711430] Lustre: DEBUG MARKER: rc=0; [ 1523.711430] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1523.711430] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1523.711430] echo $(hostname -s): $val; [ 1523.711430] rc=$val; [ 1523.711430] fi; [ 1523.711430] exit $rc [ 1524.308133] Lustre: DEBUG MARKER: dmesg [ 1524.730126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 18: open\(O_CREAT\), unlink, touch new, close, touch, unlink ===================== 08:04:09 \(1491552249\) [ 1524.896263] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ===================== 08:04:09 (1491552249) [ 1525.029058] Lustre: DEBUG MARKER: sync; sync; sync [ 1525.859132] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1526.109692] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1526.252676] Turning device dm-0 (0xfc00000) read-only [ 1526.376237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.501680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.651404] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1526.897938] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1527.140763] Removing read-only on unknown block (0xfc00000) [ 1527.269966] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1537.570712] Lustre: DEBUG MARKER: hostname [ 1537.869086] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1538.114616] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1538.358317] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1538.439045] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491552256/real 1491552256] req@ffff880078eff000 x1564004875095328/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491552263 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 1538.445802] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 1538.502703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.665174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1544.915031] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1545.408658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1545.420497] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1545.578499] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1545.578502] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1545.721213] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1545.967680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1546.212595] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1546.462159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1547.204533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.204941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.408855] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1547.433990] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.700236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1549.860439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1550.669965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1550.825902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1552.149084] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1553.020315] Lustre: DEBUG MARKER: rc=0; [ 1553.020315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1553.020315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1553.020315] echo $(hostname -s): $val; [ 1553.020315] rc=$val; [ 1553.020315] fi; [ 1553.020315] exit $rc [ 1553.611874] Lustre: DEBUG MARKER: dmesg [ 1554.036737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 19: mcreate, open, write, rename =============================================== 08:04:38 \(1491552278\) [ 1554.194696] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename =============================================== 08:04:38 (1491552278) [ 1554.335250] Lustre: DEBUG MARKER: sync; sync; sync [ 1555.243186] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1555.501198] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1555.639668] LustreError: 6783:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 1555.642446] LustreError: 6783:0:(osd_handler.c:2171:osd_ro()) Skipped 9 previous similar messages [ 1555.677660] Turning device dm-0 (0xfc00000) read-only [ 1555.812059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1555.939837] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1556.090430] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1556.337136] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1556.462388] Lustre: Failing over lustre-MDT0000 [ 1556.464588] Lustre: Skipped 9 previous similar messages [ 1556.514289] Removing read-only on unknown block (0xfc00000) [ 1556.521384] Lustre: server umount lustre-MDT0000 complete [ 1556.523481] Lustre: Skipped 9 previous similar messages [ 1556.648613] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1566.615045] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.619300] LustreError: Skipped 9 previous similar messages [ 1566.952813] Lustre: DEBUG MARKER: hostname [ 1567.260078] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1567.513775] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1567.756935] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1567.899956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.621271] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757c8981 to 0x52ac6ca1757c8fe0 [ 1572.625305] Lustre: Skipped 9 previous similar messages [ 1572.822726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1573.071243] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1573.557389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1573.567143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1573.727931] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1573.733760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1573.874190] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1574.123307] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1574.227260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 1574.229919] Lustre: Skipped 9 previous similar messages [ 1574.371966] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1574.625260] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1575.413481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.415388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.573847] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.607630] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1577.675332] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1577.681207] LustreError: Skipped 9 previous similar messages [ 1577.698241] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 1577.701262] Lustre: Skipped 9 previous similar messages [ 1578.837386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1578.902373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1578.992968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1579.093229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1580.454241] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1581.338970] Lustre: DEBUG MARKER: rc=0; [ 1581.338970] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1581.338970] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1581.338970] echo $(hostname -s): $val; [ 1581.338970] rc=$val; [ 1581.338970] fi; [ 1581.338970] exit $rc [ 1581.925898] Lustre: DEBUG MARKER: dmesg [ 1582.343470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20a: \|X\| open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ======= 08:05:07 \(1491552307\) [ 1582.504663] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ======= 08:05:07 (1491552307) [ 1582.636222] Lustre: DEBUG MARKER: sync; sync; sync [ 1583.523182] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1583.780711] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1583.932932] Turning device dm-0 (0xfc00000) read-only [ 1584.058749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1584.183840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1584.329534] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1584.576423] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1584.748095] Removing read-only on unknown block (0xfc00000) [ 1584.877118] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1595.190327] Lustre: DEBUG MARKER: hostname [ 1595.494487] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1595.738793] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1595.985282] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1596.142453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.720129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1600.982201] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1601.574821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1601.574822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1601.753278] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1601.754268] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1601.895173] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1602.144172] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1602.394948] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1602.648178] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1603.390354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.396851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.630806] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.648295] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1605.925750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1605.971004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1606.163467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1606.269429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1607.616279] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1608.495775] Lustre: DEBUG MARKER: rc=0; [ 1608.495775] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1608.495775] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1608.495775] echo $(hostname -s): $val; [ 1608.495775] rc=$val; [ 1608.495775] fi; [ 1608.495775] exit $rc [ 1609.085184] Lustre: DEBUG MARKER: dmesg [ 1609.494652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20b: write, unlink, eviction, replay, \(test mds_cleanup_orphans\) =============== 08:05:34 \(1491552334\) [ 1609.655215] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay, (test mds_cleanup_orphans) =============== 08:05:34 (1491552334) [ 1609.884069] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 [ 1610.068395] Lustre: 10986:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 at adminstrative request [ 1611.382084] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync 1 [ 1611.673165] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1611.932748] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1612.302781] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1622.609385] Lustre: DEBUG MARKER: hostname [ 1622.909599] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1623.161354] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1623.413213] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1623.568790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.756378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1630.008370] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1630.501998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1630.514272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1630.678233] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1630.688551] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1630.826723] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1631.078893] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1631.325935] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1631.577021] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1632.327917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1632.338064] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1632.539400] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1632.568586] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.817380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1634.822401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1635.009840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1635.058311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1636.388241] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1636.900181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1636.900216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1637.062567] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1637.065916] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1637.481193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 1637.808704] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before 373012, after 373012 [ 1637.972423] Lustre: DEBUG MARKER: before 373012, after 373012 [ 1638.189973] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1639.048144] Lustre: DEBUG MARKER: rc=0; [ 1639.048144] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1639.048144] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1639.048144] echo $(hostname -s): $val; [ 1639.048144] rc=$val; [ 1639.048144] fi; [ 1639.048144] exit $rc [ 1639.637765] Lustre: DEBUG MARKER: dmesg [ 1640.049800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 20c: check that client eviction does not affect file content =================== 08:06:04 \(1491552364\) [ 1640.208404] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 08:06:04 (1491552364) [ 1640.351109] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 [ 1640.479093] Lustre: 13569:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 at adminstrative request [ 1642.927234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1643.806392] Lustre: DEBUG MARKER: rc=0; [ 1643.806392] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1643.806392] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1643.806392] echo $(hostname -s): $val; [ 1643.806392] rc=$val; [ 1643.806392] fi; [ 1643.806392] exit $rc [ 1644.401516] Lustre: DEBUG MARKER: dmesg [ 1644.813645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 21: \|X\| open\(O_CREAT\), unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 08:06:09 \(1491552369\ [ 1644.975144] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 08:06:09 (1491552369) [ 1645.115392] Lustre: DEBUG MARKER: sync; sync; sync [ 1645.998932] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1646.244406] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1646.386072] Turning device dm-0 (0xfc00000) read-only [ 1646.510057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1646.635089] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1646.781742] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1647.026880] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1647.200757] Removing read-only on unknown block (0xfc00000) [ 1647.330533] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1657.632407] Lustre: DEBUG MARKER: hostname [ 1657.935305] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1658.180421] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1658.422441] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1658.603805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.958050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1663.206979] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1663.716096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1663.717647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1663.881667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1663.882350] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1664.023956] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1664.271744] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1664.519118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1664.769709] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1665.524330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1665.553275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1665.691543] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1665.785663] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1668.020834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1668.177313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1668.980776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1669.132621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1670.474948] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1671.354056] Lustre: DEBUG MARKER: rc=0; [ 1671.354056] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1671.354056] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1671.354056] echo $(hostname -s): $val; [ 1671.354056] rc=$val; [ 1671.354056] fi; [ 1671.354056] exit $rc [ 1671.952644] Lustre: DEBUG MARKER: dmesg [ 1672.350527] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 22: open\(O_CREAT\), \|X\| unlink, replay, close \(test mds_cleanup_orphans\) ======== 08:06:37 \(1491552397\) [ 1672.502547] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ======== 08:06:37 (1491552397) [ 1672.642469] Lustre: DEBUG MARKER: sync; sync; sync [ 1673.521230] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1673.768022] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1673.901984] Turning device dm-0 (0xfc00000) read-only [ 1674.034133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1674.160261] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1674.300393] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1674.548082] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1674.721508] Removing read-only on unknown block (0xfc00000) [ 1674.852153] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1685.154342] Lustre: DEBUG MARKER: hostname [ 1685.448268] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1685.691983] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1685.934446] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1686.089740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.289405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1686.539088] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1687.030126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1687.033421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1687.196497] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1687.196664] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1687.337852] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1687.583554] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1687.830797] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1688.082791] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1688.840738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1688.857332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1689.021438] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1689.127505] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1693.236734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1693.363601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1693.391861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1693.570953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1694.911725] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1695.786109] Lustre: DEBUG MARKER: rc=0; [ 1695.786109] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1695.786109] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1695.786109] echo $(hostname -s): $val; [ 1695.786109] rc=$val; [ 1695.786109] fi; [ 1695.786109] exit $rc [ 1696.373976] Lustre: DEBUG MARKER: dmesg [ 1696.777446] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 23: open\(O_CREAT\), \|X\| unlink touch new, replay, close \(test mds_cleanup_orphans\) ====================================================================================================== 08:07:01 \(1491552421\ [ 1696.936841] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 08:07:01 (1491552421) [ 1697.104077] Lustre: DEBUG MARKER: sync; sync; sync [ 1697.936997] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1698.191499] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1698.335420] Turning device dm-0 (0xfc00000) read-only [ 1698.460117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1698.586093] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1698.730310] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1698.977135] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1699.148927] Removing read-only on unknown block (0xfc00000) [ 1699.278056] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1709.585206] Lustre: DEBUG MARKER: hostname [ 1709.878633] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1710.121600] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1710.362871] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1710.506217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.141354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1715.389519] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1715.891628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1715.895725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1716.058702] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1716.059132] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1716.202106] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1716.450310] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1716.697388] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1716.948807] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1717.693692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1717.693702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1717.864395] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1717.950719] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1718.653969] Lustre: MGS: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [ 1718.656839] Lustre: Skipped 553 previous similar messages [ 1720.191980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1720.346372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1721.103263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1721.285136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1722.613101] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1723.484122] Lustre: DEBUG MARKER: rc=0; [ 1723.484122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1723.484122] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1723.484122] echo $(hostname -s): $val; [ 1723.484122] rc=$val; [ 1723.484122] fi; [ 1723.484122] exit $rc [ 1724.086410] Lustre: DEBUG MARKER: dmesg [ 1724.480612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 24: open\(O_CREAT\), replay, unlink, close \(test mds_cleanup_orphans\) ============ 08:07:29 \(1491552449\) [ 1724.648814] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ============ 08:07:29 (1491552449) [ 1724.787345] Lustre: DEBUG MARKER: sync; sync; sync [ 1725.666217] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1725.914061] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1726.056919] Turning device dm-0 (0xfc00000) read-only [ 1726.185299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1726.312681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1726.452309] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1726.698077] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1726.871443] Removing read-only on unknown block (0xfc00000) [ 1727.001358] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1737.314246] Lustre: DEBUG MARKER: hostname [ 1737.636175] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1737.883682] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1738.129410] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1738.286386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.210792] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1743.459411] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1743.947932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1743.953717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1744.113632] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1744.113634] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1744.257567] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1744.507787] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1744.754040] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1745.009107] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1745.752864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.752871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.958186] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.992547] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.262569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1748.424226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1749.216907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1749.378824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1750.717662] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1751.591221] Lustre: DEBUG MARKER: rc=0; [ 1751.591221] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1751.591221] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1751.591221] echo $(hostname -s): $val; [ 1751.591221] rc=$val; [ 1751.591221] fi; [ 1751.591221] exit $rc [ 1752.183059] Lustre: DEBUG MARKER: dmesg [ 1752.591757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 25: open\(O_CREAT\), unlink, replay, close \(test mds_cleanup_orphans\) ============ 08:07:57 \(1491552477\) [ 1752.753697] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ============ 08:07:57 (1491552477) [ 1752.892607] Lustre: DEBUG MARKER: sync; sync; sync [ 1753.764358] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1754.015813] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1754.213663] Turning device dm-0 (0xfc00000) read-only [ 1754.344179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1754.475605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1754.612681] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1754.858527] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1755.032548] Removing read-only on unknown block (0xfc00000) [ 1755.162219] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1765.469942] Lustre: DEBUG MARKER: hostname [ 1765.770551] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1766.017986] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1766.260197] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1766.413718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.291062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1771.541279] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1772.031260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1772.042824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1772.203667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1772.203668] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1772.347253] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1772.594831] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1772.842607] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1773.094285] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1773.853366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1773.854921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.057851] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.085048] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1776.329671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1776.373068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1776.506433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1776.623077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1777.942203] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1778.832260] Lustre: DEBUG MARKER: rc=0; [ 1778.832260] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1778.832260] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1778.832260] echo $(hostname -s): $val; [ 1778.832260] rc=$val; [ 1778.832260] fi; [ 1778.832260] exit $rc [ 1779.442420] Lustre: DEBUG MARKER: dmesg [ 1779.847093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 26: \|X\| open\(O_CREAT\), unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 08:08:24 \(14 [ 1780.007638] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 08:08:24 (1491552504) [ 1780.144658] Lustre: DEBUG MARKER: sync; sync; sync [ 1781.039725] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1781.300531] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1781.450170] Turning device dm-0 (0xfc00000) read-only [ 1781.600514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1781.725495] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1781.879860] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1782.131650] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1782.305060] Removing read-only on unknown block (0xfc00000) [ 1782.434516] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1792.737359] Lustre: DEBUG MARKER: hostname [ 1793.030635] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1793.275146] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1793.516946] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1793.669752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.867517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1794.117754] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1794.602757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1794.614591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1794.771610] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1794.772210] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1794.913226] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1795.159900] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1795.406048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1795.657365] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1796.407480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.409545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.589840] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.642721] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.883392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1800.952767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1801.041359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1801.147413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 1802.484567] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1803.358900] Lustre: DEBUG MARKER: rc=0; [ 1803.358900] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1803.358900] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1803.358900] echo $(hostname -s): $val; [ 1803.358900] rc=$val; [ 1803.358900] fi; [ 1803.358900] exit $rc [ 1803.963363] Lustre: DEBUG MARKER: dmesg [ 1804.408350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 27: \|X\| open\(O_CREAT\), unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 08:08:49 \(1491552529\) [ 1804.576210] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 08:08:49 (1491552529) [ 1804.708630] Lustre: DEBUG MARKER: sync; sync; sync [ 1805.580147] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1805.825871] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1805.959406] Turning device dm-0 (0xfc00000) read-only [ 1806.084030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1806.208531] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1806.363408] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1806.608703] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1806.781724] Removing read-only on unknown block (0xfc00000) [ 1806.911424] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1817.220051] Lustre: DEBUG MARKER: hostname [ 1817.536621] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1817.787232] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1818.030093] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1818.184769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.363922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1823.613300] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1824.109244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1824.113391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1824.278564] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1824.278989] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1824.422195] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1824.668428] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1824.914541] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1825.173731] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1825.917353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.917354] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.228062] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.246977] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1829.442702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1829.469341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1829.651501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1829.693960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1831.087428] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1831.974610] Lustre: DEBUG MARKER: rc=0; [ 1831.974610] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1831.974610] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1831.974610] echo $(hostname -s): $val; [ 1831.974610] rc=$val; [ 1831.974610] fi; [ 1831.974610] exit $rc [ 1832.557250] Lustre: DEBUG MARKER: dmesg [ 1832.965287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 28: open\(O_CREAT\), \|X\| unlink two, close one, replay, close one \(test mds_cleanup_orphans\) ====================================================================================================== 08:09:17 \(14 [ 1833.125188] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 08:09:17 (1491552557) [ 1833.271278] Lustre: DEBUG MARKER: sync; sync; sync [ 1834.093289] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1834.341344] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1834.483818] Turning device dm-0 (0xfc00000) read-only [ 1834.608392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1834.733633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1834.896091] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1835.143162] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1835.316502] Removing read-only on unknown block (0xfc00000) [ 1835.446382] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1845.754217] Lustre: DEBUG MARKER: hostname [ 1846.052843] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1846.297048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1846.539133] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1846.683165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.405094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1851.654241] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1852.143146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1852.154734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1852.318945] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1852.320544] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1852.460571] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1852.711979] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1852.959987] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1853.213982] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1853.969775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1853.981494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1854.172278] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1854.232795] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1856.484643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1856.644497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1857.419862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1857.572799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 1858.898889] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1859.792329] Lustre: DEBUG MARKER: rc=0; [ 1859.792329] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1859.792329] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1859.792329] echo $(hostname -s): $val; [ 1859.792329] rc=$val; [ 1859.792329] fi; [ 1859.792329] exit $rc [ 1860.399793] Lustre: DEBUG MARKER: dmesg [ 1860.814191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 29: open\(O_CREAT\), \|X\| unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 08:09:45 \(1491552585\) [ 1860.971348] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 08:09:45 (1491552585) [ 1861.127158] Lustre: DEBUG MARKER: sync; sync; sync [ 1862.013640] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1862.258705] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1862.400291] Turning device dm-0 (0xfc00000) read-only [ 1862.529829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1862.662118] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1862.810173] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1863.068815] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1863.250495] Removing read-only on unknown block (0xfc00000) [ 1863.383498] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1863.522085] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.527124] LustreError: Skipped 790 previous similar messages [ 1873.694204] Lustre: DEBUG MARKER: hostname [ 1873.988605] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1874.234484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1874.478489] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1874.641304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.445200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1880.694319] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1881.187981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1881.188484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1881.355685] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1881.357608] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1881.499298] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1881.746989] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1881.993063] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1882.243999] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1883.062881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1883.088090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1883.291108] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1883.316474] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1885.508340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1885.529692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1885.705923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1885.767954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1887.126586] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1888.014914] Lustre: DEBUG MARKER: rc=0; [ 1888.014914] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1888.014914] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1888.014914] echo $(hostname -s): $val; [ 1888.014914] rc=$val; [ 1888.014914] fi; [ 1888.014914] exit $rc [ 1888.616290] Lustre: DEBUG MARKER: dmesg [ 1889.037090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 30: open\(O_CREAT\) two, unlink two, replay, close two \(test mds_cleanup_orphans\) ====================================================================================================== 08:10:13 \(1491552613\) [ 1889.201485] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ====================================================================================================== 08:10:13 (1491552613) [ 1889.351698] Lustre: DEBUG MARKER: sync; sync; sync [ 1890.217777] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1890.480239] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1890.667457] Turning device dm-0 (0xfc00000) read-only [ 1890.795759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1890.928362] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1891.067417] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1891.314531] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1891.486692] Removing read-only on unknown block (0xfc00000) [ 1891.616291] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1901.922922] Lustre: DEBUG MARKER: hostname [ 1902.224725] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1902.475433] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1902.718545] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1902.872422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.430576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1908.687532] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1909.191741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1909.192741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1909.357245] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1909.359165] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1909.502101] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1909.748892] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1909.995791] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1910.247619] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1910.992090] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.992105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1911.208259] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1911.214056] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1913.485869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1913.555164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1913.730847] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1913.742292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 1915.128458] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1916.014239] Lustre: DEBUG MARKER: rc=0; [ 1916.014239] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1916.014239] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1916.014239] echo $(hostname -s): $val; [ 1916.014239] rc=$val; [ 1916.014239] fi; [ 1916.014239] exit $rc [ 1916.611360] Lustre: DEBUG MARKER: dmesg [ 1917.105199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 31: open\(O_CREAT\) two, unlink one, \|X\| unlink one, close two \(test mds_cleanup_orphans\) ====================================================================================================== 08:10:41 \(14915 [ 1917.260415] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ====================================================================================================== 08:10:41 (1491552641) [ 1917.408931] Lustre: DEBUG MARKER: sync; sync; sync [ 1918.299581] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1918.548260] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1918.691790] Turning device dm-0 (0xfc00000) read-only [ 1918.816254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1918.941779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1919.083837] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1919.331299] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1919.504286] Removing read-only on unknown block (0xfc00000) [ 1919.633886] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1929.933737] Lustre: DEBUG MARKER: hostname [ 1930.231262] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1930.476448] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1930.719539] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1930.874852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.073759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1931.324443] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1931.817198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1931.823252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1931.980948] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1931.986263] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1932.123864] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1932.370824] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1932.615839] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1932.868691] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1933.622432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.646024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.926843] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.944416] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.211562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 1939.224675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 1939.382471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 1939.394523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 1940.780947] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1941.667032] Lustre: DEBUG MARKER: rc=0; [ 1941.667032] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1941.667032] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1941.667032] echo $(hostname -s): $val; [ 1941.667032] rc=$val; [ 1941.667032] fi; [ 1941.667032] exit $rc [ 1942.270149] Lustre: DEBUG MARKER: dmesg [ 1942.679183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 32: close\(\) notices client eviction\; close\(\) after client eviction ============= 08:11:07 \(1491552667\) [ 1942.871382] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ============= 08:11:07 (1491552667) [ 1943.016121] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 [ 1943.146908] Lustre: 5373:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 at adminstrative request [ 1945.591746] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1946.461412] Lustre: DEBUG MARKER: rc=0; [ 1946.461412] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1946.461412] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1946.461412] echo $(hostname -s): $val; [ 1946.461412] rc=$val; [ 1946.461412] fi; [ 1946.461412] exit $rc [ 1947.048981] Lustre: DEBUG MARKER: dmesg [ 1947.455745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 33a: fid seq shouldn\'t be reused after abort recovery ========================== 08:11:12 \(1491552672\) [ 1947.612364] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================== 08:11:12 (1491552672) [ 1947.758553] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1948.006632] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1948.263477] Turning device dm-0 (0xfc00000) read-only [ 1948.388584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1948.515411] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1948.648354] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1948.904808] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1949.082194] Removing read-only on unknown block (0xfc00000) [ 1949.215045] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1949.526879] Lustre: DEBUG MARKER: hostname [ 1949.805288] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1950.057924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1950.305163] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1950.449990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.518396] LustreError: 6353:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1950.520838] LustreError: 6353:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1950.523212] Lustre: 6401:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1950.525730] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 1950.676550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1950.934971] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1951.428961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1951.432673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1951.587709] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1951.591829] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1951.736777] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1951.991437] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1952.242920] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1952.504479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1955.384879] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1956.265699] Lustre: DEBUG MARKER: rc=0; [ 1956.265699] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1956.265699] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1956.265699] echo $(hostname -s): $val; [ 1956.265699] rc=$val; [ 1956.265699] fi; [ 1956.265699] exit $rc [ 1956.864229] Lustre: DEBUG MARKER: dmesg [ 1957.260566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 33b: test fid seq allocation =================================================== 08:11:22 \(1491552682\) [ 1957.414879] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation =================================================== 08:11:22 (1491552682) [ 1957.550323] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1311 [ 1957.820238] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1958.072497] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1958.275183] Turning device dm-0 (0xfc00000) read-only [ 1958.401136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1958.525720] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1958.657980] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1958.905943] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1959.081136] Removing read-only on unknown block (0xfc00000) [ 1959.297087] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1959.886168] Lustre: DEBUG MARKER: hostname [ 1960.155758] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1960.418915] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1960.664688] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1960.813442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.891366] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1960.900564] LustreError: 8256:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1960.903062] LustreError: 8256:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1960.905525] Lustre: 8304:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1960.908106] Lustre: 8304:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 1960.910538] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 1961.057502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1961.309995] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1961.819059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1961.821905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1961.981594] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1961.986902] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1962.125046] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1962.372832] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1962.622885] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1962.878392] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1965.516977] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1965.799091] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1966.703379] Lustre: DEBUG MARKER: rc=0; [ 1966.703379] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1966.703379] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1966.703379] echo $(hostname -s): $val; [ 1966.703379] rc=$val; [ 1966.703379] fi; [ 1966.703379] exit $rc [ 1967.319388] Lustre: DEBUG MARKER: dmesg [ 1967.743923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 34: abort recovery before client does replay \(test mds_cleanup_orphans\) ======== 08:11:32 \(1491552692\) [ 1967.898222] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ======== 08:11:32 (1491552692) [ 1968.046487] Lustre: DEBUG MARKER: sync; sync; sync [ 1968.929367] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1969.176993] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1969.316681] Turning device dm-0 (0xfc00000) read-only [ 1969.441526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1969.567971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1969.701836] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1969.950260] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1970.122442] Removing read-only on unknown block (0xfc00000) [ 1970.254186] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1970.552221] Lustre: DEBUG MARKER: hostname [ 1970.831293] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1971.085324] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1971.344117] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1971.496589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.565549] LustreError: 10150:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1971.568195] LustreError: 10150:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1971.570749] Lustre: 10198:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1971.573640] Lustre: 10198:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 1971.576145] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 1971.734389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1972.001384] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1972.511595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1972.512017] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1972.671716] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1972.674775] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1972.814742] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1973.075371] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1973.325489] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1973.590397] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1976.451952] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1977.319578] Lustre: DEBUG MARKER: rc=0; [ 1977.319578] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1977.319578] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1977.319578] echo $(hostname -s): $val; [ 1977.319578] rc=$val; [ 1977.319578] fi; [ 1977.319578] exit $rc [ 1977.903433] Lustre: DEBUG MARKER: dmesg [ 1978.343091] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 35: test recovery from llog for unlink op ====================================== 08:11:43 \(1491552703\) [ 1978.494081] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ====================================== 08:11:43 (1491552703) [ 1978.641800] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [ 1978.780493] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1978.782743] LustreError: 7897:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f1c3600 x1564004848878816/t188978561028(0) o36->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 624/456 e 0 to 0 dl 1491552710 ref 1 fl Interpret:/0/0 rc 0/0 [ 1980.957127] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1981.204920] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1981.560559] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 1981.866304] Lustre: DEBUG MARKER: hostname [ 1982.143140] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 1982.394527] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 1982.637640] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 1982.792684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.859097] LustreError: 11826:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1982.861809] LustreError: 11826:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1982.864484] Lustre: 11874:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1982.867495] Lustre: 11874:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 1982.870156] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 1983.020287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 1983.269467] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 1983.765270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1983.765272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1983.930120] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1983.930515] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 1984.072855] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 1984.319655] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1984.567763] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 1984.819747] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 1987.630121] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 1988.500307] Lustre: DEBUG MARKER: rc=0; [ 1988.500307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 1988.500307] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 1988.500307] echo $(hostname -s): $val; [ 1988.500307] rc=$val; [ 1988.500307] fi; [ 1988.500307] exit $rc [ 1989.092588] Lustre: DEBUG MARKER: dmesg [ 1989.497849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 36: don\'t resend cancel ======================================================== 08:11:54 \(1491552714\) [ 1989.656588] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ======================================================== 08:11:54 (1491552714) [ 1989.793571] Lustre: DEBUG MARKER: sync; sync; sync [ 1990.680840] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 1990.934957] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 1991.079203] Turning device dm-0 (0xfc00000) read-only [ 1991.207551] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 1991.334332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1991.481935] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 1991.728464] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 1991.901226] Removing read-only on unknown block (0xfc00000) [ 1992.031990] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2002.335475] Lustre: DEBUG MARKER: hostname [ 2002.635664] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2002.881893] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2003.125607] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2003.279963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.485116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2003.745079] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2004.253994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2004.253995] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2004.419583] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2004.425266] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2004.562240] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2004.810147] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2005.056712] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2005.307469] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2008.565858] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2009.435351] Lustre: DEBUG MARKER: rc=0; [ 2009.435351] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2009.435351] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2009.435351] echo $(hostname -s): $val; [ 2009.435351] rc=$val; [ 2009.435351] fi; [ 2009.435351] exit $rc [ 2010.033025] Lustre: DEBUG MARKER: dmesg [ 2010.443893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 37: abort recovery before client does replay \(test mds_cleanup_orphans for directories\) ====================================================================================================== 08:12:15 \(149155273 [ 2010.613335] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ====================================================================================================== 08:12:15 (1491552735) [ 2010.756892] Lustre: DEBUG MARKER: sync; sync; sync [ 2011.667375] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2011.924719] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2012.067610] Turning device dm-0 (0xfc00000) read-only [ 2012.192073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2012.317039] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2012.444725] Lustre: DEBUG MARKER: dmesg -c [ 2012.716043] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2012.959318] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2013.132892] Removing read-only on unknown block (0xfc00000) [ 2013.262386] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2013.560656] Lustre: DEBUG MARKER: hostname [ 2013.851253] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2014.106246] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2014.357166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2014.501961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.565391] LustreError: 15664:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2014.568004] LustreError: 15664:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2014.570570] Lustre: 15712:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2014.573086] Lustre: 15712:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2014.575578] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2014.722819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2014.971974] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2015.464058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2015.466792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2015.626686] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2015.627699] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2015.770610] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2016.019071] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2016.265305] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2016.518096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2019.281268] Lustre: DEBUG MARKER: dmesg [ 2019.637320] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2020.518858] Lustre: DEBUG MARKER: rc=0; [ 2020.518858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2020.518858] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2020.518858] echo $(hostname -s): $val; [ 2020.518858] rc=$val; [ 2020.518858] fi; [ 2020.518858] exit $rc [ 2021.128649] Lustre: DEBUG MARKER: dmesg [ 2021.531932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 38: test recovery from unlink llog \(test llog_gen_rec\) ========================= 08:12:26 \(1491552746\) [ 2021.688094] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================= 08:12:26 (1491552746) [ 2023.109959] Lustre: DEBUG MARKER: sync; sync; sync [ 2024.026767] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2024.273944] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2024.415719] Turning device dm-0 (0xfc00000) read-only [ 2024.540254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2024.665515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2024.804127] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2025.051739] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2025.227585] Removing read-only on unknown block (0xfc00000) [ 2025.357087] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2029.549241] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 2029.551813] LustreError: Skipped 57 previous similar messages [ 2029.554102] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.558661] Lustre: Skipped 43 previous similar messages [ 2035.666590] Lustre: DEBUG MARKER: hostname [ 2035.971361] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2036.223193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2036.466582] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2036.620649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.684442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.688072] Lustre: Skipped 21 previous similar messages [ 2036.836257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2037.085983] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2037.573234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2037.574778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2037.738654] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2037.739747] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2037.881457] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2038.128024] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2038.374599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2038.625377] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2039.376628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2039.376629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2039.593270] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2039.598475] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2041.870299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2042.027925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2042.860891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2043.011429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2046.793717] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2047.672110] Lustre: DEBUG MARKER: rc=0; [ 2047.672110] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2047.672110] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2047.672110] echo $(hostname -s): $val; [ 2047.672110] rc=$val; [ 2047.672110] fi; [ 2047.672110] exit $rc [ 2048.278166] Lustre: DEBUG MARKER: dmesg [ 2048.722066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 39: test recovery from unlink llog \(test llog_gen_rec\) ========================= 08:12:53 \(1491552773\) [ 2048.882545] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================= 08:12:53 (1491552773) [ 2049.883186] Lustre: DEBUG MARKER: sync; sync; sync [ 2050.778789] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2051.031356] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2051.177302] Turning device dm-0 (0xfc00000) read-only [ 2051.301828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2051.425625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2052.019251] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2052.269224] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2052.533506] Removing read-only on unknown block (0xfc00000) [ 2052.669800] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2062.980164] Lustre: DEBUG MARKER: hostname [ 2063.284474] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2063.531234] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2063.775237] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2063.928493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.148492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2064.399041] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2064.890351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2064.900605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2065.057365] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2065.061389] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2065.199412] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2065.446651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2065.691951] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2065.947422] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2066.708618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2066.724929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2066.912749] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2066.963897] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2070.169185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2070.285113] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2070.470286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2070.566320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2074.320572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2075.194565] Lustre: DEBUG MARKER: rc=0; [ 2075.194565] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2075.194565] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2075.194565] echo $(hostname -s): $val; [ 2075.194565] rc=$val; [ 2075.194565] fi; [ 2075.194565] exit $rc [ 2075.781694] Lustre: DEBUG MARKER: dmesg [ 2076.212202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 08:13:21 \(1491552801\) [ 2076.369999] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ============================== 08:13:21 (1491552801) [ 2076.560780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2076.720477] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2076.910309] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2077.801535] Lustre: DEBUG MARKER: rc=0; [ 2077.801535] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2077.801535] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2077.801535] echo $(hostname -s): $val; [ 2077.801535] rc=$val; [ 2077.801535] fi; [ 2077.801535] exit $rc [ 2078.404219] Lustre: DEBUG MARKER: dmesg [ 2078.801488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 08:13:23 \(1491552803\) [ 2078.954929] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================= 08:13:23 (1491552803) [ 2079.129860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2079.129860] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2079.129860] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2079.390937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2079.390937] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2079.390937] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2079.650653] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2079.898841] Lustre: DEBUG MARKER: lctl get_param -n devices [ 2080.146294] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 21 deactivate [ 2080.269603] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2080.417435] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 21 activate [ 2080.723517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2081.593486] Lustre: DEBUG MARKER: rc=0; [ 2081.593486] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2081.593486] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2081.593486] echo $(hostname -s): $val; [ 2081.593486] rc=$val; [ 2081.593486] fi; [ 2081.593486] exit $rc [ 2082.183642] Lustre: DEBUG MARKER: dmesg [ 2082.578542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 42: recovery after ost failure ================================================= 08:13:27 \(1491552807\) [ 2082.740794] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ================================================= 08:13:27 (1491552807) [ 2087.210989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 2100.262807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2100.414040] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2141.643656] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 2142.455760] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2143.318746] Lustre: DEBUG MARKER: rc=0; [ 2143.318746] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2143.318746] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2143.318746] echo $(hostname -s): $val; [ 2143.318746] rc=$val; [ 2143.318746] fi; [ 2143.318746] exit $rc [ 2143.922599] Lustre: DEBUG MARKER: dmesg [ 2144.310799] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 43: mds osc import failure during recovery\; don\'t LBUG ========================= 08:14:29 \(1491552869\) [ 2144.487763] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================= 08:14:29 (1491552869) [ 2144.643827] Lustre: DEBUG MARKER: sync; sync; sync [ 2145.485304] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2145.730180] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2145.865602] LustreError: 22921:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 2145.867938] LustreError: 22921:0:(osd_handler.c:2171:osd_ro()) Skipped 19 previous similar messages [ 2145.903789] Turning device dm-0 (0xfc00000) read-only [ 2146.027969] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2146.152295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2146.549270] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2146.795185] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2146.922220] Lustre: Failing over lustre-MDT0000 [ 2146.924143] Lustre: Skipped 21 previous similar messages [ 2146.974214] Removing read-only on unknown block (0xfc00000) [ 2146.981117] Lustre: server umount lustre-MDT0000 complete [ 2146.983025] Lustre: Skipped 21 previous similar messages [ 2147.106663] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2157.405370] Lustre: DEBUG MARKER: hostname [ 2157.709276] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2157.751051] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491552874/real 1491552874] req@ffff880064bacf00 x1564004875228832/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491552881 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2157.757702] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 2157.760131] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.764337] LustreError: Skipped 21 previous similar messages [ 2157.964464] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2158.207109] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2158.349648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.766212] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1757ee02b to 0x52ac6ca175805b9e [ 2163.770401] Lustre: Skipped 21 previous similar messages [ 2163.978117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2164.226557] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2164.716306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2164.718202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2164.880414] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2164.881510] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2164.982003] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 2164.984677] Lustre: Skipped 16 previous similar messages [ 2165.024943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2165.278768] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2165.525609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2165.784651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2166.553638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2166.569707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2166.824199] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2166.874924] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2168.816729] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2168.822282] LustreError: Skipped 27 previous similar messages [ 2168.839247] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 2168.842396] Lustre: Skipped 16 previous similar messages [ 2169.051688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2169.082554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2169.233737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2169.338407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2175.854068] LustreError: 23509:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0003-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2180.674594] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2181.554683] Lustre: DEBUG MARKER: rc=0; [ 2181.554683] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2181.554683] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2181.554683] echo $(hostname -s): $val; [ 2181.554683] rc=$val; [ 2181.554683] fi; [ 2181.554683] exit $rc [ 2182.145027] Lustre: DEBUG MARKER: dmesg [ 2182.554269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 44a: race in target handle connect ============================================= 08:15:07 \(1491552907\) [ 2182.708647] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ============================================= 08:15:07 (1491552907) [ 2182.854302] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 2183.109106] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 2183.408537] Lustre: DEBUG MARKER: lctl set_param at_max=40 [ 2183.675900] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2183.932835] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2184.062194] LustreError: 5342:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2194.981909] LustreError: 25221:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2194.984495] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2194.987100] LustreError: 5342:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2195.120968] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2195.369597] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2195.495976] LustreError: 5342:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2204.981736] LustreError: 25221:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2204.984228] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2204.986751] Lustre: Skipped 1 previous similar message [ 2204.987259] LustreError: 5342:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2204.987268] Lustre: lustre-MDT0000: Export ffff88005fcd4800 already connecting from 10.9.5.237@tcp [ 2205.119820] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2205.367104] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2205.496501] LustreError: 25221:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2214.982091] LustreError: 5342:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2214.984431] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2214.984440] LustreError: 25221:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2214.984449] Lustre: lustre-MDT0000: Export ffff88005fcd4800 already connecting from 10.9.5.237@tcp [ 2215.118703] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2215.367062] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2215.494030] LustreError: 5342:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2224.981968] LustreError: 5340:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2224.984237] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2224.984242] LustreError: 5342:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2224.984250] Lustre: lustre-MDT0000: Export ffff88005fcd4800 already connecting from 10.9.5.237@tcp [ 2225.123140] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2225.372029] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2225.500029] LustreError: 5340:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2234.981767] LustreError: 5341:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2234.984043] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2234.986435] LustreError: 5340:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2235.115213] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2235.362113] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2235.489858] LustreError: 5340:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2244.982035] LustreError: 5341:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2244.984353] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2244.984356] LustreError: 5340:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2244.984365] Lustre: lustre-MDT0000: Export ffff88005fcd4800 already connecting from 10.9.5.237@tcp [ 2244.991562] Lustre: Skipped 1 previous similar message [ 2245.126643] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2245.372904] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2255.113749] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2255.365730] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2255.492650] LustreError: 5341:0:(libcfs_fail.h:165:cfs_race()) cfs_race id 701 sleeping [ 2255.494972] LustreError: 5341:0:(libcfs_fail.h:165:cfs_race()) Skipped 1 previous similar message [ 2264.982418] LustreError: 25221:0:(libcfs_fail.h:170:cfs_race()) cfs_fail_race id 701 waking [ 2264.984747] LustreError: 25221:0:(libcfs_fail.h:170:cfs_race()) Skipped 1 previous similar message [ 2264.987072] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2264.989467] Lustre: Skipped 2 previous similar messages [ 2264.991579] LustreError: 5341:0:(libcfs_fail.h:168:cfs_race()) cfs_fail_race id 701 awake, rc=0 [ 2264.993945] LustreError: 5341:0:(libcfs_fail.h:168:cfs_race()) Skipped 1 previous similar message [ 2265.121782] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2265.368800] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2275.112997] Lustre: DEBUG MARKER: lctl get_param -n md[ts].*.mdt.timeouts | grep service [ 2275.365540] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000701 [ 2285.112219] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2285.416593] Lustre: DEBUG MARKER: lctl set_param at_max=600 [ 2285.756230] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2286.656658] Lustre: DEBUG MARKER: rc=0; [ 2286.656658] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2286.656658] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2286.656658] echo $(hostname -s): $val; [ 2286.656658] rc=$val; [ 2286.656658] fi; [ 2286.656658] exit $rc [ 2287.258493] Lustre: DEBUG MARKER: dmesg [ 2287.687643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_44b skipping SLOW test 44b [ 2287.871580] Lustre: DEBUG MARKER: SKIP: replay-single test_44b skipping SLOW test 44b [ 2288.083648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 44c: race in target handle connect ============================================= 08:16:52 \(1491553012\) [ 2288.245439] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ============================================= 08:16:52 (1491553012) [ 2288.381069] Lustre: DEBUG MARKER: sync; sync; sync [ 2289.213894] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2289.460407] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2289.603191] Turning device dm-0 (0xfc00000) read-only [ 2289.726875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2289.851758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2290.046984] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000712 [ 2290.303846] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2290.548694] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2290.721235] Removing read-only on unknown block (0xfc00000) [ 2290.850184] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2291.147614] Lustre: DEBUG MARKER: hostname [ 2291.416573] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2291.658382] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2291.899972] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2292.053951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.123371] LustreError: 27813:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2292.125694] LustreError: 27813:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2292.128187] Lustre: 27861:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2292.130639] Lustre: 27861:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 2292.132856] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 2292.280976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2292.529451] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2293.015221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2293.016571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2293.175440] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2293.179107] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2293.318385] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2293.565089] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2293.810930] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2294.063753] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2296.834240] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2297.081106] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2297.422921] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2307.651145] LustreError: 5337:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [ 2307.731769] Lustre: DEBUG MARKER: hostname [ 2308.034696] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2308.282512] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2308.527748] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2308.681968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.836465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2314.086434] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2314.583563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2314.587540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2314.748274] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2314.754200] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2314.892037] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2315.137755] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2315.384336] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2315.636215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2316.387324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2316.402613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2316.611984] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2316.631664] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2318.684328] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [ 2318.687788] Lustre: Skipped 529 previous similar messages [ 2318.887758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2318.911195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2319.074199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2319.154158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2320.514267] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2321.408947] Lustre: DEBUG MARKER: rc=0; [ 2321.408947] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2321.408947] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2321.408947] echo $(hostname -s): $val; [ 2321.408947] rc=$val; [ 2321.408947] fi; [ 2321.408947] exit $rc [ 2322.000313] Lustre: DEBUG MARKER: dmesg [ 2322.399731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 45: Handle failed close ======================================================== 08:17:27 \(1491553047\) [ 2322.560616] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ======================================================== 08:17:27 (1491553047) [ 2322.572918] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2322.575721] Lustre: Skipped 5 previous similar messages [ 2323.775508] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2324.671548] Lustre: DEBUG MARKER: rc=0; [ 2324.671548] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2324.671548] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2324.671548] echo $(hostname -s): $val; [ 2324.671548] rc=$val; [ 2324.671548] fi; [ 2324.671548] exit $rc [ 2325.266936] Lustre: DEBUG MARKER: dmesg [ 2325.670862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 46: Don\'t leak file handle after open resend \(3325\) ============================ 08:17:30 \(1491553050\) [ 2325.822627] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ============================ 08:17:30 (1491553050) [ 2325.966513] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x122 [ 2326.093142] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2326.095245] LustreError: 5352:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005e689e00 x1564004849051744/t0(0) o700->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 264/248 e 0 to 0 dl 1491553057 ref 1 fl Interpret:/0/0 rc 0/0 [ 2333.223937] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2333.491698] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2333.750573] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2334.084633] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2344.388466] Lustre: DEBUG MARKER: hostname [ 2344.691040] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2344.949395] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2345.198425] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2345.355441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.555072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2345.806915] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2346.301848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2346.301932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2346.471352] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2346.473341] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2346.614515] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2346.861285] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2347.109745] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2347.367851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2348.116834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2348.119702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2348.292988] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2348.385441] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2350.620884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2350.801630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2351.507242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2351.664265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2353.001370] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2353.872504] Lustre: DEBUG MARKER: rc=0; [ 2353.872504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2353.872504] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2353.872504] echo $(hostname -s): $val; [ 2353.872504] rc=$val; [ 2353.872504] fi; [ 2353.872504] exit $rc [ 2354.472862] Lustre: DEBUG MARKER: dmesg [ 2354.885434] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 47: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 08:17:59 \(1491553079\) [ 2355.047940] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) =========================== 08:17:59 (1491553079) [ 2368.141169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2368.308938] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2369.747228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2369.773584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2370.052796] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2370.095715] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2371.286305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2371.435397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2371.508362] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2371.671663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 2434.455680] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2435.333601] Lustre: DEBUG MARKER: rc=0; [ 2435.333601] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2435.333601] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2435.333601] echo $(hostname -s): $val; [ 2435.333601] rc=$val; [ 2435.333601] fi; [ 2435.333601] exit $rc [ 2435.931310] Lustre: DEBUG MARKER: dmesg [ 2436.352333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 48: MDS-\>OSC failure during precreate cleanup \(2824\) =========================== 08:19:21 \(1491553161\) [ 2436.515900] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) =========================== 08:19:21 (1491553161) [ 2436.684616] Lustre: DEBUG MARKER: sync; sync; sync [ 2437.547696] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2437.807078] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2437.946250] Turning device dm-0 (0xfc00000) read-only [ 2438.076566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2438.202918] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2438.362072] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2438.608411] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2438.779704] Removing read-only on unknown block (0xfc00000) [ 2438.909909] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2449.216578] Lustre: DEBUG MARKER: hostname [ 2449.528695] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2449.774107] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2450.016776] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2450.170397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.510314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2454.760688] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2455.253433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.254303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.421718] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.422496] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2455.565870] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2455.813387] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.059698] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2456.309854] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2459.343654] LustreError: 1878:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0005-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2459.708276] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2460.590864] Lustre: DEBUG MARKER: rc=0; [ 2460.590864] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2460.590864] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2460.590864] echo $(hostname -s): $val; [ 2460.590864] rc=$val; [ 2460.590864] fi; [ 2460.590864] exit $rc [ 2461.181650] Lustre: DEBUG MARKER: dmesg [ 2461.583824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 50: Double OSC recovery, don\'t LASSERT \(3812\) ================================== 08:19:46 \(1491553186\) [ 2461.742110] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ================================== 08:19:46 (1491553186) [ 2461.874979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2461.874979] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2461.874979] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2462.141112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2462.141112] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2462.141112] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2462.402394] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 2462.652252] Lustre: DEBUG MARKER: lctl get_param -n devices [ 2462.899711] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 19 recover [ 2463.148875] Lustre: DEBUG MARKER: /usr/sbin/lctl --device 19 recover [ 2468.458421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2469.319208] Lustre: DEBUG MARKER: rc=0; [ 2469.319208] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2469.319208] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2469.319208] echo $(hostname -s): $val; [ 2469.319208] rc=$val; [ 2469.319208] fi; [ 2469.319208] exit $rc [ 2469.912907] Lustre: DEBUG MARKER: dmesg [ 2470.344430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 52: time out lock replay \(3764\) ================================================ 08:19:55 \(1491553195\) [ 2470.509371] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ================================================ 08:19:55 (1491553195) [ 2470.656449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 2470.656449] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 2470.656449] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 2470.936355] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000157 [ 2471.191703] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2471.437663] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2471.779784] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2474.215614] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.219982] LustreError: Skipped 408 previous similar messages [ 2482.093578] Lustre: DEBUG MARKER: hostname [ 2482.402854] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2482.648866] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2482.892314] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2483.036189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.177419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2488.438465] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2489.044168] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2489.044964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2489.209118] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2489.209332] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2489.352292] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2489.599565] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2489.846618] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2490.099785] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2490.853609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.865569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2491.075542] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2491.090750] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2493.014807] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2493.017549] LustreError: 4349:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f83a700 x1564004849063584/t0(0) o101->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 328/344 e 1 to 0 dl 1491553231 ref 1 fl Complete:/40/0 rc 0/0 [ 2507.093758] Lustre: lustre-MDT0000: Recovery already passed deadline 0:48. If you do not want to wait more, please abort the recovery by force. [ 2507.407613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 2507.408500] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 2507.562356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 2507.648555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 2508.954431] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 2509.283975] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2510.159333] Lustre: DEBUG MARKER: rc=0; [ 2510.159333] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2510.159333] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2510.159333] echo $(hostname -s): $val; [ 2510.159333] rc=$val; [ 2510.159333] fi; [ 2510.159333] exit $rc [ 2510.757191] Lustre: DEBUG MARKER: dmesg [ 2511.177934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53a: \|X\| close request while two MDC requests in flight ======================== 08:20:36 \(1491553236\) [ 2511.335343] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ======================== 08:20:36 (1491553236) [ 2512.495506] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 2512.641024] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2512.773288] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2513.027490] Lustre: DEBUG MARKER: sync; sync; sync [ 2513.306662] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2513.555656] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2513.693340] Turning device dm-0 (0xfc00000) read-only [ 2513.817366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2513.941603] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2514.078474] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2514.323558] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2514.496157] Removing read-only on unknown block (0xfc00000) [ 2514.625467] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2524.934491] Lustre: DEBUG MARKER: hostname [ 2525.242091] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2525.487895] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2525.731282] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2525.875305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.094232] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (not set up) [ 2531.096558] Lustre: Skipped 7 previous similar messages [ 2531.303749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2531.554575] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2532.049081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2532.049839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2532.209869] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2532.212694] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2532.351373] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2532.597735] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2532.842769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2533.094393] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2533.837564] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2533.837997] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2534.001417] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2534.073375] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.354912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2536.505227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2537.230882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2537.396989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2538.750780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2539.658787] Lustre: DEBUG MARKER: rc=0; [ 2539.658787] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2539.658787] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2539.658787] echo $(hostname -s): $val; [ 2539.658787] rc=$val; [ 2539.658787] fi; [ 2539.658787] exit $rc [ 2540.266565] Lustre: DEBUG MARKER: dmesg [ 2540.669130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53b: \|X\| open request while two MDC requests in flight ========================= 08:21:05 \(1491553265\) [ 2540.822271] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================= 08:21:05 (1491553265) [ 2540.974571] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 2541.110755] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2542.231246] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2542.488716] Lustre: DEBUG MARKER: sync; sync; sync [ 2542.761543] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2543.007655] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2543.135876] Turning device dm-0 (0xfc00000) read-only [ 2543.260438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2543.386534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2543.524051] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2543.777463] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2543.949881] Removing read-only on unknown block (0xfc00000) [ 2544.079273] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2554.377408] Lustre: DEBUG MARKER: hostname [ 2554.681583] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2554.927047] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2555.169144] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2555.321973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.562361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2559.812826] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2560.301496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2560.314107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2560.472980] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2560.472983] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2560.617247] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2560.867859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2561.113323] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2561.363469] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2562.114739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2562.127699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2562.330590] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2562.345262] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2565.601775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2565.650895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2565.774523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2565.889975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2567.231134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2568.117422] Lustre: DEBUG MARKER: rc=0; [ 2568.117422] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2568.117422] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2568.117422] echo $(hostname -s): $val; [ 2568.117422] rc=$val; [ 2568.117422] fi; [ 2568.117422] exit $rc [ 2568.711209] Lustre: DEBUG MARKER: dmesg [ 2569.114029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53c: \|X\| open request and close request while two MDC requests in flight ======= 08:21:33 \(1491553293\) [ 2569.267176] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ======= 08:21:33 (1491553293) [ 2569.415362] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 2569.543243] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2570.668625] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 2570.918736] Lustre: DEBUG MARKER: sync; sync; sync [ 2571.232051] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2571.489438] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2571.633921] Turning device dm-0 (0xfc00000) read-only [ 2571.763850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2571.891163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2572.032663] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2572.285325] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2572.459456] Removing read-only on unknown block (0xfc00000) [ 2572.590509] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2582.888908] Lustre: DEBUG MARKER: hostname [ 2583.184912] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2583.430465] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2583.673337] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2583.827598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.332908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2588.582366] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2589.071983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2589.084177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2589.241505] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2589.251759] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2589.387437] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2589.636048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2589.883544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2590.135445] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2595.318495] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2595.649358] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2596.545195] Lustre: DEBUG MARKER: rc=0; [ 2596.545195] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2596.545195] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2596.545195] echo $(hostname -s): $val; [ 2596.545195] rc=$val; [ 2596.545195] fi; [ 2596.545195] exit $rc [ 2597.144975] Lustre: DEBUG MARKER: dmesg [ 2597.545472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53d: close reply while two MDC requests in flight ============================== 08:22:02 \(1491553322\) [ 2597.710666] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ============================== 08:22:02 (1491553322) [ 2598.861072] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 2598.989560] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2598.991696] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2598.993796] LustreError: 3046:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880079899500 x1564004849070192/t249108103185(0) o35->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 512/456 e 0 to 0 dl 1491553330 ref 1 fl Interpret:/0/0 rc 0/0 [ 2599.119941] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2599.378855] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2599.625686] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2599.966993] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2610.276063] Lustre: DEBUG MARKER: hostname [ 2610.576819] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2610.822600] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2611.069829] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2611.222385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.441554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2616.691223] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2617.188057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2617.188102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2617.351261] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2617.353020] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2617.492417] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2617.739702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2617.985747] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2618.239875] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2618.990291] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2619.002187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2619.168596] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2619.229133] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.503607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2621.660623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2622.414203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2622.564891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2623.917494] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2624.810091] Lustre: DEBUG MARKER: rc=0; [ 2624.810091] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2624.810091] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2624.810091] echo $(hostname -s): $val; [ 2624.810091] rc=$val; [ 2624.810091] fi; [ 2624.810091] exit $rc [ 2625.450613] Lustre: DEBUG MARKER: dmesg [ 2625.869191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53e: \|X\| open reply while two MDC requests in flight =========================== 08:22:30 \(1491553350\) [ 2626.030607] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight =========================== 08:22:30 (1491553350) [ 2626.181249] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 2626.318985] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2626.322310] LustreError: 25221:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880064e50900 x1564004849071952/t253403070480(0) o36->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491553357 ref 1 fl Interpret:/0/0 rc 0/0 [ 2627.457419] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2627.721401] Lustre: DEBUG MARKER: sync; sync; sync [ 2628.020127] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2628.267973] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2628.397129] Turning device dm-0 (0xfc00000) read-only [ 2628.520909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2628.644355] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2628.780936] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2629.034156] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2629.209768] Removing read-only on unknown block (0xfc00000) [ 2629.345490] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2631.255280] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 2631.257916] LustreError: Skipped 34 previous similar messages [ 2631.260061] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.264628] Lustre: Skipped 31 previous similar messages [ 2639.667601] Lustre: DEBUG MARKER: hostname [ 2639.976185] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2640.226864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2640.469472] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2640.625169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.310381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.312752] Lustre: Skipped 11 previous similar messages [ 2644.459593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2644.710484] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2645.209505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2645.222921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2645.385727] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2645.391346] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2645.529245] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2645.777845] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2646.022746] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2646.274664] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2647.016377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.029212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.229346] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.246806] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2649.533637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2649.690811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2650.474029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2650.627426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2651.967866] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2652.845738] Lustre: DEBUG MARKER: rc=0; [ 2652.845738] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2652.845738] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2652.845738] echo $(hostname -s): $val; [ 2652.845738] rc=$val; [ 2652.845738] fi; [ 2652.845738] exit $rc [ 2653.449574] Lustre: DEBUG MARKER: dmesg [ 2653.855187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53f: \|X\| open reply and close reply while two MDC requests in flight =========== 08:22:58 \(1491553378\) [ 2654.012322] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight =========== 08:22:58 (1491553378) [ 2654.165511] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 2654.295051] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2654.297494] LustreError: 5342:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880064bac000 x1564004849073696/t257698037776(0) o36->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491553385 ref 1 fl Interpret:/0/0 rc 0/0 [ 2655.419760] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 2655.554002] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2655.679536] Lustre: DEBUG MARKER: sync; sync; sync [ 2656.003509] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2656.248932] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2656.386608] Turning device dm-0 (0xfc00000) read-only [ 2656.509855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2656.633341] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2656.773483] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2657.019934] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2657.191923] Removing read-only on unknown block (0xfc00000) [ 2657.321367] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2667.623619] Lustre: DEBUG MARKER: hostname [ 2667.924991] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2668.182527] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2668.436620] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2668.598019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.697406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2671.945278] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2672.437954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2672.453115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2672.611158] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2672.612004] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2672.753904] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2673.002413] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2673.248458] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2673.499362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2678.684071] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2679.011783] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2679.899987] Lustre: DEBUG MARKER: rc=0; [ 2679.899987] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2679.899987] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2679.899987] echo $(hostname -s): $val; [ 2679.899987] rc=$val; [ 2679.899987] fi; [ 2679.899987] exit $rc [ 2680.499085] Lustre: DEBUG MARKER: dmesg [ 2680.910677] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53g: \|X\| drop open reply and close request while close and open are both in flight ====================================================================================================== 08:23:25 \(1491553405\) [ 2681.068123] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ====================================================================================================== 08:23:25 (1491553405) [ 2681.214626] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 2681.350600] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2681.353824] Lustre: Skipped 1 previous similar message [ 2681.356195] LustreError: 5340:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f83b900 x1564004849075088/t261993005072(0) o36->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 616/448 e 0 to 0 dl 1491553412 ref 1 fl Interpret:/0/0 rc 0/0 [ 2681.363444] LustreError: 5340:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 1 previous similar message [ 2682.471438] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [ 2682.598910] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2682.601389] Lustre: Skipped 1 previous similar message [ 2682.723915] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2682.972045] Lustre: DEBUG MARKER: sync; sync; sync [ 2683.371841] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2683.616634] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2683.754359] Turning device dm-0 (0xfc00000) read-only [ 2683.878444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2684.002381] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2684.139723] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2684.384141] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2684.555642] Removing read-only on unknown block (0xfc00000) [ 2684.685567] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2694.987149] Lustre: DEBUG MARKER: hostname [ 2695.293657] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2695.537333] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2695.778731] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2695.932548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.705200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2699.963303] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2700.453165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.457074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.618786] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.620313] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2700.760140] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2701.007527] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2701.253971] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2701.506106] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2706.773933] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2707.662864] Lustre: DEBUG MARKER: rc=0; [ 2707.662864] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2707.662864] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2707.662864] echo $(hostname -s): $val; [ 2707.662864] rc=$val; [ 2707.662864] fi; [ 2707.662864] exit $rc [ 2708.259472] Lustre: DEBUG MARKER: dmesg [ 2708.678063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 53h: open request and close reply while two MDC requests in flight ============= 08:23:53 \(1491553433\) [ 2708.829964] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ============= 08:23:53 (1491553433) [ 2708.984664] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107 [ 2709.113835] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2710.239962] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013b [ 2710.366575] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2711.492421] Lustre: DEBUG MARKER: sync; sync; sync [ 2711.814850] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2712.061127] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2712.198523] Turning device dm-0 (0xfc00000) read-only [ 2712.321860] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2712.445718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2712.581964] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2712.827423] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2712.999476] Removing read-only on unknown block (0xfc00000) [ 2713.129366] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2723.427671] Lustre: DEBUG MARKER: hostname [ 2723.732604] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2723.978155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2724.219700] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2724.376222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.557881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2728.808471] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2729.306726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2729.306766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2729.468841] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2729.473240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2729.610353] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2729.858445] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2730.105200] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2730.355863] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2735.543160] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2735.874206] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2736.765624] Lustre: DEBUG MARKER: rc=0; [ 2736.765624] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2736.765624] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2736.765624] echo $(hostname -s): $val; [ 2736.765624] rc=$val; [ 2736.765624] fi; [ 2736.765624] exit $rc [ 2737.361368] Lustre: DEBUG MARKER: dmesg [ 2737.774983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 08:24:22 \(1491553462\) [ 2737.938528] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========== 08:24:22 (1491553462) [ 2738.083357] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012b [ 2738.212334] Lustre: *** cfs_fail_loc=12b, val=2147484428*** [ 2743.334857] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 2743.646170] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2744.523258] Lustre: DEBUG MARKER: rc=0; [ 2744.523258] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2744.523258] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2744.523258] echo $(hostname -s): $val; [ 2744.523258] rc=$val; [ 2744.523258] fi; [ 2744.523258] exit $rc [ 2745.119556] Lustre: DEBUG MARKER: dmesg [ 2745.516346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 56: don\'t replay a symlink open request \(3440\) ================================= 08:24:30 \(1491553470\) [ 2745.670590] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ================================= 08:24:30 (1491553470) [ 2745.853712] Lustre: DEBUG MARKER: sync; sync; sync [ 2746.691393] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2746.938525] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2747.073912] LustreError: 23758:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 2747.076320] LustreError: 23758:0:(osd_handler.c:2171:osd_ro()) Skipped 9 previous similar messages [ 2747.128189] Turning device dm-0 (0xfc00000) read-only [ 2747.252852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2747.377408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2747.514695] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2747.758900] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2747.883781] Lustre: Failing over lustre-MDT0000 [ 2747.885553] Lustre: Skipped 13 previous similar messages [ 2747.933930] Removing read-only on unknown block (0xfc00000) [ 2747.940843] Lustre: server umount lustre-MDT0000 complete [ 2747.942546] Lustre: Skipped 13 previous similar messages [ 2748.066192] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2758.374143] Lustre: DEBUG MARKER: hostname [ 2758.679872] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2758.925559] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2759.167184] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2759.320941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.402039] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491553481/real 1491553481] req@ffff88005ff3a700 x1564004875299712/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491553487 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 2761.408463] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [ 2761.606715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2761.857563] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2762.347890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2762.355333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2762.509611] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2762.512212] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2762.651209] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2762.898510] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2763.144283] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2763.397490] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2764.225246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.226408] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.466882] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.483412] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2766.691131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2766.705915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2766.895294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2766.943790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2778.349505] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2779.232874] Lustre: DEBUG MARKER: rc=0; [ 2779.232874] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2779.232874] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2779.232874] echo $(hostname -s): $val; [ 2779.232874] rc=$val; [ 2779.232874] fi; [ 2779.232874] exit $rc [ 2779.832035] Lustre: DEBUG MARKER: dmesg [ 2780.247729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 57: test recovery from llog for setattr op ===================================== 08:25:05 \(1491553505\) [ 2780.405056] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ===================================== 08:25:05 (1491553505) [ 2780.537780] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012c [ 2780.791102] Lustre: DEBUG MARKER: sync; sync; sync [ 2781.662497] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2781.910050] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2782.052839] Turning device dm-0 (0xfc00000) read-only [ 2782.176846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2782.300633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2782.439706] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2782.685395] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2782.859175] Removing read-only on unknown block (0xfc00000) [ 2782.990472] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2793.295222] Lustre: DEBUG MARKER: hostname [ 2793.603531] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2793.672052] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.676281] LustreError: Skipped 14 previous similar messages [ 2793.854937] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2794.098160] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2794.252519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.678292] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca17580ce6f to 0x52ac6ca17580d450 [ 2799.682578] Lustre: Skipped 14 previous similar messages [ 2799.905658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2800.156095] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2800.670331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2800.689494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2800.857186] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2800.870935] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2801.005296] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2801.252303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2801.497380] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2801.749612] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2802.089710] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 2802.092376] Lustre: Skipped 13 previous similar messages [ 2802.512252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.514789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.697820] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.747292] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2804.720364] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2804.725568] LustreError: Skipped 15 previous similar messages [ 2804.740640] Lustre: lustre-MDT0000: Recovery over after 0:03, of 5 clients 5 recovered and 0 were evicted. [ 2804.743645] Lustre: Skipped 13 previous similar messages [ 2804.994197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2805.025820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2805.196794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2805.234434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2807.549530] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 2807.870480] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2808.769100] Lustre: DEBUG MARKER: rc=0; [ 2808.769100] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2808.769100] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2808.769100] echo $(hostname -s): $val; [ 2808.769100] rc=$val; [ 2808.769100] fi; [ 2808.769100] exit $rc [ 2809.377601] Lustre: DEBUG MARKER: dmesg [ 2809.787862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58a: test recovery from llog for setattr op \(test llog_gen_rec\) ================ 08:25:34 \(1491553534\) [ 2809.944885] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ================ 08:25:34 (1491553534) [ 2810.094586] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000012c [ 2812.980334] Lustre: DEBUG MARKER: sync; sync; sync [ 2813.950451] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2814.196241] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2814.340602] Turning device dm-0 (0xfc00000) read-only [ 2814.463959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2814.588648] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2814.727812] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2814.974447] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2815.159308] Removing read-only on unknown block (0xfc00000) [ 2815.290718] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2825.591345] Lustre: DEBUG MARKER: hostname [ 2825.891074] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2826.137304] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2826.380607] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2826.535678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.730766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2826.979074] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2827.466186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2827.469256] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2827.629423] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2827.629529] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2827.773916] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2828.020825] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2828.267954] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2828.524224] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2829.272165] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.272179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.458290] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.512132] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.675194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2832.777227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2832.832218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2832.965630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2838.820113] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 2840.560296] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2841.451708] Lustre: DEBUG MARKER: rc=0; [ 2841.451708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2841.451708] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2841.451708] echo $(hostname -s): $val; [ 2841.451708] rc=$val; [ 2841.451708] fi; [ 2841.451708] exit $rc [ 2842.310405] Lustre: DEBUG MARKER: dmesg [ 2842.913092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58b: test replay of setxattr op ================================================ 08:26:07 \(1491553567\) [ 2843.070696] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ================================================ 08:26:07 (1491553567) [ 2843.207489] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [ 2843.207489] grep -E -q '(ea_inode|large_xattr)' [ 2843.572189] Lustre: DEBUG MARKER: sync; sync; sync [ 2844.450373] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2844.714119] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2844.864475] Turning device dm-0 (0xfc00000) read-only [ 2844.995744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2845.131477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2845.273934] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2845.521411] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2845.702880] Removing read-only on unknown block (0xfc00000) [ 2845.832619] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2856.139030] Lustre: DEBUG MARKER: hostname [ 2856.445167] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2856.691608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2856.936198] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2857.078992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.126197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2860.376519] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2860.879184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2860.882034] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2861.041615] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2861.045540] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2861.184025] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2861.432716] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2861.678952] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2861.931205] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2862.680442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.707885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.856113] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.962248] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.079319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2865.247997] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2865.263833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2865.455782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2867.488703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2867.491529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2867.687253] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2867.722862] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2867.955110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2867.986361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2868.106760] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2868.222271] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2868.433978] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2869.316138] Lustre: DEBUG MARKER: rc=0; [ 2869.316138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2869.316138] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2869.316138] echo $(hostname -s): $val; [ 2869.316138] rc=$val; [ 2869.316138] fi; [ 2869.316138] exit $rc [ 2869.938918] Lustre: DEBUG MARKER: dmesg [ 2870.351467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 58c: resend\/reconstruct setxattr op ============================================ 08:26:35 \(1491553595\) [ 2870.508883] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ============================================ 08:26:35 (1491553595) [ 2870.654421] Lustre: DEBUG MARKER: dumpe2fs -h /dev/lvm-Role_MDS/P1 2>&1 | [ 2870.654421] grep -E -q '(ea_inode|large_xattr)' [ 2875.980390] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x123 [ 2876.123953] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 2883.123746] Lustre: lustre-MDT0000: Client 52ca55ea-bc04-c94b-13c1-152cf6cc74f0 (at 10.9.5.237@tcp) reconnecting [ 2883.126473] Lustre: Skipped 2 previous similar messages [ 2883.259143] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 2883.519844] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x119 [ 2883.649901] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2883.652215] Lustre: Skipped 1 previous similar message [ 2883.654458] LustreError: 5340:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880064c32a00 x1564004849328736/t287762808840(0) o36->52ca55ea-bc04-c94b-13c1-152cf6cc74f0@10.9.5.237@tcp:-1/-1 lens 624/440 e 0 to 0 dl 1491553615 ref 1 fl Interpret:/0/0 rc 0/0 [ 2883.661761] LustreError: 5340:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 1 previous similar message [ 2890.776405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 2891.245614] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2892.121889] Lustre: DEBUG MARKER: rc=0; [ 2892.121889] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2892.121889] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2892.121889] echo $(hostname -s): $val; [ 2892.121889] rc=$val; [ 2892.121889] fi; [ 2892.121889] exit $rc [ 2892.724483] Lustre: DEBUG MARKER: dmesg [ 2893.126621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 08:26:57 \(1491553617\) [ 2893.292847] Lustre: DEBUG MARKER: == replay-single test 59: test log_commit_thread vs filter_destroy race ============================== 08:26:57 (1491553617) [ 2907.070582] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2907.235926] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2908.666408] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2908.666437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2908.863814] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2908.896395] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 2911.117246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 2911.186916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 2911.289634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 2911.411649] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 2912.690221] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2912.938929] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2913.308438] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2923.611376] Lustre: DEBUG MARKER: hostname [ 2923.915545] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2924.162499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2924.406941] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2924.561841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.656432] Lustre: MGS: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [ 2925.659064] Lustre: Skipped 411 previous similar messages [ 2927.033399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2927.283161] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2927.777788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2927.778767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2927.943117] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2927.943118] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2928.088960] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2928.338505] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2928.585557] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2928.835967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2929.601973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.618417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.817555] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.851942] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2933.054304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2933.159628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2933.206415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2933.368338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2954.982205] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2955.866974] Lustre: DEBUG MARKER: rc=0; [ 2955.866974] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2955.866974] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2955.866974] echo $(hostname -s): $val; [ 2955.866974] rc=$val; [ 2955.866974] fi; [ 2955.866974] exit $rc [ 2956.474893] Lustre: DEBUG MARKER: dmesg [ 2956.880552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 60: test llog post recovery init vs llog unlink ================================ 08:28:01 \(1491553681\) [ 2957.042967] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ================================ 08:28:01 (1491553681) [ 2957.382238] Lustre: DEBUG MARKER: sync; sync; sync [ 2958.246798] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 2958.494832] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 2958.627345] Turning device dm-0 (0xfc00000) read-only [ 2958.751893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 2958.875665] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2959.120133] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 2959.366456] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 2959.544398] Removing read-only on unknown block (0xfc00000) [ 2959.674417] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 2969.974881] Lustre: DEBUG MARKER: hostname [ 2970.274660] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 2970.522680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 2970.765115] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 2970.919770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.084757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 2975.335854] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 2975.838375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2975.849807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2976.009965] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2976.015801] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 2976.151156] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 2976.398436] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2976.644719] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 2976.898655] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 2977.654622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.671236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.847311] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.918704] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2981.086562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2981.173935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2981.242394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2981.370753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2982.841373] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 2983.726366] Lustre: DEBUG MARKER: rc=0; [ 2983.726366] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 2983.726366] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 2983.726366] echo $(hostname -s): $val; [ 2983.726366] rc=$val; [ 2983.726366] fi; [ 2983.726366] exit $rc [ 2984.332990] Lustre: DEBUG MARKER: dmesg [ 2984.751362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 08:28:29 \(1491553709\) [ 2984.903556] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup =================================== 08:28:29 (1491553709) [ 3001.964911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3002.132320] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3025.915906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3026.074558] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3027.522483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3027.522484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3027.714137] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3027.748362] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3030.946025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 3031.066448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 3031.113928] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 3031.245631] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 3062.923301] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3063.842407] Lustre: DEBUG MARKER: rc=0; [ 3063.842407] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3063.842407] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3063.842407] echo $(hostname -s): $val; [ 3063.842407] rc=$val; [ 3063.842407] fi; [ 3063.842407] exit $rc [ 3064.457231] Lustre: DEBUG MARKER: dmesg [ 3064.872505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 08:29:49 \(1491553789\) [ 3065.030591] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup =================================== 08:29:49 (1491553789) [ 3065.171873] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000013a [ 3065.426637] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3065.671120] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3065.996120] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3074.931181] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.936277] LustreError: Skipped 640 previous similar messages [ 3076.300574] Lustre: DEBUG MARKER: hostname [ 3076.608782] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3076.861885] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3077.106600] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3077.261937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.457353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3077.707557] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3078.202595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3078.204234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3078.370886] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3078.371962] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3078.516692] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3078.764346] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3079.010922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3079.267076] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3089.533906] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3089.781114] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3090.133950] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3100.447668] Lustre: DEBUG MARKER: hostname [ 3100.747574] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3100.992916] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3101.236396] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3101.390759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.223954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3105.474145] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3105.970334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3105.972238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3106.137429] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3106.143650] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3106.282956] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3106.532103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3106.779465] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3107.039317] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3107.811966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3107.818618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.075361] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.195548] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3110.324242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3110.458696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3110.478613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3110.645472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3112.008308] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3112.888079] Lustre: DEBUG MARKER: rc=0; [ 3112.888079] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3112.888079] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3112.888079] echo $(hostname -s): $val; [ 3112.888079] rc=$val; [ 3112.888079] fi; [ 3112.888079] exit $rc [ 3113.485134] Lustre: DEBUG MARKER: dmesg [ 3113.899496] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 08:30:38 \(1491553838\) [ 3114.057292] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup =================================== 08:30:38 (1491553838) [ 3137.348108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3137.512561] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3138.968761] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3139.011796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3139.193740] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3139.211092] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 3140.480327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 3140.507641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 3140.668946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 3140.730408] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 3142.438532] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3143.345958] Lustre: DEBUG MARKER: rc=0; [ 3143.345958] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3143.345958] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3143.345958] echo $(hostname -s): $val; [ 3143.345958] rc=$val; [ 3143.345958] fi; [ 3143.345958] exit $rc [ 3143.953721] Lustre: DEBUG MARKER: dmesg [ 3144.361148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-single test_61d skipping ALWAYS excluded test 61d [ 3144.518778] Lustre: DEBUG MARKER: SKIP: replay-single test_61d skipping ALWAYS excluded test 61d [ 3144.729277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 62: don\'t mis-drop resent replay =============================================== 08:31:09 \(1491553869\) [ 3144.895905] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay =============================================== 08:31:09 (1491553869) [ 3145.044587] Lustre: DEBUG MARKER: sync; sync; sync [ 3146.424084] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3146.680524] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3146.829683] Turning device dm-0 (0xfc00000) read-only [ 3146.956524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3147.090335] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3147.258785] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000707 [ 3147.526555] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3147.778926] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3147.959187] Removing read-only on unknown block (0xfc00000) [ 3148.094477] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3158.400292] Lustre: DEBUG MARKER: hostname [ 3158.698419] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3158.943691] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3159.188209] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3159.332771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.162447] Lustre: *** cfs_fail_loc=707, val=0*** [ 3164.280958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3164.537143] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3165.025905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3165.034141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3165.193354] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3165.200260] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3165.336985] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3165.584906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3165.831179] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3166.081978] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3166.847313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3166.861936] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3167.057987] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3167.096316] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.171383] Lustre: lustre-MDT0000: Recovery already passed deadline 0:59. If you do not want to wait more, please abort the recovery by force. [ 3172.300331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3172.408571] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3172.504476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3172.610362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 3173.891750] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [ 3174.234043] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3175.140146] Lustre: DEBUG MARKER: rc=0; [ 3175.140146] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3175.140146] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3175.140146] echo $(hostname -s): $val; [ 3175.140146] rc=$val; [ 3175.140146] fi; [ 3175.140146] exit $rc [ 3175.749719] Lustre: DEBUG MARKER: dmesg [ 3176.157839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 65a: AT: verify early replies ================================================== 08:31:40 \(1491553900\) [ 3176.321836] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ================================================== 08:31:40 (1491553900) [ 3176.492026] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3177.019773] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3177.554728] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3177.814552] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3198.471316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3198.722939] Lustre: DEBUG MARKER: lctl set_param fail_val=6000 [ 3198.971189] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3199.098212] LustreError: 7897:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 3205.100050] LustreError: 7897:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3205.550540] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3214.909867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3215.789999] Lustre: DEBUG MARKER: rc=0; [ 3215.789999] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3215.789999] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3215.789999] echo $(hostname -s): $val; [ 3215.789999] rc=$val; [ 3215.789999] fi; [ 3215.789999] exit $rc [ 3216.395279] Lustre: DEBUG MARKER: dmesg [ 3216.830480] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 65b: AT: verify early replies on packed reply \/ bulk =========================== 08:32:21 \(1491553941\) [ 3216.977206] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk =========================== 08:32:21 (1491553941) [ 3217.142592] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3217.670470] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3218.198427] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3218.444849] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3238.953765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [ 3246.504145] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [ 3246.838896] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3247.735120] Lustre: DEBUG MARKER: rc=0; [ 3247.735120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3247.735120] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3247.735120] echo $(hostname -s): $val; [ 3247.735120] rc=$val; [ 3247.735120] fi; [ 3247.735120] exit $rc [ 3248.327399] Lustre: DEBUG MARKER: dmesg [ 3248.747928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 08:32:53 \(1491553973\) [ 3248.908042] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ================= 08:32:53 (1491553973) [ 3249.069179] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3249.586902] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3250.106169] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3250.351468] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3270.868077] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=5000 [ 3271.115605] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3271.243247] LustreError: 7897:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3276.245045] LustreError: 7897:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3276.415172] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10000 [ 3276.662391] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x8000050a [ 3276.789932] LustreError: 7897:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3286.792036] LustreError: 7897:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3286.960078] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 3296.318475] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3297.214777] Lustre: DEBUG MARKER: rc=0; [ 3297.214777] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3297.214777] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3297.214777] echo $(hostname -s): $val; [ 3297.214777] rc=$val; [ 3297.214777] fi; [ 3297.214777] exit $rc [ 3297.810411] Lustre: DEBUG MARKER: dmesg [ 3298.229252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 66b: AT: verify net latency adjusts ============================================ 08:33:43 \(1491554023\) [ 3298.389553] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ============================================ 08:33:43 (1491554023) [ 3298.546183] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3299.067895] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3299.593869] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3299.841838] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3332.451853] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3333.320690] Lustre: DEBUG MARKER: rc=0; [ 3333.320690] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3333.320690] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3333.320690] echo $(hostname -s): $val; [ 3333.320690] rc=$val; [ 3333.320690] fi; [ 3333.320690] exit $rc [ 3333.915810] Lustre: DEBUG MARKER: dmesg [ 3334.349942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 67a: AT: verify slow request processing doesn\'t induce reconnects ============== 08:34:19 \(1491554059\) [ 3334.500791] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ============== 08:34:19 (1491554059) [ 3334.677125] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3335.194921] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3335.714726] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3335.961163] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3357.362622] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3358.230606] Lustre: DEBUG MARKER: rc=0; [ 3358.230606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3358.230606] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3358.230606] echo $(hostname -s): $val; [ 3358.230606] rc=$val; [ 3358.230606] fi; [ 3358.230606] exit $rc [ 3358.834947] Lustre: DEBUG MARKER: dmesg [ 3359.248100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 67b: AT: verify instant slowdown doesn\'t induce reconnects ===================== 08:34:44 \(1491554084\) [ 3359.406807] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ===================== 08:34:44 (1491554084) [ 3359.575376] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3360.097856] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3360.623524] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3360.872188] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3381.414135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3381.414135] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3381.414135] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3381.689478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 3381.689478] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 3381.689478] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 3381.952387] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 3382.200368] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 3382.448957] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 3383.863777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark phase 2 [ 3384.025662] Lustre: DEBUG MARKER: phase 2 [ 3385.378076] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3386.284911] Lustre: DEBUG MARKER: rc=0; [ 3386.284911] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3386.284911] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3386.284911] echo $(hostname -s): $val; [ 3386.284911] rc=$val; [ 3386.284911] fi; [ 3386.284911] exit $rc [ 3386.879215] Lustre: DEBUG MARKER: dmesg [ 3387.303383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 68: AT: verify slowing locks =================================================== 08:35:12 \(1491554112\) [ 3387.463701] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks =================================================== 08:35:12 (1491554112) [ 3387.625744] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3388.156359] Lustre: DEBUG MARKER: lctl get_param -n at_max [ 3388.682660] Lustre: DEBUG MARKER: lctl get_param -n at_history [ 3388.928604] Lustre: DEBUG MARKER: lctl set_param at_history=8 [ 3454.694289] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3455.638086] Lustre: DEBUG MARKER: rc=0; [ 3455.638086] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3455.638086] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3455.638086] echo $(hostname -s): $val; [ 3455.638086] rc=$val; [ 3455.638086] fi; [ 3455.638086] exit $rc [ 3456.248908] Lustre: DEBUG MARKER: dmesg [ 3456.680965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70a: check multi client t-f ==================================================== 08:36:21 \(1491554181\) [ 3456.839084] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ==================================================== 08:36:21 (1491554181) [ 3458.003426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3458.158645] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3459.837449] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3460.725948] Lustre: DEBUG MARKER: rc=0; [ 3460.725948] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3460.725948] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3460.725948] echo $(hostname -s): $val; [ 3460.725948] rc=$val; [ 3460.725948] fi; [ 3460.725948] exit $rc [ 3461.330320] Lustre: DEBUG MARKER: dmesg [ 3461.756399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70b: dbench 4mdts recovery\; 2 clients ========================================== 08:36:26 \(1491554186\) [ 3461.915121] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 4mdts recovery; 2 clients ========================================== 08:36:26 (1491554186) [ 3463.117379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3463.278548] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3470.760780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=20919 ... [ 3470.949692] Lustre: DEBUG MARKER: Started rundbench load pid=20919 ... [ 3472.426518] Lustre: DEBUG MARKER: sync; sync; sync [ 3473.619934] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3473.908638] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3474.051556] LustreError: 447:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 3474.054141] LustreError: 447:0:(osd_handler.c:2171:osd_ro()) Skipped 5 previous similar messages [ 3474.096503] Turning device dm-0 (0xfc00000) read-only [ 3474.269887] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3474.425371] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3475.650319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds1 1 times [ 3475.849136] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3476.013266] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3476.334264] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3476.508926] Lustre: Failing over lustre-MDT0000 [ 3476.511188] Lustre: Skipped 8 previous similar messages [ 3476.519901] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 3476.735086] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005ac22d00 x1564004875493616/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 3476.740160] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 3478.433217] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 3478.435638] Lustre: Skipped 1 previous similar message [ 3479.113145] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 3479.115757] LustreError: Skipped 35 previous similar messages [ 3479.117981] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.123148] Lustre: Skipped 33 previous similar messages [ 3481.752075] Removing read-only on unknown block (0xfc00000) [ 3481.759355] Lustre: server umount lustre-MDT0000 complete [ 3481.761445] Lustre: Skipped 8 previous similar messages [ 3481.892739] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3491.687047] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491554209/real 1491554209] req@ffff88005ff39500 x1564004875494016/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491554216 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 3491.694068] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 3491.696553] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3491.701069] LustreError: Skipped 7 previous similar messages [ 3492.201745] Lustre: DEBUG MARKER: hostname [ 3492.508806] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3492.762501] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3493.008219] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3493.162749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.703218] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca17585cba9 to 0x52ac6ca175874f65 [ 3497.707541] Lustre: Skipped 7 previous similar messages [ 3497.765281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.767673] Lustre: Skipped 12 previous similar messages [ 3497.937448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3498.187770] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3498.684575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3498.689138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3498.855221] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3498.856752] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3499.002197] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3499.250387] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3499.523555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3499.821171] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3500.574255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3500.575919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3500.748661] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3500.835342] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3501.170103] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 3501.173130] Lustre: Skipped 7 previous similar messages [ 3502.754615] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3502.760250] LustreError: Skipped 7 previous similar messages [ 3502.874606] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [ 3502.877930] Lustre: Skipped 7 previous similar messages [ 3503.106488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3503.280626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3504.077272] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3504.247569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3510.381233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds2 2 times [ 3510.566341] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3528.265126] Lustre: lustre-MDT0000: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 3528.265128] Lustre: lustre-MDT0002: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 3528.265133] Lustre: Skipped 168 previous similar messages [ 3529.206396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3529.430781] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3531.163451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3531.178542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3531.358809] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3531.414625] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3533.632079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3533.744172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3533.802731] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3534.007269] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 3536.693086] Lustre: DEBUG MARKER: sync; sync; sync [ 3537.900459] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 3538.191744] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 3538.368472] Turning device dm-2 (0xfc00002) read-only [ 3538.526824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 3538.665629] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 3539.891694] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds3 3 times [ 3540.054642] Lustre: DEBUG MARKER: test_70b fail mds3 3 times [ 3540.212573] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 3540.488257] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 3540.635192] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.237@tcp (stopping) [ 3540.637963] Lustre: Skipped 10 previous similar messages [ 3540.655065] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f703300 x1564004875536368/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 3540.660516] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 3542.639053] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f700900 x1564004875536416/t0(0) o13->lustre-OST0002-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 3542.644523] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 3545.448334] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [ 3545.450940] Lustre: Skipped 1 previous similar message [ 3545.757273] Removing read-only on unknown block (0xfc00002) [ 3545.778163] LustreError: 6346:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491554271 with bad export cookie 5957255847821748128 [ 3545.784742] LustreError: 6346:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3545.899802] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3556.191037] Lustre: DEBUG MARKER: hostname [ 3556.503057] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 3556.752593] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 3556.997396] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 3557.154042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.380874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3557.632501] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3558.131318] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3558.131376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3558.298133] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3558.298139] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3558.440567] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3558.691238] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3558.938447] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3559.191225] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 3559.945137] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3559.947099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3560.149818] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3560.189943] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3563.454419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3563.494782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3563.695361] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3563.774542] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3569.596750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70b fail mds4 4 times [ 3569.829788] Lustre: DEBUG MARKER: test_70b fail mds4 4 times [ 3582.883073] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3583.050787] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3584.700952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3584.704513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3584.918603] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3584.935869] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 3588.197547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3588.303394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3588.356546] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3588.503484] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3614.057809] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3614.969730] Lustre: DEBUG MARKER: rc=0; [ 3614.969730] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3614.969730] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3614.969730] echo $(hostname -s): $val; [ 3614.969730] rc=$val; [ 3614.969730] fi; [ 3614.969730] exit $rc [ 3615.585218] Lustre: DEBUG MARKER: dmesg [ 3616.039834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70c: tar 4mdts recovery ======================================================== 08:39:00 \(1491554340\) [ 3616.199084] Lustre: DEBUG MARKER: == replay-single test 70c: tar 4mdts recovery ======================================================== 08:39:00 (1491554340) [ 3617.363380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3617.519023] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3737.737792] Lustre: DEBUG MARKER: sync; sync; sync [ 3739.148967] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 3739.441948] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 3739.635432] Turning device dm-2 (0xfc00002) read-only [ 3739.782190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 3739.945329] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 3750.164606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70c fail mds3 1 times [ 3750.332032] Lustre: DEBUG MARKER: test_70c fail mds3 1 times [ 3750.492239] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 3750.761333] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 3750.919950] Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) [ 3750.922388] Lustre: Skipped 7 previous similar messages [ 3750.958205] Removing read-only on unknown block (0xfc00002) [ 3751.090081] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3752.716384] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.721330] LustreError: Skipped 166 previous similar messages [ 3756.782908] LustreError: 5060:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491554482 with bad export cookie 5957255847821927454 [ 3761.384967] Lustre: DEBUG MARKER: hostname [ 3761.683495] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 3761.931736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 3762.178798] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 3762.334709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.588345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3762.839389] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3763.331368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3763.333358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3763.495288] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3763.500850] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3763.640125] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3763.889305] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3764.140926] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3764.394687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 3765.164707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3765.191869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3765.365045] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3765.499595] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 3768.668903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3768.790514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3768.871320] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3768.970024] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3890.308855] Lustre: DEBUG MARKER: sync; sync; sync [ 3892.300493] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 3892.573297] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 3892.773306] Turning device dm-0 (0xfc00000) read-only [ 3892.941743] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 3893.103680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3903.334132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70c fail mds1 2 times [ 3903.515475] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 3903.665347] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 3903.935385] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 3904.083639] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 3904.147670] Removing read-only on unknown block (0xfc00000) [ 3904.279741] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 3909.723551] LustreError: 5060:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1491554635 with bad export cookie 5957255847823246464 [ 3909.728984] LustreError: 5060:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3914.589803] Lustre: DEBUG MARKER: hostname [ 3914.905991] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 3915.153870] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 3915.421624] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 3915.581192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.672328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 3920.925999] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 3921.448055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3921.448362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3921.616259] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3921.623537] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3921.762697] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 3922.013576] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3922.262977] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 3922.517661] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 3923.271181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.273790] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.459297] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3923.533367] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3926.761176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3926.840135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3926.916165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3927.028858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3977.804718] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 3978.708844] Lustre: DEBUG MARKER: rc=0; [ 3978.708844] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 3978.708844] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 3978.708844] echo $(hostname -s): $val; [ 3978.708844] rc=$val; [ 3978.708844] fi; [ 3978.708844] exit $rc [ 3979.339799] Lustre: DEBUG MARKER: dmesg [ 3979.868146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70d: mkdir\/rmdir striped dir 4mdts recovery ==================================== 08:45:04 \(1491554704\) [ 3980.020544] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 4mdts recovery ==================================== 08:45:04 (1491554704) [ 3981.232680] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 3981.393675] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4101.557659] Lustre: DEBUG MARKER: sync; sync; sync [ 4102.709691] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 4103.024991] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 4103.179069] LustreError: 9933:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0002 read-only *** [ 4103.182928] LustreError: 9933:0:(osd_handler.c:2171:osd_ro()) Skipped 3 previous similar messages [ 4103.205070] Turning device dm-2 (0xfc00002) read-only [ 4103.348495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 4103.479766] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 4113.736155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70d fail mds3 1 times [ 4113.931212] Lustre: DEBUG MARKER: test_70d fail mds3 1 times [ 4114.091642] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 4114.395643] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 4114.552540] Lustre: Failing over lustre-MDT0002 [ 4114.554660] Lustre: Skipped 3 previous similar messages [ 4114.560696] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.237@tcp (stopping) [ 4114.610502] Removing read-only on unknown block (0xfc00002) [ 4114.617700] Lustre: server umount lustre-MDT0002 complete [ 4114.619762] Lustre: Skipped 3 previous similar messages [ 4114.746219] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4115.497298] LustreError: 11-0: lustre-MDT0002-osp-MDT0000: operation obd_ping to node 0@lo failed: rc = -107 [ 4115.500047] LustreError: Skipped 7 previous similar messages [ 4115.502083] Lustre: lustre-MDT0002-osp-MDT0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.506662] Lustre: Skipped 9 previous similar messages [ 4119.487414] LustreError: 5326:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491554845 with bad export cookie 5957255847823246352 [ 4125.048331] Lustre: DEBUG MARKER: hostname [ 4125.359033] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 4125.607193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 4125.852662] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 4126.009279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.087420] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 4126.089854] Lustre: Skipped 3 previous similar messages [ 4126.246595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4126.499602] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4126.999376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4127.009867] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4127.176443] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4127.180607] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4127.321633] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4127.572975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4127.822733] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4128.077434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 4128.265977] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 4128.268693] Lustre: Skipped 3 previous similar messages [ 4128.271081] Lustre: lustre-MDT0002: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [ 4128.273774] Lustre: Skipped 76 previous similar messages [ 4128.830670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4128.832749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4129.024371] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4129.057785] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 4131.297133] Lustre: lustre-MDT0002: Recovery over after 0:03, of 5 clients 5 recovered and 0 were evicted. [ 4131.300281] Lustre: Skipped 3 previous similar messages [ 4132.336328] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4132.344599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4132.567888] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4132.581855] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4253.918947] Lustre: DEBUG MARKER: sync; sync; sync [ 4254.988423] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4255.269025] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4255.461200] Turning device dm-0 (0xfc00000) read-only [ 4255.635459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4255.814917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4266.055907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70d fail mds1 2 times [ 4266.221271] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4266.443321] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4266.753951] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4266.949133] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 4267.008776] Removing read-only on unknown block (0xfc00000) [ 4267.141437] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4272.783289] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491554998 with bad export cookie 5957255847824367570 [ 4277.462286] Lustre: DEBUG MARKER: hostname [ 4277.765172] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4277.815042] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491554996/real 1491554996] req@ffff880061f14900 x1564004878226272/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491555003 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4277.822618] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4277.825246] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.829788] LustreError: Skipped 1 previous similar message [ 4278.011154] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4278.255749] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4278.410433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.832305] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca175b09f5b to 0x52ac6ca175c325c6 [ 4283.837015] Lustre: Skipped 1 previous similar message [ 4284.038256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4284.288826] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4284.790362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4284.795164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4284.958853] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4284.958854] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4285.104440] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4285.353195] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4285.600367] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4285.853439] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4286.611129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4286.614053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4286.826793] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4286.858778] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4288.871094] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4288.879761] LustreError: Skipped 1 previous similar message [ 4290.146456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4290.156912] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4290.337346] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4290.447769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4291.786873] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4292.695614] Lustre: DEBUG MARKER: rc=0; [ 4292.695614] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4292.695614] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4292.695614] echo $(hostname -s): $val; [ 4292.695614] rc=$val; [ 4292.695614] fi; [ 4292.695614] exit $rc [ 4293.305572] Lustre: DEBUG MARKER: dmesg [ 4293.729826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70e: rename cross-MDT with random fails ======================================== 08:50:18 \(1491555018\) [ 4293.890854] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ======================================== 08:50:18 (1491555018) [ 4295.118036] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4295.278544] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4415.507559] Lustre: DEBUG MARKER: sync; sync; sync [ 4416.556442] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4416.899540] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4417.126823] Turning device dm-0 (0xfc00000) read-only [ 4417.296337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4417.427831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4427.658537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70e fail mds1 1 times [ 4427.835465] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4427.982673] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4428.513026] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4428.663505] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 4428.717766] Removing read-only on unknown block (0xfc00000) [ 4428.849551] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4428.872404] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.877175] LustreError: Skipped 129 previous similar messages [ 4439.152500] Lustre: DEBUG MARKER: hostname [ 4439.457097] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4439.705388] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4439.950041] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4440.095150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.278431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4442.531400] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4443.036303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4443.039231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4443.204569] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4443.210294] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4443.352318] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 4443.602023] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4443.852651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 4444.106430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 4444.920552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.922232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4445.081379] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4445.196539] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4450.421938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4450.542462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4450.633866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4450.765584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4583.869795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70e fail mds2 2 times [ 4584.036429] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 4597.130845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4597.288392] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4599.017876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.037101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.209908] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.308445] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.493824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4601.561720] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4601.683403] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4601.819788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4603.181763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4604.100668] Lustre: DEBUG MARKER: rc=0; [ 4604.100668] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4604.100668] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4604.100668] echo $(hostname -s): $val; [ 4604.100668] rc=$val; [ 4604.100668] fi; [ 4604.100668] exit $rc [ 4604.718790] Lustre: DEBUG MARKER: dmesg [ 4605.150339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 70f: OSS O_DIRECT recovery with 2 clients ====================================== 08:55:29 \(1491555329\) [ 4605.307331] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 2 clients ====================================== 08:55:29 (1491555329) [ 4606.498901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4606.651916] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4616.250902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 1 times [ 4616.434514] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4629.540619] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4629.714538] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4631.170324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4631.171903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4631.364330] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4631.407176] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4632.650602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4632.731691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4632.874805] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4632.978874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4643.149285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 2 times [ 4643.305604] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 4656.318575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4656.490436] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4657.980722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4657.991928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4658.212930] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4658.214567] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4659.495310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4659.546894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4659.802349] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4659.902040] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 4669.758026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 3 times [ 4669.910809] Lustre: DEBUG MARKER: test_70f failing OST 3 times [ 4682.910172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4683.097685] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4684.574862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4684.574880] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4684.797597] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4684.799036] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 4687.066526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 4687.124654] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 4687.316048] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 4687.435557] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 4691.620732] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 4692.532490] Lustre: DEBUG MARKER: rc=0; [ 4692.532490] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 4692.532490] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 4692.532490] echo $(hostname -s): $val; [ 4692.532490] rc=$val; [ 4692.532490] fi; [ 4692.532490] exit $rc [ 4693.146850] Lustre: DEBUG MARKER: dmesg [ 4693.607911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 71a: mkdir\/rmdir striped dir with 2 mdts recovery ============================== 08:56:58 \(1491555418\) [ 4693.771594] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ============================== 08:56:58 (1491555418) [ 4694.960233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4695.129414] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4817.538182] Lustre: DEBUG MARKER: sync; sync; sync [ 4818.538736] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 4818.837678] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 4818.986650] LustreError: 18516:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 4818.990104] LustreError: 18516:0:(osd_handler.c:2171:osd_ro()) Skipped 2 previous similar messages [ 4819.026976] Turning device dm-0 (0xfc00000) read-only [ 4819.161807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 4819.306339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4829.540450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark fail mds2 mds1 1 times [ 4829.719410] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 4830.697698] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation out_update to node 10.9.5.239@tcp failed: rc = -107 [ 4830.700512] LustreError: Skipped 12 previous similar messages [ 4830.702774] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4830.707636] Lustre: Skipped 12 previous similar messages [ 4831.151968] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 4831.400461] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 4831.528099] Lustre: Failing over lustre-MDT0000 [ 4831.530233] Lustre: Skipped 2 previous similar messages [ 4831.547853] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [ 4835.887072] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880052b43300 x1564004879246864/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 4835.892177] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 4836.447084] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005ff3b600 x1564004879246992/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 4836.452185] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 4836.754932] Removing read-only on unknown block (0xfc00000) [ 4836.762355] Lustre: server umount lustre-MDT0000 complete [ 4836.764428] Lustre: Skipped 2 previous similar messages [ 4836.891618] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 4884.799037] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491555594/real 1491555594] req@ffff880065db7300 x1564004879248336/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491555610 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4884.806207] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4905.158052] Lustre: lustre-MDT0002: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 4905.160731] Lustre: Skipped 74 previous similar messages [ 4970.301973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4971.558145] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4988.930180] Lustre: DEBUG MARKER: hostname [ 4989.238798] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 4989.488627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 4989.733582] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 4989.891084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.799271] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca175f6abcc to 0x52ac6ca1763a102c [ 4998.803982] Lustre: Skipped 1 previous similar message [ 4998.851743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.854344] Lustre: Skipped 2 previous similar messages [ 4999.033606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 4999.285121] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 4999.301391] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 4999.304213] Lustre: Skipped 2 previous similar messages [ 4999.785841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 4999.785843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5000.432724] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5000.438769] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5000.580558] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5000.832575] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5001.081480] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5001.333643] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5002.104655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5002.121338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5002.315798] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5002.353225] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5003.836791] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5003.842253] LustreError: Skipped 1 previous similar message [ 5013.430920] Lustre: lustre-MDT0000: Recovery over after 0:15, of 5 clients 5 recovered and 0 were evicted. [ 5013.434300] Lustre: Skipped 2 previous similar messages [ 5013.724093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec [ 5013.912597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec [ 5014.118289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5014.277987] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5014.667572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 12 sec [ 5014.822355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 12 sec [ 5015.134356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5015.361633] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5016.716950] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5017.681937] Lustre: DEBUG MARKER: rc=0; [ 5017.681937] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5017.681937] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5017.681937] echo $(hostname -s): $val; [ 5017.681937] rc=$val; [ 5017.681937] fi; [ 5017.681937] exit $rc [ 5018.310446] Lustre: DEBUG MARKER: dmesg [ 5018.872846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 73a: open\(O_CREAT\), unlink, replay, reconnect before open replay, close ======== 09:02:23 \(1491555743\) [ 5019.106053] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ======== 09:02:23 (1491555743) [ 5019.273697] Lustre: DEBUG MARKER: sync; sync; sync [ 5020.283083] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5020.542955] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5020.681794] Turning device dm-0 (0xfc00000) read-only [ 5020.812220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5020.937303] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5021.066701] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000302 [ 5021.324134] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5021.570364] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5021.745753] Removing read-only on unknown block (0xfc00000) [ 5021.876320] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5028.987464] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.992536] LustreError: Skipped 526 previous similar messages [ 5030.836033] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.840431] LustreError: Skipped 2 previous similar messages [ 5032.183998] Lustre: DEBUG MARKER: hostname [ 5032.494783] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5032.741787] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5032.988182] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5033.144529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.049197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5037.312844] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5037.838269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5037.841674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5038.005102] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5038.005422] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5038.153552] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5038.404373] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5038.652888] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5038.907883] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5039.744107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5039.757217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5039.923006] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5040.022789] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.301692] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5048.309535] Lustre: lustre-MDT0000: Recovery already passed deadline 0:54. If you do not want to wait more, please abort the recovery by force. [ 5049.285198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5049.402119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5049.517986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5049.693394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 5051.042146] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5051.960678] Lustre: DEBUG MARKER: rc=0; [ 5051.960678] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5051.960678] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5051.960678] echo $(hostname -s): $val; [ 5051.960678] rc=$val; [ 5051.960678] fi; [ 5051.960678] exit $rc [ 5052.587990] Lustre: DEBUG MARKER: dmesg [ 5053.032050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 73b: open\(O_CREAT\), unlink, replay, reconnect at open_replay reply, close ====== 09:02:57 \(1491555777\) [ 5053.189915] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ====== 09:02:57 (1491555777) [ 5053.343613] Lustre: DEBUG MARKER: sync; sync; sync [ 5054.273265] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5054.522893] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5054.667087] Turning device dm-0 (0xfc00000) read-only [ 5054.792282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5054.917839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5055.048731] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000030c [ 5055.313056] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5055.572332] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5055.758619] Removing read-only on unknown block (0xfc00000) [ 5055.895115] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5066.201824] Lustre: DEBUG MARKER: hostname [ 5066.501673] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5066.748636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5066.993031] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5067.148118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.095667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5070.347108] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5070.843675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5070.849765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5071.008767] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5071.011228] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5071.151501] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5071.402169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5071.650378] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5071.904408] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5072.665325] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5072.684023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5072.884089] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5072.920680] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5075.166917] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5075.199848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5075.395584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5075.413256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5076.821651] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5077.746238] Lustre: DEBUG MARKER: rc=0; [ 5077.746238] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5077.746238] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5077.746238] echo $(hostname -s): $val; [ 5077.746238] rc=$val; [ 5077.746238] fi; [ 5077.746238] exit $rc [ 5078.382489] Lustre: DEBUG MARKER: dmesg [ 5078.821719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 74: Ensure applications don\'t fail waiting for OST recovery ==================== 09:03:23 \(1491555803\) [ 5078.991300] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ==================== 09:03:23 (1491555803) [ 5081.052833] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5081.303138] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5081.632235] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5091.941371] Lustre: DEBUG MARKER: hostname [ 5092.246565] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5092.494361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5092.739573] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5092.884453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.100195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5093.353328] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5093.848625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5093.849171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5094.013714] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5094.015287] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5094.157365] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5094.408760] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5094.660714] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5094.916387] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5095.340352] Lustre: lustre-MDT0000: Denying connection for new client 7bc6477b-52d7-9e0f-3a83-b76b8ff6cf6f(at 10.9.5.237@tcp), waiting for 3 known clients (1 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 5101.238179] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5101.398072] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5103.279486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5103.444658] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5105.757047] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5106.676481] Lustre: DEBUG MARKER: rc=0; [ 5106.676481] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5106.676481] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5106.676481] echo $(hostname -s): $val; [ 5106.676481] rc=$val; [ 5106.676481] fi; [ 5106.676481] exit $rc [ 5107.300778] Lustre: DEBUG MARKER: dmesg [ 5107.739164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ============== 09:03:52 \(1491555832\) [ 5107.904205] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ============== 09:03:52 (1491555832) [ 5108.052555] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5108.193155] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5108.195630] LustreError: 32630:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88007a2de850 x1564005164085744/t343597383687(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491555839 ref 1 fl Interpret:/0/0 rc 0/0 [ 5108.313796] Lustre: DEBUG MARKER: sync; sync; sync [ 5109.155203] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5109.404592] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5109.549246] Turning device dm-0 (0xfc00000) read-only [ 5109.675497] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5109.801882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5109.943360] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5110.191633] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5110.365774] Removing read-only on unknown block (0xfc00000) [ 5110.495966] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5120.804181] Lustre: DEBUG MARKER: hostname [ 5121.102003] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5121.348706] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5121.593173] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5121.747798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.465495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5126.728402] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5127.224232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5127.224234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5127.391603] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5127.392321] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5127.535291] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5127.784499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5128.033380] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5128.287193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5129.034144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5129.201378] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5129.331838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5129.492427] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5131.709816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5131.862230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5132.422615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5132.572032] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5133.991208] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5134.919557] Lustre: DEBUG MARKER: rc=0; [ 5134.919557] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5134.919557] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5134.919557] echo $(hostname -s): $val; [ 5134.919557] rc=$val; [ 5134.919557] fi; [ 5134.919557] exit $rc [ 5135.542711] Lustre: DEBUG MARKER: dmesg [ 5135.977966] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ============== 09:04:20 \(1491555860\) [ 5136.143282] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ============== 09:04:20 (1491555860) [ 5136.292473] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5136.428592] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5136.431070] LustreError: 5347:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88007a2dcc50 x1564005164089440/t347892350988(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 2088/4320 e 0 to 0 dl 1491555868 ref 1 fl Interpret:/0/0 rc 0/0 [ 5136.551438] Lustre: DEBUG MARKER: sync; sync; sync [ 5137.454504] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5137.704736] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5137.850176] Turning device dm-0 (0xfc00000) read-only [ 5137.975337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5138.101558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5158.143154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5158.311029] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5159.979330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5159.979332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5160.200777] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5160.203083] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5162.478598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5162.481468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5162.688090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5162.704372] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5164.174100] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5165.102843] Lustre: DEBUG MARKER: rc=0; [ 5165.102843] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5165.102843] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5165.102843] echo $(hostname -s): $val; [ 5165.102843] rc=$val; [ 5165.102843] fi; [ 5165.102843] exit $rc [ 5165.723075] Lustre: DEBUG MARKER: dmesg [ 5166.191776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========== 09:04:51 \(1491555891\) [ 5166.361645] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========== 09:04:51 (1491555891) [ 5166.502325] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5166.640448] LustreError: 5348:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88007a2db050 x1564005164094448/t347892351008(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491555898 ref 1 fl Interpret:/0/0 rc 0/0 [ 5166.766613] Lustre: DEBUG MARKER: sync; sync; sync [ 5167.602123] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5167.854308] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5168.105397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5168.231597] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5170.029155] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5170.285564] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5170.460214] Removing read-only on unknown block (0xfc00000) [ 5170.590734] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5180.893867] Lustre: DEBUG MARKER: hostname [ 5181.208598] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5181.456699] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5181.702729] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5181.857459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.558878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5185.812130] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5186.318260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5186.319810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5186.484825] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5186.484827] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5186.628424] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5186.876361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5187.124527] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5187.377533] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5188.134885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5188.136821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5188.300673] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5188.402752] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5191.560282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5191.646239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5191.724370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5191.851537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5205.922350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5206.081589] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5207.767229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.774285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5207.969243] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5208.054182] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5210.193873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5210.317310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5210.358867] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5210.488195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5211.924430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5212.853369] Lustre: DEBUG MARKER: rc=0; [ 5212.853369] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5212.853369] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5212.853369] echo $(hostname -s): $val; [ 5212.853369] rc=$val; [ 5212.853369] fi; [ 5212.853369] exit $rc [ 5213.471395] Lustre: DEBUG MARKER: dmesg [ 5213.922585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ============ 09:05:38 \(1491555938\) [ 5214.085835] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ============ 09:05:38 (1491555938) [ 5214.229417] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5214.368448] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5214.370767] Lustre: Skipped 1 previous similar message [ 5214.372992] LustreError: 5348:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88007a9f9850 x1564005164101408/t352187318285(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491555946 ref 1 fl Interpret:/0/0 rc 0/0 [ 5217.483355] Lustre: DEBUG MARKER: sync; sync; sync [ 5218.304852] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5218.554345] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5218.698796] Turning device dm-0 (0xfc00000) read-only [ 5218.824094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5218.949379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5220.757574] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5221.016656] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5221.191328] Removing read-only on unknown block (0xfc00000) [ 5221.322761] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5238.252424] Lustre: DEBUG MARKER: hostname [ 5238.554656] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5238.801890] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5239.045203] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5239.210056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.427928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5239.679820] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5240.190638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5240.193703] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5240.354395] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5240.357734] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5240.503986] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5240.753133] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5241.001113] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5241.254437] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5253.465410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5253.616391] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5255.301722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5255.313412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5255.507449] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5255.533608] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5257.811927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5257.964399] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5258.156991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5258.311798] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5258.759044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5258.913316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5259.103366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5259.271194] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5260.686110] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5261.607715] Lustre: DEBUG MARKER: rc=0; [ 5261.607715] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5261.607715] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5261.607715] echo $(hostname -s): $val; [ 5261.607715] rc=$val; [ 5261.607715] fi; [ 5261.607715] exit $rc [ 5262.235684] Lustre: DEBUG MARKER: dmesg [ 5262.687760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================== 09:06:27 \(1491555987\) [ 5262.851127] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================== 09:06:27 (1491555987) [ 5266.263215] Lustre: DEBUG MARKER: sync; sync; sync [ 5267.178028] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5267.438853] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5267.590853] Turning device dm-0 (0xfc00000) read-only [ 5267.723236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5267.854173] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5267.993726] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5268.242497] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5268.416952] Removing read-only on unknown block (0xfc00000) [ 5268.547825] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5278.850609] Lustre: DEBUG MARKER: hostname [ 5279.148401] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5279.396360] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5279.642149] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5279.797331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5282.467979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5282.721692] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5283.223163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5283.229413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5283.388974] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5283.395424] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5283.533611] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5283.782278] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5284.031103] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5284.295153] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5285.052245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5285.052275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5285.260651] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5285.283597] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5287.537394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5287.562906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5287.703762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5287.814742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5307.408070] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5308.347314] Lustre: DEBUG MARKER: rc=0; [ 5308.347314] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5308.347314] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5308.347314] echo $(hostname -s): $val; [ 5308.347314] rc=$val; [ 5308.347314] fi; [ 5308.347314] exit $rc [ 5308.968307] Lustre: DEBUG MARKER: dmesg [ 5309.435080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================== 09:07:14 \(1491556034\) [ 5309.613961] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================== 09:07:14 (1491556034) [ 5324.457961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5324.622541] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5326.302250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5326.319400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5326.529056] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5326.532234] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5328.800951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5328.828795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5328.986187] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5329.034216] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5330.500454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5331.413901] Lustre: DEBUG MARKER: rc=0; [ 5331.413901] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5331.413901] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5331.413901] echo $(hostname -s): $val; [ 5331.413901] rc=$val; [ 5331.413901] fi; [ 5331.413901] exit $rc [ 5332.039635] Lustre: DEBUG MARKER: dmesg [ 5332.493832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 =============== 09:07:37 \(1491556057\) [ 5332.655014] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 =============== 09:07:37 (1491556057) [ 5336.053351] Lustre: DEBUG MARKER: sync; sync; sync [ 5336.928540] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5337.176563] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5337.322614] Turning device dm-0 (0xfc00000) read-only [ 5337.448711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5337.574173] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5339.340603] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5339.589937] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5339.762942] Removing read-only on unknown block (0xfc00000) [ 5339.893695] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5350.198865] Lustre: DEBUG MARKER: hostname [ 5350.498623] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5350.745666] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5350.990652] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5351.134945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.569121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5354.831553] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5355.343243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5355.345593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5355.504573] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5355.515102] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5355.652317] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5355.901757] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5356.149625] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5356.404852] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5357.173436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5357.173437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5357.384312] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5357.403086] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5359.671197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5359.671754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5359.867629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5359.888898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5373.997028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5374.157655] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5375.816531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5375.837617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5375.975426] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5376.103789] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5381.209959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5381.400483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5381.424133] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5381.553873] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 5 sec [ 5382.999475] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5383.913723] Lustre: DEBUG MARKER: rc=0; [ 5383.913723] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5383.913723] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5383.913723] echo $(hostname -s): $val; [ 5383.913723] rc=$val; [ 5383.913723] fi; [ 5383.913723] exit $rc [ 5384.557924] Lustre: DEBUG MARKER: dmesg [ 5385.029921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ======================== 09:08:29 \(1491556109\) [ 5385.207264] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ======================== 09:08:29 (1491556109) [ 5388.612285] Lustre: DEBUG MARKER: sync; sync; sync [ 5389.452086] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5389.702771] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5389.847498] Turning device dm-0 (0xfc00000) read-only [ 5389.973938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5390.100156] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5391.938764] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5392.187925] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5392.330351] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 5392.333114] Lustre: Skipped 12 previous similar messages [ 5392.361912] Removing read-only on unknown block (0xfc00000) [ 5392.492480] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5403.631636] Lustre: DEBUG MARKER: hostname [ 5403.933503] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5404.180572] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5404.425205] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5404.580468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.595471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5407.847125] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5408.342883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5408.344087] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5408.506911] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5408.507175] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5408.652808] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5408.901852] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5409.151048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5409.405932] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5421.612993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5421.780544] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5423.499736] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5423.538791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5423.706592] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5423.859926] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.011808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5426.137646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5426.172287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5426.334959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5426.435845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5426.562908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5426.663384] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5426.757797] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5428.217877] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5429.136559] Lustre: DEBUG MARKER: rc=0; [ 5429.136559] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5429.136559] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5429.136559] echo $(hostname -s): $val; [ 5429.136559] rc=$val; [ 5429.136559] fi; [ 5429.136559] exit $rc [ 5429.774422] Lustre: DEBUG MARKER: dmesg [ 5430.230603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ================== 09:09:15 \(1491556155\) [ 5430.399819] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ================== 09:09:15 (1491556155) [ 5430.576741] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5430.712112] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5430.714432] LustreError: 5347:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88007a2dd050 x1564005164131648/t369367187473(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1488/4320 e 0 to 0 dl 1491556166 ref 1 fl Interpret:/0/0 rc 0/0 [ 5437.427567] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 5437.430342] LustreError: Skipped 35 previous similar messages [ 5437.430374] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5437.430376] Lustre: Skipped 32 previous similar messages [ 5451.137458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5451.306301] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5452.981639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5452.998176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5453.177016] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5453.204907] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5455.438767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5455.507617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5455.653324] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5455.701862] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5457.103253] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5458.029781] Lustre: DEBUG MARKER: rc=0; [ 5458.029781] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5458.029781] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5458.029781] echo $(hostname -s): $val; [ 5458.029781] rc=$val; [ 5458.029781] fi; [ 5458.029781] exit $rc [ 5458.659061] Lustre: DEBUG MARKER: dmesg [ 5459.167641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ================ 09:09:43 \(1491556183\) [ 5459.321385] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ================ 09:09:43 (1491556183) [ 5459.560500] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5459.813409] Lustre: DEBUG MARKER: sync; sync; sync [ 5460.638747] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5460.897208] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5461.041525] LustreError: 11540:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 5461.044276] LustreError: 11540:0:(osd_handler.c:2171:osd_ro()) Skipped 9 previous similar messages [ 5461.085511] Turning device dm-0 (0xfc00000) read-only [ 5461.216526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5461.349261] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5461.492865] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5461.752666] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5461.886359] Lustre: Failing over lustre-MDT0000 [ 5461.888881] Lustre: Skipped 9 previous similar messages [ 5461.939895] Removing read-only on unknown block (0xfc00000) [ 5461.947179] Lustre: server umount lustre-MDT0000 complete [ 5461.949318] Lustre: Skipped 9 previous similar messages [ 5462.077599] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5472.381756] Lustre: DEBUG MARKER: hostname [ 5472.679033] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5472.925922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5473.170319] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5473.315008] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.533369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5473.797844] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5474.308948] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5474.314697] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5474.475327] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5474.475568] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5474.618432] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5474.868582] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5475.117151] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5475.371098] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5476.123481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.126254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.316066] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.398493] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5479.578986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5479.667127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5479.762501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5479.849239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5481.219586] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5482.152616] Lustre: DEBUG MARKER: rc=0; [ 5482.152616] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5482.152616] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5482.152616] echo $(hostname -s): $val; [ 5482.152616] rc=$val; [ 5482.152616] fi; [ 5482.152616] exit $rc [ 5482.768529] Lustre: DEBUG MARKER: dmesg [ 5483.245050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ============ 09:10:08 \(1491556208\) [ 5483.411482] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ============ 09:10:08 (1491556208) [ 5483.563245] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5483.813494] Lustre: DEBUG MARKER: sync; sync; sync [ 5484.734498] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5484.988121] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5485.122213] Turning device dm-0 (0xfc00000) read-only [ 5485.248857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5485.374416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5487.141967] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5487.391702] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5487.576876] Removing read-only on unknown block (0xfc00000) [ 5487.708038] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5496.599034] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491556214/real 1491556214] req@ffff88005fb09200 x1564004879356912/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491556221 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 5496.606246] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 5498.016416] Lustre: DEBUG MARKER: hostname [ 5498.325721] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5498.571084] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5498.814882] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5498.969787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.816432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5503.069175] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5503.563536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5503.564625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5503.724840] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5503.730483] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5503.867846] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5504.117515] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5504.365547] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5504.620918] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5505.383953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.419652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.551188] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.635519] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5506.331705] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 5506.334619] Lustre: Skipped 338 previous similar messages [ 5508.834022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5508.923218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5509.008644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5509.116438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5523.210835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5523.368394] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5525.051505] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5525.056937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5525.222067] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5525.316477] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5527.550511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5527.720947] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5528.453536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5528.612597] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5529.956679] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5530.864573] Lustre: DEBUG MARKER: rc=0; [ 5530.864573] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5530.864573] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5530.864573] echo $(hostname -s): $val; [ 5530.864573] rc=$val; [ 5530.864573] fi; [ 5530.864573] exit $rc [ 5531.502845] Lustre: DEBUG MARKER: dmesg [ 5531.995338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ============== 09:10:56 \(1491556256\) [ 5532.174604] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ============== 09:10:56 (1491556256) [ 5532.376138] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 5532.629254] Lustre: DEBUG MARKER: sync; sync; sync [ 5533.469558] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5533.730206] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5533.880754] Turning device dm-0 (0xfc00000) read-only [ 5534.012149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5534.138379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5535.979260] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5536.233998] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5536.408479] Removing read-only on unknown block (0xfc00000) [ 5536.538502] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5553.253425] Lustre: DEBUG MARKER: hostname [ 5553.566803] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5553.820554] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5554.072139] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5554.232588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5554.446304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5554.705792] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5555.205116] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5555.207353] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5555.373204] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5555.373781] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5555.522547] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5555.779452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5556.036485] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5556.298978] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5568.513989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5568.689899] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5570.365659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.381776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.571913] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.619065] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5572.852439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5572.892099] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5573.079515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5573.166855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5573.268690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5573.406373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5573.440654] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5573.649346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5574.996134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5575.922242] Lustre: DEBUG MARKER: rc=0; [ 5575.922242] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5575.922242] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5575.922242] echo $(hostname -s): $val; [ 5575.922242] rc=$val; [ 5575.922242] fi; [ 5575.922242] exit $rc [ 5576.565890] Lustre: DEBUG MARKER: dmesg [ 5577.049842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ==================== 09:11:41 \(1491556301\) [ 5577.213276] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ==================== 09:11:41 (1491556301) [ 5577.912133] Lustre: DEBUG MARKER: sync; sync; sync [ 5578.785833] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5579.042273] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5579.190031] Turning device dm-0 (0xfc00000) read-only [ 5579.320148] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5579.449373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5579.591297] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5579.845684] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5580.023651] Removing read-only on unknown block (0xfc00000) [ 5580.158345] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5590.470171] Lustre: DEBUG MARKER: hostname [ 5590.784826] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5591.040309] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5591.292727] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5591.452892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.671370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5591.929751] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5592.430903] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5592.432961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5592.593694] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5592.598784] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5592.749143] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5593.007758] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5593.275083] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5593.552702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5594.423426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5594.429788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5594.600823] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5594.600829] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5596.813393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5596.949889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5597.073809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5597.150422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5621.777815] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5622.718100] Lustre: DEBUG MARKER: rc=0; [ 5622.718100] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5622.718100] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5622.718100] echo $(hostname -s): $val; [ 5622.718100] rc=$val; [ 5622.718100] fi; [ 5622.718100] exit $rc [ 5623.346025] Lustre: DEBUG MARKER: dmesg [ 5623.818363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ==================== 09:12:28 \(1491556348\) [ 5623.987126] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ==================== 09:12:28 (1491556348) [ 5638.794792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5638.968379] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5640.630313] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.630382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.792260] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.880462] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.113110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5643.280940] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5644.016754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5644.172252] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5645.507304] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5646.443249] Lustre: DEBUG MARKER: rc=0; [ 5646.443249] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5646.443249] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5646.443249] echo $(hostname -s): $val; [ 5646.443249] rc=$val; [ 5646.443249] fi; [ 5646.443249] exit $rc [ 5647.080505] Lustre: DEBUG MARKER: dmesg [ 5647.568702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ================== 09:12:52 \(1491556372\) [ 5647.741672] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ================== 09:12:52 (1491556372) [ 5648.211102] Lustre: DEBUG MARKER: sync; sync; sync [ 5649.053169] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5649.327880] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5649.479179] Turning device dm-0 (0xfc00000) read-only [ 5649.608754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5649.737088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5651.554560] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5651.811728] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5651.989304] Removing read-only on unknown block (0xfc00000) [ 5652.123586] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5652.886887] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.891901] LustreError: Skipped 466 previous similar messages [ 5662.441676] Lustre: DEBUG MARKER: hostname [ 5662.746451] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5662.999668] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5663.255484] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5663.405833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.425125] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.429573] LustreError: Skipped 12 previous similar messages [ 5663.432230] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763ad6fd to 0x52ac6ca1763ae011 [ 5663.436516] Lustre: Skipped 13 previous similar messages [ 5663.473553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.475957] Lustre: Skipped 13 previous similar messages [ 5663.638197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5663.897212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5664.398568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5664.398569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5664.566626] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5664.567354] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5664.713169] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5664.971115] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5664.976502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 5664.979283] Lustre: Skipped 13 previous similar messages [ 5665.232048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5665.493147] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5666.280326] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.285977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.446533] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.507133] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5668.468314] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5668.474013] LustreError: Skipped 13 previous similar messages [ 5668.550131] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 5668.553090] Lustre: Skipped 13 previous similar messages [ 5668.757737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5668.911873] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5669.696521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5669.857298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5683.945648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5684.110346] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5685.773663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5685.780732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5685.989192] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5685.995659] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5688.268236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5688.273991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5688.472578] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5688.570614] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5689.928810] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5690.860725] Lustre: DEBUG MARKER: rc=0; [ 5690.860725] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5690.860725] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5690.860725] echo $(hostname -s): $val; [ 5690.860725] rc=$val; [ 5690.860725] fi; [ 5690.860725] exit $rc [ 5691.505213] Lustre: DEBUG MARKER: dmesg [ 5691.976818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs =================== 09:13:36 \(1491556416\) [ 5692.135108] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs =================== 09:13:36 (1491556416) [ 5692.594457] Lustre: DEBUG MARKER: sync; sync; sync [ 5693.430910] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5693.690189] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5693.838968] Turning device dm-0 (0xfc00000) read-only [ 5693.967949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5694.098257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5695.894752] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5696.149973] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5696.328080] Removing read-only on unknown block (0xfc00000) [ 5696.462951] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5707.611361] Lustre: DEBUG MARKER: hostname [ 5707.923332] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5708.179845] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5708.431807] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5708.588955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.199888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5713.459908] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5713.963898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5713.969607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5714.130215] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5714.131281] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5714.277596] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5714.536041] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5714.792525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5715.051980] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5727.291972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5727.455219] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5729.144126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5729.146881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5729.351238] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5729.365704] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5731.602759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5731.685527] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5731.855682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5731.896862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5732.086952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5732.191771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5732.280314] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5732.385037] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5733.742518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5734.667686] Lustre: DEBUG MARKER: rc=0; [ 5734.667686] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5734.667686] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5734.667686] echo $(hostname -s): $val; [ 5734.667686] rc=$val; [ 5734.667686] fi; [ 5734.667686] exit $rc [ 5735.301887] Lustre: DEBUG MARKER: dmesg [ 5735.771981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 84a: stale open during export disconnect ======================================= 09:14:20 \(1491556460\) [ 5735.932737] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ======================================= 09:14:20 (1491556460) [ 5736.085960] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000144 [ 5736.349390] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 7bc6477b-52d7-9e0f-3a83-b76b8ff6cf6f [ 5736.481313] Lustre: 26725:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7bc6477b-52d7-9e0f-3a83-b76b8ff6cf6f at adminstrative request [ 5738.935633] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5739.859819] Lustre: DEBUG MARKER: rc=0; [ 5739.859819] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5739.859819] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5739.859819] echo $(hostname -s): $val; [ 5739.859819] rc=$val; [ 5739.859819] fi; [ 5739.859819] exit $rc [ 5740.474858] Lustre: DEBUG MARKER: dmesg [ 5740.950609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 85a: check the cancellation of unused locks during recovery\(IBITS\) ============= 09:14:25 \(1491556465\) [ 5741.121073] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ============= 09:14:25 (1491556465) [ 5741.698282] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5741.955049] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5742.296749] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5752.605298] Lustre: DEBUG MARKER: hostname [ 5752.909159] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5753.161497] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5753.414934] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5753.572520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.481316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5756.755238] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5757.255054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5757.255586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5757.422538] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5757.426610] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5757.570035] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5757.827784] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5758.084208] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5758.344727] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5759.114265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5759.119181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5759.320706] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5759.336518] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.591742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5761.615454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5761.820721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5761.820745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5763.227415] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5764.135367] Lustre: DEBUG MARKER: rc=0; [ 5764.135367] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5764.135367] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5764.135367] echo $(hostname -s): $val; [ 5764.135367] rc=$val; [ 5764.135367] fi; [ 5764.135367] exit $rc [ 5764.774424] Lustre: DEBUG MARKER: dmesg [ 5765.260884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 85b: check the cancellation of unused locks during recovery\(EXTENT\) ============ 09:14:50 \(1491556490\) [ 5765.436703] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ============ 09:14:50 (1491556490) [ 5765.574669] Lustre: DEBUG MARKER: lctl pool_new lustre.test_85b [ 5769.849571] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 5770.108142] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 5770.877898] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 5771.138831] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.test_85b 2>/dev/null || echo foo [ 5771.941247] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.test_85b lustre-OST0000 [ 5794.313057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5794.475693] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5795.986205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5796.003071] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5796.201208] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5796.215623] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5798.464278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5798.541297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5798.638190] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5798.745394] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [ 5800.036230] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_remove lustre.test_85b lustre-OST0000 [ 5806.302761] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_destroy lustre.test_85b [ 5812.625227] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5813.549487] Lustre: DEBUG MARKER: rc=0; [ 5813.549487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5813.549487] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5813.549487] echo $(hostname -s): $val; [ 5813.549487] rc=$val; [ 5813.549487] fi; [ 5813.549487] exit $rc [ 5814.189699] Lustre: DEBUG MARKER: dmesg [ 5814.603901] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [ 5814.946159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 09:15:39 \(1491556539\) [ 5815.125023] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ==================== 09:15:39 (1491556539) [ 5815.679892] Lustre: DEBUG MARKER: lctl set_param mdt.lustre-MDT*.exports.clear=0 [ 5815.940500] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5816.203393] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5816.546945] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5816.822689] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5817.078164] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5817.329920] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5817.489794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.708920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5817.970549] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5818.472980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5818.473519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5818.639314] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5818.641353] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5818.800800] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5819.057745] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5819.315101] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5819.575817] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5820.002389] Lustre: lustre-MDT0000: Denying connection for new client 3745d42a-9438-1bf2-0c74-82555ebcaf64(at 10.9.5.238@tcp), waiting for 3 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5820.007584] Lustre: Skipped 1 previous similar message [ 5825.919777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5826.074622] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5826.289543] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5827.198785] Lustre: DEBUG MARKER: rc=0; [ 5827.198785] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5827.198785] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5827.198785] echo $(hostname -s): $val; [ 5827.198785] rc=$val; [ 5827.198785] fi; [ 5827.198785] exit $rc [ 5827.836578] Lustre: DEBUG MARKER: dmesg [ 5828.321337] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 87a: write replay ============================================================== 09:15:53 \(1491556553\) [ 5828.511882] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ============================================================== 09:15:53 (1491556553) [ 5844.826057] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5844.992720] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5846.491205] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5846.492832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5846.699090] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5846.723657] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5847.973771] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5848.003955] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5848.160384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5848.235652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [ 5849.802830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5850.722420] Lustre: DEBUG MARKER: rc=0; [ 5850.722420] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5850.722420] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5850.722420] echo $(hostname -s): $val; [ 5850.722420] rc=$val; [ 5850.722420] fi; [ 5850.722420] exit $rc [ 5851.353052] Lustre: DEBUG MARKER: dmesg [ 5851.840324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 87b: write replay with changed data \(checksum resend\) ========================== 09:16:16 \(1491556576\) [ 5852.012980] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================== 09:16:16 (1491556576) [ 5868.769937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5868.934321] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5870.417061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5870.445473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5870.602682] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5870.694109] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 5873.920289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 5874.018942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 5874.085163] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 5874.211891] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [ 5875.568912] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5876.490330] Lustre: DEBUG MARKER: rc=0; [ 5876.490330] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5876.490330] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5876.490330] echo $(hostname -s): $val; [ 5876.490330] rc=$val; [ 5876.490330] fi; [ 5876.490330] exit $rc [ 5877.124026] Lustre: DEBUG MARKER: dmesg [ 5877.585033] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 88: MDS should not assign same objid to different files ======================== 09:16:42 \(1491556602\) [ 5877.752421] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ======================== 09:16:42 (1491556602) [ 5879.617418] Lustre: DEBUG MARKER: sync; sync; sync [ 5880.478147] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 5880.733523] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 5880.881556] Turning device dm-0 (0xfc00000) read-only [ 5881.010250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 5881.138286] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5881.280466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5881.280466] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5881.280466] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5881.551900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 5881.551900] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 5881.551900] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 5881.823094] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [ 5882.078458] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 5882.335153] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 5882.638588] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 5882.894555] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 5883.159601] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5883.414555] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5883.591360] Removing read-only on unknown block (0xfc00000) [ 5883.724741] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5894.910891] Lustre: DEBUG MARKER: hostname [ 5895.199537] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5895.453579] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5895.704458] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5895.862249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.744289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5902.000585] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5902.498439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5902.500246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5902.663552] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5902.663615] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5902.807729] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5903.064932] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5903.321436] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5903.583196] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5915.853752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5916.028967] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5918.463609] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id [ 5918.737548] Lustre: DEBUG MARKER: lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id [ 5920.665454] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 5921.582594] Lustre: DEBUG MARKER: rc=0; [ 5921.582594] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 5921.582594] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 5921.582594] echo $(hostname -s): $val; [ 5921.582594] rc=$val; [ 5921.582594] fi; [ 5921.582594] exit $rc [ 5922.209859] Lustre: DEBUG MARKER: dmesg [ 5922.765226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 89: no disk space leak on late ost connection ================================== 09:17:27 \(1491556647\) [ 5922.938312] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ================================== 09:17:27 (1491556647) [ 5923.208476] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 5923.469869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 5926.050101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 5928.499138] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 5928.755139] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5929.096159] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 5939.410645] Lustre: DEBUG MARKER: hostname [ 5939.725575] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 5939.977700] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 5940.230199] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 5940.388437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.761652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 5945.020088] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 5945.516888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5945.519797] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5945.685902] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5945.686265] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5945.833940] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 5946.089643] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5946.345239] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 5946.605931] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 5951.557749] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 5951.818667] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6022.696392] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [ 6022.954581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [ 6025.514327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6025.777361] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 6026.061626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6027.337421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6028.613604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6029.894111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6031.175191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6032.455552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 6033.833024] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6034.807037] Lustre: DEBUG MARKER: rc=0; [ 6034.807037] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6034.807037] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6034.807037] echo $(hostname -s): $val; [ 6034.807037] rc=$val; [ 6034.807037] fi; [ 6034.807037] exit $rc [ 6035.426360] Lustre: DEBUG MARKER: dmesg [ 6035.899630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 90: lfs find identifies the missing striped file segments ====================== 09:19:20 \(1491556760\) [ 6036.066503] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ====================== 09:19:20 (1491556760) [ 6036.248243] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [ 6036.248243] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [ 6036.516902] Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param -n lod.lustre-MDT*-*.target_obd | sort -u | [ 6036.516902] awk 'BEGIN {c = 0} /ACTIVE/{c += 1} END {printf "%d", c}' [ 6039.679712] LustreError: 11-0: lustre-OST0004-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 6039.682352] LustreError: Skipped 43 previous similar messages [ 6039.684491] Lustre: lustre-OST0004-osc-MDT0002: Connection to lustre-OST0004 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6039.689328] Lustre: Skipped 38 previous similar messages [ 6050.549020] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6050.715307] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6054.327754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6055.249255] Lustre: DEBUG MARKER: rc=0; [ 6055.249255] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6055.249255] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6055.249255] echo $(hostname -s): $val; [ 6055.249255] rc=$val; [ 6055.249255] fi; [ 6055.249255] exit $rc [ 6055.880749] Lustre: DEBUG MARKER: dmesg [ 6056.355078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 93a: replay + reconnect ======================================================== 09:19:41 \(1491556781\) [ 6056.525882] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ======================================================== 09:19:41 (1491556781) [ 6056.671992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6056.671992] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6056.671992] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6070.498429] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6070.664990] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6072.159777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6072.161243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6072.331439] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6072.396070] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [ 6097.618047] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491556816/real 1491556816] req@ffff880064372700 x1564004879422256/t0(0) o400->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 224/224 e 0 to 1 dl 1491556823 ref 1 fl Rpc:X/c0/ffffffff rc 0/-1 [ 6097.626451] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 6114.015105] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 6114.018082] Lustre: Skipped 269 previous similar messages [ 6114.689455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6114.815171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6114.848802] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6115.034954] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 42 sec [ 6116.381517] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6117.313413] Lustre: DEBUG MARKER: rc=0; [ 6117.313413] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6117.313413] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6117.313413] echo $(hostname -s): $val; [ 6117.313413] rc=$val; [ 6117.313413] fi; [ 6117.313413] exit $rc [ 6117.942457] Lustre: DEBUG MARKER: dmesg [ 6118.414964] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 93b: replay + reconnect on mds ================================================= 09:20:43 \(1491556843\) [ 6118.576110] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ================================================= 09:20:43 (1491556843) [ 6118.736259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 6118.736259] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 6118.736259] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 6119.044336] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=80 [ 6119.318749] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x715 [ 6119.598530] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6119.872118] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6120.004598] Lustre: Failing over lustre-MDT0000 [ 6120.006830] Lustre: Skipped 9 previous similar messages [ 6120.092021] Lustre: server umount lustre-MDT0000 complete [ 6120.094236] Lustre: Skipped 9 previous similar messages [ 6120.223656] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6130.541723] Lustre: DEBUG MARKER: hostname [ 6130.857814] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6131.112171] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6131.362785] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6131.520843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.553277] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (not set up) [ 6131.555613] Lustre: Skipped 18 previous similar messages [ 6131.777293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6132.048036] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6132.538590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6132.542469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6132.708815] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6132.714474] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6132.854298] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6133.110726] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6133.365949] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6133.624196] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6134.393590] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6134.404731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6134.590464] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6134.623619] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6134.677059] Lustre: *** cfs_fail_loc=715, val=0*** [ 6135.815038] Lustre: *** cfs_fail_loc=715, val=0*** [ 6135.817262] Lustre: Skipped 1 previous similar message [ 6136.570215] LustreError: 8505:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6137.005029] Lustre: *** cfs_fail_loc=715, val=0*** [ 6140.673282] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6141.679052] Lustre: *** cfs_fail_loc=715, val=0*** [ 6141.681327] Lustre: Skipped 2 previous similar messages [ 6141.811728] Lustre: lustre-MDT0000: Recovery already passed deadline 0:57. If you do not want to wait more, please abort the recovery by force. [ 6143.002592] Lustre: lustre-MDT0000: Recovery already passed deadline 0:56. If you do not want to wait more, please abort the recovery by force. [ 6143.569148] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6147.676213] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6147.679200] Lustre: Skipped 1 previous similar message [ 6148.683041] Lustre: *** cfs_fail_loc=715, val=0*** [ 6148.685467] Lustre: Skipped 4 previous similar messages [ 6148.816678] Lustre: lustre-MDT0000: Recovery already passed deadline 0:50. If you do not want to wait more, please abort the recovery by force. [ 6150.572203] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6154.681341] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6154.684326] Lustre: Skipped 1 previous similar message [ 6155.821721] Lustre: lustre-MDT0000: Recovery already passed deadline 0:43. If you do not want to wait more, please abort the recovery by force. [ 6155.826922] Lustre: Skipped 1 previous similar message [ 6156.831034] Lustre: *** cfs_fail_loc=715, val=0*** [ 6156.833457] Lustre: Skipped 5 previous similar messages [ 6162.829705] Lustre: lustre-MDT0000: Recovery already passed deadline 0:36. If you do not want to wait more, please abort the recovery by force. [ 6162.834636] Lustre: Skipped 1 previous similar message [ 6164.575153] Lustre: lustre-MDT0000: Received new LWP connection from 0@lo, removing former export from same NID [ 6164.578058] Lustre: Skipped 3 previous similar messages [ 6171.008529] Lustre: lustre-MDT0000: Recovery already passed deadline 0:28. If you do not want to wait more, please abort the recovery by force. [ 6171.013377] Lustre: Skipped 2 previous similar messages [ 6176.687034] Lustre: *** cfs_fail_loc=715, val=0*** [ 6176.689337] Lustre: Skipped 13 previous similar messages [ 6182.686227] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.239@tcp, removing former export from same NID [ 6182.689100] Lustre: Skipped 7 previous similar messages [ 6190.836690] Lustre: lustre-MDT0000: Recovery already passed deadline 0:08. If you do not want to wait more, please abort the recovery by force. [ 6190.841554] Lustre: Skipped 4 previous similar messages [ 6204.844665] Lustre: lustre-MDT0000: Recovery already passed deadline 0:05, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6206.015580] Lustre: lustre-MDT0000: Recovery already passed deadline 0:06, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6211.693027] Lustre: *** cfs_fail_loc=715, val=0*** [ 6211.695263] Lustre: Skipped 24 previous similar messages [ 6211.849670] Lustre: lustre-MDT0000: Recovery already passed deadline 0:12, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. [ 6216.572037] LustreError: 8505:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6216.592179] Lustre: 8505:0:(ldlm_lib.c:2550:target_recovery_thread()) too long recovery - read logs [ 6216.595495] LustreError: dumping log to /tmp/lustre-log.1491556942.8505 [ 6217.111796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 82 sec [ 6217.186202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 82 sec [ 6217.269538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 82 sec [ 6217.393102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 82 sec [ 6218.767192] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6219.696440] Lustre: DEBUG MARKER: rc=0; [ 6219.696440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6219.696440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6219.696440] echo $(hostname -s): $val; [ 6219.696440] rc=$val; [ 6219.696440] fi; [ 6219.696440] exit $rc [ 6220.337491] Lustre: DEBUG MARKER: dmesg [ 6220.812833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ============ 09:22:25 \(1491556945\) [ 6220.979144] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ============ 09:22:25 (1491556945) [ 6240.220118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6240.389327] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6242.093502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6242.133671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6242.390206] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6242.413060] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6244.713510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6244.727079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6244.879659] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6244.897376] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6246.470770] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6247.386505] Lustre: DEBUG MARKER: rc=0; [ 6247.386505] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6247.386505] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6247.386505] echo $(hostname -s): $val; [ 6247.386505] rc=$val; [ 6247.386505] fi; [ 6247.386505] exit $rc [ 6248.021683] Lustre: DEBUG MARKER: dmesg [ 6248.485481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 100b: DNE: create striped dir, fail MDT0 ======================================= 09:22:53 \(1491556973\) [ 6248.648851] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ======================================= 09:22:53 (1491556973) [ 6248.851426] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 [ 6249.050998] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6249.053114] Lustre: Skipped 3 previous similar messages [ 6249.055147] LustreError: 5341:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f7c6700 x1564004864601360/t416611827779(0) o36->98d70537-db2e-f853-3baf-ad9e5965d15a@10.9.5.237@tcp:-1/-1 lens 672/448 e 0 to 0 dl 1491557007 ref 1 fl Interpret:/0/0 rc 0/0 [ 6249.062051] LustreError: 5341:0:(ldlm_lib.c:2916:target_send_reply_msg()) Skipped 3 previous similar messages [ 6249.117541] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6249.371645] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6249.719353] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6251.764048] LustreError: 5060:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491556977 with bad export cookie 5957255847833465127 [ 6254.282076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.287307] LustreError: Skipped 221 previous similar messages [ 6260.037828] Lustre: DEBUG MARKER: hostname [ 6260.350221] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6260.611162] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6260.860236] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6261.020634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.103183] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763b6bbd to 0x52ac6ca1763b76e5 [ 6265.107667] Lustre: Skipped 6 previous similar messages [ 6265.162344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.164816] Lustre: Skipped 6 previous similar messages [ 6265.333484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6265.400453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 6265.402912] Lustre: Skipped 6 previous similar messages [ 6265.592521] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6266.097197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6266.107376] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6266.267455] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6266.267705] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6266.415258] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6266.672846] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6266.930649] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6267.191765] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6267.950495] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6267.950561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6268.155595] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6268.182846] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6270.157776] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6270.163272] LustreError: Skipped 6 previous similar messages [ 6270.179227] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 6270.182378] Lustre: Skipped 6 previous similar messages [ 6270.443843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6270.444350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6270.672874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6270.719686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6272.222745] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6273.153365] Lustre: DEBUG MARKER: rc=0; [ 6273.153365] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6273.153365] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6273.153365] echo $(hostname -s): $val; [ 6273.153365] rc=$val; [ 6273.153365] fi; [ 6273.153365] exit $rc [ 6273.782591] Lustre: DEBUG MARKER: dmesg [ 6274.261514] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 101: Shouldn\'t reassign precreated objs to other files after recovery ========== 09:23:19 \(1491556999\) [ 6274.428998] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========== 09:23:19 (1491556999) [ 6274.580544] Lustre: DEBUG MARKER: sync; sync; sync [ 6275.546295] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6275.803675] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6275.946113] LustreError: 12833:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 6275.948778] LustreError: 12833:0:(osd_handler.c:2171:osd_ro()) Skipped 6 previous similar messages [ 6275.981548] Turning device dm-0 (0xfc00000) read-only [ 6276.110932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6276.240165] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6278.081783] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6278.355955] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6280.223100] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f76f900 x1564004879442352/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 6280.228262] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 6283.754477] Removing read-only on unknown block (0xfc00000) [ 6283.896182] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6284.222720] Lustre: DEBUG MARKER: hostname [ 6284.508016] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6284.764395] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6285.023313] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6285.182071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.199536] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6285.203735] LustreError: Skipped 7 previous similar messages [ 6285.257311] LustreError: 13424:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 6285.261528] LustreError: 13424:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6285.265175] Lustre: 13472:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6285.268825] Lustre: 13472:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 6285.288023] LustreError: 13469:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 getting update log failed: rc = -108 [ 6285.292108] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 6285.449298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6285.710939] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6286.218544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6286.240078] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6286.404544] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6286.409709] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6286.553749] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6286.810087] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6287.067733] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6287.331490] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6294.411468] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6295.358347] Lustre: DEBUG MARKER: rc=0; [ 6295.358347] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6295.358347] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6295.358347] echo $(hostname -s): $val; [ 6295.358347] rc=$val; [ 6295.358347] fi; [ 6295.358347] exit $rc [ 6295.998506] Lustre: DEBUG MARKER: dmesg [ 6296.468369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102a: check resend \(request lost\) with multiple modify RPCs in flight ========== 09:23:41 \(1491557021\) [ 6296.625107] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========== 09:23:41 (1491557021) [ 6296.807406] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x159 [ 6296.944788] Lustre: *** cfs_fail_loc=159, val=0*** [ 6298.086919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6303.944797] Lustre: lustre-MDT0000: Client 98d70537-db2e-f853-3baf-ad9e5965d15a (at 10.9.5.237@tcp) reconnecting [ 6303.947560] Lustre: Skipped 1 previous similar message [ 6304.173118] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6305.133487] Lustre: DEBUG MARKER: rc=0; [ 6305.133487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6305.133487] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6305.133487] echo $(hostname -s): $val; [ 6305.133487] rc=$val; [ 6305.133487] fi; [ 6305.133487] exit $rc [ 6305.766909] Lustre: DEBUG MARKER: dmesg [ 6306.242770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102b: check resend \(reply lost\) with multiple modify RPCs in flight ============ 09:23:51 \(1491557031\) [ 6306.410237] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ============ 09:23:51 (1491557031) [ 6306.590929] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6306.728927] Lustre: *** cfs_fail_loc=15a, val=0*** [ 6307.869855] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6313.951409] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6314.856387] Lustre: DEBUG MARKER: rc=0; [ 6314.856387] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6314.856387] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6314.856387] echo $(hostname -s): $val; [ 6314.856387] rc=$val; [ 6314.856387] fi; [ 6314.856387] exit $rc [ 6315.489643] Lustre: DEBUG MARKER: dmesg [ 6315.973264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102c: check replay w\/o reconstruction with multiple mod RPCs in flight ========= 09:24:00 \(1491557040\) [ 6316.136644] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========= 09:24:00 (1491557040) [ 6316.309442] Lustre: DEBUG MARKER: sync; sync; sync [ 6317.245305] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6317.497851] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6317.644750] Turning device dm-0 (0xfc00000) read-only [ 6317.773840] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6317.902696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6318.035459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6319.297855] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6319.560904] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6319.814772] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6319.992128] Removing read-only on unknown block (0xfc00000) [ 6320.124792] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6330.439594] Lustre: DEBUG MARKER: hostname [ 6330.754349] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6331.007707] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6331.261708] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6331.420399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.430132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6333.688483] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6334.185644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6334.185646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6334.351925] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6334.352205] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6334.500511] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6334.757176] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6335.011378] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6335.275181] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6336.062302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6336.068225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6336.255044] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6336.321094] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6338.528114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6338.553716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6338.760764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6338.765071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6340.179916] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6341.111122] Lustre: DEBUG MARKER: rc=0; [ 6341.111122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6341.111122] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6341.111122] echo $(hostname -s): $val; [ 6341.111122] rc=$val; [ 6341.111122] fi; [ 6341.111122] exit $rc [ 6341.757267] Lustre: DEBUG MARKER: dmesg [ 6342.230095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight =========== 09:24:27 \(1491557067\) [ 6342.401134] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [ 6342.577691] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x15a [ 6342.716226] Lustre: *** cfs_fail_loc=15a, val=0*** [ 6342.718492] Lustre: Skipped 13 previous similar messages [ 6343.844538] Lustre: DEBUG MARKER: sync; sync; sync [ 6344.183859] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [ 6344.449890] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6344.706700] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6345.031781] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6355.349494] Lustre: DEBUG MARKER: hostname [ 6355.664867] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6355.919062] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6356.169772] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6356.326451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.514157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6362.771776] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6363.273684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6363.273706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6363.440447] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6363.440448] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6363.587341] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6363.846851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6364.104427] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6364.364931] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6365.129761] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6365.129764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6365.345590] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6365.355310] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6367.591372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6367.634625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6367.812890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6367.832947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6369.245881] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6370.178092] Lustre: DEBUG MARKER: rc=0; [ 6370.178092] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6370.178092] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6370.178092] echo $(hostname -s): $val; [ 6370.178092] rc=$val; [ 6370.178092] fi; [ 6370.178092] exit $rc [ 6370.816210] Lustre: DEBUG MARKER: dmesg [ 6371.325909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 103: Check otr_next_id overflow ================================================ 09:24:56 \(1491557096\) [ 6371.485435] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ================================================ 09:24:56 (1491557096) [ 6371.662582] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000162 [ 6371.902238] Lustre: *** cfs_fail_loc=162, val=0*** [ 6372.087159] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6372.343235] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6372.757253] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6383.074527] Lustre: DEBUG MARKER: hostname [ 6383.393669] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6383.648100] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6383.900750] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6384.059039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.282647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6384.538682] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6385.039327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6385.039349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6385.210575] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6385.213477] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6385.358527] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6385.614605] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6385.871237] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6386.134894] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6386.903198] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6386.920580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6387.124626] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6387.128312] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6389.399131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6389.416153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6389.604057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6389.631372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6391.000971] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6391.919799] Lustre: DEBUG MARKER: rc=0; [ 6391.919799] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6391.919799] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6391.919799] echo $(hostname -s): $val; [ 6391.919799] rc=$val; [ 6391.919799] fi; [ 6391.919799] exit $rc [ 6392.551379] Lustre: DEBUG MARKER: dmesg [ 6393.035363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110a: DNE: create striped dir, fail MDT1 ======================================= 09:25:17 \(1491557117\) [ 6393.205676] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ======================================= 09:25:17 (1491557117) [ 6393.365255] Lustre: DEBUG MARKER: sync; sync; sync [ 6394.264216] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6394.522421] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6394.672341] Turning device dm-0 (0xfc00000) read-only [ 6394.801883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6394.931030] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6395.139574] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6395.628816] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6395.819085] Removing read-only on unknown block (0xfc00000) [ 6395.952612] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6406.268203] Lustre: DEBUG MARKER: hostname [ 6406.582056] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6406.841977] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6407.095229] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6407.243643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.581563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6412.840420] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6413.337794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6413.350536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6413.512280] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6413.512282] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6413.659111] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6413.916803] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6414.175907] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6414.434851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6415.200184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6415.238117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6415.472486] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6415.486507] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6418.690270] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6418.744612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6418.858621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6418.974484] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6420.432245] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6421.352237] Lustre: DEBUG MARKER: rc=0; [ 6421.352237] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6421.352237] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6421.352237] echo $(hostname -s): $val; [ 6421.352237] rc=$val; [ 6421.352237] fi; [ 6421.352237] exit $rc [ 6421.979548] Lustre: DEBUG MARKER: dmesg [ 6422.466144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ============================ 09:25:47 \(1491557147\) [ 6422.619733] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ============================ 09:25:47 (1491557147) [ 6422.770293] Lustre: DEBUG MARKER: sync; sync; sync [ 6423.776963] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6424.033290] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6424.181433] Turning device dm-0 (0xfc00000) read-only [ 6424.310759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6424.440265] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6424.646743] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6424.906894] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6425.086210] Removing read-only on unknown block (0xfc00000) [ 6425.221379] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6429.774040] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491557155 with bad export cookie 5957255847833608256 [ 6429.778752] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6435.537320] Lustre: DEBUG MARKER: hostname [ 6435.844211] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6436.097152] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6436.350303] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6436.507668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.638659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6440.900104] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6441.404732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6441.411647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6441.572546] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6441.577805] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6441.717040] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6441.973018] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6442.230159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6442.493133] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6443.253222] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6443.259716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6443.466063] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6443.483395] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6509.279033] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 6509.281703] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 6510.000833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 66 sec [ 6510.155304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 66 sec [ 6512.804468] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6513.800896] Lustre: DEBUG MARKER: rc=0; [ 6513.800896] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6513.800896] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6513.800896] echo $(hostname -s): $val; [ 6513.800896] rc=$val; [ 6513.800896] fi; [ 6513.800896] exit $rc [ 6514.448531] Lustre: DEBUG MARKER: dmesg [ 6514.927499] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110c: DNE: create striped dir, fail MDT2 ======================================= 09:27:19 \(1491557239\) [ 6515.096853] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ======================================= 09:27:19 (1491557239) [ 6529.838447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6529.995740] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6531.687902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6531.715277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6531.893033] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6531.981894] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6534.128646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6534.257510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6534.284302] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6534.467688] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6535.961785] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6536.905638] Lustre: DEBUG MARKER: rc=0; [ 6536.905638] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6536.905638] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6536.905638] echo $(hostname -s): $val; [ 6536.905638] rc=$val; [ 6536.905638] fi; [ 6536.905638] exit $rc [ 6537.544745] Lustre: DEBUG MARKER: dmesg [ 6538.036530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ============================ 09:27:42 \(1491557262\) [ 6538.203340] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ============================ 09:27:42 (1491557262) [ 6553.089417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6553.255749] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6555.003776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.036905] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.184364] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6555.192786] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6654.799798] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 99 sec [ 6654.967253] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 99 sec [ 6657.649242] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6658.572189] Lustre: DEBUG MARKER: rc=0; [ 6658.572189] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6658.572189] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6658.572189] echo $(hostname -s): $val; [ 6658.572189] rc=$val; [ 6658.572189] fi; [ 6658.572189] exit $rc [ 6659.208740] Lustre: DEBUG MARKER: dmesg [ 6659.704609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client\/MDT1\/MDT2 ========= 09:29:44 \(1491557384\) [ 6659.863415] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========= 09:29:44 (1491557384) [ 6661.713844] Lustre: DEBUG MARKER: sync; sync; sync [ 6662.630334] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6662.885771] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6663.033659] Turning device dm-0 (0xfc00000) read-only [ 6663.163145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6663.292492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6663.439573] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6663.696179] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6663.874318] Removing read-only on unknown block (0xfc00000) [ 6664.010033] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6664.709274] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491557390 with bad export cookie 5957255847833611861 [ 6665.462150] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 6665.464866] LustreError: Skipped 29 previous similar messages [ 6665.467143] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.472121] Lustre: Skipped 25 previous similar messages [ 6675.212746] Lustre: DEBUG MARKER: hostname [ 6675.534603] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6675.789656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6676.041666] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6676.189130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.016044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6679.271414] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6679.771518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6679.773875] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6679.936883] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6679.938482] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6680.084375] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6680.341912] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6680.597777] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6680.858751] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6693.100646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6693.255295] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6694.961737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6694.963245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6695.188963] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6695.206817] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6697.475295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6697.631217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6794.236580] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 10.9.5.239@tcp (at 10.9.5.239@tcp) [ 6794.239565] Lustre: Skipped 310 previous similar messages [ 6795.198519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 97 sec [ 6795.366745] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 97 sec [ 6798.064692] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6798.995715] Lustre: DEBUG MARKER: rc=0; [ 6798.995715] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6798.995715] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6798.995715] echo $(hostname -s): $val; [ 6798.995715] rc=$val; [ 6798.995715] fi; [ 6798.995715] exit $rc [ 6799.627939] Lustre: DEBUG MARKER: dmesg [ 6800.142451] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110f: DNE: create striped dir, fail MDT1\/MDT2 ================================== 09:32:04 \(1491557524\) [ 6800.305713] Lustre: DEBUG MARKER: == replay-single test 110f: DNE: create striped dir, fail MDT1/MDT2 ================================== 09:32:04 (1491557524) [ 6800.462698] Lustre: DEBUG MARKER: sync; sync; sync [ 6801.364765] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6801.620049] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6801.758651] Turning device dm-0 (0xfc00000) read-only [ 6801.889438] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6802.020775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6803.899396] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6804.156444] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6804.288255] Lustre: Failing over lustre-MDT0000 [ 6804.290466] Lustre: Skipped 8 previous similar messages [ 6804.341298] Removing read-only on unknown block (0xfc00000) [ 6804.348410] Lustre: server umount lustre-MDT0000 complete [ 6804.350555] Lustre: Skipped 8 previous similar messages [ 6804.479629] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6805.160294] LustreError: 5060:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491557530 with bad export cookie 5957255847833612694 [ 6815.664884] Lustre: DEBUG MARKER: hostname [ 6815.982278] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6816.250280] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6816.502152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6816.661207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.872430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6817.130336] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6817.634404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6817.635411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6817.800171] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6817.804760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6817.947947] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6818.206568] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6818.461610] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6818.721894] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6830.988707] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6831.169428] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6832.917923] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6832.917926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.130127] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.144291] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6835.406980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6835.421341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6835.599359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6835.655632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6835.816297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6835.922752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6835.999630] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6836.163330] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6837.626560] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6838.566795] Lustre: DEBUG MARKER: rc=0; [ 6838.566795] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6838.566795] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6838.566795] echo $(hostname -s): $val; [ 6838.566795] rc=$val; [ 6838.566795] fi; [ 6838.566795] exit $rc [ 6839.197717] Lustre: DEBUG MARKER: dmesg [ 6839.790031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client\/MDT1\/MDT2 ========= 09:32:44 \(1491557564\) [ 6839.957064] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========= 09:32:44 (1491557564) [ 6840.111477] Lustre: DEBUG MARKER: sync; sync; sync [ 6841.131035] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6841.385879] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6841.523724] Turning device dm-0 (0xfc00000) read-only [ 6841.653543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6841.780921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6843.841516] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6844.100104] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6844.276509] Removing read-only on unknown block (0xfc00000) [ 6844.410150] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6853.711086] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491557572/real 1491557572] req@ffff88005f230000 x1564004879518720/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491557579 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 6853.718108] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 29 previous similar messages [ 6855.557599] Lustre: DEBUG MARKER: hostname [ 6855.870584] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6856.126339] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6856.378083] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6856.537156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.369387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.374038] LustreError: Skipped 302 previous similar messages [ 6859.928369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6860.186964] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6860.687902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6860.687904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6860.853731] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6860.871733] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6861.011658] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6861.268107] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6861.524194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6861.783120] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6874.093849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6874.278912] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6876.062127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6876.074675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6876.258848] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6876.274424] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6929.375043] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 6929.377851] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 6929.713870] Lustre: lustre-MDT0000: Recovery over after 1:10, of 5 clients 4 recovered and 1 was evicted. [ 6929.717057] Lustre: Skipped 7 previous similar messages [ 6930.756653] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 54 sec [ 6930.913651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 54 sec [ 6931.107227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6931.263114] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6933.912149] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6934.855666] Lustre: DEBUG MARKER: rc=0; [ 6934.855666] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6934.855666] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6934.855666] echo $(hostname -s): $val; [ 6934.855666] rc=$val; [ 6934.855666] fi; [ 6934.855666] exit $rc [ 6935.483231] Lustre: DEBUG MARKER: dmesg [ 6935.981741] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ======================================= 09:34:20 \(1491557660\) [ 6936.148611] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ======================================= 09:34:20 (1491557660) [ 6936.303040] Lustre: DEBUG MARKER: sync; sync; sync [ 6937.216579] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 6937.473181] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 6937.613970] LustreError: 2749:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 6937.616682] LustreError: 2749:0:(osd_handler.c:2171:osd_ro()) Skipped 6 previous similar messages [ 6937.655493] Turning device dm-0 (0xfc00000) read-only [ 6937.784383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 6937.914314] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6938.092755] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 6938.348104] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 6938.525200] Removing read-only on unknown block (0xfc00000) [ 6938.659202] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 6938.756284] LustreError: 5327:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491557664 with bad export cookie 5957255847833616537 [ 6938.761113] LustreError: 5327:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6947.255086] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.259584] LustreError: Skipped 8 previous similar messages [ 6948.982565] Lustre: DEBUG MARKER: hostname [ 6949.294607] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 6949.555649] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 6949.807648] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 6949.964557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.261174] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763dc029 to 0x52ac6ca1763dc7bc [ 6953.265277] Lustre: Skipped 9 previous similar messages [ 6953.318805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.321169] Lustre: Skipped 9 previous similar messages [ 6953.493292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 6953.750305] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 6954.250439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6954.262273] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6954.408704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 6954.411269] Lustre: Skipped 8 previous similar messages [ 6954.421892] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6954.430991] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6954.572579] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 6954.828801] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6955.084460] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 6955.345592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 6956.113630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.129024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.278422] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.412232] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6958.310041] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6958.315795] LustreError: Skipped 10 previous similar messages [ 6958.633249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6958.790552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6959.497468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6959.656756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6961.005884] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 6961.947252] Lustre: DEBUG MARKER: rc=0; [ 6961.947252] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 6961.947252] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 6961.947252] echo $(hostname -s): $val; [ 6961.947252] rc=$val; [ 6961.947252] fi; [ 6961.947252] exit $rc [ 6962.589713] Lustre: DEBUG MARKER: dmesg [ 6963.091863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ======================================= 09:34:47 \(1491557687\) [ 6963.265935] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ======================================= 09:34:47 (1491557687) [ 6977.988946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6978.155389] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 6979.843533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6979.843535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6980.034980] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6980.064259] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7078.682945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 98 sec [ 7078.831779] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 98 sec [ 7081.358712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7082.336144] Lustre: DEBUG MARKER: rc=0; [ 7082.336144] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7082.336144] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7082.336144] echo $(hostname -s): $val; [ 7082.336144] rc=$val; [ 7082.336144] fi; [ 7082.336144] exit $rc [ 7082.973035] Lustre: DEBUG MARKER: dmesg [ 7083.462302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client\/MDT1\/MDT2 ========= 09:36:48 \(1491557808\) [ 7083.647680] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========= 09:36:48 (1491557808) [ 7083.887192] Lustre: DEBUG MARKER: sync; sync; sync [ 7084.850080] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7085.103839] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7085.252444] Turning device dm-0 (0xfc00000) read-only [ 7085.382937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7085.511452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7087.482385] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7087.739680] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7087.918110] Removing read-only on unknown block (0xfc00000) [ 7088.052455] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7104.248083] Lustre: DEBUG MARKER: hostname [ 7104.557992] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7104.810825] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7105.063718] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7105.222639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.430235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7105.689464] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7106.187684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7106.187739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7106.357414] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7106.357416] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7106.503684] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7106.764339] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7107.018215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7107.278528] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7119.551641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7119.713519] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7121.428126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7121.472643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7121.693823] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7121.859125] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7178.719041] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 7178.721731] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 7179.372336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 57 sec [ 7179.534724] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 57 sec [ 7179.733598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7179.900231] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7182.436261] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7183.367612] Lustre: DEBUG MARKER: rc=0; [ 7183.367612] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7183.367612] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7183.367612] echo $(hostname -s): $val; [ 7183.367612] rc=$val; [ 7183.367612] fi; [ 7183.367612] exit $rc [ 7184.038557] Lustre: DEBUG MARKER: dmesg [ 7184.548241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client\/MDT1\/MDT2 ========= 09:38:29 \(1491557909\) [ 7184.705751] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========= 09:38:29 (1491557909) [ 7186.677144] Lustre: DEBUG MARKER: sync; sync; sync [ 7187.626045] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7187.883819] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7188.030727] Turning device dm-0 (0xfc00000) read-only [ 7188.160891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7188.292406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7188.439397] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7188.696160] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7188.873320] Removing read-only on unknown block (0xfc00000) [ 7189.007365] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7189.669041] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491557915 with bad export cookie 5957255847833620723 [ 7189.673930] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7200.173130] Lustre: DEBUG MARKER: hostname [ 7200.489864] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7200.744322] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7200.996416] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7201.155849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.886428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7204.152221] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7204.678397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7204.678507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7204.839441] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7204.844385] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7204.995082] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7205.252526] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7205.507891] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7205.768688] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7218.026974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7218.190112] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7219.931531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7220.014669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7220.171811] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7220.200422] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7222.462684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7222.619650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7320.193701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 97 sec [ 7320.352030] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 97 sec [ 7322.903299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7323.845719] Lustre: DEBUG MARKER: rc=0; [ 7323.845719] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7323.845719] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7323.845719] echo $(hostname -s): $val; [ 7323.845719] rc=$val; [ 7323.845719] fi; [ 7323.845719] exit $rc [ 7324.475466] Lustre: DEBUG MARKER: dmesg [ 7324.971788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1\/MDT2 ================ 09:40:49 \(1491558049\) [ 7325.136546] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ================ 09:40:49 (1491558049) [ 7327.072470] Lustre: DEBUG MARKER: sync; sync; sync [ 7328.044705] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7328.302878] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7328.451537] Turning device dm-0 (0xfc00000) read-only [ 7328.580562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7328.710979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7328.859400] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7329.115258] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7329.293351] Removing read-only on unknown block (0xfc00000) [ 7329.427697] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7330.096090] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491558055 with bad export cookie 5957255847833622585 [ 7330.101049] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7333.689229] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 7333.689263] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.689264] Lustre: Skipped 18 previous similar messages [ 7333.699131] LustreError: Skipped 22 previous similar messages [ 7340.612586] Lustre: DEBUG MARKER: hostname [ 7340.932326] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7341.190318] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7341.446558] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7341.608906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.985109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7347.241064] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7347.746121] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7347.750228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7347.920775] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7347.920998] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7348.067606] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7348.325562] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7348.582337] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7348.840688] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7361.137284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7361.291238] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7363.002052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7363.004967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7363.211685] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7363.261328] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7365.444791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7365.551485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7365.604203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7365.737362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7365.905238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7365.978331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.086507] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.161351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7367.567152] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7368.514638] Lustre: DEBUG MARKER: rc=0; [ 7368.514638] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7368.514638] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7368.514638] echo $(hostname -s): $val; [ 7368.514638] rc=$val; [ 7368.514638] fi; [ 7368.514638] exit $rc [ 7369.156539] Lustre: DEBUG MARKER: dmesg [ 7369.650943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1\/MDT2 ================ 09:41:34 \(1491558094\) [ 7369.816174] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ================ 09:41:34 (1491558094) [ 7369.995980] Lustre: DEBUG MARKER: sync; sync; sync [ 7370.987109] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7371.244160] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7371.390882] Turning device dm-0 (0xfc00000) read-only [ 7371.519213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7371.647609] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7373.574042] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7373.831099] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7374.008170] Removing read-only on unknown block (0xfc00000) [ 7374.142746] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7391.269554] Lustre: DEBUG MARKER: hostname [ 7391.582860] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7391.850162] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7392.120842] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7392.281827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.485724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7392.743313] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7393.248134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7393.249306] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7393.413772] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7393.413812] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7393.563141] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7393.819609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7394.078620] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7394.339636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7395.030974] Lustre: lustre-MDT0000: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [ 7395.030978] Lustre: MGS: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [ 7395.030980] Lustre: Skipped 198 previous similar messages [ 7406.617396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7406.789156] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7408.487650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7408.491710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7408.677878] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7408.725443] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7410.969975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7410.991600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7411.176784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7411.216427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7411.414402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7411.496410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7411.573415] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7411.713462] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7413.074329] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7414.010582] Lustre: DEBUG MARKER: rc=0; [ 7414.010582] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7414.010582] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7414.010582] echo $(hostname -s): $val; [ 7414.010582] rc=$val; [ 7414.010582] fi; [ 7414.010582] exit $rc [ 7414.664445] Lustre: DEBUG MARKER: dmesg [ 7415.167658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 111g: DNE: unlink striped dir, fail MDT1\/MDT2 ================================== 09:42:19 \(1491558139\) [ 7415.351404] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ================================== 09:42:19 (1491558139) [ 7415.532959] Lustre: DEBUG MARKER: sync; sync; sync [ 7416.527503] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7416.798782] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7416.984190] Turning device dm-0 (0xfc00000) read-only [ 7417.114646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7417.243939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7419.139031] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7419.392817] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7419.522214] Lustre: Failing over lustre-MDT0000 [ 7419.524257] Lustre: Skipped 6 previous similar messages [ 7419.573443] Removing read-only on unknown block (0xfc00000) [ 7419.580791] Lustre: server umount lustre-MDT0000 complete [ 7419.582846] Lustre: Skipped 6 previous similar messages [ 7419.711465] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7420.370408] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491558146 with bad export cookie 5957255847833626323 [ 7420.375020] LustreError: 5325:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7430.894476] Lustre: DEBUG MARKER: hostname [ 7431.200080] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7431.452197] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7431.702914] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7431.860795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.247312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7435.507110] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7436.004479] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7436.015304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7436.175541] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7436.180890] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7436.322612] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7436.578174] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7436.834205] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7437.094839] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7449.349089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7449.524166] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7451.224143] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7451.228287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7451.410242] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7451.455403] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7453.744133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7453.898688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7454.089397] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7454.240280] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7454.626436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7454.786749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7454.980452] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7455.132657] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7456.482551] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7457.414547] Lustre: DEBUG MARKER: rc=0; [ 7457.414547] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7457.414547] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7457.414547] echo $(hostname -s): $val; [ 7457.414547] rc=$val; [ 7457.414547] fi; [ 7457.414547] exit $rc [ 7458.046788] Lustre: DEBUG MARKER: dmesg [ 7458.558146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================= 09:43:03 \(1491558183\) [ 7458.731470] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================= 09:43:03 (1491558183) [ 7458.917934] Lustre: DEBUG MARKER: sync; sync; sync [ 7459.861050] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7460.118939] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7460.265972] Turning device dm-0 (0xfc00000) read-only [ 7460.397640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7460.536744] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7460.715308] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7460.970580] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7461.148756] Removing read-only on unknown block (0xfc00000) [ 7461.282654] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7463.419629] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.424606] LustreError: Skipped 234 previous similar messages [ 7471.601263] Lustre: DEBUG MARKER: hostname [ 7471.920091] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7472.178190] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7472.430681] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7472.589114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.807115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7473.066256] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7473.569576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7473.571241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7473.732317] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7473.734831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7473.876890] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7474.131362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7474.387298] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7474.649158] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7475.404931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7475.438842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7475.594823] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7475.674274] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7477.926602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7478.091329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7478.831362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7478.999420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7480.384904] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7481.318365] Lustre: DEBUG MARKER: rc=0; [ 7481.318365] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7481.318365] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7481.318365] echo $(hostname -s): $val; [ 7481.318365] rc=$val; [ 7481.318365] fi; [ 7481.318365] exit $rc [ 7481.959271] Lustre: DEBUG MARKER: dmesg [ 7482.459181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================= 09:43:27 \(1491558207\) [ 7482.632568] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================= 09:43:27 (1491558207) [ 7497.656972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7497.818381] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7499.520511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7499.526602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7499.711151] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7499.784612] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7501.964202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7502.061257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7502.130524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7502.315495] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7503.731130] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7504.666060] Lustre: DEBUG MARKER: rc=0; [ 7504.666060] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7504.666060] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7504.666060] echo $(hostname -s): $val; [ 7504.666060] rc=$val; [ 7504.666060] fi; [ 7504.666060] exit $rc [ 7505.318269] Lustre: DEBUG MARKER: dmesg [ 7505.812224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================= 09:43:50 \(1491558230\) [ 7505.976412] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================= 09:43:50 (1491558230) [ 7506.206480] Lustre: DEBUG MARKER: sync; sync; sync [ 7507.218235] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7507.492141] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7507.632028] Turning device dm-2 (0xfc00002) read-only [ 7507.770215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7507.903435] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7508.063240] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7508.318368] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7508.498190] Removing read-only on unknown block (0xfc00002) [ 7508.633721] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7518.930457] Lustre: DEBUG MARKER: hostname [ 7519.245276] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7519.497357] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7519.749243] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7519.908089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.117747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7520.376757] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7520.881108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7520.898967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7521.058214] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7521.060087] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7521.207999] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7521.467858] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7521.726042] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7521.987117] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7522.767340] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7522.823778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7522.995957] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7523.101396] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7525.281967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7525.341566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7525.506723] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7525.509671] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7526.972416] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7527.920770] Lustre: DEBUG MARKER: rc=0; [ 7527.920770] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7527.920770] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7527.920770] echo $(hostname -s): $val; [ 7527.920770] rc=$val; [ 7527.920770] fi; [ 7527.920770] exit $rc [ 7528.552560] Lustre: DEBUG MARKER: dmesg [ 7529.061751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================= 09:44:13 \(1491558253\) [ 7529.228945] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================= 09:44:13 (1491558253) [ 7544.011949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7544.176895] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7546.043311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7546.045300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7546.275612] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7546.282160] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7548.509461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7548.509483] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7548.705407] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7548.732180] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7550.149561] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7551.091703] Lustre: DEBUG MARKER: rc=0; [ 7551.091703] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7551.091703] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7551.091703] echo $(hostname -s): $val; [ 7551.091703] rc=$val; [ 7551.091703] fi; [ 7551.091703] exit $rc [ 7551.726306] Lustre: DEBUG MARKER: dmesg [ 7552.255407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ================================ 09:44:37 \(1491558277\) [ 7552.432393] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ================================ 09:44:37 (1491558277) [ 7552.650881] Lustre: DEBUG MARKER: sync; sync; sync [ 7553.635161] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7553.890691] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7554.032033] LustreError: 23572:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 7554.034498] LustreError: 23572:0:(osd_handler.c:2171:osd_ro()) Skipped 7 previous similar messages [ 7554.070862] Turning device dm-0 (0xfc00000) read-only [ 7554.201466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7554.329538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7556.284287] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7556.543665] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7556.721567] Removing read-only on unknown block (0xfc00000) [ 7556.858781] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7561.773493] LustreError: 6346:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491558287 with bad export cookie 5957255847833635017 [ 7561.778613] LustreError: 6346:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7567.071043] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491558285/real 1491558285] req@ffff88005f233c00 x1564004879591520/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491558292 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 7567.078536] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 7567.081116] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7567.085581] LustreError: Skipped 6 previous similar messages [ 7573.266512] Lustre: DEBUG MARKER: hostname [ 7573.576825] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7573.830992] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7574.087093] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7574.245093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.262871] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763df5cf to 0x52ac6ca1763e0d6f [ 7574.266956] Lustre: Skipped 6 previous similar messages [ 7574.303213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7574.305574] Lustre: Skipped 7 previous similar messages [ 7574.463872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7574.723745] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7575.144064] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 7575.146676] Lustre: Skipped 7 previous similar messages [ 7575.218191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7575.231920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7575.387823] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7575.388155] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7575.536610] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7575.794138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7576.051084] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7576.313478] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7579.298822] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 7579.304206] LustreError: Skipped 6 previous similar messages [ 7588.586442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7588.743207] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7590.453183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7590.470367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7590.644669] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7590.713411] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7592.675214] Lustre: lustre-MDT0000: Recovery over after 0:18, of 5 clients 5 recovered and 0 were evicted. [ 7592.679877] Lustre: Skipped 8 previous similar messages [ 7592.882300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7592.974409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7593.077142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7593.175711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7593.304721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7593.434283] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7593.462391] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7593.624692] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7595.024079] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7595.962748] Lustre: DEBUG MARKER: rc=0; [ 7595.962748] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7595.962748] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7595.962748] echo $(hostname -s): $val; [ 7595.962748] rc=$val; [ 7595.962748] fi; [ 7595.962748] exit $rc [ 7596.602401] Lustre: DEBUG MARKER: dmesg [ 7597.136359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ================================ 09:45:21 \(1491558321\) [ 7597.299750] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ================================ 09:45:21 (1491558321) [ 7597.564047] Lustre: DEBUG MARKER: sync; sync; sync [ 7598.596223] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7598.852876] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7598.989474] Turning device dm-0 (0xfc00000) read-only [ 7599.118226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7599.247694] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7599.381487] Lustre: DEBUG MARKER: sync; sync; sync [ 7600.238441] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7600.497270] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7600.628869] Turning device dm-2 (0xfc00002) read-only [ 7600.758059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7600.886944] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7601.068717] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7601.322906] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7601.501466] Removing read-only on unknown block (0xfc00000) [ 7601.634537] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7601.901322] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7602.156412] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7608.327677] Removing read-only on unknown block (0xfc00002) [ 7608.493576] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7618.801893] Lustre: DEBUG MARKER: hostname [ 7619.105298] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7619.351141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7619.595680] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7619.748683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.044600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7620.298337] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7620.795485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7620.795502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7620.956300] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7620.961605] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7621.101543] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7621.352742] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7621.607316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7621.864637] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7632.154410] Lustre: DEBUG MARKER: hostname [ 7632.462330] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7632.714338] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7632.966877] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7633.124854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.344067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7633.601526] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7634.120888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7634.122245] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7634.291615] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7634.292043] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7634.439302] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7634.696594] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7634.955221] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7635.217372] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7636.002902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7636.028289] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7636.230780] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7636.246261] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7638.483437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7638.510512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7638.671889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7638.749734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7638.887331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7638.995816] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7639.058739] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7639.214954] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7640.677526] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7641.602352] Lustre: DEBUG MARKER: rc=0; [ 7641.602352] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7641.602352] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7641.602352] echo $(hostname -s): $val; [ 7641.602352] rc=$val; [ 7641.602352] fi; [ 7641.602352] exit $rc [ 7642.241700] Lustre: DEBUG MARKER: dmesg [ 7642.774335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ================================ 09:46:07 \(1491558367\) [ 7642.942167] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ================================ 09:46:07 (1491558367) [ 7643.155997] Lustre: DEBUG MARKER: sync; sync; sync [ 7644.420352] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7644.676116] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7644.855065] Turning device dm-0 (0xfc00000) read-only [ 7644.985629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7645.116125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7646.964154] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7647.219639] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7647.404764] Removing read-only on unknown block (0xfc00000) [ 7647.538478] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7658.701377] Lustre: DEBUG MARKER: hostname [ 7659.008465] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7659.259696] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7659.510114] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7659.669224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.472978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7661.728583] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7662.232410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7662.239442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7662.394760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7662.400657] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7662.541775] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7662.798444] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7663.051285] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7663.310701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7675.601660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7675.765237] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7677.482502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7677.482507] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7677.678817] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7677.719726] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7679.922401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7680.034088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7680.125199] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7680.244952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7680.401666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7680.512129] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7680.603023] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7680.740136] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7682.128763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7683.060376] Lustre: DEBUG MARKER: rc=0; [ 7683.060376] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7683.060376] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7683.060376] echo $(hostname -s): $val; [ 7683.060376] rc=$val; [ 7683.060376] fi; [ 7683.060376] exit $rc [ 7683.710752] Lustre: DEBUG MARKER: dmesg [ 7684.234673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ================================ 09:46:49 \(1491558409\) [ 7684.398764] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ================================ 09:46:49 (1491558409) [ 7686.360587] Lustre: DEBUG MARKER: sync; sync; sync [ 7687.375179] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7687.632740] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7687.774850] Turning device dm-2 (0xfc00002) read-only [ 7687.904075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7688.034339] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7689.057679] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7689.329657] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7689.505353] Removing read-only on unknown block (0xfc00002) [ 7689.641110] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7701.899586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7702.060854] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7713.280055] Lustre: DEBUG MARKER: hostname [ 7713.587977] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7713.839740] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7714.091141] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7714.249478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.453023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7714.711610] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7715.211387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7715.217373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7715.374918] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7715.374949] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7715.522655] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7715.777934] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7716.048291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7716.308514] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7717.068581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7717.072805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7717.278210] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7717.305924] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7719.547871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7719.645244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7719.763271] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7719.854368] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7719.990197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.077320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.152324] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.257356] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7721.709381] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7722.655452] Lustre: DEBUG MARKER: rc=0; [ 7722.655452] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7722.655452] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7722.655452] echo $(hostname -s): $val; [ 7722.655452] rc=$val; [ 7722.655452] fi; [ 7722.655452] exit $rc [ 7723.280677] Lustre: DEBUG MARKER: dmesg [ 7723.801336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ================================ 09:47:28 \(1491558448\) [ 7723.963811] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ================================ 09:47:28 (1491558448) [ 7747.056319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7747.219619] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7760.462187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7760.623325] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7762.319624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7762.320068] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7762.501187] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7762.556784] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7764.736956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7764.841404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7764.896340] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7765.035404] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7765.168726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7765.274591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7765.348772] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7765.478309] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7766.918620] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7767.846499] Lustre: DEBUG MARKER: rc=0; [ 7767.846499] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7767.846499] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7767.846499] echo $(hostname -s): $val; [ 7767.846499] rc=$val; [ 7767.846499] fi; [ 7767.846499] exit $rc [ 7768.475663] Lustre: DEBUG MARKER: dmesg [ 7769.090081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ================================ 09:48:13 \(1491558493\) [ 7769.257261] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ================================ 09:48:13 (1491558493) [ 7769.478514] Lustre: DEBUG MARKER: sync; sync; sync [ 7770.487246] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7770.744471] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7770.885635] Turning device dm-2 (0xfc00002) read-only [ 7771.015232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7771.144604] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7773.128770] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7773.396782] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7773.572621] Removing read-only on unknown block (0xfc00002) [ 7773.707968] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7784.837453] Lustre: DEBUG MARKER: hostname [ 7785.146643] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7785.400072] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7785.652797] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7785.812288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.035654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7786.292672] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7786.792747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7786.793084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7786.954635] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7786.956383] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7787.100538] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7787.356456] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7787.611990] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7787.870399] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7800.125888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7800.295813] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7801.979355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7801.979752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7802.193018] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7802.194513] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7804.481559] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7804.534377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7804.737786] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7804.758146] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7805.026098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7805.037043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7805.189727] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7805.277707] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7806.733079] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7807.657117] Lustre: DEBUG MARKER: rc=0; [ 7807.657117] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7807.657117] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7807.657117] echo $(hostname -s): $val; [ 7807.657117] rc=$val; [ 7807.657117] fi; [ 7807.657117] exit $rc [ 7808.300495] Lustre: DEBUG MARKER: dmesg [ 7808.815126] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 =============================== 09:48:53 \(1491558533\) [ 7808.975906] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 =============================== 09:48:53 (1491558533) [ 7809.255777] Lustre: DEBUG MARKER: sync; sync; sync [ 7810.343263] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7810.599846] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7810.747166] Turning device dm-0 (0xfc00000) read-only [ 7810.876856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7811.004288] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7812.794464] Lustre: DEBUG MARKER: sync; sync; sync [ 7813.662269] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7813.917978] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7814.050323] Turning device dm-2 (0xfc00002) read-only [ 7814.179097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7814.306331] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7814.477154] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7814.732807] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7814.911636] Removing read-only on unknown block (0xfc00000) [ 7815.044804] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7819.773390] LustreError: 27167:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491558545 with bad export cookie 5957255847833648317 [ 7819.777783] LustreError: 27167:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7821.208924] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7821.469559] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7827.639558] Removing read-only on unknown block (0xfc00002) [ 7827.793644] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7838.111398] Lustre: DEBUG MARKER: hostname [ 7838.417386] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7838.663107] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7838.906472] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7839.062070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.284183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7839.537566] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7840.038852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7840.040987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7840.200090] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7840.210004] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7840.348242] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7840.602880] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7840.854514] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7841.110771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7853.398639] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7853.562228] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7864.778734] Lustre: DEBUG MARKER: hostname [ 7865.102637] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7865.371217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7865.632616] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7865.791403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7865.995152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7866.254027] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7866.753883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7866.754167] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7866.915849] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7866.916192] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7867.062884] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7867.318459] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7867.581762] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7867.842413] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 7868.622664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7868.624261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7868.853074] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7868.854877] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 7871.125641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7871.136216] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7871.327497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7871.372059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7871.582163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7871.597711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7871.776785] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7871.845985] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7871.982184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7872.094316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7872.137539] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7872.300197] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7873.739237] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7874.666709] Lustre: DEBUG MARKER: rc=0; [ 7874.666709] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7874.666709] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7874.666709] echo $(hostname -s): $val; [ 7874.666709] rc=$val; [ 7874.666709] fi; [ 7874.666709] exit $rc [ 7875.299980] Lustre: DEBUG MARKER: dmesg [ 7875.810768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 =============================== 09:50:00 \(1491558600\) [ 7875.976935] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 =============================== 09:50:00 (1491558600) [ 7876.165562] Lustre: DEBUG MARKER: sync; sync; sync [ 7877.248325] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7877.514849] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7877.659801] Turning device dm-0 (0xfc00000) read-only [ 7877.787792] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7877.916636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7881.441344] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7881.696766] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7881.874324] Removing read-only on unknown block (0xfc00000) [ 7882.007407] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7900.007257] Lustre: DEBUG MARKER: hostname [ 7900.330220] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7900.582663] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7900.833467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7900.981365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.180764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7901.435920] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7901.934530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7901.948202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7902.105221] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7902.111611] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7902.254391] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7902.512599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7902.767921] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7903.029095] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7915.301646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7915.462022] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7928.634491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7928.794606] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7930.516660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7930.516662] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7930.717417] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7930.746661] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 7933.037988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7933.277665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7933.548909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7933.731356] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7933.951373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7933.964842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7934.120222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7934.155326] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7934.335553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7934.518838] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7934.713631] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7934.880756] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7936.321194] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 7937.259196] Lustre: DEBUG MARKER: rc=0; [ 7937.259196] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 7937.259196] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 7937.259196] echo $(hostname -s): $val; [ 7937.259196] rc=$val; [ 7937.259196] fi; [ 7937.259196] exit $rc [ 7937.904844] Lustre: DEBUG MARKER: dmesg [ 7938.441140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 =============================== 09:51:03 \(1491558663\) [ 7938.601703] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 =============================== 09:51:03 (1491558663) [ 7938.838581] Lustre: DEBUG MARKER: sync; sync; sync [ 7939.732808] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 7939.989460] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 7940.126954] Turning device dm-0 (0xfc00000) read-only [ 7940.254616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 7940.385435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7940.517738] Lustre: DEBUG MARKER: sync; sync; sync [ 7941.355704] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 7941.611043] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 7941.809342] Turning device dm-2 (0xfc00002) read-only [ 7941.939600] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 7942.070050] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 7944.046931] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 7944.303332] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 7944.482529] Removing read-only on unknown block (0xfc00000) [ 7944.616211] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7944.883622] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 7945.135935] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 7945.262900] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 [ 7945.265304] LustreError: Skipped 42 previous similar messages [ 7951.311624] Removing read-only on unknown block (0xfc00002) [ 7951.456348] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 7974.254873] Lustre: DEBUG MARKER: hostname [ 7974.560744] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 7974.810944] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 7975.056403] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 7975.200400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.455903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7975.712361] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7976.205908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7976.208153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7976.373119] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7976.374192] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7976.517656] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7976.776165] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7977.028097] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7977.283991] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 7987.575403] Lustre: DEBUG MARKER: hostname [ 7987.905428] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 7988.162325] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 7988.415248] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 7988.569457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.783347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 7989.040968] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 7989.551745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7989.551747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7989.716836] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7989.733898] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 7989.874613] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 7990.129854] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7990.385794] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 7990.649028] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8001.990479] Lustre: lustre-MDT0000: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [ 8001.993590] Lustre: Skipped 345 previous similar messages [ 8002.938503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8003.104454] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8004.817892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8004.835672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8005.012156] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8005.075457] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8007.285637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8007.352529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8007.615357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8007.719961] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8007.884329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8007.940118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.084229] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.169537] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.351316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.397978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.533287] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8008.603595] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8010.074232] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8011.014269] Lustre: DEBUG MARKER: rc=0; [ 8011.014269] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8011.014269] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8011.014269] echo $(hostname -s): $val; [ 8011.014269] rc=$val; [ 8011.014269] fi; [ 8011.014269] exit $rc [ 8011.639272] Lustre: DEBUG MARKER: dmesg [ 8012.160184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 =============================== 09:52:16 \(1491558736\) [ 8012.324717] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 =============================== 09:52:16 (1491558736) [ 8014.261391] Lustre: DEBUG MARKER: sync; sync; sync [ 8015.307602] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8015.563583] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8015.706803] Turning device dm-2 (0xfc00002) read-only [ 8015.835572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8015.963125] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8018.607376] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8018.627795] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8018.632455] Lustre: Skipped 36 previous similar messages [ 8018.880283] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8019.065444] Removing read-only on unknown block (0xfc00002) [ 8019.202860] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8032.300866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8032.487906] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8043.710394] Lustre: DEBUG MARKER: hostname [ 8044.031476] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8044.287125] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8044.540250] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8044.685806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.881080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8045.136096] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8045.631361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8045.633768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8045.794909] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8045.797658] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8045.941654] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8046.196082] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8046.452235] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8046.714340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8058.975914] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8059.136393] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8060.886585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8060.893239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8061.064138] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8061.125357] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8063.396332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8063.421215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8063.656997] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8063.674935] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8063.844351] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8063.900022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.051297] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.098160] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.249123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.389423] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.408322] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8064.594791] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8066.022838] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8066.947440] Lustre: DEBUG MARKER: rc=0; [ 8066.947440] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8066.947440] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8066.947440] echo $(hostname -s): $val; [ 8066.947440] rc=$val; [ 8066.947440] fi; [ 8066.947440] exit $rc [ 8067.581820] Lustre: DEBUG MARKER: dmesg [ 8068.098522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 115: failover for create\/unlink striped directory ============================== 09:53:12 \(1491558792\) [ 8068.267470] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ============================== 09:53:12 (1491558792) [ 8083.231737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8083.390030] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8085.076976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8085.080197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8085.293137] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8085.297450] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8087.559454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8087.650444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8087.776394] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8087.857446] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8089.507559] Lustre: DEBUG MARKER: sync; sync; sync [ 8090.501552] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8090.756122] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8090.897917] Turning device dm-2 (0xfc00002) read-only [ 8091.027248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8091.155168] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8091.394654] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8091.647322] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8091.775997] Lustre: Failing over lustre-MDT0002 [ 8091.778029] Lustre: Skipped 14 previous similar messages [ 8092.281654] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 8092.283990] Lustre: Skipped 22 previous similar messages [ 8093.183087] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f545800 x1564004879654208/t0(0) o13->lustre-OST0004-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8093.188115] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 8097.746383] Removing read-only on unknown block (0xfc00002) [ 8097.753503] Lustre: server umount lustre-MDT0002 complete [ 8097.755668] Lustre: Skipped 14 previous similar messages [ 8097.885019] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8098.077619] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.082505] LustreError: Skipped 511 previous similar messages [ 8108.176458] Lustre: DEBUG MARKER: hostname [ 8108.484641] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8108.738862] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8108.990078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8109.149584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.360440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8109.619112] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8110.115963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8110.123147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8110.281836] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8110.281947] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8110.427841] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8110.683798] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8110.940279] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8111.202345] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8111.966140] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8112.121257] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8112.134856] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8112.314451] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [ 8114.536307] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8114.697948] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8115.400118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8115.566207] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8131.892865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8132.051556] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8133.729491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8133.769963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8133.924810] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8133.993206] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8136.262627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8136.424516] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8137.159820] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8137.313313] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8139.032915] Lustre: DEBUG MARKER: sync; sync; sync [ 8140.154819] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8140.409841] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8140.558954] Turning device dm-0 (0xfc00000) read-only [ 8140.690688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8140.830129] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8141.135133] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8141.391911] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8142.271065] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007897fc00 x1564004879664368/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8142.277160] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 8146.758521] Removing read-only on unknown block (0xfc00000) [ 8146.893772] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8157.214529] Lustre: DEBUG MARKER: hostname [ 8157.537672] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8157.790912] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8158.040412] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8158.257243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.767100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8162.026942] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8162.528627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8162.551043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8162.704686] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8162.705512] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8162.854278] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8163.108735] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8163.361908] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8163.621621] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8164.387342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8164.405096] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8164.656828] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8164.693168] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8166.880472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8166.920896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8167.077564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8167.132515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8168.960151] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8169.915315] Lustre: DEBUG MARKER: rc=0; [ 8169.915315] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8169.915315] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8169.915315] echo $(hostname -s): $val; [ 8169.915315] rc=$val; [ 8169.915315] fi; [ 8169.915315] exit $rc [ 8170.556468] Lustre: DEBUG MARKER: dmesg [ 8171.070851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 116a: large update log master MDT recovery ===================================== 09:54:55 \(1491558895\) [ 8171.240625] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ===================================== 09:54:55 (1491558895) [ 8171.397847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8171.397847] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8171.397847] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8171.691706] Lustre: DEBUG MARKER: sync; sync; sync [ 8172.668342] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8172.925427] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8173.056383] LustreError: 24984:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 8173.059037] LustreError: 24984:0:(osd_handler.c:2171:osd_ro()) Skipped 13 previous similar messages [ 8173.069863] Turning device dm-0 (0xfc00000) read-only [ 8173.199803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8173.328893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8173.459904] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80001702 [ 8173.592640] Lustre: *** cfs_fail_loc=1702, val=0*** [ 8173.858834] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8174.114166] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8174.289938] Removing read-only on unknown block (0xfc00000) [ 8174.424207] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8183.524051] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491558902/real 1491558902] req@ffff8800653fd800 x1564004879670784/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491558909 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 8183.530892] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 29 previous similar messages [ 8183.533318] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8183.537536] LustreError: Skipped 3 previous similar messages [ 8184.742663] Lustre: DEBUG MARKER: hostname [ 8185.055122] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8185.309110] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8185.561516] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8185.721056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.539234] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763e7e48 to 0x52ac6ca1763e8747 [ 8189.543253] Lustre: Skipped 3 previous similar messages [ 8189.594264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.596601] Lustre: Skipped 13 previous similar messages [ 8189.803924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8190.077477] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8190.480206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 8190.482881] Lustre: Skipped 13 previous similar messages [ 8190.594070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8190.595252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8190.755017] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8190.758564] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8190.902037] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8191.158088] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8191.413347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8191.673771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8192.447731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8192.454180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8192.688863] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8192.715371] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8194.591499] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 8194.597063] LustreError: Skipped 3 previous similar messages [ 8194.723328] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [ 8194.726475] Lustre: Skipped 13 previous similar messages [ 8194.970035] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8195.067329] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8195.153476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8195.220827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8196.590894] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8197.510114] Lustre: DEBUG MARKER: rc=0; [ 8197.510114] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8197.510114] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8197.510114] echo $(hostname -s): $val; [ 8197.510114] rc=$val; [ 8197.510114] fi; [ 8197.510114] exit $rc [ 8198.134298] Lustre: DEBUG MARKER: dmesg [ 8198.675960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 116b: large update log slave MDT recovery ====================================== 09:55:23 \(1491558923\) [ 8198.835879] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ====================================== 09:55:23 (1491558923) [ 8198.978415] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8198.978415] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8198.978415] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8214.101370] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8214.261372] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8215.971259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8215.982345] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8216.143361] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8216.267372] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8218.401900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8218.498390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8218.577585] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8218.710871] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8220.071053] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8221.014125] Lustre: DEBUG MARKER: rc=0; [ 8221.014125] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8221.014125] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8221.014125] echo $(hostname -s): $val; [ 8221.014125] rc=$val; [ 8221.014125] fi; [ 8221.014125] exit $rc [ 8221.642384] Lustre: DEBUG MARKER: dmesg [ 8222.158515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ================================= 09:55:46 \(1491558946\) [ 8222.331372] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ================================= 09:55:46 (1491558946) [ 8224.585317] Lustre: DEBUG MARKER: sync; sync; sync [ 8225.539728] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8225.795211] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8225.942019] Turning device dm-0 (0xfc00000) read-only [ 8226.070744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8226.200380] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8228.049395] Lustre: DEBUG MARKER: sync; sync; sync [ 8228.866969] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 notransno [ 8229.130188] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 readonly [ 8229.275360] Turning device dm-2 (0xfc00002) read-only [ 8229.403673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds3 REPLAY BARRIER on lustre-MDT0002 [ 8229.531935] Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 [ 8231.477427] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8231.749463] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8231.938322] Removing read-only on unknown block (0xfc00000) [ 8232.075117] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8233.188147] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 8233.444695] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [ 8239.616412] Removing read-only on unknown block (0xfc00002) [ 8239.764047] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8250.918868] Lustre: DEBUG MARKER: hostname [ 8251.219278] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8251.463576] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8251.709298] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8251.864709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.078934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8252.334386] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8252.828566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8252.828568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8252.988494] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8252.989332] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8253.131966] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8253.386415] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8253.640163] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8253.895053] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8266.104050] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8266.308078] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8277.526489] Lustre: DEBUG MARKER: hostname [ 8277.846533] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 8278.115718] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 8278.382733] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 8278.552439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.749759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8279.008157] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8279.504489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8279.504491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8279.740599] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8279.747975] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8279.902476] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8280.161470] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8280.433416] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8280.695964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8292.969818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8293.137679] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8294.865444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_u [ 8294.867843] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_u [ 8295.094507] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8295.096380] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,mdc.lustre-MDT0003-mdc-*.mds_server_uuid [ 8297.363519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8297.387865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8297.694722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8297.695777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8297.900281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8297.947416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.093786] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.159466] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.425131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.435503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.623698] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.642248] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.861512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8298.904401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8299.041726] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8299.146486] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8300.504963] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8301.426871] Lustre: DEBUG MARKER: rc=0; [ 8301.426871] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8301.426871] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8301.426871] echo $(hostname -s): $val; [ 8301.426871] rc=$val; [ 8301.426871] fi; [ 8301.426871] exit $rc [ 8302.059236] Lustre: DEBUG MARKER: dmesg [ 8302.578051] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 118: invalidate osp update will not cause update log corruption ================ 09:57:07 \(1491559027\) [ 8302.742372] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ================ 09:57:07 (1491559027) [ 8302.888250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8302.888250] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8302.888250] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8303.264486] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1705 [ 8303.400076] Lustre: *** cfs_fail_loc=1705, val=0*** [ 8303.543356] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x0 [ 8303.799094] Lustre: DEBUG MARKER: sync; sync; sync [ 8304.039846] LustreError: 29213:0:(llog_cat.c:764:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 8304.851709] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8305.106197] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8305.243501] Turning device dm-0 (0xfc00000) read-only [ 8305.371690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8305.501498] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8305.666756] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8305.921705] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8306.063744] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [ 8306.065906] Lustre: Skipped 32 previous similar messages [ 8307.215069] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880060d9cc00 x1564004879687392/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8307.219522] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 11 previous similar messages [ 8311.749719] Removing read-only on unknown block (0xfc00000) [ 8311.883037] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8322.198298] Lustre: DEBUG MARKER: hostname [ 8322.504097] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8322.755976] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8323.007957] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8323.166331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.357787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8327.616648] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8328.113238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8328.114560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8328.279665] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8328.287506] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8328.434379] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8328.693354] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8328.950544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8329.209094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8329.970827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8329.994027] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8330.184083] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8330.203787] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8332.468669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8332.546907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8332.624066] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8332.732870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8334.085754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8335.010898] Lustre: DEBUG MARKER: rc=0; [ 8335.010898] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8335.010898] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8335.010898] echo $(hostname -s): $val; [ 8335.010898] rc=$val; [ 8335.010898] fi; [ 8335.010898] exit $rc [ 8335.652803] Lustre: DEBUG MARKER: dmesg [ 8336.174349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 119: timeout of normal replay does not cause DNE replay fails ================= 09:57:40 \(1491559060\) [ 8336.338978] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ================= 09:57:40 (1491559060) [ 8336.481179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8336.481179] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8336.481179] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8336.758230] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.recovery_time_hard [ 8337.027694] Lustre: DEBUG MARKER: sync; sync; sync [ 8338.035322] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8338.292335] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8338.441571] Turning device dm-0 (0xfc00000) read-only [ 8338.571455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8338.710777] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8338.922458] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8339.179136] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8339.357815] Removing read-only on unknown block (0xfc00000) [ 8339.492907] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8339.809762] Lustre: DEBUG MARKER: hostname [ 8340.081493] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000714 [ 8340.345178] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=65 [ 8340.612084] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8340.865350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8341.118767] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o recovery_time_hard=60 /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8341.277778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.508852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8341.766067] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8342.258518] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8342.258910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8342.424215] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8342.425442] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8342.572924] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8342.830848] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8343.088467] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8343.348533] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8344.114755] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8344.127982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8344.308289] Lustre: 29179:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [ 8344.321658] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8344.345556] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8344.565493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8344.597029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8344.735994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8344.836352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8345.992124] Lustre: 30103:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 0 [ 8345.997138] Lustre: 30103:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 1 previous similar message [ 8346.327453] LustreError: 3578:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 714 sleeping for 65000ms [ 8411.330036] LustreError: 3578:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 714 awake [ 8411.332829] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 8411.335502] Lustre: 3578:0:(ldlm_lib.c:1625:abort_req_replay_queue()) @@@ aborted: req@ffff880051e9d800 x1564004864928960/t0(532575944715) o36->c36fff13-4af7-2c83-1c38-d82b4c834851@10.9.5.237@tcp:-1/-1 lens 616/0 e 7 to 0 dl 1491559147 ref 1 fl Complete:/4/ffffffff rc 0/-1 [ 8411.343845] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 8411.344376] Lustre: lustre-MDT0000: Denying connection for new client c36fff13-4af7-2c83-1c38-d82b4c834851(at 10.9.5.237@tcp), waiting for 5 known clients (4 recovered, 0 in progress, and 1 evicted) to recover in 21188505:12 [ 8411.344377] Lustre: Skipped 1 previous similar message [ 8411.358097] Lustre: 3578:0:(ldlm_lib.c:1779:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reaching hard limit: 60, extend: 1 [ 8411.363443] Lustre: 3578:0:(ldlm_lib.c:1779:extend_recovery_timer()) Skipped 2 previous similar messages [ 8411.396018] Lustre: 3578:0:(ldlm_lib.c:2044:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 8411.399051] Lustre: 3578:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 8411.402043] Lustre: 3578:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [ 8411.420071] Lustre: 3578:0:(ldlm_lib.c:2550:target_recovery_thread()) too long recovery - read logs [ 8411.423112] LustreError: dumping log to /tmp/lustre-log.1491559137.3578 [ 8417.880981] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.recovery_time_hard=180 [ 8418.347663] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8419.255310] Lustre: DEBUG MARKER: rc=0; [ 8419.255310] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8419.255310] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8419.255310] echo $(hostname -s): $val; [ 8419.255310] rc=$val; [ 8419.255310] fi; [ 8419.255310] exit $rc [ 8419.879615] Lustre: DEBUG MARKER: dmesg [ 8420.400447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 120: DNE fail abort should stop both normal and DNE replay ===================== 09:59:05 \(1491559145\) [ 8420.572407] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ===================== 09:59:05 (1491559145) [ 8420.717622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [ 8420.717622] /usr/sbin/lctl lustre_build_version 2>/dev/null || [ 8420.717622] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [ 8420.995350] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [ 8421.251926] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [ 8421.441233] Turning device dm-0 (0xfc00000) read-only [ 8421.571608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [ 8421.699282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8421.974119] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 8422.227920] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 8426.479064] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064f2f000 x1564004879702208/t0(0) o13->lustre-OST0004-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 8426.484207] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 8427.753504] Removing read-only on unknown block (0xfc00000) [ 8427.888133] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 8428.201191] Lustre: DEBUG MARKER: hostname [ 8428.484338] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 8428.739676] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 8428.989800] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o abort_recovery /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 8429.149450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.222042] LustreError: 5881:0:(mdt_handler.c:5950:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 8429.225137] LustreError: 5881:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 8429.228064] Lustre: 5929:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 8429.230678] Lustre: 5929:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 1 previous similar message [ 8429.256078] LustreError: 5926:0:(lod_dev.c:419:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 getting update log failed: rc = -108 [ 8429.260218] Lustre: lustre-MDT0000: disconnecting 5 stale clients [ 8429.411987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 8429.666490] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [ 8430.164183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8430.164190] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8430.328430] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8430.334292] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8430.476626] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8430.735427] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8430.991762] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 8431.253555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8434.106240] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8435.044541] Lustre: DEBUG MARKER: rc=0; [ 8435.044541] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8435.044541] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8435.044541] echo $(hostname -s): $val; [ 8435.044541] rc=$val; [ 8435.044541] fi; [ 8435.044541] exit $rc [ 8435.673363] Lustre: DEBUG MARKER: dmesg [ 8436.181591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test complete, duration 7751 sec ==================================================== 09:59:21 \(1491559161\) [ 8436.356784] Lustre: DEBUG MARKER: == replay-single test complete, duration 7751 sec ==================================================== 09:59:21 (1491559161) [ 8440.100430] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-quota ============----- Fri Apr 7 09:59:24 UTC 2017 [ 8440.268714] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-quota ============----- Fri Apr 7 09:59:24 UTC 2017 [ 8440.657133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test complete, duration -o sec ======================================================= 09:59:25 \(1491559165\) [ 8440.820950] Lustre: DEBUG MARKER: == sanity-quota test complete, duration -o sec ======================================================= 09:59:25 (1491559165) [ 8441.126170] Lustre: DEBUG MARKER: ! debugfs -c -R supported_features | [ 8441.126170] grep -q 'quota' [ 8441.898766] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 8441.933281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client \/mnt\/lustre [ 8442.100179] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 8442.149976] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_config_client /mnt/lustre [ 8444.785700] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8444.785700] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8444.785700] if [ $running -ne $mpts ]; then [ 8444.785700] echo $(hostname) env are INSANE!; [ 8444.785700] exit 1; [ 8444.785700] fi [ 8445.381497] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 8445.381497] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 8445.381497] if [ $running -ne $mpts ]; then [ 8445.381497] echo $(hostname) env are INSANE!; [ 8445.381497] exit 1; [ 8445.381497] fi [ 8445.970863] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 8445.970863] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 8445.970863] if [ $running -ne $mpts ]; then [ 8445.970863] echo $(hostname) env are INSANE!; [ 8445.970863] exit 1; [ 8445.970863] fi [ 8446.862288] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 8447.120376] Lustre: DEBUG MARKER: cat /proc/mounts [ 8447.911404] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 8448.168212] Lustre: DEBUG MARKER: cat /proc/mounts [ 8453.474178] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 8453.792812] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 8453.958650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8454.108059] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 8454.455054] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 8455.100439] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.107557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.111516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.129901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.211154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.368188] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.426178] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.465139] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.532618] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.573398] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8455.854203] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [ 8456.143932] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8456.660485] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 8457.230299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [ 8457.395139] Lustre: DEBUG MARKER: excepting tests: [ 8457.763803] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8462.186633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 0: Test basic quota performance ================================================= 09:59:47 \(1491559187\) [ 8462.347275] Lustre: DEBUG MARKER: == sanity-quota test 0: Test basic quota performance ================================================= 09:59:47 (1491559187) [ 8462.773543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8463.040281] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8463.315446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8464.592398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8465.922352] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8466.220127] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8473.886250] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8486.549135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8486.815960] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8487.092950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8488.370919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8489.648849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8490.924305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8492.199752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8493.480115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8494.757999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8496.055771] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8497.740031] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8498.636887] Lustre: DEBUG MARKER: rc=0; [ 8498.636887] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8498.636887] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8498.636887] echo $(hostname -s): $val; [ 8498.636887] rc=$val; [ 8498.636887] fi; [ 8498.636887] exit $rc [ 8499.251935] Lustre: DEBUG MARKER: dmesg [ 8499.761437] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 1: Block hard limit \(normal use and out of quota\) =============================== 10:00:24 \(1491559224\) [ 8499.925970] Lustre: DEBUG MARKER: == sanity-quota test 1: Block hard limit (normal use and out of quota) =============================== 10:00:24 (1491559224) [ 8500.339842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8501.652347] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8501.947318] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8508.822973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(block hardlimit:10 MB\) [ 8508.998381] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB) [ 8509.247705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write... [ 8509.397635] Lustre: DEBUG MARKER: Write... [ 8509.826695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8509.988049] Lustre: DEBUG MARKER: Write out of block quota ... [ 8510.611050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8510.873167] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8511.147352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8512.426578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8513.710462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8514.986968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8516.280183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8517.563288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8518.843742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8520.127510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8521.435582] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8523.183626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8523.344669] Lustre: DEBUG MARKER: -------------------------------------- [ 8523.539219] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(block hardlimit:10 MB\) [ 8523.701904] Lustre: DEBUG MARKER: Group quota (block hardlimit:10 MB) [ 8523.951542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write ... [ 8524.111217] Lustre: DEBUG MARKER: Write ... [ 8524.478776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [ 8524.655057] Lustre: DEBUG MARKER: Write out of block quota ... [ 8525.360204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8525.639707] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8525.929756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8527.222519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8528.504689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8529.776363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8531.049287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8532.325355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8533.612510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8534.889148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8536.195119] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8537.881669] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8538.803104] Lustre: DEBUG MARKER: rc=0; [ 8538.803104] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8538.803104] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8538.803104] echo $(hostname -s): $val; [ 8538.803104] rc=$val; [ 8538.803104] fi; [ 8538.803104] exit $rc [ 8539.398100] Lustre: DEBUG MARKER: dmesg [ 8540.004130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 2: File hard limit \(normal use and out of quota\) ================================ 10:01:04 \(1491559264\) [ 8540.177488] Lustre: DEBUG MARKER: == sanity-quota test 2: File hard limit (normal use and out of quota) ================================ 10:01:04 (1491559264) [ 8540.609403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8541.933362] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8542.237188] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8542.501743] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8542.753913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8543.008847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8544.264939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8545.518917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8546.772745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8548.032642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8549.288659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8550.550669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8552.857256] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8553.115535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8553.368248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8554.453311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(inode hardlimit:1024 files\) [ 8554.612051] Lustre: DEBUG MARKER: User quota (inode hardlimit:1024 files) [ 8554.889281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8555.056194] Lustre: DEBUG MARKER: Create 1024 files ... [ 8555.756906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8555.922137] Lustre: DEBUG MARKER: Create out of file quota ... [ 8557.321652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8559.720927] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -------------------------------------- [ 8559.881831] Lustre: DEBUG MARKER: -------------------------------------- [ 8560.079779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(inode hardlimit:1024 files\) [ 8560.256316] Lustre: DEBUG MARKER: Group quota (inode hardlimit:1024 files) [ 8560.495119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create 1024 files ... [ 8560.652593] Lustre: DEBUG MARKER: Create 1024 files ... [ 8561.368598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Create out of file quota ... [ 8561.535267] Lustre: DEBUG MARKER: Create out of file quota ... [ 8562.918634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8564.478744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8565.786556] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8567.471167] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8568.382089] Lustre: DEBUG MARKER: rc=0; [ 8568.382089] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8568.382089] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8568.382089] echo $(hostname -s): $val; [ 8568.382089] rc=$val; [ 8568.382089] fi; [ 8568.382089] exit $rc [ 8568.982299] Lustre: DEBUG MARKER: dmesg [ 8569.486958] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 3: Block soft limit \(start timer, timer goes off, stop timer\) =================== 10:01:34 \(1491559294\) [ 8569.654724] Lustre: DEBUG MARKER: == sanity-quota test 3: Block soft limit (start timer, timer goes off, stop timer) =================== 10:01:34 (1491559294) [ 8569.799461] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8576.955207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8578.269864] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8609.104129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8609.366115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8609.686677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8610.965431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8612.256536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8613.536691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8614.808544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8616.115503] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8617.131288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8617.391088] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8617.674586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8618.954898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8620.234652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8621.517319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8622.806009] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8624.089550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8625.369333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8626.671104] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8628.619514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8629.940443] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8660.787574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8661.047854] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8661.331074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8662.606897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8663.880577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8665.153469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8666.430800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8667.708299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8668.983322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8670.256665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8671.555095] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8672.704753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8672.968442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8673.241141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8674.520652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8675.797321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8677.077704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8678.368687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8679.645156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8680.947512] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8682.655689] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8683.550742] Lustre: DEBUG MARKER: rc=0; [ 8683.550742] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8683.550742] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8683.550742] echo $(hostname -s): $val; [ 8683.550742] rc=$val; [ 8683.550742] fi; [ 8683.550742] exit $rc [ 8684.153204] Lustre: DEBUG MARKER: dmesg [ 8684.666778] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 4a: File soft limit \(start timer, timer goes off, stop timer\) =================== 10:03:29 \(1491559409\) [ 8684.835088] Lustre: DEBUG MARKER: == sanity-quota test 4a: File soft limit (start timer, timer goes off, stop timer) =================== 10:03:29 (1491559409) [ 8684.982378] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8685.248255] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8685.502297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8685.760901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8686.803436] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8687.056982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8687.309533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8688.661907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8689.989722] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8690.314250] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8691.041491] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8709.725232] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8710.679108] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8711.609706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8711.868987] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8712.146128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8713.422679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8714.699929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8715.989141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8717.266670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8718.542937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8719.818658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8721.137640] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8722.025934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8722.292112] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8722.573308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8723.850286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8725.126722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8726.410767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8727.691731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8728.965702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8730.242534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8731.561756] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8733.518696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8734.835279] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8735.162590] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8735.836419] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8754.514502] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8755.216219] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8756.141154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8756.405857] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8756.689189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8757.968348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8759.250307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8760.532536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8761.805038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8763.083669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8764.357295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8765.637718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8766.952732] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8767.874207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8768.136637] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8768.421502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8769.705466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8770.991115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8772.266374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8773.554442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8774.831083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8776.140119] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8777.813412] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8778.706861] Lustre: DEBUG MARKER: rc=0; [ 8778.706861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8778.706861] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8778.706861] echo $(hostname -s): $val; [ 8778.706861] rc=$val; [ 8778.706861] fi; [ 8778.706861] exit $rc [ 8779.300925] Lustre: DEBUG MARKER: dmesg [ 8779.820093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 4b: Grace time strings handling ================================================= 10:05:04 \(1491559504\) [ 8779.982705] Lustre: DEBUG MARKER: == sanity-quota test 4b: Grace time strings handling ================================================= 10:05:04 (1491559504) [ 8780.403718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8781.777090] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8782.684217] Lustre: DEBUG MARKER: rc=0; [ 8782.684217] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8782.684217] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8782.684217] echo $(hostname -s): $val; [ 8782.684217] rc=$val; [ 8782.684217] fi; [ 8782.684217] exit $rc [ 8783.299557] Lustre: DEBUG MARKER: dmesg [ 8783.794893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 5: Chown & chgrp successfully even out of block\/file quota ====================== 10:05:08 \(1491559508\) [ 8783.962477] Lustre: DEBUG MARKER: == sanity-quota test 5: Chown [ 8784.526400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8785.854936] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8786.151739] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 8786.418189] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8786.672289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8786.926246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 8787.962596] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 8788.218964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8788.470842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 8789.507974] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8797.439827] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8797.700545] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8797.990913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8799.275083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8800.547864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8801.821758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8803.097291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8804.376171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8805.653568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8806.959139] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8809.659032] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8810.555786] Lustre: DEBUG MARKER: rc=0; [ 8810.555786] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8810.555786] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8810.555786] echo $(hostname -s): $val; [ 8810.555786] rc=$val; [ 8810.555786] fi; [ 8810.555786] exit $rc [ 8811.158287] Lustre: DEBUG MARKER: dmesg [ 8811.670613] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 6: Test dropping acquire request on master ====================================== 10:05:36 \(1491559536\) [ 8811.832097] Lustre: DEBUG MARKER: == sanity-quota test 6: Test dropping acquire request on master ====================================== 10:05:36 (1491559536) [ 8812.618832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8813.945057] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8814.254765] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8822.361517] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8823.047761] Lustre: DEBUG MARKER: lctl set_param fail_val=601 fail_loc=0x513 [ 8823.200691] Lustre: *** cfs_fail_loc=513, val=0*** [ 8823.202195] Lustre: Skipped 4 previous similar messages [ 8823.763265] LustreError: 29174:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1564004913173088 [ 8831.084035] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 8831.086156] Lustre: Skipped 14 previous similar messages [ 8831.087784] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [ 8831.089975] Lustre: Skipped 234 previous similar messages [ 8831.579737] Lustre: *** cfs_fail_loc=513, val=0*** [ 8831.581483] Lustre: Skipped 116 previous similar messages [ 8832.581441] LustreError: 29174:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1564004913173664 [ 8839.580977] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 8841.580897] LustreError: 29174:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1564004913174224 [ 8848.580975] Lustre: *** cfs_fail_loc=513, val=0*** [ 8848.582606] Lustre: Skipped 92 previous similar messages [ 8848.584275] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 8851.580724] LustreError: 29174:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1564004913174784 [ 8862.580248] LustreError: 29174:0:(service.c:1908:ptlrpc_server_handle_req_in()) drop incoming rpc opc 601, x1564004913175856 [ 8864.584097] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8869.579915] Lustre: lustre-MDT0000: Received new LWP connection from 10.9.5.244@tcp, removing former export from same NID [ 8869.582206] Lustre: Skipped 1 previous similar message [ 8875.505362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8875.766903] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8876.051434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8877.334976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8878.628655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8879.912196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8881.192111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8882.467901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8883.749865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8885.027564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8886.325104] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8887.991199] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8888.922121] Lustre: DEBUG MARKER: rc=0; [ 8888.922121] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8888.922121] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8888.922121] echo $(hostname -s): $val; [ 8888.922121] rc=$val; [ 8888.922121] fi; [ 8888.922121] exit $rc [ 8889.531304] Lustre: DEBUG MARKER: dmesg [ 8890.022161] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7a: Quota reintegration \(global index\) ========================================== 10:06:54 \(1491559614\) [ 8890.187140] Lustre: DEBUG MARKER: == sanity-quota test 7a: Quota reintegration (global index) ========================================== 10:06:54 (1491559614) [ 8890.615130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8891.925303] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8892.259474] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8900.166188] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 8909.235253] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 8909.968038] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [ 8909.970340] LustreError: Skipped 32 previous similar messages [ 8909.972118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8909.976014] Lustre: Skipped 20 previous similar messages [ 8923.939319] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8924.115606] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8925.161568] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8943.757129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8944.028308] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8944.351035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8945.626743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8946.905915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8948.188759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8949.467359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8950.743248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8952.027914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8953.329388] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8959.933746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8960.104983] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 8961.145461] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 8975.484210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8975.765681] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 8976.047176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8977.328844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8978.621512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8979.909516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8981.199287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8982.480973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8983.758665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8985.036746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8986.341470] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 8988.072599] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 8988.979293] Lustre: DEBUG MARKER: rc=0; [ 8988.979293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 8988.979293] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 8988.979293] echo $(hostname -s): $val; [ 8988.979293] rc=$val; [ 8988.979293] fi; [ 8988.979293] exit $rc [ 8989.605786] Lustre: DEBUG MARKER: dmesg [ 8990.145176] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7b: Quota reintegration \(slave index\) =========================================== 10:08:34 \(1491559714\) [ 8990.310153] Lustre: DEBUG MARKER: == sanity-quota test 7b: Quota reintegration (slave index) =========================================== 10:08:34 (1491559714) [ 8990.740978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 8992.055512] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 8992.373662] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9000.279835] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9007.281979] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0xa02 [ 9007.560828] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9018.168410] Lustre: *** cfs_fail_loc=a02, val=0*** [ 9018.442094] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9021.658643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9021.833326] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9022.870246] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9040.679580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9040.943091] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9041.234521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9042.518766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9043.796749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9045.075063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9046.350914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9047.625452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9048.929684] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9050.638547] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9051.540322] Lustre: DEBUG MARKER: rc=0; [ 9051.540322] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9051.540322] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9051.540322] echo $(hostname -s): $val; [ 9051.540322] rc=$val; [ 9051.540322] fi; [ 9051.540322] exit $rc [ 9052.146500] Lustre: DEBUG MARKER: dmesg [ 9052.633808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7c: Quota reintegration \(restart mds during reintegration\) ====================== 10:09:37 \(1491559777\) [ 9052.808760] Lustre: DEBUG MARKER: == sanity-quota test 7c: Quota reintegration (restart mds during reintegration) ====================== 10:09:37 (1491559777) [ 9053.241740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9054.556325] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9054.937281] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9068.528595] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9079.495889] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 9079.752760] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 9079.884457] Lustre: Failing over lustre-MDT0000 [ 9079.886206] Lustre: Skipped 7 previous similar messages [ 9079.994960] Lustre: server umount lustre-MDT0000 complete [ 9079.996746] Lustre: Skipped 7 previous similar messages [ 9080.126529] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 9080.698350] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9080.702443] LustreError: Skipped 240 previous similar messages [ 9080.712123] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 9080.977778] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 9081.233508] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 9081.485922] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 9081.634426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.653914] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9081.657489] LustreError: Skipped 3 previous similar messages [ 9081.659650] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1763eb3bd to 0x52ac6ca176413f6c [ 9081.663098] Lustre: Skipped 3 previous similar messages [ 9081.700953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.703067] Lustre: Skipped 5 previous similar messages [ 9081.888367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 9082.103607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [ 9082.105892] Lustre: Skipped 4 previous similar messages [ 9082.146442] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [ 9082.646972] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9082.649897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9082.812757] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9082.813855] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9082.960951] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9083.216621] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9083.475691] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 9083.735425] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 9083.992213] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 9084.312872] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9086.696421] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 9086.701424] LustreError: Skipped 4 previous similar messages [ 9086.815957] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [ 9086.818969] Lustre: Skipped 4 previous similar messages [ 9207.165893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9207.446526] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9207.747714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9209.039676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9210.323382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9211.604778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9212.886226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9214.170790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9215.456276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9216.734027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9218.008953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9219.327928] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9221.041635] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9221.963052] Lustre: DEBUG MARKER: rc=0; [ 9221.963052] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9221.963052] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9221.963052] echo $(hostname -s): $val; [ 9221.963052] rc=$val; [ 9221.963052] fi; [ 9221.963052] exit $rc [ 9222.598098] Lustre: DEBUG MARKER: dmesg [ 9223.084624] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7d: Quota reintegration \(Transfer index in multiple bulks\) ====================== 10:12:27 \(1491559947\) [ 9223.252497] Lustre: DEBUG MARKER: == sanity-quota test 7d: Quota reintegration (Transfer index in multiple bulks) ====================== 10:12:27 (1491559947) [ 9223.669376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9225.001087] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9225.308142] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [ 9241.162216] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0x608 [ 9241.453231] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9262.627948] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9295.450714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9295.716134] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9295.996825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9297.276437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9298.551126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9299.825118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9301.105697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9302.386188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9303.688106] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9306.398170] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9307.321962] Lustre: DEBUG MARKER: rc=0; [ 9307.321962] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9307.321962] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9307.321962] echo $(hostname -s): $val; [ 9307.321962] rc=$val; [ 9307.321962] fi; [ 9307.321962] exit $rc [ 9307.932780] Lustre: DEBUG MARKER: dmesg [ 9308.411896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 7e: Quota reintegration \(inode limits\) ========================================== 10:13:53 \(1491560033\) [ 9308.582457] Lustre: DEBUG MARKER: == sanity-quota test 7e: Quota reintegration (inode limits) ========================================== 10:13:53 (1491560033) [ 9309.019646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9310.347273] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9310.678553] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9310.946765] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9311.209694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9311.465694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9312.509226] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9312.765037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9313.019331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9315.078371] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=none [ 9315.374824] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9315.633416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9315.888233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9317.143992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9318.397049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9319.651534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9320.907556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9321.946093] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9322.203900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9322.459214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9324.356259] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9324.619183] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9324.873739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9325.129141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9326.384402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9327.644587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9328.914480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9330.178358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9331.438240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9332.698947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9333.955897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9335.208725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9336.242765] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9336.500915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9336.757383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9339.308060] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9339.469256] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9340.949190] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9341.261727] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9341.519716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9341.519716] grep user | awk '{ print $3 }' [ 9341.779184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9341.779184] grep user | awk '{ print $3 }' [ 9342.833053] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9343.094386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9343.094386] grep user | awk '{ print $3 }' [ 9343.351393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9343.351393] grep user | awk '{ print $3 }' [ 9344.431478] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9344.688119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9344.688119] grep group | awk '{ print $3 }' [ 9344.945647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9344.945647] grep group | awk '{ print $3 }' [ 9345.995240] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9346.248690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9346.248690] grep group | awk '{ print $3 }' [ 9346.507275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9346.507275] grep group | awk '{ print $3 }' [ 9356.678136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9357.975153] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9361.487723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9361.650632] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9363.153516] Lustre: DEBUG MARKER: lctl set_param debug=+quota [ 9363.503420] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9363.759767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9363.759767] grep user | awk '{ print $3 }' [ 9364.017650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9364.017650] grep user | awk '{ print $3 }' [ 9365.079941] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9365.337571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9365.337571] grep user | awk '{ print $3 }' [ 9365.595062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9365.595062] grep user | awk '{ print $3 }' [ 9366.652088] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9366.912575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9366.912575] grep group | awk '{ print $3 }' [ 9367.173723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.info | [ 9367.173723] grep group | awk '{ print $3 }' [ 9368.231540] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9368.487300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9368.487300] grep group | awk '{ print $3 }' [ 9368.744856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.info | [ 9368.744856] grep group | awk '{ print $3 }' [ 9378.310780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9379.600690] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9381.291242] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9382.198851] Lustre: DEBUG MARKER: rc=0; [ 9382.198851] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9382.198851] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9382.198851] echo $(hostname -s): $val; [ 9382.198851] rc=$val; [ 9382.198851] fi; [ 9382.198851] exit $rc [ 9382.822584] Lustre: DEBUG MARKER: dmesg [ 9383.303298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 8: Run dbench with quota enabled ================================================ 10:15:08 \(1491560108\) [ 9383.461399] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ================================================ 10:15:08 (1491560108) [ 9383.872324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9385.191142] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9385.503344] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9385.772323] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9386.029633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9386.283267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9387.337136] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9387.604310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9387.858612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9388.900441] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9542.337480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9542.603599] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9542.903552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9544.229453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9545.511340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9546.794055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9548.089628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9549.363838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9550.640961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9551.918861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9553.193706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9554.496098] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9557.219625] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9558.134387] Lustre: DEBUG MARKER: rc=0; [ 9558.134387] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9558.134387] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9558.134387] echo $(hostname -s): $val; [ 9558.134387] rc=$val; [ 9558.134387] fi; [ 9558.134387] exit $rc [ 9558.743637] Lustre: DEBUG MARKER: dmesg [ 9559.223169] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 9: Block limit larger than 4GB \(b10707\) ========================================= 10:18:04 \(1491560284\) [ 9559.389700] Lustre: DEBUG MARKER: == sanity-quota test 9: Block limit larger than 4GB (b10707) ========================================= 10:18:04 (1491560284) [ 9559.582492] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 15019304 required: 4900000 [ 9559.743659] Lustre: DEBUG MARKER: OST0_SIZE: 15019304 required: 4900000 [ 9560.156877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9561.488775] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9561.787410] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9568.764664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set enough high limit\(block:100G\; file: 1000000\) for user: quota_usr [ 9568.931622] Lustre: DEBUG MARKER: Set enough high limit(block:100G; file: 1000000) for user: quota_usr [ 9569.159163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set enough high limit\(block:100G\; file: 1000000\) for group: quota_usr [ 9569.327100] Lustre: DEBUG MARKER: Set enough high limit(block:100G; file: 1000000) for group: quota_usr [ 9569.703066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write the big file of 4.5G ... [ 9569.863378] Lustre: DEBUG MARKER: Write the big file of 4.5G ... [ 9767.010737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9767.278584] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9767.898661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9769.185873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9770.471075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9771.763471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9773.056235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9774.362830] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9777.348615] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9778.242599] Lustre: DEBUG MARKER: rc=0; [ 9778.242599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9778.242599] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9778.242599] echo $(hostname -s): $val; [ 9778.242599] rc=$val; [ 9778.242599] fi; [ 9778.242599] exit $rc [ 9778.850767] Lustre: DEBUG MARKER: dmesg [ 9779.340911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 10: Test quota for root user ==================================================== 10:21:44 \(1491560504\) [ 9779.506542] Lustre: DEBUG MARKER: == sanity-quota test 10: Test quota for root user ==================================================== 10:21:44 (1491560504) [ 9779.938258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9781.255898] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9781.566166] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9789.013527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9789.280964] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9789.570732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9790.859738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9792.138115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9793.428759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9794.723101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9796.007562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9797.298263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9798.588776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9799.907780] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9801.599853] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9802.490642] Lustre: DEBUG MARKER: rc=0; [ 9802.490642] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9802.490642] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9802.490642] echo $(hostname -s): $val; [ 9802.490642] rc=$val; [ 9802.490642] fi; [ 9802.490642] exit $rc [ 9803.096599] Lustre: DEBUG MARKER: dmesg [ 9803.572911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 11: Chown\/chgrp ignores quota =================================================== 10:22:08 \(1491560528\) [ 9803.740251] Lustre: DEBUG MARKER: == sanity-quota test 11: Chown/chgrp ignores quota =================================================== 10:22:08 (1491560528) [ 9804.157716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9805.472377] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9805.771205] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [ 9806.034984] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9806.291658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9806.548416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9807.586691] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9807.842277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9808.099609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9809.470411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9809.730723] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9810.014946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9811.287341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9812.570668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9813.854600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9815.144392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9816.433146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9817.712148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9819.003115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9820.314023] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9821.995156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9822.892302] Lustre: DEBUG MARKER: rc=0; [ 9822.892302] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9822.892302] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9822.892302] echo $(hostname -s): $val; [ 9822.892302] rc=$val; [ 9822.892302] fi; [ 9822.892302] exit $rc [ 9823.498853] Lustre: DEBUG MARKER: dmesg [ 9823.980535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 12a: Block quota rebalancing ==================================================== 10:22:28 \(1491560548\) [ 9824.148710] Lustre: DEBUG MARKER: == sanity-quota test 12a: Block quota rebalancing ==================================================== 10:22:28 (1491560548) [ 9824.570902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9825.887220] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9826.193536] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9854.893092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9855.157146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9855.440911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9856.726489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9858.009847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9859.286524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9860.568078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9861.844164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9863.121099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9864.419197] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9878.954189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9879.218883] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9879.501004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9880.778613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9882.051400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9883.328330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9884.604696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9885.887333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9887.161563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9888.443072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9889.747135] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9891.418261] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9892.329357] Lustre: DEBUG MARKER: rc=0; [ 9892.329357] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9892.329357] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9892.329357] echo $(hostname -s): $val; [ 9892.329357] rc=$val; [ 9892.329357] fi; [ 9892.329357] exit $rc [ 9892.937311] Lustre: DEBUG MARKER: dmesg [ 9893.416849] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 12b: Inode quota rebalancing ==================================================== 10:23:38 \(1491560618\) [ 9893.582124] Lustre: DEBUG MARKER: == sanity-quota test 12b: Inode quota rebalancing ==================================================== 10:23:38 (1491560618) [ 9893.997130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9895.326725] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9895.660104] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=u [ 9895.923868] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9896.177454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9896.433602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9897.688521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9898.946138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9900.201820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9901.456906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9902.717718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9903.980367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9905.240221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [ 9906.280632] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [ 9906.536486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9906.790213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [ 9911.364059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9912.659690] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9917.916296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9919.222781] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9920.912203] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9921.806342] Lustre: DEBUG MARKER: rc=0; [ 9921.806342] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9921.806342] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9921.806342] echo $(hostname -s): $val; [ 9921.806342] rc=$val; [ 9921.806342] fi; [ 9921.806342] exit $rc [ 9922.403921] Lustre: DEBUG MARKER: dmesg [ 9922.888163] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 10:24:07 \(1491560647\) [ 9923.066296] Lustre: DEBUG MARKER: == sanity-quota test 13: Cancel per-ID lock in the LRU list ========================================== 10:24:07 (1491560647) [ 9923.473851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9924.791327] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9925.097655] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [ 9936.199338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9936.464964] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9936.750360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9938.029869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9939.311416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9940.594702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9941.872950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9943.152793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9944.444922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9945.766401] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9946.485572] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9947.385068] Lustre: DEBUG MARKER: rc=0; [ 9947.385068] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9947.385068] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9947.385068] echo $(hostname -s): $val; [ 9947.385068] rc=$val; [ 9947.385068] fi; [ 9947.385068] exit $rc [ 9947.994615] Lustre: DEBUG MARKER: dmesg [ 9948.473182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 15: Set over 4T block quota ===================================================== 10:24:33 \(1491560673\) [ 9948.638410] Lustre: DEBUG MARKER: == sanity-quota test 15: Set over 4T block quota ===================================================== 10:24:33 (1491560673) [ 9949.063696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9950.361079] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9953.151920] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [ 9954.051423] Lustre: DEBUG MARKER: rc=0; [ 9954.051423] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [ 9954.051423] if [[ $? -eq 0 && $val -ne 0 ]]; then [ 9954.051423] echo $(hostname -s): $val; [ 9954.051423] rc=$val; [ 9954.051423] fi; [ 9954.051423] exit $rc [ 9954.670138] Lustre: DEBUG MARKER: dmesg [ 9955.144538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 17: DQACQ return recoverable error ============================================== 10:24:39 \(1491560679\) [ 9955.323317] Lustre: DEBUG MARKER: == sanity-quota test 17: DQACQ return recoverable error ============================================== 10:24:39 (1491560679) [ 9955.766312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9957.083323] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9957.396554] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9971.769056] Lustre: DEBUG MARKER: lctl set_param fail_val=37 fail_loc=0xa04 [ 9971.957946] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9972.997960] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9974.083378] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9975.423773] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9976.462303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9976.727934] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [ 9977.012865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9978.297916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9979.580600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9980.866996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9982.145628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9983.422769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9984.726924] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [ 9986.625587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [ 9987.943925] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9988.260802] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [ 9996.286580] Lustre: DEBUG MARKER: lctl set_param fail_val=11 fail_loc=0xa04 [ 9996.470892] Lustre: *** cfs_fail_loc=a04, val=0*** [ 9998.600092] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [ 9999.925196] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10000.853996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10001.120552] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10001.407670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10002.693703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10003.971713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10005.258059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10006.543783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10007.820907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10009.099927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10010.416644] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10012.318927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10013.643678] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10013.968740] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10021.933169] Lustre: DEBUG MARKER: lctl set_param fail_val=110 fail_loc=0xa04 [10022.128722] Lustre: *** cfs_fail_loc=a04, val=0*** [10022.130159] Lustre: Skipped 1 previous similar message [10024.255752] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10025.589075] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10026.535895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10026.814663] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10027.114762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10028.404449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10029.677614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10030.952184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10032.241126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10033.520336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10034.825661] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10036.730046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10038.050195] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10038.381937] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10046.388663] Lustre: DEBUG MARKER: lctl set_param fail_val=107 fail_loc=0xa04 [10046.576131] Lustre: *** cfs_fail_loc=a04, val=0*** [10046.577393] Lustre: Skipped 1 previous similar message [10048.707424] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10050.048193] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10051.041499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10051.305138] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10051.588483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10052.866939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10054.146494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10055.426211] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10056.718214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10058.007452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10059.294151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10060.619964] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10062.333022] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10063.237154] Lustre: DEBUG MARKER: rc=0; [10063.237154] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10063.237154] if [[ $? -eq 0 && $val -ne 0 ]]; then [10063.237154] echo $(hostname -s): $val; [10063.237154] rc=$val; [10063.237154] fi; [10063.237154] exit $rc [10063.844389] Lustre: DEBUG MARKER: dmesg [10064.348491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 18: MDS failover while writing, no watchdog triggered \(b14840\) ================== 10:26:29 \(1491560789\) [10064.511150] Lustre: DEBUG MARKER: == sanity-quota test 18: MDS failover while writing, no watchdog triggered (b14840) ================== 10:26:29 (1491560789) [10065.213890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10066.531090] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10066.840341] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10082.679926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [10082.836063] Lustre: DEBUG MARKER: User quota (limit: 200m) [10083.108876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(buffered\) ... [10083.268916] Lustre: DEBUG MARKER: Write 100M (buffered) ... [10083.416516] Lustre: DEBUG MARKER: sync; sync; sync [10084.783524] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [10085.053829] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [10085.204955] LustreError: 26662:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [10085.206856] LustreError: 26662:0:(osd_handler.c:2171:osd_ro()) Skipped 5 previous similar messages [10085.245821] Turning device dm-0 (0xfc00000) read-only [10085.423413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [10085.565187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10085.911789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [10086.144061] Lustre: DEBUG MARKER: Fail mds for 0 seconds [10086.304477] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10086.556889] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [10086.685983] Lustre: Failing over lustre-MDT0000 [10086.703233] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [10086.705241] LustreError: Skipped 13 previous similar messages [10086.706796] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.710237] Lustre: Skipped 11 previous similar messages [10086.712216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10086.714011] Lustre: Skipped 31 previous similar messages [10086.740587] Removing read-only on unknown block (0xfc00000) [10086.747224] Lustre: server umount lustre-MDT0000 complete [10086.876762] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10090.554039] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10090.558268] LustreError: Skipped 9 previous similar messages [10097.197605] Lustre: DEBUG MARKER: hostname [10097.500620] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10097.754463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10098.009697] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10098.231394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.249323] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.253528] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [10098.255802] Lustre: Skipped 52 previous similar messages [10098.257808] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176413f6c to 0x52ac6ca1769326aa [10098.300364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10098.466980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10098.702220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [10098.728051] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10099.242232] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10099.244504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10099.409382] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10099.409389] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10099.555742] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10099.817119] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10100.075484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10100.338373] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10101.110224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10101.112597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10101.324063] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10101.330050] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10103.296352] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [10103.311872] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [10103.565911] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10103.608323] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10103.745111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10103.853655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10105.137370] Lustre: DEBUG MARKER: lctl get_param -n at_max [10105.393690] Lustre: DEBUG MARKER: lctl get_param -n at_max [10105.718302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=7488, time=0, timeout=600\) [10105.888586] Lustre: DEBUG MARKER: (dd_pid=7488, time=0, timeout=600) [10106.440751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10106.703904] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10107.005399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10108.282709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10109.593530] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10111.533999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10112.856362] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10113.166608] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10120.148930] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 200m\) [10120.319423] Lustre: DEBUG MARKER: User quota (limit: 200m) [10120.601747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write 100M \(directio\) ... [10120.773816] Lustre: DEBUG MARKER: Write 100M (directio) ... [10120.958046] Lustre: DEBUG MARKER: sync; sync; sync [10122.256412] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [10122.521933] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [10122.670888] Turning device dm-0 (0xfc00000) read-only [10122.802183] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [10122.941436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10123.202105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail mds for 0 seconds [10123.582490] Lustre: DEBUG MARKER: Fail mds for 0 seconds [10123.806799] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10124.088550] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [10124.274593] Removing read-only on unknown block (0xfc00000) [10124.427951] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10134.423044] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491560853/real 1491560853] req@ffff88005f81cf00 x1564004880111200/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491560860 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10134.429586] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [10134.748807] Lustre: DEBUG MARKER: hostname [10135.063295] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10135.317799] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10135.570768] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10135.730210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.659267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10140.932324] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10141.432838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10141.435159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10141.592831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10141.593987] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10141.741163] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10142.011462] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10142.265961] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10142.529310] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10143.298244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10143.362626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10143.569575] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10143.615512] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10145.895543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10145.966501] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10146.165852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10146.253431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [10147.551969] Lustre: DEBUG MARKER: lctl get_param -n at_max [10147.813905] Lustre: DEBUG MARKER: lctl get_param -n at_max [10153.193153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark \(dd_pid=10422, time=5, timeout=600\) [10153.361444] Lustre: DEBUG MARKER: (dd_pid=10422, time=5, timeout=600) [10153.861898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10154.129205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10154.421214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10155.701584] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10156.983197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10158.265109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10159.546250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10160.827112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10162.138055] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10164.132861] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10165.071454] Lustre: DEBUG MARKER: rc=0; [10165.071454] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10165.071454] if [[ $? -eq 0 && $val -ne 0 ]]; then [10165.071454] echo $(hostname -s): $val; [10165.071454] rc=$val; [10165.071454] fi; [10165.071454] exit $rc [10165.697698] Lustre: DEBUG MARKER: dmesg [10166.228893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 19: Updating admin limits doesn\'t zero operational limits\(b14790\) =============== 10:28:11 \(1491560891\) [10166.386126] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) =============== 10:28:11 (1491560891) [10166.815767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10168.139139] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10168.439880] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10184.958043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10185.239434] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10185.534111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10186.818665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10188.094284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10189.376585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10190.659524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10191.960801] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10193.642177] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10194.596874] Lustre: DEBUG MARKER: rc=0; [10194.596874] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10194.596874] if [[ $? -eq 0 && $val -ne 0 ]]; then [10194.596874] echo $(hostname -s): $val; [10194.596874] rc=$val; [10194.596874] fi; [10194.596874] exit $rc [10195.216771] Lustre: DEBUG MARKER: dmesg [10195.700244] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 20: Test if setquota specifiers work properly \(b15754\) ========================== 10:28:40 \(1491560920\) [10195.862656] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================== 10:28:40 (1491560920) [10198.198135] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10199.117036] Lustre: DEBUG MARKER: rc=0; [10199.117036] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10199.117036] if [[ $? -eq 0 && $val -ne 0 ]]; then [10199.117036] echo $(hostname -s): $val; [10199.117036] rc=$val; [10199.117036] fi; [10199.117036] exit $rc [10199.722952] Lustre: DEBUG MARKER: dmesg [10200.219211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 21: Setquota while writing & deleting \(b16053\) ================================== 10:28:45 \(1491560925\) [10200.390326] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing [10200.816619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10202.129618] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10202.440879] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10209.369506] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for user: quota_usr [10209.535796] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr [10209.772457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set limit\(block:10G\; file:1000000\) for group: quota_usr [10209.945523] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr [10210.339119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 1 times [10210.689891] Lustre: DEBUG MARKER: Set quota for 1 times [10212.002407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 2 times [10212.206646] Lustre: DEBUG MARKER: Set quota for 2 times [10213.414864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 3 times [10213.690949] Lustre: DEBUG MARKER: Set quota for 3 times [10214.935942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 4 times [10215.110100] Lustre: DEBUG MARKER: Set quota for 4 times [10216.438331] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 5 times [10216.640365] Lustre: DEBUG MARKER: Set quota for 5 times [10217.874915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 6 times [10218.062088] Lustre: DEBUG MARKER: Set quota for 6 times [10219.401993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 7 times [10219.693077] Lustre: DEBUG MARKER: Set quota for 7 times [10221.010515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 8 times [10221.173624] Lustre: DEBUG MARKER: Set quota for 8 times [10222.594897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 9 times [10222.753476] Lustre: DEBUG MARKER: Set quota for 9 times [10224.047552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 10 times [10224.217737] Lustre: DEBUG MARKER: Set quota for 10 times [10225.474681] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 11 times [10226.082955] Lustre: DEBUG MARKER: Set quota for 11 times [10227.334189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 12 times [10227.537895] Lustre: DEBUG MARKER: Set quota for 12 times [10228.788214] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 13 times [10228.950695] Lustre: DEBUG MARKER: Set quota for 13 times [10230.239836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 14 times [10230.460096] Lustre: DEBUG MARKER: Set quota for 14 times [10231.675027] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 15 times [10232.042361] Lustre: DEBUG MARKER: Set quota for 15 times [10233.308026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 16 times [10233.461470] Lustre: DEBUG MARKER: Set quota for 16 times [10234.725081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 17 times [10234.901984] Lustre: DEBUG MARKER: Set quota for 17 times [10236.434731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 18 times [10236.764094] Lustre: DEBUG MARKER: Set quota for 18 times [10238.043382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Set quota for 19 times [10238.270524] Lustre: DEBUG MARKER: Set quota for 19 times [10243.849168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10244.182744] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10244.473600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10245.767493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10247.053351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10248.339312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10249.627648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10250.915074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10252.224580] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10254.921061] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10255.863531] Lustre: DEBUG MARKER: rc=0; [10255.863531] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10255.863531] if [[ $? -eq 0 && $val -ne 0 ]]; then [10255.863531] echo $(hostname -s): $val; [10255.863531] rc=$val; [10255.863531] fi; [10255.863531] exit $rc [10256.484918] Lustre: DEBUG MARKER: dmesg [10256.958643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 22: enable\/disable quota by \'lctl conf_param\' =================================== 10:29:41 \(1491560981\) [10257.123911] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param' =================================== 10:29:41 (1491560981) [10257.282048] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug [10257.546923] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10257.803711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10258.060214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10259.323108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10260.586242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10261.844113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10263.106142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10264.366902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10265.624168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10266.888017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10267.942426] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10268.201274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10268.457680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10269.494058] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10277.282621] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10277.539764] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10279.733385] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [10279.735153] Lustre: Skipped 8 previous similar messages [10280.735089] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005e478c00 x1564004880136176/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10280.738933] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [10283.895518] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10291.043727] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10291.313673] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10292.477058] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491561011/real 1491561011] req@ffff88005e479500 x1564004880136416/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491561018 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10292.482616] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10293.679085] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006274ad00 x1564004880136880/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10293.683452] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [10295.359082] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006274ad00 x1564004880136896/t0(0) o13->lustre-OST0004-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10297.673212] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10319.896980] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10319.896980] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10319.896980] if [ $running -ne $mpts ]; then [10319.896980] echo $(hostname) env are INSANE!; [10319.896980] exit 1; [10319.896980] fi [10320.450055] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [10320.450055] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [10320.450055] if [ $running -ne $mpts ]; then [10320.450055] echo $(hostname) env are INSANE!; [10320.450055] exit 1; [10320.450055] fi [10321.029743] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10321.029743] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10321.029743] if [ $running -ne $mpts ]; then [10321.029743] echo $(hostname) env are INSANE!; [10321.029743] exit 1; [10321.029743] fi [10321.953849] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [10322.213775] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10322.461856] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10322.705159] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10322.875110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.896878] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [10322.899401] Lustre: Skipped 51 previous similar messages [10323.113127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10323.366485] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10323.859845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10323.863267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10324.024882] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10324.024891] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10324.169527] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10324.422671] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10324.673876] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10324.928206] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10325.180915] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10327.368259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10327.528551] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10328.945986] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [10329.211191] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [10329.464915] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [10329.719188] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [10329.867693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.095401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10330.353646] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10330.856059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10330.869225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10331.030557] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10331.034393] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10331.177446] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10331.431518] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10331.694893] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10331.952658] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [10332.210361] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10334.408751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10334.578700] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10338.559208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10338.718175] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10341.748432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10341.907123] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10345.426493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10345.594611] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10348.771937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10348.935544] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10351.857131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10352.020513] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10355.128775] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10355.283526] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10358.425474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10358.582503] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10361.608881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10361.858732] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10364.006934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10364.160868] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10365.580495] Lustre: DEBUG MARKER: lctl get_param -n timeout [10365.896608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [10366.067738] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10366.201155] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [10371.484945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10372.027118] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=none [10372.295314] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10372.550328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10372.804174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10374.061398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10375.316196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10376.572314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10377.830869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10381.394214] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [10381.651253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10381.905814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.quota_slave.enabled [10382.947113] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=none [10393.963806] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10394.224417] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10394.482161] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [10394.482162] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [10394.482163] Lustre: Skipped 15 previous similar messages [10394.489632] Lustre: Skipped 2 previous similar messages [10394.991047] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800616db900 x1564004880147920/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10394.996444] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [10399.359100] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800615cf600 x1564004880148112/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10399.364471] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [10400.579514] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10407.736457] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10407.990197] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10409.391086] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880077c33300 x1564004880148720/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10414.367199] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10436.316391] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10436.316391] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10436.316391] if [ $running -ne $mpts ]; then [10436.316391] echo $(hostname) env are INSANE!; [10436.316391] exit 1; [10436.316391] fi [10436.873953] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [10436.873953] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [10436.873953] if [ $running -ne $mpts ]; then [10436.873953] echo $(hostname) env are INSANE!; [10436.873953] exit 1; [10436.873953] fi [10437.446675] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10437.446675] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10437.446675] if [ $running -ne $mpts ]; then [10437.446675] echo $(hostname) env are INSANE!; [10437.446675] exit 1; [10437.446675] fi [10438.367546] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [10438.624695] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10438.872333] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10439.117337] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10439.270339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.514932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10439.770102] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10440.265555] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10440.290110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10440.445667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10440.457976] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10440.595142] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10440.849964] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10441.103361] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10441.360114] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10441.611330] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10443.366066] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491561164/real 1491561164] req@ffff880060fd4000 x1564004880149024/t0(0) o38->lustre-MDT0001-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491561169 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10443.373283] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [10443.784784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10443.939776] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10445.371558] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [10445.633763] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [10445.886358] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [10446.137663] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [10446.296965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10446.511555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10446.769844] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10447.265544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.265688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.432158] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.432350] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10447.581132] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10447.837997] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10448.091193] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10448.351786] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [10448.607299] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10450.809485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10450.968692] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10454.303960] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10454.462579] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10457.294764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10457.454930] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10460.277941] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10460.438190] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10463.287178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10463.442853] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10466.305358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10466.463365] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10469.312620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10469.479654] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10472.319352] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10472.484792] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10474.413135] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [10474.416418] Lustre: Skipped 62 previous similar messages [10475.335230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10475.498725] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10477.576414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10477.735711] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10479.161183] Lustre: DEBUG MARKER: lctl get_param -n timeout [10479.482663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [10479.641460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10479.779484] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [10485.115398] Lustre: DEBUG MARKER: lctl set_param debug=+quota [10485.402400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled [10485.978746] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10486.876001] Lustre: DEBUG MARKER: rc=0; [10486.876001] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10486.876001] if [[ $? -eq 0 && $val -ne 0 ]]; then [10486.876001] echo $(hostname -s): $val; [10486.876001] rc=$val; [10486.876001] fi; [10486.876001] exit $rc [10487.495327] Lustre: DEBUG MARKER: dmesg [10487.976537] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 23: Quota should be honored with directIO \(b16125\) ============================== 10:33:32 \(1491561212\) [10488.132632] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ============================== 10:33:32 (1491561212) [10488.329366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 15019884 required: 6144 [10488.489951] Lustre: DEBUG MARKER: OST0_SIZE: 15019884 required: 6144 [10488.674532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark run for 4MB test file [10488.829351] Lustre: DEBUG MARKER: run for 4MB test file [10489.239999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10490.555939] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10490.842678] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10503.732595] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 4 MB\) [10503.880255] Lustre: DEBUG MARKER: User quota (limit: 4 MB) [10504.235490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: trigger EDQUOT with O_DIRECT [10504.388163] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [10504.585706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write half of file [10504.745244] Lustre: DEBUG MARKER: Write half of file [10505.010611] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [10505.159403] Lustre: DEBUG MARKER: Write out of block quota ... [10505.399950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: done [10505.562987] Lustre: DEBUG MARKER: Step1: done [10505.753939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: rewrite should succeed [10505.912424] Lustre: DEBUG MARKER: Step2: rewrite should succeed [10506.141223] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: done [10506.298159] Lustre: DEBUG MARKER: Step2: done [10506.733626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10507.006353] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10507.300723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10508.583641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10509.856137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10511.137524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10512.418203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10513.693215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10514.967857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10516.242955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10517.519761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10518.837778] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10520.648754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OST0_SIZE: 15019884 required: 61440 [10520.803098] Lustre: DEBUG MARKER: OST0_SIZE: 15019884 required: 61440 [10520.990779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark run for 40MB test file [10521.147222] Lustre: DEBUG MARKER: run for 40MB test file [10521.573611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10522.903900] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10523.191767] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10529.779394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(limit: 40 MB\) [10529.934369] Lustre: DEBUG MARKER: User quota (limit: 40 MB) [10530.218194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: trigger EDQUOT with O_DIRECT [10530.387554] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT [10530.604447] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write half of file [10530.765832] Lustre: DEBUG MARKER: Write half of file [10531.805146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ... [10531.990352] Lustre: DEBUG MARKER: Write out of block quota ... [10532.717390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step1: done [10532.873499] Lustre: DEBUG MARKER: Step1: done [10533.076118] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: rewrite should succeed [10533.249569] Lustre: DEBUG MARKER: Step2: rewrite should succeed [10533.501989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Step2: done [10533.652210] Lustre: DEBUG MARKER: Step2: done [10534.064694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10534.330948] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10534.615758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10535.905657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10537.189710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10538.475127] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10539.759223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10541.039826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10542.322874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10543.623177] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10545.332134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10546.206954] Lustre: DEBUG MARKER: rc=0; [10546.206954] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10546.206954] if [[ $? -eq 0 && $val -ne 0 ]]; then [10546.206954] echo $(hostname -s): $val; [10546.206954] rc=$val; [10546.206954] fi; [10546.206954] exit $rc [10546.794631] Lustre: DEBUG MARKER: dmesg [10547.280672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 24: lfs draws an asterix when limit is reached \(b16646\) ========================= 10:34:32 \(1491561272\) [10547.443957] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================= 10:34:32 (1491561272) [10547.863152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10549.181561] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10549.469553] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [10556.304875] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10557.233114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10557.497637] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10557.780096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10559.072894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10560.358117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10561.636891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10562.925554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10564.202476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10565.484311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10566.763118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10568.073018] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10569.813366] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10570.687858] Lustre: DEBUG MARKER: rc=0; [10570.687858] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10570.687858] if [[ $? -eq 0 && $val -ne 0 ]]; then [10570.687858] echo $(hostname -s): $val; [10570.687858] rc=$val; [10570.687858] fi; [10570.687858] exit $rc [10571.308124] Lustre: DEBUG MARKER: dmesg [10571.783895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27a: lfs quota\/setquota should handle wrong arguments \(b19612\) ================== 10:34:56 \(1491561296\) [10571.962293] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ================== 10:34:56 (1491561296) [10572.143324] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10573.034395] Lustre: DEBUG MARKER: rc=0; [10573.034395] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10573.034395] if [[ $? -eq 0 && $val -ne 0 ]]; then [10573.034395] echo $(hostname -s): $val; [10573.034395] rc=$val; [10573.034395] fi; [10573.034395] exit $rc [10573.620424] Lustre: DEBUG MARKER: dmesg [10574.119643] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27b: lfs quota\/setquota should handle user\/group ID \(b20200\) ==================== 10:34:58 \(1491561298\) [10574.276550] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group ID (b20200) ==================== 10:34:58 (1491561298) [10576.619887] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10577.505148] Lustre: DEBUG MARKER: rc=0; [10577.505148] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10577.505148] if [[ $? -eq 0 && $val -ne 0 ]]; then [10577.505148] echo $(hostname -s): $val; [10577.505148] rc=$val; [10577.505148] fi; [10577.505148] exit $rc [10578.093534] Lustre: DEBUG MARKER: dmesg [10578.572742] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27c: lfs quota should support human-readable output ============================= 10:35:03 \(1491561303\) [10578.732708] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ============================= 10:35:03 (1491561303) [10580.029760] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10580.919359] Lustre: DEBUG MARKER: rc=0; [10580.919359] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10580.919359] if [[ $? -eq 0 && $val -ne 0 ]]; then [10580.919359] echo $(hostname -s): $val; [10580.919359] rc=$val; [10580.919359] fi; [10580.919359] exit $rc [10581.506261] Lustre: DEBUG MARKER: dmesg [10581.995970] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 27d: lfs setquota should support fraction block limit =========================== 10:35:06 \(1491561306\) [10582.160721] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit =========================== 10:35:06 (1491561306) [10583.408898] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10584.288523] Lustre: DEBUG MARKER: rc=0; [10584.288523] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10584.288523] if [[ $? -eq 0 && $val -ne 0 ]]; then [10584.288523] echo $(hostname -s): $val; [10584.288523] rc=$val; [10584.288523] fi; [10584.288523] exit $rc [10584.889892] Lustre: DEBUG MARKER: dmesg [10585.368312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 30: Hard limit updates should not reset grace times ============================= 10:35:10 \(1491561310\) [10585.539046] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ============================= 10:35:10 (1491561310) [10585.944180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10587.258032] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10587.541144] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=u [10612.482926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10612.751133] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10613.025056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10614.375050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10615.651770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10616.925845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10618.215737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10619.509566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10620.789797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10622.069466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10623.374912] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10625.065283] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10625.949826] Lustre: DEBUG MARKER: rc=0; [10625.949826] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10625.949826] if [[ $? -eq 0 && $val -ne 0 ]]; then [10625.949826] echo $(hostname -s): $val; [10625.949826] rc=$val; [10625.949826] fi; [10625.949826] exit $rc [10626.539755] Lustre: DEBUG MARKER: dmesg [10627.044141] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 33: Basic usage tracking for user & group ======================================= 10:35:51 \(1491561351\) [10627.194339] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user [10627.603752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10628.920392] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10630.576619] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10631.545245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10631.803022] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10632.085435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10633.358671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10634.639023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10635.911243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10637.201811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10638.480728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10639.756554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10641.030167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10642.314218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10643.611557] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10644.325035] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10645.206944] Lustre: DEBUG MARKER: rc=0; [10645.206944] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10645.206944] if [[ $? -eq 0 && $val -ne 0 ]]; then [10645.206944] echo $(hostname -s): $val; [10645.206944] rc=$val; [10645.206944] fi; [10645.206944] exit $rc [10645.797561] Lustre: DEBUG MARKER: dmesg [10646.286432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 34: Usage transfer for user & group ============================================= 10:36:11 \(1491561371\) [10646.446194] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user [10646.847034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10648.158246] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10648.660681] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10649.545967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10649.803970] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10650.084968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10651.362111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10652.638406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10653.914785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10655.197291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10656.478304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10658.090317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10658.349721] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10658.624029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10659.897075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10661.174459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10662.455328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10663.742414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10665.022227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10666.313950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10667.919846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10668.177514] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10668.465082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10669.739106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10671.016281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10672.288178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10673.561207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10674.836610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10676.113295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10677.733064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10677.993938] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10678.291805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10679.568444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10680.846276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10682.121753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10683.394522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10684.670362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10685.948421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10687.254154] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10687.898897] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10688.782942] Lustre: DEBUG MARKER: rc=0; [10688.782942] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10688.782942] if [[ $? -eq 0 && $val -ne 0 ]]; then [10688.782942] echo $(hostname -s): $val; [10688.782942] rc=$val; [10688.782942] fi; [10688.782942] exit $rc [10689.394680] Lustre: DEBUG MARKER: dmesg [10689.909326] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 35: Usage is still accessible across reboot ===================================== 10:36:54 \(1491561414\) [10690.067217] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ===================================== 10:36:54 (1491561414) [10690.479965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10691.815480] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10692.275558] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10693.023097] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Restart... [10693.170702] Lustre: DEBUG MARKER: Restart... [10694.006674] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10694.260151] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10694.428907] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [10694.430847] Lustre: Skipped 26 previous similar messages [10695.487094] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800619bcc00 x1564004880178800/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10695.490803] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [10696.353194] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [10696.353194] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10696.353194] Lustre: Skipped 9 previous similar messages [10696.359909] LustreError: Skipped 12 previous similar messages [10700.490917] Lustre: server umount lustre-MDT0000 complete [10700.492591] Lustre: Skipped 5 previous similar messages [10700.624679] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10700.868591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.872497] LustreError: Skipped 133 previous similar messages [10707.775080] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10708.034580] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10708.071053] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10708.074754] LustreError: Skipped 3 previous similar messages [10714.398494] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10742.050637] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10742.050637] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10742.050637] if [ $running -ne $mpts ]; then [10742.050637] echo $(hostname) env are INSANE!; [10742.050637] exit 1; [10742.050637] fi [10742.596838] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [10742.596838] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [10742.596838] if [ $running -ne $mpts ]; then [10742.596838] echo $(hostname) env are INSANE!; [10742.596838] exit 1; [10742.596838] fi [10743.167112] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [10743.167112] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [10743.167112] if [ $running -ne $mpts ]; then [10743.167112] echo $(hostname) env are INSANE!; [10743.167112] exit 1; [10743.167112] fi [10746.442052] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [10746.701254] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [10746.947158] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [10747.190688] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [10747.344605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10747.415105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10747.417328] Lustre: Skipped 5 previous similar messages [10747.564739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10747.817860] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10748.310275] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10748.316215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10748.472670] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10748.473774] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10748.616284] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10748.869217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10749.121774] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10749.379112] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [10749.643226] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10751.839473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10751.999494] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10752.411114] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491561473/real 1491561473] req@ffff88007a0e0000 x1564004880180176/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491561478 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10752.418463] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [10753.438252] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [10753.705405] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [10753.961259] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [10754.211934] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [10754.369239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.593648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [10754.848738] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [10755.352300] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10755.355385] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10755.517621] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10755.520352] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10755.666762] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10755.922099] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10756.182222] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [10756.442143] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [10756.700433] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10758.932655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10759.099334] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10762.565554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10762.763695] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10765.559466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10765.717324] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10768.532807] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10768.693603] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10771.524457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10771.685069] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10774.487652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10774.659666] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10776.567870] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [10776.571280] Lustre: Skipped 30 previous similar messages [10777.493082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10777.664788] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10780.513082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10780.672067] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10783.540235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10783.705198] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10785.808425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10785.959202] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [10787.381849] Lustre: DEBUG MARKER: lctl get_param -n timeout [10787.708226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [10787.859215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10787.991667] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [10793.424859] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [10793.702310] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10794.222236] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [10794.800444] Lustre: DEBUG MARKER: lctl set_param debug=+quota [10795.322083] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10796.289911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10796.558687] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10796.834947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10798.118067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10799.397627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10800.679737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10801.957594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10803.237318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10804.522114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10805.850911] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10806.552405] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10807.449690] Lustre: DEBUG MARKER: rc=0; [10807.449690] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10807.449690] if [[ $? -eq 0 && $val -ne 0 ]]; then [10807.449690] echo $(hostname -s): $val; [10807.449690] rc=$val; [10807.449690] fi; [10807.449690] exit $rc [10808.052809] Lustre: DEBUG MARKER: dmesg [10808.565678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 37: Quota accounted properly for file created by \'lfs setstripe\' ================ 10:38:53 \(1491561533\) [10808.734088] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ================ 10:38:53 (1491561533) [10808.867828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10808.867828] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10808.867828] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10809.414781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10810.738506] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10811.438902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10811.698146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10811.982159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10813.262579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10814.543311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10815.818394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10817.112930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10818.702544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10818.968558] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10819.250374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10820.524556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10821.797140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10823.081734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10824.356303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10825.634992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10826.916465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10828.203705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10829.508951] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10830.165765] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10831.050239] Lustre: DEBUG MARKER: rc=0; [10831.050239] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10831.050239] if [[ $? -eq 0 && $val -ne 0 ]]; then [10831.050239] echo $(hostname -s): $val; [10831.050239] rc=$val; [10831.050239] fi; [10831.050239] exit $rc [10831.639434] Lustre: DEBUG MARKER: dmesg [10832.128701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test 38: Quota accounting iterator doesn\'t skip id entries =========================== 10:39:16 \(1491561556\) [10832.286778] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries =========================== 10:39:16 (1491561556) [10832.423947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10832.423947] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10832.423947] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10832.967644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10834.276457] Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0 [10931.136081] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10931.816222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osd-ldiskfs.lustre-MDT0000.quota_slave.acct_user [10943.560288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10943.822854] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [10944.228222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10945.547739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10946.825776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10948.106128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10949.385132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10950.667773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [10951.975650] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [10952.637169] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [10953.565643] Lustre: DEBUG MARKER: rc=0; [10953.565643] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [10953.565643] if [[ $? -eq 0 && $val -ne 0 ]]; then [10953.565643] echo $(hostname -s): $val; [10953.565643] rc=$val; [10953.565643] fi; [10953.565643] exit $rc [10954.175735] Lustre: DEBUG MARKER: dmesg [10954.650671] Lustre: DEBUG MARKER: lctl set_param debug=-quota [10955.009933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-quota test complete, duration 2514 sec ===================================================== 10:41:19 \(1491561679\) [10955.171522] Lustre: DEBUG MARKER: == sanity-quota test complete, duration 2514 sec ===================================================== 10:41:19 (1491561679) [10955.405236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-scrub ============----- Fri Apr 7 10:41:20 UTC 2017 [10955.575187] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Fri Apr 7 10:41:20 UTC 2017 [10955.881171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test complete, duration -o sec ======================================================= 10:41:20 \(1491561680\) [10956.043393] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration -o sec ======================================================= 10:41:20 (1491561680) [10956.391183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10956.391183] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10956.391183] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10956.971315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10956.971315] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10956.971315] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10957.994656] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10958.250146] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [10958.687133] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005e762100 x1564004880527792/t0(0) o13->lustre-OST0003-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [10958.691951] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [10964.760531] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10971.890218] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10972.144426] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [10978.510880] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10992.862814] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10993.105876] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10993.865118] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [10994.111855] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10996.958775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [10996.958775] /usr/sbin/lctl lustre_build_version 2>/dev/null || [10996.958775] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [10997.272048] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [10997.891484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [10997.898784] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [10997.919268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [10997.931251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [10998.013768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [10998.235546] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [10998.250196] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [10998.262906] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [10998.275463] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [10998.346015] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [10998.617752] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [10998.861984] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [10999.138740] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [10999.297091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11000.416167] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11000.660060] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11000.910178] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11001.096399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11006.912169] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11006.912169] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11006.912169] if [ $running -ne $mpts ]; then [11006.912169] echo $(hostname) env are INSANE!; [11006.912169] exit 1; [11006.912169] fi [11007.460168] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11007.460168] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11007.460168] if [ $running -ne $mpts ]; then [11007.460168] echo $(hostname) env are INSANE!; [11007.460168] exit 1; [11007.460168] fi [11008.032665] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11008.032665] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11008.032665] if [ $running -ne $mpts ]; then [11008.032665] echo $(hostname) env are INSANE!; [11008.032665] exit 1; [11008.032665] fi [11008.953545] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11009.217375] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11009.464922] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11009.709924] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11009.852846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11009.864538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.893089] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11009.895191] Lustre: Skipped 4 previous similar messages [11009.912464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [11009.927782] Lustre: lustre-MDT0000: new disk, initializing [11009.952197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11009.954472] Lustre: Skipped 3 previous similar messages [11010.116750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11010.369227] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11010.854898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11010.862597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11011.020059] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11011.021298] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11011.162862] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11011.414319] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11011.664123] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11011.915276] Lustre: DEBUG MARKER: sync; sync; sync [11017.219745] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11017.475801] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11018.715822] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [11018.718226] Lustre: Skipped 4 previous similar messages [11018.783993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11019.719280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11019.891910] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11026.587100] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11026.849267] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11027.099839] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11027.349534] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11027.494332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11027.507427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.522119] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [11027.524739] Lustre: Skipped 4 previous similar messages [11027.531568] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [11027.534108] Lustre: Skipped 1 previous similar message [11027.549739] Lustre: lustre-MDT0002: new disk, initializing [11027.573724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [11027.576550] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [11027.734086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11027.986701] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11028.477473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11028.491803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11028.653271] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11028.663409] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11028.802607] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11029.052153] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11029.304210] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11029.554726] Lustre: DEBUG MARKER: sync; sync; sync [11034.874994] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11035.129279] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11036.363122] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [11036.365804] Lustre: Skipped 4 previous similar messages [11037.353767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11037.508523] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11046.191243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11046.347972] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11048.067436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [11048.070581] Lustre: Skipped 1 previous similar message [11054.486108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11054.648315] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11062.854977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11063.013321] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11071.234242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11071.409429] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11078.798206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11078.956710] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11080.370259] Lustre: DEBUG MARKER: lctl get_param -n timeout [11080.683934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11080.837424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11080.967657] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11086.244442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11086.244442] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11086.244442] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11086.515013] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11086.515013] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11086.515013] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11086.786433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11086.786433] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11086.786433] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11087.058432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11087.058432] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11087.058432] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11087.603840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11087.603840] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11087.603840] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11087.933957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [11088.094640] Lustre: DEBUG MARKER: excepting tests: [11088.291924] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup\/restore case ===================== 10:43:33 \(1491561813\) [11088.467205] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ===================== 10:43:33 (1491561813) [11091.626360] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11094.945083] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11095.234436] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11095.487461] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11095.617048] Lustre: Failing over lustre-MDT0000 [11095.619335] Lustre: Skipped 1 previous similar message [11095.831455] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11096.979902] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11097.233421] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11103.617776] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11110.733776] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11110.996725] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11111.246036] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11111.491390] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11111.639922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.868573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11112.121192] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11112.611688] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11112.612832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11112.775362] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11112.782537] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11112.923135] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11113.173488] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11113.425694] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11113.679526] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11113.931736] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11116.142520] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11116.300875] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11117.771555] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11118.032986] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11118.284194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11118.534997] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11118.695716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11118.884872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11119.138207] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11119.626896] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11119.629770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11119.799939] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11119.802949] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11119.950848] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11120.206025] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11120.220387] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [11120.223484] Lustre: Skipped 1 previous similar message [11120.484473] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11120.804882] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11121.063159] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11123.259844] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11123.421223] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11124.869657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11124.869657] awk '/^status/ { print $2 }' [11125.124574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11125.124574] awk '/^status/ { print $2 }' [11125.899343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11125.899343] awk '/^status/ { print $2 }' [11126.156399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11126.156399] awk '/^status/ { print $2 }' [11126.933529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11127.353056] Lustre: lustre-MDT0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [11127.356123] Lustre: Skipped 1 previous similar message [11127.450482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11128.263443] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11129.138619] Lustre: DEBUG MARKER: rc=0; [11129.138619] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11129.138619] if [[ $? -eq 0 && $val -ne 0 ]]; then [11129.138619] echo $(hostname -s): $val; [11129.138619] rc=$val; [11129.138619] fi; [11129.138619] exit $rc [11129.743849] Lustre: DEBUG MARKER: dmesg [11130.233723] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ============================ 10:44:15 \(1491561855\) [11130.388914] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ============================ 10:44:15 (1491561855) [11133.775707] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11137.095423] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11137.396813] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11137.656105] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11138.023430] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11139.162016] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11139.412702] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11145.800403] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11152.902511] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11153.160876] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11153.408668] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11153.653886] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11153.812486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.028374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11154.282043] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11154.775904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11154.777231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11154.942058] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11154.948327] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11155.084951] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11155.336678] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11155.585369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11155.840077] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11156.092256] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11158.222472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11158.386619] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11159.819490] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11160.091078] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11160.351065] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11160.606200] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11160.763400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.963708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11161.219182] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11161.711931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11161.718892] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11161.878103] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11161.882181] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11162.026143] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11162.280482] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11162.535910] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11162.792946] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11163.045146] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11165.196218] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11165.363515] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11166.794970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11166.955404] Lustre: lustre-MDT0000: Denying connection for new client f8f58a5c-02f8-d150-704a-81e8d0b470d5(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [11172.101190] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x193 [11172.240655] Lustre: *** cfs_fail_loc=193, val=0*** [11172.365744] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [11172.758163] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11173.011851] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11173.339364] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11173.599146] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11173.867580] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11174.123422] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11174.376114] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11174.532928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.550728] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176a6641f to 0x52ac6ca176a669eb [11174.555529] Lustre: Skipped 1 previous similar message [11174.747681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11175.021544] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11175.529322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11175.529324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11175.695385] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11175.695387] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11175.843747] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11176.100612] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11176.356096] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11176.614364] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11176.867839] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11177.119291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11177.425344] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11178.314778] Lustre: DEBUG MARKER: rc=0; [11178.314778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11178.314778] if [[ $? -eq 0 && $val -ne 0 ]]; then [11178.314778] echo $(hostname -s): $val; [11178.314778] rc=$val; [11178.314778] fi; [11178.314778] exit $rc [11178.914379] Lustre: DEBUG MARKER: dmesg [11179.403293] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove\/recreate case ========== 10:45:04 \(1491561904\) [11179.567428] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========== 10:45:04 (1491561904) [11179.583640] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [11179.588406] LustreError: Skipped 1 previous similar message [11180.878164] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11180.878164] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11180.878164] if [ $running -ne $mpts ]; then [11180.878164] echo $(hostname) env are INSANE!; [11180.878164] exit 1; [11180.878164] fi [11181.444229] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11181.444229] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11181.444229] if [ $running -ne $mpts ]; then [11181.444229] echo $(hostname) env are INSANE!; [11181.444229] exit 1; [11181.444229] fi [11182.029661] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11182.029661] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11182.029661] if [ $running -ne $mpts ]; then [11182.029661] echo $(hostname) env are INSANE!; [11182.029661] exit 1; [11182.029661] fi [11182.952844] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11183.214051] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11183.473680] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11183.727786] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11183.985447] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11185.555953] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11185.825712] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11186.081238] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11186.338108] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11186.595169] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11193.553776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11193.712998] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11195.134787] Lustre: DEBUG MARKER: lctl get_param -n timeout [11195.437464] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11195.599401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11195.734734] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11204.155544] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11207.476431] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11207.766695] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11208.021923] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11208.364211] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11209.502858] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11209.774627] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11216.129482] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11217.259775] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11217.501039] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11217.745400] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11217.870473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11217.995792] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16* [11218.241740] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11219.788554] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11220.031447] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11220.275737] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11220.400338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11220.526098] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16* [11220.772840] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11222.325199] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11222.582293] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11222.828139] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11223.073583] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11223.227335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.454453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11223.707790] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11224.201838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11224.203777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11224.361998] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11224.363563] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11224.508783] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11224.777043] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11225.038795] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11225.301664] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11225.555364] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11227.691644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11227.849959] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11229.276215] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11229.538237] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11229.790897] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11230.047869] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11230.195364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.400410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11230.654695] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11231.141638] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11231.152292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11231.309808] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11231.312473] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11231.455703] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11231.709133] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11231.979454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11232.236191] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11232.491191] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11234.656456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11234.822877] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11236.269656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11236.269656] awk '/^status/ { print $2 }' [11236.524843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11236.524843] awk '/^status/ { print $2 }' [11237.306755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11237.306755] awk '/^status/ { print $2 }' [11237.571775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11237.571775] awk '/^status/ { print $2 }' [11238.253909] Lustre: lustre-MDT0000: Denying connection for new client aef0ba1d-b11c-254e-da3a-601600f71997(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [11238.259514] Lustre: Skipped 1 previous similar message [11243.549350] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11244.446045] Lustre: DEBUG MARKER: rc=0; [11244.446045] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11244.446045] if [[ $? -eq 0 && $val -ne 0 ]]; then [11244.446045] echo $(hostname -s): $val; [11244.446045] rc=$val; [11244.446045] fi; [11244.446045] exit $rc [11245.047605] Lustre: DEBUG MARKER: dmesg [11245.629383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 1c: Auto detect kinds of OI file\(s\) removed\/recreated cases ===================== 10:46:10 \(1491561970\) [11245.800330] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ===================== 10:46:10 (1491561970) [11249.053257] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11252.382080] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11252.732361] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11252.989396] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11253.343532] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11254.492125] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11254.742630] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11261.058822] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11262.194324] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11262.435505] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11262.678968] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11262.812940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11262.938554] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 [11263.183435] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11264.764503] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11265.010943] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11265.255441] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11265.380307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11265.505528] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 [11265.750976] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11267.298787] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11267.566050] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11267.825463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11268.086945] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11268.233468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.448255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11268.715118] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11269.204225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11269.206678] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11269.361637] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11269.377601] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11269.516180] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11269.768494] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11270.024495] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11270.280049] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11270.530174] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11272.704414] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11272.886269] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11274.288042] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11274.549337] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11274.800261] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11275.053078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11275.210441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.397548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11275.650927] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11276.140754] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11276.143759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11276.301093] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11276.302865] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11276.447916] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11276.703155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11276.978569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11277.235639] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11277.489926] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11279.666908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11279.821712] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11281.253065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11281.767037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11282.282230] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11282.800307] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11283.324088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11283.324088] awk '/^status/ { print $2 }' [11283.583905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11283.583905] awk '/^status/ { print $2 }' [11284.377715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11284.377715] awk '/^status/ { print $2 }' [11284.633895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11284.633895] awk '/^status/ { print $2 }' [11285.403708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11285.913254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11287.601371] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11287.601371] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11287.601371] if [ $running -ne $mpts ]; then [11287.601371] echo $(hostname) env are INSANE!; [11287.601371] exit 1; [11287.601371] fi [11288.172925] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11288.172925] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11288.172925] if [ $running -ne $mpts ]; then [11288.172925] echo $(hostname) env are INSANE!; [11288.172925] exit 1; [11288.172925] fi [11288.769031] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11288.769031] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11288.769031] if [ $running -ne $mpts ]; then [11288.769031] echo $(hostname) env are INSANE!; [11288.769031] exit 1; [11288.769031] fi [11289.715768] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11289.979960] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11290.233841] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11290.489859] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11290.745908] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11292.330900] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11292.593654] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11292.845034] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11293.098450] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11293.355268] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11300.266334] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11300.416583] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11301.837505] Lustre: DEBUG MARKER: lctl get_param -n timeout [11302.157554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11302.310407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11302.440853] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11310.968200] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11314.288716] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11314.578797] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11314.829125] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11315.036231] Lustre: server umount lustre-MDT0000 complete [11315.038029] Lustre: Skipped 12 previous similar messages [11315.166317] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11315.256046] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [11315.258126] LustreError: Skipped 27 previous similar messages [11315.259847] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.263628] Lustre: Skipped 11 previous similar messages [11315.265498] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.269591] LustreError: Skipped 131 previous similar messages [11316.290125] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11316.543495] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11322.924642] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11330.074667] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11330.318658] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11330.562841] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11330.698740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11330.823682] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 [11331.067618] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 [11331.313191] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 [11331.558464] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 [11331.804149] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 [11332.052294] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11334.563798] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11334.823781] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11335.084296] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11335.221858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11335.345063] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 [11335.590665] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 [11335.834924] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 [11336.080726] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 [11336.326050] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 [11336.568857] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11339.100269] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11339.353755] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11339.601188] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11339.844896] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11340.000143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11340.210359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11340.461200] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11340.953519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11340.955474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11341.112504] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11341.114258] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11341.254923] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11341.505060] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11341.755290] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11342.009907] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11342.268615] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11344.390946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11344.548577] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11345.960914] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11346.222405] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11346.472659] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11346.721380] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11346.868607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.060483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11347.332675] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11347.828610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11347.831079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11347.994413] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11347.994726] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11348.140803] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11348.394174] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11348.662608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11348.921301] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11349.177096] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11351.373765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11351.533731] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11353.003137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11353.528628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11354.050811] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11354.561322] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11355.075187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11355.075187] awk '/^status/ { print $2 }' [11355.330477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11355.330477] awk '/^status/ { print $2 }' [11356.137756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11356.137756] awk '/^status/ { print $2 }' [11356.397229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11356.397229] awk '/^status/ { print $2 }' [11357.169661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11357.680118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11359.380845] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11359.380845] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11359.380845] if [ $running -ne $mpts ]; then [11359.380845] echo $(hostname) env are INSANE!; [11359.380845] exit 1; [11359.380845] fi [11359.957538] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11359.957538] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11359.957538] if [ $running -ne $mpts ]; then [11359.957538] echo $(hostname) env are INSANE!; [11359.957538] exit 1; [11359.957538] fi [11360.567370] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11360.567370] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11360.567370] if [ $running -ne $mpts ]; then [11360.567370] echo $(hostname) env are INSANE!; [11360.567370] exit 1; [11360.567370] fi [11361.502182] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11361.777122] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11362.034420] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11362.289702] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11362.549999] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11364.105259] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11364.368063] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11364.620691] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11364.871391] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11365.128681] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11372.039943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11372.197896] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11373.617337] Lustre: DEBUG MARKER: lctl get_param -n timeout [11373.927717] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11374.085785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11374.219730] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11382.601778] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11385.918376] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11386.212550] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11386.463614] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11386.807180] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11387.938126] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11388.193911] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11393.912069] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491562112/real 1491562112] req@ffff880062b8d500 x1564004880591280/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491562119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11393.918094] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 43 previous similar messages [11394.567717] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11401.695919] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11401.943166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11402.187919] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11402.326016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11402.452509] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 [11402.699668] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 [11402.944541] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 [11403.191298] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11405.234991] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11405.477421] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11405.719333] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11405.843769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11405.968764] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 [11406.212508] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 [11406.456578] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 [11406.700687] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11408.734495] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11408.989957] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11409.248397] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11409.491951] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11409.646600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11409.663537] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [11409.665778] Lustre: Skipped 207 previous similar messages [11409.710827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11409.713029] Lustre: Skipped 14 previous similar messages [11409.857832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11410.109670] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11410.599159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11410.608144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11410.761406] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11410.762771] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11410.905476] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11411.155653] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11411.404543] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11411.658591] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11411.908847] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11414.110504] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11414.277654] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11415.693640] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11415.953772] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11416.209205] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11416.458971] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11416.616242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.819191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11417.075498] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11417.567846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11417.578918] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11417.735263] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11417.735265] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11417.884511] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11418.140552] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11418.428054] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11418.689390] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11418.941745] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11421.106213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11421.284181] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11422.730898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11423.251543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11423.770186] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 [11424.276692] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 [11424.791605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11424.791605] awk '/^status/ { print $2 }' [11425.046930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11425.046930] awk '/^status/ { print $2 }' [11425.872444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11425.872444] awk '/^status/ { print $2 }' [11426.147046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11426.147046] awk '/^status/ { print $2 }' [11426.932456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11427.439114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11428.001299] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11428.893172] Lustre: DEBUG MARKER: rc=0; [11428.893172] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11428.893172] if [[ $? -eq 0 && $val -ne 0 ]]; then [11428.893172] echo $(hostname -s): $val; [11428.893172] rc=$val; [11428.893172] fi; [11428.893172] exit $rc [11429.495710] Lustre: DEBUG MARKER: dmesg [11429.971509] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup\/restore case ===================== 10:49:14 \(1491562154\) [11430.124583] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ===================== 10:49:14 (1491562154) [11431.429302] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11431.429302] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11431.429302] if [ $running -ne $mpts ]; then [11431.429302] echo $(hostname) env are INSANE!; [11431.429302] exit 1; [11431.429302] fi [11431.997422] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11431.997422] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11431.997422] if [ $running -ne $mpts ]; then [11431.997422] echo $(hostname) env are INSANE!; [11431.997422] exit 1; [11431.997422] fi [11432.584173] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11432.584173] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11432.584173] if [ $running -ne $mpts ]; then [11432.584173] echo $(hostname) env are INSANE!; [11432.584173] exit 1; [11432.584173] fi [11433.540694] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11433.804459] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11434.058526] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11434.314572] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11434.573817] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11436.120742] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11436.384660] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11436.638215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11436.892392] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11437.150901] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11444.109522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11444.264325] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11445.679875] Lustre: DEBUG MARKER: lctl get_param -n timeout [11445.996049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11446.144841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11446.285745] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11454.785357] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11458.096083] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11458.385412] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11458.641123] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11458.986193] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11460.093417] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11460.350217] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11466.713633] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11467.862968] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11468.108739] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11468.352477] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11468.596409] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11468.722317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11468.848856] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11469.120706] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11469.398683] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11469.719574] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11469.962241] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11470.233136] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11470.427857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11470.568496] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11470.694799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11470.818603] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11471.085723] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11471.333412] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11471.577069] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11471.909633] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11472.154126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11477.033427] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11477.296181] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11477.538722] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11477.781940] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11477.917203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11478.043284] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11478.306079] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11478.569185] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11478.879170] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11479.123776] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11479.373588] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11479.559729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11479.701423] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11479.827665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11479.952683] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11480.219265] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11480.469283] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11480.714387] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11481.068767] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11481.315336] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11486.055774] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11486.316566] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11486.562693] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11486.807501] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11486.963835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.972245] LustreError: 15701:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from e178c6da-6f64-9ddb-4f55-be2219096806 to 245ac678-6794-9fcc-48ec-307448f8199b [11487.222576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11487.474020] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11487.961404] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11487.965236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11488.123174] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11488.123313] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11488.266422] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11488.520444] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11488.771519] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11489.027677] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11489.279177] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11491.496584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11491.653633] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11493.089166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11493.354657] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11493.621943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11493.870998] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11494.029901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.038999] LustreError: 17019:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0002: UUID has been changed from 6f2f325c-e808-b8d6-45df-0135e1810b27 to 307def78-0624-aae0-46e3-03509f1e6a52 [11494.252294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11494.507286] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11495.005667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11495.008108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11495.166520] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11495.172087] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11495.313120] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11495.570769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11495.843277] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11496.103358] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11496.358983] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11498.634419] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11498.791585] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11500.236856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11500.236856] awk '/^status/ { print $2 }' [11500.504268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11500.504268] awk '/^status/ { print $2 }' [11501.319999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11501.319999] awk '/^status/ { print $2 }' [11501.575932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11501.575932] awk '/^status/ { print $2 }' [11502.277941] Lustre: lustre-MDT0000: Denying connection for new client 3d4007b5-8577-881f-ac7c-699d0aa269c3(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [11502.283709] Lustre: Skipped 1 previous similar message [11507.561031] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11508.455776] Lustre: DEBUG MARKER: rc=0; [11508.455776] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11508.455776] if [[ $? -eq 0 && $val -ne 0 ]]; then [11508.455776] echo $(hostname -s): $val; [11508.455776] rc=$val; [11508.455776] fi; [11508.455776] exit $rc [11509.060663] Lustre: DEBUG MARKER: dmesg [11509.541449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found \(1\) ======================= 10:50:34 \(1491562234\) [11509.713532] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ======================= 10:50:34 (1491562234) [11512.962285] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11516.276750] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11516.573919] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11516.825394] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11517.160336] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11518.283620] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11518.538858] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11524.858881] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11526.075827] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11526.321851] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11526.568071] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11526.812734] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11526.948320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11527.071847] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11527.330825] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11527.598561] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11527.917876] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11528.162727] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11528.439086] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11528.630526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11528.775795] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11528.911258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11529.044720] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11529.321127] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11529.576222] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11529.822615] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11530.180738] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11530.425901] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11535.241507] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11535.483970] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11535.728945] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11535.973569] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11536.101404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11536.226414] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11536.483289] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11536.748672] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11537.075071] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11537.319766] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11537.570419] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11537.755959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11537.898477] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11538.023833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11538.148821] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11538.410721] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11538.659487] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11538.909723] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11539.267196] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11539.513371] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11544.315258] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11544.569236] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11544.814769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11545.058869] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11545.211701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.220184] LustreError: 22028:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 245ac678-6794-9fcc-48ec-307448f8199b to 0b9e58fb-e61c-8aa1-440d-4363468f1942 [11545.460713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11545.713457] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11546.206395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11546.207286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11546.366747] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11546.379787] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11546.518123] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11546.768605] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11547.020548] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11547.273713] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11547.525935] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11549.657902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11549.821632] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11551.250116] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11551.522046] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11551.776869] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11552.029593] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11552.221534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.230456] LustreError: 23342:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0002: UUID has been changed from 307def78-0624-aae0-46e3-03509f1e6a52 to 8afe77f7-29fc-b500-4d3f-19deeed6490c [11552.489870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11552.743031] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11553.242598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.243597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.408936] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.409882] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11553.558604] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11553.814866] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11554.074303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11554.332439] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11554.588103] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11556.757691] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11556.924633] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11558.392499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11558.904812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11559.327578] Lustre: lustre-MDT0000: Denying connection for new client d23bd574-6f9b-6279-b3c1-ae725147ae05(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [11559.333302] Lustre: Skipped 1 previous similar message [11560.855701] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11564.512774] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11564.783412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11564.783412] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11564.783412] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11565.082739] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [11565.236851] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004284:0x1:0x0]: rc = 0 [11565.277257] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280003ab0:0x2:0x0] with flags 0x52, rc = 0 [11568.482328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11568.482328] awk '/^status/ { print $2 }' [11568.741433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11568.741433] awk '/^status/ { print $2 }' [11569.520808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11569.520808] awk '/^status/ { print $2 }' [11569.779491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11569.779491] awk '/^status/ { print $2 }' [11570.558526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11571.075712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11571.605577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11572.116982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11575.676654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11576.192754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11576.751270] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11577.647028] Lustre: DEBUG MARKER: rc=0; [11577.647028] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11577.647028] if [[ $? -eq 0 && $val -ne 0 ]]; then [11577.647028] echo $(hostname -s): $val; [11577.647028] rc=$val; [11577.647028] fi; [11577.647028] exit $rc [11578.256755] Lustre: DEBUG MARKER: dmesg [11578.759901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found \(2\) ======================= 10:51:43 \(1491562303\) [11578.915668] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ======================= 10:51:43 (1491562303) [11582.275628] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11585.602268] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11585.899723] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11586.155073] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11586.493344] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11587.601147] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11587.850937] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11594.263459] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11601.370079] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11601.613272] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11601.856349] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11602.100338] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11602.235175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11602.359787] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11602.624704] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11602.907522] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11603.228922] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11603.476395] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11603.751771] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11603.942853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11604.081700] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11604.205301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11604.329493] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11604.601162] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11604.852593] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11605.101570] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11605.463554] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11605.708876] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11610.470969] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11610.714541] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11610.975831] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11611.237593] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11611.381637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11611.513895] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11611.781838] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11612.049327] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11612.384276] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11612.644127] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11612.912440] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11613.078999] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11613.219245] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11613.342851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11613.468121] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11613.735910] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11613.983368] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11614.230021] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11614.546051] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11614.790195] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11619.575856] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11619.828473] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11620.073627] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11620.317657] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11620.471332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11620.479827] LustreError: 29157:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 0b9e58fb-e61c-8aa1-440d-4363468f1942 to 77befd6d-d6d0-b0a3-419d-a8a394ea4bd4 [11620.728138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11620.979282] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11621.488783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11621.491762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11621.647574] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11621.649079] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11621.791837] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11622.044920] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11622.300932] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11622.561749] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11622.816683] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11624.995795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11625.155548] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11626.577760] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11626.840258] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11627.091159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11627.341788] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11627.488520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.694610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11627.955089] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11628.450818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11628.451045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11628.620462] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11628.623747] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11628.772079] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11629.042703] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11629.318093] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11629.589493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11629.858152] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11632.027893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11632.187211] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11633.633520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11634.145834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11634.573039] Lustre: lustre-MDT0000: Denying connection for new client f31a13ca-8373-bcae-d6ce-3b76f4187fe4(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [11634.578809] Lustre: Skipped 2 previous similar messages [11636.112695] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11639.747010] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11640.018221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11640.018221] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11640.018221] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11640.320931] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=10 [11640.600844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11640.600844] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11640.600844] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11640.892292] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_threshold_rate=10000 [11641.044865] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004a54:0x1:0x0]: rc = 0 [11641.085198] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280004281:0x2:0x0] with flags 0x52, rc = 0 [11644.302503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11644.302503] awk '/^status/ { print $2 }' [11644.560418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11644.560418] awk '/^status/ { print $2 }' [11645.343663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11645.343663] awk '/^status/ { print $2 }' [11645.596867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11645.596867] awk '/^status/ { print $2 }' [11646.391098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11646.906989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11647.419806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11647.672926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11648.448242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11648.702263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11652.505504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11652.505504] awk '/^status/ { print $2 }' [11652.758116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11652.758116] awk '/^status/ { print $2 }' [11653.529482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11653.529482] awk '/^status/ { print $2 }' [11653.782675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11653.782675] awk '/^status/ { print $2 }' [11654.553576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11655.071909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11655.588627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11655.840475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11656.612073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11656.865455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11660.989566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11660.989566] awk '/^status/ { print $2 }' [11661.247614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11661.247614] awk '/^status/ { print $2 }' [11662.027540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11662.027540] awk '/^status/ { print $2 }' [11662.284308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11662.284308] awk '/^status/ { print $2 }' [11663.055158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11663.566130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11664.080047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11664.332163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11665.099346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11665.353998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11669.145271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11669.656513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11670.223408] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11671.110835] Lustre: DEBUG MARKER: rc=0; [11671.110835] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11671.110835] if [[ $? -eq 0 && $val -ne 0 ]]; then [11671.110835] echo $(hostname -s): $val; [11671.110835] rc=$val; [11671.110835] fi; [11671.110835] exit $rc [11671.715813] Lustre: DEBUG MARKER: dmesg [11672.323482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found \(3\) ======================= 10:53:17 \(1491562397\) [11672.478236] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ======================= 10:53:17 (1491562397) [11677.402552] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11680.744034] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11681.030905] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11681.285880] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11681.633804] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11682.748899] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11683.015468] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11689.394207] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11696.562150] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11696.807057] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11697.049140] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11697.293155] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11697.417814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11697.542293] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11697.824311] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11698.107387] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11698.452604] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11698.697034] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11698.967869] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11699.142818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11699.281629] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11699.407243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11699.533510] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11699.816341] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11700.069991] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11700.315879] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11700.658183] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11700.903603] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11705.655823] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11705.897755] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11706.142255] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11706.385274] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11706.512608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11706.636920] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11706.906348] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11707.177922] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11707.490461] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11707.735082] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11707.987298] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11708.176521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11708.317732] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11708.443814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11708.569820] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11708.846758] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11709.096068] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11709.340136] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11709.680987] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11709.926926] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11714.747279] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11715.001147] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11715.249203] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11715.492825] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11715.646513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.655418] LustreError: 5706:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 77befd6d-d6d0-b0a3-419d-a8a394ea4bd4 to ab69cdf4-277c-b3f9-4a4d-cd40e0b93ac2 [11715.660075] LustreError: 5706:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 1 previous similar message [11715.909219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11716.174246] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11716.694952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11716.700568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11716.859769] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11716.862678] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11717.007667] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11717.258482] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11717.509491] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11717.766435] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11718.021880] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11720.180768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11720.339388] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11721.763751] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11722.027327] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11722.281299] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11722.533113] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11722.696655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.921193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11723.187319] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11723.685898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11723.688728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11723.850294] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11723.857239] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11724.000495] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11724.254641] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11724.282431] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [11724.285440] Lustre: Skipped 18 previous similar messages [11724.534311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11724.792687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11725.048999] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11727.243510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11727.410813] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11728.833834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11729.357083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11729.783078] Lustre: lustre-MDT0000: Denying connection for new client e8fe2e5b-9580-9ee7-7d78-d40c5a535765(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [11731.311188] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11731.317401] Lustre: Skipped 1 previous similar message [11731.332186] Lustre: lustre-MDT0002: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [11731.335410] Lustre: Skipped 18 previous similar messages [11734.957655] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11735.230651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11735.230651] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11735.230651] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11735.528840] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=2 [11735.803360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11735.803360] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11735.803360] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11736.096042] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_threshold_rate=20 [11736.250560] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200005224:0x1:0x0]: rc = 0 [11736.316729] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280004a52:0x26:0x0]: rc = 0 [11736.343532] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280004a52:0x125:0x0] with flags 0x52, rc = 0 [11736.346416] Lustre: Skipped 2 previous similar messages [11739.533567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11739.533567] awk '/^status/ { print $2 }' [11739.792435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11739.792435] awk '/^status/ { print $2 }' [11740.579379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11740.579379] awk '/^status/ { print $2 }' [11740.836621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11740.836621] awk '/^status/ { print $2 }' [11741.612047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11742.123404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11742.635674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11742.889961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11743.654743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11743.911164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11747.712865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11747.712865] awk '/^status/ { print $2 }' [11747.967171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11747.967171] awk '/^status/ { print $2 }' [11748.745160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11748.745160] awk '/^status/ { print $2 }' [11749.000263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11749.000263] awk '/^status/ { print $2 }' [11749.774544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11750.288894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11750.802162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11751.056589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11751.837668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11752.089674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11756.282225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11756.282225] awk '/^status/ { print $2 }' [11756.541036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11756.541036] awk '/^status/ { print $2 }' [11757.325139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11757.325139] awk '/^status/ { print $2 }' [11757.597962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11757.597962] awk '/^status/ { print $2 }' [11758.386805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11758.897972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11759.408206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11759.660293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11760.428343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11760.679766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11764.463946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11764.981767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11765.553429] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [11766.443307] Lustre: DEBUG MARKER: rc=0; [11766.443307] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [11766.443307] if [[ $? -eq 0 && $val -ne 0 ]]; then [11766.443307] echo $(hostname -s): $val; [11766.443307] rc=$val; [11766.443307] fi; [11766.443307] exit $rc [11767.031392] Lustre: DEBUG MARKER: dmesg [11767.514396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 5: OI scrub state machine ======================================================= 10:54:52 \(1491562492\) [11767.668535] Lustre: DEBUG MARKER: == sanity-scrub test 5: OI scrub state machine ======================================================= 10:54:52 (1491562492) [11768.489965] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11768.745541] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [11770.357953] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11770.357957] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [11770.357958] Lustre: Skipped 48 previous similar messages [11770.363657] Lustre: Skipped 2 previous similar messages [11771.439112] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079c30c00 x1564004880645536/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11771.443400] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [11775.108246] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11782.208102] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11782.458604] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [11784.455061] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11784.458698] LustreError: Skipped 2 previous similar messages [11786.463047] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880035955800 x1564004880646048/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [11786.467618] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [11788.809971] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11805.285488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11805.285488] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11805.285488] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11805.593892] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11806.255608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11806.279709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [11806.283474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11806.284822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11806.295690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11806.590366] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [11806.597130] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [11806.608087] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [11806.625614] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11806.659398] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [11806.969339] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11807.214863] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11807.488970] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11807.684195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11808.751953] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11809.003324] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11809.257898] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11809.414599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11815.014992] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11815.014992] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11815.014992] if [ $running -ne $mpts ]; then [11815.014992] echo $(hostname) env are INSANE!; [11815.014992] exit 1; [11815.014992] fi [11815.563079] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [11815.563079] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [11815.563079] if [ $running -ne $mpts ]; then [11815.563079] echo $(hostname) env are INSANE!; [11815.563079] exit 1; [11815.563079] fi [11816.135382] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [11816.135382] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [11816.135382] if [ $running -ne $mpts ]; then [11816.135382] echo $(hostname) env are INSANE!; [11816.135382] exit 1; [11816.135382] fi [11817.045937] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11817.304560] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11817.553269] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11817.797923] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11817.951889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11817.991298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11818.020131] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [11818.022235] Lustre: Skipped 4 previous similar messages [11818.039687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [11818.054923] Lustre: lustre-MDT0000: new disk, initializing [11818.084135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11818.086385] Lustre: Skipped 3 previous similar messages [11818.246118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11818.498993] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11818.987252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11818.993607] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11819.153593] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11819.153595] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11819.296515] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11819.546787] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11819.795477] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11820.048269] Lustre: DEBUG MARKER: sync; sync; sync [11825.319684] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11825.572135] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11826.794396] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [11826.796842] Lustre: Skipped 4 previous similar messages [11826.879659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11827.776689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11827.935989] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11834.640393] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11834.902125] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11835.151730] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11835.402386] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11835.556493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11835.598636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.613549] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [11835.616233] Lustre: Skipped 4 previous similar messages [11835.624056] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [11835.626597] Lustre: Skipped 1 previous similar message [11835.642321] Lustre: lustre-MDT0002: new disk, initializing [11835.667559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [11835.670788] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [11835.835364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11836.087649] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11836.577789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11836.591042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11836.749179] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11836.752705] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11836.895133] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11837.146930] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11837.395355] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11837.649088] Lustre: DEBUG MARKER: sync; sync; sync [11842.941105] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11843.196169] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11844.414679] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [11844.417350] Lustre: Skipped 4 previous similar messages [11845.385629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11845.549324] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11854.288366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11854.455123] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11856.343708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [11856.346709] Lustre: Skipped 1 previous similar message [11862.654458] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11862.809307] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11871.011832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11871.172052] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11879.397895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11879.565527] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11886.981530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11887.184886] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11888.602420] Lustre: DEBUG MARKER: lctl get_param -n timeout [11888.908468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [11889.059872] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11889.197439] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [11900.464304] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11903.815361] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [11904.161758] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11904.415870] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11904.547221] Lustre: Failing over lustre-MDT0000 [11904.549738] Lustre: Skipped 20 previous similar messages [11904.758993] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11905.890230] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11906.144949] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11912.547084] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11919.678216] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11919.923393] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11920.167738] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11920.414250] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11920.550479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11920.677597] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11920.953295] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11921.234453] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11921.570078] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11921.818045] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11922.095033] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [11922.327098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11922.465716] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [11922.592263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11922.717188] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11923.013707] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11923.279367] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11923.526220] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11923.873253] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11924.117823] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [11929.217330] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11929.460999] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [11929.705174] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11929.950684] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11930.077578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11930.202746] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [11930.479738] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [11930.758401] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11931.082730] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11931.327346] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11931.578576] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [11931.775986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11931.916389] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [11932.042134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [11932.167534] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [11932.461150] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [11932.715511] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [11932.960752] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [11933.306756] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [11933.550227] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [11938.393880] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11938.650756] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11938.900219] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11939.143741] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11939.296190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11939.305302] LustreError: 19897:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 70670b3d-1fb5-90fc-43b2-eb6c2067142c to 875d40e2-2d75-a825-4860-ee53c3b08b7f [11939.309952] LustreError: 19897:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 1 previous similar message [11939.443454] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.448308] LustreError: Skipped 148 previous similar messages [11939.655762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11939.909072] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11940.405578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11940.408080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11940.567993] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11940.569482] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11940.716375] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11940.967522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11941.219682] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11941.477887] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11941.731351] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11943.898695] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11944.055514] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11945.483943] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11945.751203] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11946.005763] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11946.258706] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11946.417905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.449146] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_connect to node 0@lo failed: rc = -114 [11946.452013] LustreError: Skipped 35 previous similar messages [11946.621138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11946.883042] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11947.403726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11947.405529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11947.569049] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11947.569057] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11947.714964] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11947.968586] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11948.237960] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11948.498895] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11948.754260] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11951.366870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11951.548726] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11952.983543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11952.983543] awk '/^status/ { print $2 }' [11953.241131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11953.241131] awk '/^status/ { print $2 }' [11954.025388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11954.025388] awk '/^status/ { print $2 }' [11954.282770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11954.282770] awk '/^status/ { print $2 }' [11955.059094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [11955.078910] LustreError: 21245:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200000404:0x1:0x0]: rc = -78 [11955.084296] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x200000404:0x1:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [11955.090332] Lustre: Skipped 3 previous similar messages [11955.601436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [11956.207478] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [11956.487279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [11956.487279] /usr/sbin/lctl lustre_build_version 2>/dev/null || [11956.487279] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [11956.782136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [11957.080136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [11957.234407] Lustre: *** cfs_fail_loc=190, val=0*** [11957.236956] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200000404:0x1:0x0]: rc = 0 [11957.270021] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280000404:0x26:0x0]: rc = 0 [11957.376921] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200000403:0x2:0x0] with flags 0x52, rc = 0 [11957.379735] Lustre: Skipped 1 previous similar message [11957.687184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11957.687184] awk '/^status/ { print $2 }' [11957.945344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11957.945344] awk '/^status/ { print $2 }' [11958.717872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11958.717872] awk '/^status/ { print $2 }' [11958.973670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11958.973670] awk '/^status/ { print $2 }' [11959.770824] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x191 [11960.274055] Lustre: *** cfs_fail_loc=191, val=0*** [11960.276322] Lustre: Skipped 8 previous similar messages [11964.053686] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11964.309425] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11964.555849] Lustre: server umount lustre-MDT0000 complete [11964.558152] Lustre: Skipped 15 previous similar messages [11964.686652] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11965.784069] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11966.035743] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11972.372510] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11973.524737] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [11973.796236] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [11974.051982] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [11974.297967] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [11974.542078] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [11974.695913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.908238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11975.162299] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11975.657258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11975.657260] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11975.820807] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11975.820808] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11975.965727] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11976.217859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11976.468936] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11976.724644] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [11976.976977] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11979.145262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11979.385298] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11980.974127] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [11981.234267] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [11981.485377] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [11981.745861] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [11981.915363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11982.109892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [11982.365728] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [11982.859558] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11982.870769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11983.032255] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11983.034314] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11983.178205] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11983.446170] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11983.699190] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [11983.956989] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [11984.213643] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [11986.373974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11986.541256] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [11987.984116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11987.984116] awk '/^status/ { print $2 }' [11988.237952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [11988.237952] awk '/^status/ { print $2 }' [11989.015718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11989.015718] awk '/^status/ { print $2 }' [11989.273148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [11989.273148] awk '/^status/ { print $2 }' [11990.057690] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [11990.311778] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [11990.441186] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [11990.449833] LustreError: 26882:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [11990.452688] Lustre: 24200:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [11990.455578] Lustre: 24200:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [11990.459411] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [11990.462225] Lustre: Skipped 11 previous similar messages [11990.467489] LustreError: 24185:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005ef6aa00 x1564004880679248/t0(0) o5->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [11990.467591] LustreError: 24187:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0002-osc-MDT0000: cannot cleanup orphans: rc = -5 [11990.478411] LustreError: 24185:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [11991.674197] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [11993.860536] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [11994.125549] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [11994.263719] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [11994.263721] LustreError: 25480:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200000404:0x1:0x0]: rc = -5 [11994.271756] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) Skipped 1 previous similar message [11994.278973] LustreError: 25463:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0002: cannot cleanup orphans: rc = -5 [11994.284134] LustreError: 25463:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [12001.290043] Lustre: 27107:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491562720/real 1491562720] req@ffff88005af38c00 x1564004880679648/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491562726 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12001.298378] Lustre: 27107:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [12001.498546] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12002.659755] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12002.937111] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12003.191395] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12003.436951] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12003.680543] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12003.834974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.850964] Lustre: *** cfs_fail_loc=190, val=0*** [12003.853339] Lustre: Skipped 1 previous similar message [12004.044207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12004.296536] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12004.793777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12004.794686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12004.956358] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12004.956508] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12005.104309] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12005.356772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12005.608159] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12005.865168] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12006.118483] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12006.855093] Lustre: *** cfs_fail_loc=190, val=0*** [12008.369937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12008.542644] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12009.961119] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12010.220422] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12010.471648] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12010.723560] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12010.940166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12010.955102] Lustre: *** cfs_fail_loc=190, val=0*** [12010.957516] Lustre: Skipped 1 previous similar message [12010.989343] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [12010.992107] Lustre: Skipped 222 previous similar messages [12011.010652] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [12011.013195] Lustre: Skipped 16 previous similar messages [12011.156703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12011.413143] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12011.926888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12011.940203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12012.096328] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12012.097127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12012.242943] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12012.502945] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12012.755059] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12013.015404] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12013.271036] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12015.444908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12015.608965] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12017.078074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12017.078074] awk '/^status/ { print $2 }' [12017.423374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12017.423374] awk '/^status/ { print $2 }' [12018.199716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12018.199716] awk '/^status/ { print $2 }' [12018.455773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12018.455773] awk '/^status/ { print $2 }' [12019.263900] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x192 [12019.519370] Lustre: *** cfs_fail_loc=192, val=0*** [12019.521901] Lustre: Skipped 5 previous similar messages [12019.524674] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280000401:0x2:0x0] with flags 0x52, rc = 0 [12019.545395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12019.545395] awk '/^status/ { print $2 }' [12019.815966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12019.815966] awk '/^status/ { print $2 }' [12021.072471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12021.072471] awk '/^status/ { print $2 }' [12022.331587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12022.331587] awk '/^status/ { print $2 }' [12023.108326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12023.108326] awk '/^status/ { print $2 }' [12023.361620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12023.361620] awk '/^status/ { print $2 }' [12024.199373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12024.199373] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12024.199373] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12024.493125] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12024.795215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12024.970043] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200000404:0x1b8:0x0]: rc = 0 [12024.975167] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280000404:0x3d5:0x0]: rc = 0 [12028.078122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12028.078122] awk '/^status/ { print $2 }' [12028.331909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12028.331909] awk '/^status/ { print $2 }' [12029.108140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12029.108140] awk '/^status/ { print $2 }' [12029.364476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12029.364476] awk '/^status/ { print $2 }' [12030.162111] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12030.439071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12030.439071] awk '/^status/ { print $2 }' [12030.694530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12030.694530] awk '/^status/ { print $2 }' [12031.950332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12031.950332] awk '/^status/ { print $2 }' [12032.729531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12032.729531] awk '/^status/ { print $2 }' [12032.982924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12032.982924] awk '/^status/ { print $2 }' [12033.758119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12034.274054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12034.838860] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12035.761171] Lustre: DEBUG MARKER: rc=0; [12035.761171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12035.761171] if [[ $? -eq 0 && $val -ne 0 ]]; then [12035.761171] echo $(hostname -s): $val; [12035.761171] rc=$val; [12035.761171] fi; [12035.761171] exit $rc [12036.347890] Lustre: DEBUG MARKER: dmesg [12036.857138] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 6: OI scrub resumes from last checkpoint ======================================== 10:59:21 \(1491562761\) [12037.016057] Lustre: DEBUG MARKER: == sanity-scrub test 6: OI scrub resumes from last checkpoint ======================================== 10:59:21 (1491562761) [12048.809100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12052.160803] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12052.448362] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12052.703546] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12053.059484] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12054.201579] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12054.452040] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12060.828269] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12068.031865] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12068.278031] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12068.522176] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12068.768078] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12068.903934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12069.028506] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12069.311449] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12069.591706] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12069.899289] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12070.145897] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12070.417455] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12070.604151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12070.742316] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12070.868681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12070.993982] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12071.290114] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12071.544724] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12071.790524] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12072.139071] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12072.382752] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12077.210049] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12077.454989] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12077.696791] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12077.939097] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12078.066167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12078.190701] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12078.467564] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12078.741484] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12079.067368] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12079.310666] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12079.564117] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12079.723393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12079.862706] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12079.986840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12080.112893] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12080.406220] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12080.660542] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12080.906882] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12081.198145] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12081.442552] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12086.273924] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12086.529342] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12086.775270] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12087.019132] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12087.174031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.182849] LustreError: 3144:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 875d40e2-2d75-a825-4860-ee53c3b08b7f to a9667392-23b0-9d88-48f8-9e5e7b44886e [12087.187524] LustreError: 3144:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 1 previous similar message [12087.441808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12087.695556] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12088.187339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12088.192030] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12088.352292] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12088.353699] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12088.495243] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12088.746633] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12088.999588] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12089.257033] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12089.508521] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12091.666866] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12091.830264] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12093.256058] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12093.518866] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12093.772704] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12094.024077] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12094.170708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.366112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12094.623187] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12095.117114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12095.128308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12095.286911] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12095.286921] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12095.435220] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12095.690054] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12095.960685] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12096.221745] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12096.475844] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12098.646721] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12098.802544] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12100.244468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12100.760855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12101.183064] Lustre: lustre-MDT0002: Denying connection for new client f92bdfd3-2f04-d1c0-817c-42b51404d500(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [12101.188731] Lustre: Skipped 1 previous similar message [12102.725986] LustreError: 4495:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200001b74:0x1:0x0]: rc = -78 [12102.731310] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x200001b74:0x1:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12106.354984] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12106.626024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12106.626024] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12106.626024] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12106.926460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12107.233037] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=2 fail_loc=0x190 [12107.389319] Lustre: *** cfs_fail_loc=190, val=0*** [12107.389325] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200001b74:0x1:0x0]: rc = 0 [12107.394885] Lustre: Skipped 8 previous similar messages [12107.439822] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280001b72:0x26:0x0]: rc = 0 [12107.494660] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200001b71:0x3:0x0] with flags 0x52, rc = 0 [12112.666919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x192 [12112.940859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12112.940859] awk '/^status/ { print $2 }' [12113.196066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12113.196066] awk '/^status/ { print $2 }' [12114.451504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12114.451504] awk '/^status/ { print $2 }' [12115.226399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12115.226399] awk '/^status/ { print $2 }' [12115.483183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12115.483183] awk '/^status/ { print $2 }' [12116.288177] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12116.449714] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280001b72:0x1b8:0x0]: rc = 0 [12116.464642] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200001b74:0x3d6:0x0]: rc = 0 [12116.732193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12116.732193] awk '/^status/ { print $2 }' [12116.987804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12116.987804] awk '/^status/ { print $2 }' [12117.764852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12117.764852] awk '/^status/ { print $2 }' [12118.023674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12118.023674] awk '/^status/ { print $2 }' [12118.824132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x191 [12123.098875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12123.615159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12124.127586] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12124.384173] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12124.698123] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12125.796497] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12126.047410] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12132.374366] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12139.508845] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12139.783772] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12140.043491] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12140.292549] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12140.536656] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12140.691895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.709203] Lustre: *** cfs_fail_loc=190, val=0*** [12140.711397] Lustre: Skipped 20 previous similar messages [12140.912257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12141.176244] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12141.673120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12141.679661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12141.835143] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12141.835930] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12141.983480] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12142.234929] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12142.486305] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12142.740067] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12142.990817] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12145.170615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12145.329065] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12146.785030] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12147.044901] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12147.296913] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12147.547792] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12147.694173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.886283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12148.139960] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12148.632756] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12148.641181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12148.800783] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12148.801515] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12148.947352] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12149.201799] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12149.463427] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12149.721106] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12149.975059] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12152.139028] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12152.302430] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12153.755667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12153.755667] awk '/^status/ { print $2 }' [12154.012429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12154.012429] awk '/^status/ { print $2 }' [12154.793923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12154.793923] awk '/^status/ { print $2 }' [12155.046738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12155.046738] awk '/^status/ { print $2 }' [12155.825773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12156.353700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12156.912249] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12157.187160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12157.187160] awk '/^status/ { print $2 }' [12157.440225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12157.440225] awk '/^status/ { print $2 }' [12158.703853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12158.703853] awk '/^status/ { print $2 }' [12159.510646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12159.510646] awk '/^status/ { print $2 }' [12159.769959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12159.769959] awk '/^status/ { print $2 }' [12160.543869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12161.052992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12161.636986] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12162.513204] Lustre: DEBUG MARKER: rc=0; [12162.513204] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12162.513204] if [[ $? -eq 0 && $val -ne 0 ]]; then [12162.513204] echo $(hostname -s): $val; [12162.513204] rc=$val; [12162.513204] fi; [12162.513204] exit $rc [12163.112166] Lustre: DEBUG MARKER: dmesg [12163.641189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 7: System is available during OI scrub scanning ================================= 11:01:28 \(1491562888\) [12163.818423] Lustre: DEBUG MARKER: == sanity-scrub test 7: System is available during OI scrub scanning ================================= 11:01:28 (1491562888) [12165.139959] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12165.139959] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12165.139959] if [ $running -ne $mpts ]; then [12165.139959] echo $(hostname) env are INSANE!; [12165.139959] exit 1; [12165.139959] fi [12165.705833] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12165.705833] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12165.705833] if [ $running -ne $mpts ]; then [12165.705833] echo $(hostname) env are INSANE!; [12165.705833] exit 1; [12165.705833] fi [12166.300317] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12166.300317] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12166.300317] if [ $running -ne $mpts ]; then [12166.300317] echo $(hostname) env are INSANE!; [12166.300317] exit 1; [12166.300317] fi [12167.223668] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12167.487468] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12167.741852] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12167.996399] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12168.255096] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12169.811482] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12170.076485] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12170.332851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12170.583560] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12170.846219] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12177.813848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12178.004065] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12179.416468] Lustre: DEBUG MARKER: lctl get_param -n timeout [12179.720632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12179.880560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12180.025503] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12195.025100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12198.340469] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12198.635938] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12198.892249] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12199.224313] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12200.357490] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12200.610607] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12206.952171] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12214.087371] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12214.333940] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12214.577852] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12214.821151] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12214.957544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12215.082768] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12215.350412] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12215.622699] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12215.944953] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12216.191129] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12216.464423] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12216.656094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12216.795251] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12216.923316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12217.046890] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12217.328946] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12217.579591] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12217.824055] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12218.128332] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12218.375217] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12223.174801] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12223.422116] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12223.664169] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12223.906933] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12224.031822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12224.154268] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12224.422630] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12224.693657] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12225.020166] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12225.264747] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12225.519086] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12225.717832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12225.857517] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12225.982367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12226.107112] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12226.388606] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12226.639988] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12226.885124] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12227.236917] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12227.482731] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12232.291526] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12232.546403] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12232.793150] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12233.037533] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12233.190266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12233.199090] LustreError: 16117:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from a9667392-23b0-9d88-48f8-9e5e7b44886e to d230c003-1761-a0e7-4d89-7c1ddf70fbff [12233.203840] LustreError: 16117:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 1 previous similar message [12233.448244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12233.701452] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12234.193237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12234.204609] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12234.358494] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12234.361250] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12234.502668] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12234.753520] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12235.003312] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12235.257048] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12235.510287] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12237.667829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12237.823506] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12239.261101] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12239.537718] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12239.804551] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12240.061673] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12240.224140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.461641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12240.720340] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12241.212617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12241.218831] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12241.379127] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12241.379308] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12241.524747] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12241.778108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12242.060224] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12242.335233] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12242.607376] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12244.838130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12244.994519] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12246.440275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12246.958977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12247.383045] Lustre: lustre-MDT0002: Denying connection for new client e9e5db76-9b80-1f67-a1f6-44e51438fa5a(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [12247.388774] Lustre: Skipped 2 previous similar messages [12248.919455] LustreError: 17466:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200002b14:0x1:0x0]: rc = -78 [12248.924659] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x200002b14:0x1:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12252.552637] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12252.823655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12252.823655] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12252.823655] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12253.116659] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12253.417334] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=3 fail_loc=0x190 [12253.573405] Lustre: *** cfs_fail_loc=190, val=0*** [12253.575888] Lustre: Skipped 11 previous similar messages [12253.576027] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200002b14:0x1:0x0]: rc = 0 [12253.617775] Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency checking FID [0x280002b12:0x26:0x0]: rc = 0 [12253.673202] Lustre: lustre-MDT0000: trigger OI scrub by RPC for the [0x200002b13:0x3:0x0] with flags 0x52, rc = 0 [12253.832576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12253.832576] awk '/^status/ { print $2 }' [12254.094994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12254.094994] awk '/^status/ { print $2 }' [12254.867298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12254.867298] awk '/^status/ { print $2 }' [12255.123371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12255.123371] awk '/^status/ { print $2 }' [12255.902458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12256.426568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12256.984442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12257.256971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12257.256971] awk '/^status/ { print $2 }' [12257.509429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12257.509429] awk '/^status/ { print $2 }' [12258.766429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12258.766429] awk '/^status/ { print $2 }' [12260.031051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12260.031051] awk '/^status/ { print $2 }' [12260.805100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12260.805100] awk '/^status/ { print $2 }' [12261.061085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12261.061085] awk '/^status/ { print $2 }' [12261.840964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12262.352433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12262.951284] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12263.855042] Lustre: DEBUG MARKER: rc=0; [12263.855042] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12263.855042] if [[ $? -eq 0 && $val -ne 0 ]]; then [12263.855042] echo $(hostname -s): $val; [12263.855042] rc=$val; [12263.855042] fi; [12263.855042] exit $rc [12264.455437] Lustre: DEBUG MARKER: dmesg [12264.991989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 8: Control OI scrub manually ==================================================== 11:03:09 \(1491562989\) [12265.150929] Lustre: DEBUG MARKER: == sanity-scrub test 8: Control OI scrub manually ==================================================== 11:03:09 (1491562989) [12271.017697] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12274.334488] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12274.633174] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12274.884272] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12275.221434] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12275.304369] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12275.311983] Lustre: Skipped 10 previous similar messages [12276.363963] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12276.616629] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12282.957694] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12290.093840] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12290.341853] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12290.586125] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12290.828300] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12290.965297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12291.090345] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12291.351257] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12291.620657] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12291.942634] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12292.187412] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12292.456619] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12292.639763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12292.778373] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12292.902969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12293.028110] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12293.301068] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12293.549038] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12293.793418] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12294.154658] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12294.397877] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12299.174805] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12299.418983] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12299.663984] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12299.907280] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12300.032929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12300.157579] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12300.419543] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12300.692246] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12301.009449] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12301.252671] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12301.504858] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12301.712469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12301.850704] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12301.974347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12302.098479] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12302.364791] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12302.612412] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12302.856969] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12303.228061] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12303.474172] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12308.243332] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12308.498245] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12308.743787] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12308.987638] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12309.142415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.401993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12309.653508] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12310.145310] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12310.145530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12310.306670] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12310.310811] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12310.449410] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12310.700954] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12310.954531] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12311.209417] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12311.458844] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12313.618494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12313.814702] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12315.293312] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12315.558821] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12315.814947] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12316.066171] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12316.215076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12316.421449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12316.677602] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12317.176796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12317.181383] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12317.346819] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12317.351090] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12317.490465] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12317.748636] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12318.011292] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12318.268119] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12318.522110] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12320.682738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12320.844746] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12322.298397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12322.816924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12323.353543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190 [12323.626872] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 [12324.776248] Lustre: lustre-MDT0002: hit invalid OI mapping for [0x280002b12:0x125:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12326.178660] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 [12328.695747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12328.695747] awk '/^status/ { print $2 }' [12328.951285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12328.951285] awk '/^status/ { print $2 }' [12329.738459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12329.738459] awk '/^status/ { print $2 }' [12329.997414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12329.997414] awk '/^status/ { print $2 }' [12330.772830] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0000 [12334.171601] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_stop -M lustre-MDT0002 [12337.692590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12337.692590] awk '/^status/ { print $2 }' [12337.946765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12337.946765] awk '/^status/ { print $2 }' [12338.723400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12338.723400] awk '/^status/ { print $2 }' [12338.977971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12338.977971] awk '/^status/ { print $2 }' [12339.746880] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 9 [12340.260412] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 9 [12340.770214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12340.770214] awk '/^status/ { print $2 }' [12341.026277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12341.026277] awk '/^status/ { print $2 }' [12341.797331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12341.797331] awk '/^status/ { print $2 }' [12342.052114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12342.052114] awk '/^status/ { print $2 }' [12342.852419] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12343.129669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12343.129669] awk '/^status/ { print $2 }' [12343.382760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12343.382760] awk '/^status/ { print $2 }' [12344.638155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12344.638155] awk '/^status/ { print $2 }' [12345.410849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12345.410849] awk '/^status/ { print $2 }' [12345.663946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12345.663946] awk '/^status/ { print $2 }' [12346.437165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12346.962878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12347.529350] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12348.451302] Lustre: DEBUG MARKER: rc=0; [12348.451302] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12348.451302] if [[ $? -eq 0 && $val -ne 0 ]]; then [12348.451302] echo $(hostname -s): $val; [12348.451302] rc=$val; [12348.451302] fi; [12348.451302] exit $rc [12349.054360] Lustre: DEBUG MARKER: dmesg [12349.559909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 9: OI scrub speed control ======================================================= 11:04:34 \(1491563074\) [12349.717557] Lustre: DEBUG MARKER: == sanity-scrub test 9: OI scrub speed control ======================================================= 11:04:34 (1491563074) [12351.031755] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12351.031755] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12351.031755] if [ $running -ne $mpts ]; then [12351.031755] echo $(hostname) env are INSANE!; [12351.031755] exit 1; [12351.031755] fi [12351.609903] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12351.609903] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12351.609903] if [ $running -ne $mpts ]; then [12351.609903] echo $(hostname) env are INSANE!; [12351.609903] exit 1; [12351.609903] fi [12352.198067] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12352.198067] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12352.198067] if [ $running -ne $mpts ]; then [12352.198067] echo $(hostname) env are INSANE!; [12352.198067] exit 1; [12352.198067] fi [12353.123195] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12353.385564] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12353.640647] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12353.896917] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12354.165857] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12355.717144] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12355.982395] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12356.251974] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12356.517282] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12356.774972] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12363.739864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12363.903819] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12365.319311] Lustre: DEBUG MARKER: lctl get_param -n timeout [12365.629173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12365.782619] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12365.922948] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12396.259482] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12399.572210] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12399.863401] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12400.118327] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12400.485730] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12401.631839] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12401.883513] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12408.275925] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12415.415446] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12415.658819] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12415.904156] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12416.146658] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12416.283647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12416.408515] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12416.769664] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12417.091416] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12417.463177] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12417.709444] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12417.985607] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12418.177783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12418.317716] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12418.444055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12418.568920] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12419.023062] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12419.309789] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12419.638165] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12419.966369] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12420.211407] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12425.628851] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12425.873973] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12426.118748] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12426.361753] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12426.489137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12426.615042] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12426.972038] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12427.283869] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12427.641458] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12427.885899] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12428.140726] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12428.386076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12428.524931] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12428.649849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12428.775932] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12429.228845] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12429.515716] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12429.763537] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12430.117419] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12430.361697] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12435.581628] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12435.837836] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12436.083055] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12436.327781] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12436.483045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12436.491488] LustreError: 737:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from 65c69202-c09b-a81d-4c22-de79286c2f11 to a3a398f7-66ea-a2ee-4421-e687c73765af [12436.496164] LustreError: 737:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 3 previous similar messages [12436.756521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12437.020547] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12437.512625] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12437.516712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12437.741413] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12437.748976] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12437.886290] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12438.141782] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12438.396812] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12438.653730] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12438.909122] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12441.074355] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12441.226516] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12442.649939] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12442.912384] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12443.166160] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12443.418638] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12443.576449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12443.790025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12444.045068] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12444.538879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12444.539626] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12444.707105] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12444.709806] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12444.855431] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12445.110772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12445.176374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12445.179385] Lustre: Skipped 15 previous similar messages [12445.366037] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12445.627771] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12445.883530] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12448.065379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12448.221102] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12449.671072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12450.186855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12450.698417] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 -s 100 [12451.213878] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 -s 100 [12452.166909] Lustre: lustre-MDT0002: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [12452.170028] Lustre: Skipped 16 previous similar messages [12461.734366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12461.734366] awk '/^status/ { print $2 }' [12461.989861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12461.989861] awk '/^status/ { print $2 }' [12462.769833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12462.769833] awk '/^status/ { print $2 }' [12463.025959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12463.025959] awk '/^status/ { print $2 }' [12463.800837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12464.316862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12464.828824] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 8 -s 100 -r [12465.340287] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 8 -s 100 -r [12475.860813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12475.860813] awk '/^status/ { print $2 }' [12476.118447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12476.118447] awk '/^status/ { print $2 }' [12476.895675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12476.895675] awk '/^status/ { print $2 }' [12477.153343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12477.153343] awk '/^status/ { print $2 }' [12477.932872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12478.453237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12478.964793] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0000.lfsck_speed_limit 300 [12479.476445] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0002.lfsck_speed_limit 300 [12489.989688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12490.246774] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0000.lfsck_speed_limit 0 [12491.014081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12491.265617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdd.lustre-MDT0002.lfsck_speed_limit 0 [12492.032695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12492.032695] awk '/^status/ { print $2 }' [12492.285490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12492.285490] awk '/^status/ { print $2 }' [12493.051511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12493.051511] awk '/^status/ { print $2 }' [12493.307655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12493.307655] awk '/^status/ { print $2 }' [12494.127832] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12495.042074] Lustre: DEBUG MARKER: rc=0; [12495.042074] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12495.042074] if [[ $? -eq 0 && $val -ne 0 ]]; then [12495.042074] echo $(hostname -s): $val; [12495.042074] rc=$val; [12495.042074] fi; [12495.042074] exit $rc [12495.645879] Lustre: DEBUG MARKER: dmesg [12496.161486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount \(1\) ============ 11:07:00 \(1491563220\) [12496.321273] Lustre: DEBUG MARKER: == sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1) ============ 11:07:00 (1491563220) [12497.675458] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12497.675458] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12497.675458] if [ $running -ne $mpts ]; then [12497.675458] echo $(hostname) env are INSANE!; [12497.675458] exit 1; [12497.675458] fi [12498.252554] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12498.252554] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12498.252554] if [ $running -ne $mpts ]; then [12498.252554] echo $(hostname) env are INSANE!; [12498.252554] exit 1; [12498.252554] fi [12498.834467] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12498.834467] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12498.834467] if [ $running -ne $mpts ]; then [12498.834467] echo $(hostname) env are INSANE!; [12498.834467] exit 1; [12498.834467] fi [12499.772347] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12500.037798] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12500.292735] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12500.545701] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12500.803447] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12502.352620] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12502.616750] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12502.870385] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12503.122781] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12503.379836] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12510.311442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12510.471267] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12511.891527] Lustre: DEBUG MARKER: lctl get_param -n timeout [12512.197302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12512.354848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12512.500871] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12547.160979] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12550.477269] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12550.766642] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12551.019973] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12551.148426] Lustre: Failing over lustre-MDT0000 [12551.150200] Lustre: Skipped 15 previous similar messages [12551.389762] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12552.152021] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12552.156087] LustreError: Skipped 138 previous similar messages [12552.558760] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12552.809503] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12552.936273] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 [12552.938541] LustreError: Skipped 88835 previous similar messages [12559.269933] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12560.402766] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12560.645681] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12560.887440] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12561.131677] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12561.266732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12561.391713] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12561.662561] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12561.945990] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12562.257513] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12562.502504] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12562.779376] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12562.965774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12563.105947] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12563.232283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12563.359364] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12563.629956] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12563.880279] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12564.124999] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12564.459219] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12564.701898] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12569.445768] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12569.689641] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12569.931475] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12570.177335] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12570.313889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12570.439423] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12570.700541] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12570.971848] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12571.304711] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12571.550435] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12571.802851] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12571.963469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12572.106958] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12572.231656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12572.355118] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12572.622314] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12572.870495] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12573.114315] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12573.422437] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12573.666888] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12578.533941] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12578.790374] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12579.035143] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12579.281162] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12579.436528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.715156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12579.969298] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12580.460616] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12580.460618] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12580.622170] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12580.623680] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12580.765130] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12581.015494] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12581.265975] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12581.521116] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12581.770922] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12583.924471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12584.085765] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12585.562152] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12585.828719] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12586.084471] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12586.335689] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12586.482649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12586.690748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12586.945970] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12587.441181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12587.455092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12587.614405] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12587.615927] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12587.760746] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12588.014943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12588.277094] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12588.533542] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12588.788939] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12590.941858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12591.108637] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12592.554636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12593.069586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12593.487377] Lustre: lustre-MDT0000: Denying connection for new client e631895a-0c4b-7558-baa8-3be41b6e3460(at 10.9.5.237@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [12595.047067] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x200004284:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12595.047403] LustreError: 11511:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200003ab4:0x1:0x0]: rc = -78 [12595.058564] Lustre: Skipped 2 previous similar messages [12598.679693] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.auto_scrub=1 [12598.957790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12598.957790] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12598.957790] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12599.253561] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-ldiskfs.*.full_scrub_ratio=0 [12599.551873] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190 [12599.703968] Lustre: *** cfs_fail_loc=190, val=0*** [12599.704035] Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency checking FID [0x200004284:0x1:0x0]: rc = 0 [12599.709488] Lustre: Skipped 42 previous similar messages [12599.746111] Lustre: lustre-MDT0002: trigger OI scrub by RPC for the [0x280004280:0x3:0x0] with flags 0x52, rc = 0 [12599.953894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12599.953894] awk '/^status/ { print $2 }' [12600.212149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12600.212149] awk '/^status/ { print $2 }' [12601.000807] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12601.000807] awk '/^status/ { print $2 }' [12601.258115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12601.258115] awk '/^status/ { print $2 }' [12602.254941] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12602.506170] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12602.716317] Lustre: server umount lustre-MDT0000 complete [12602.718749] Lustre: Skipped 15 previous similar messages [12602.846445] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12603.971672] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12604.225581] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12610.404034] Lustre: 13583:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491563330/real 1491563330] req@ffff8800690b8000 x1564004880866528/t0(0) o251->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491563336 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12610.411556] Lustre: 13583:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 21 previous similar messages [12610.613892] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12617.771418] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12618.024970] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12618.274378] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12618.518602] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12618.663899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.684196] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [12618.686761] Lustre: Skipped 199 previous similar messages [12618.730505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12618.733046] Lustre: Skipped 12 previous similar messages [12618.877948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12619.130457] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12619.617672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12619.620095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12619.777380] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12619.780123] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12619.920577] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12620.170906] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12620.422788] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12620.678514] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12620.929376] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12623.070544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12623.229786] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12624.646110] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12624.910576] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12625.161934] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12625.411967] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12625.570882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.771181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12626.027957] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12626.522277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12626.530076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12626.687701] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12626.688261] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12626.834482] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12627.089487] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12627.347669] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12627.605692] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12627.861255] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12630.038967] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12630.199369] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12631.637834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12631.637834] awk '/^status/ { print $2 }' [12631.895758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12631.895758] awk '/^status/ { print $2 }' [12632.668106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12632.668106] awk '/^status/ { print $2 }' [12632.920852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12632.920852] awk '/^status/ { print $2 }' [12633.699901] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12633.955100] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12634.085553] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [12634.088352] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) Skipped 1 previous similar message [12634.097214] LustreError: 16703:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [12634.100072] Lustre: 14023:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [12634.102921] Lustre: 14023:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [12634.113698] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [12634.116539] Lustre: Skipped 2 previous similar messages [12634.119571] LustreError: 14006:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880067234600 x1564004880869184/t0(0) o5->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [12634.120319] LustreError: 14012:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0003-osc-MDT0000: cannot cleanup orphans: rc = -5 [12634.130327] LustreError: 14006:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [12635.333427] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12637.535906] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12637.805160] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12642.615037] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12642.620496] LustreError: 15303:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x200003ab4:0x1:0x0]: rc = -5 [12642.625664] LustreError: 15303:0:(osp_object.c:527:osp_attr_get()) Skipped 1 previous similar message [12642.628473] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) can't connect to a closed import [12642.631250] LustreError: 5313:0:(import.c:662:ptlrpc_connect_import()) Skipped 1 previous similar message [12642.632745] LustreError: 15303:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880066674600 x1564004880869488/t0(0) o1000->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 [12642.632746] LustreError: 15303:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [12642.650833] LustreError: 15286:0:(osp_precreate.c:619:osp_precreate_send()) lustre-OST0000-osc-MDT0002: can't precreate: rc = -5 [12642.655820] LustreError: 15286:0:(osp_precreate.c:1273:osp_precreate_thread()) lustre-OST0000-osc-MDT0002: cannot precreate objects: rc = -5 [12642.660951] LustreError: 15286:0:(osp_precreate.c:1273:osp_precreate_thread()) Skipped 1 previous similar message [12642.891265] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12650.057416] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12650.313574] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12650.560245] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12650.804951] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12650.961840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12651.172102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12651.424469] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12651.915292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12651.921220] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12652.080810] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12652.080961] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12652.226369] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12652.480826] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12652.733328] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12652.987899] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12653.237856] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12655.375994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12655.546625] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12656.980213] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12657.242095] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12657.498748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12657.752585] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12657.911337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.140205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12658.396673] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12658.901789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12658.901804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12659.066298] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12659.066592] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12659.212886] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12659.467180] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12659.723301] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12659.982675] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12660.238905] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12662.434574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12662.602617] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12664.059366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12664.059366] awk '/^status/ { print $2 }' [12664.318350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12664.318350] awk '/^status/ { print $2 }' [12665.093788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12665.093788] awk '/^status/ { print $2 }' [12665.352545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12665.352545] awk '/^status/ { print $2 }' [12666.148434] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 fail_val=0 [12666.424112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12666.424112] awk '/^status/ { print $2 }' [12666.698462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12666.698462] awk '/^status/ { print $2 }' [12667.479604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12667.479604] awk '/^status/ { print $2 }' [12667.749579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12667.749579] awk '/^status/ { print $2 }' [12668.547870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12669.065999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12669.623024] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12670.545415] Lustre: DEBUG MARKER: rc=0; [12670.545415] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12670.545415] if [[ $? -eq 0 && $val -ne 0 ]]; then [12670.545415] echo $(hostname -s): $val; [12670.545415] rc=$val; [12670.545415] fi; [12670.545415] exit $rc [12671.154706] Lustre: DEBUG MARKER: dmesg [12671.696975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 11: OI scrub skips the new created objects only once ============================ 11:09:56 \(1491563396\) [12671.860438] Lustre: DEBUG MARKER: == sanity-scrub test 11: OI scrub skips the new created objects only once ============================ 11:09:56 (1491563396) [12673.190867] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12673.190867] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12673.190867] if [ $running -ne $mpts ]; then [12673.190867] echo $(hostname) env are INSANE!; [12673.190867] exit 1; [12673.190867] fi [12673.752494] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12673.752494] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12673.752494] if [ $running -ne $mpts ]; then [12673.752494] echo $(hostname) env are INSANE!; [12673.752494] exit 1; [12673.752494] fi [12674.331051] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12674.331051] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12674.331051] if [ $running -ne $mpts ]; then [12674.331051] echo $(hostname) env are INSANE!; [12674.331051] exit 1; [12674.331051] fi [12675.261358] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12675.537079] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12675.806432] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12676.075281] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12676.336556] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12677.903642] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12678.168549] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12678.421718] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12678.673348] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12678.932675] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12685.847999] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12686.007275] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12687.418637] Lustre: DEBUG MARKER: lctl get_param -n timeout [12687.728101] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [12687.879278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12688.014871] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [12694.685960] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 3 -r [12695.201987] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 3 -r [12695.712149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12695.712149] awk '/^status/ { print $2 }' [12695.969595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12695.969595] awk '/^status/ { print $2 }' [12696.741054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12696.741054] awk '/^status/ { print $2 }' [12696.996679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12696.996679] awk '/^status/ { print $2 }' [12697.767175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12698.021467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12698.790840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12699.046963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12699.816384] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 6 -r [12700.328088] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 6 -r [12700.843407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12700.843407] awk '/^status/ { print $2 }' [12701.096744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12701.096744] awk '/^status/ { print $2 }' [12701.873431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12701.873431] awk '/^status/ { print $2 }' [12702.127476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12702.127476] awk '/^status/ { print $2 }' [12702.931159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12703.186512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12703.957196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12704.211116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12705.034207] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12705.913143] Lustre: DEBUG MARKER: rc=0; [12705.913143] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12705.913143] if [[ $? -eq 0 && $val -ne 0 ]]; then [12705.913143] echo $(hostname -s): $val; [12705.913143] rc=$val; [12705.913143] fi; [12705.913143] exit $rc [12706.507216] Lustre: DEBUG MARKER: dmesg [12707.064226] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 12: OI scrub can rebuild invalid \/O entries ===================================== 11:10:31 \(1491563431\) [12707.230644] Lustre: DEBUG MARKER: == sanity-scrub test 12: OI scrub can rebuild invalid /O entries ===================================== 11:10:31 (1491563431) [12708.280956] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [12708.534313] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [12712.090606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12712.255592] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12719.425214] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12720.309320] Lustre: DEBUG MARKER: rc=0; [12720.309320] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12720.309320] if [[ $? -eq 0 && $val -ne 0 ]]; then [12720.309320] echo $(hostname -s): $val; [12720.309320] rc=$val; [12720.309320] fi; [12720.309320] exit $rc [12720.919738] Lustre: DEBUG MARKER: dmesg [12721.461278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 13: OI scrub can rebuild missed \/O entries ====================================== 11:10:46 \(1491563446\) [12721.610132] Lustre: DEBUG MARKER: == sanity-scrub test 13: OI scrub can rebuild missed /O entries ====================================== 11:10:46 (1491563446) [12722.088465] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [12722.342200] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [12725.826570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12726.004149] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12737.881332] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12738.769165] Lustre: DEBUG MARKER: rc=0; [12738.769165] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12738.769165] if [[ $? -eq 0 && $val -ne 0 ]]; then [12738.769165] echo $(hostname -s): $val; [12738.769165] rc=$val; [12738.769165] fi; [12738.769165] exit $rc [12739.361923] Lustre: DEBUG MARKER: dmesg [12739.920290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 14: OI scrub can repair objects under lost+found ================================ 11:11:04 \(1491563464\) [12740.085402] Lustre: DEBUG MARKER: == sanity-scrub test 14: OI scrub can repair objects under lost+found ================================ 11:11:04 (1491563464) [12740.549752] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id [12740.801472] Lustre: DEBUG MARKER: lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id [12744.693919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12744.855191] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12755.907865] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12756.791792] Lustre: DEBUG MARKER: rc=0; [12756.791792] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12756.791792] if [[ $? -eq 0 && $val -ne 0 ]]; then [12756.791792] echo $(hostname -s): $val; [12756.791792] rc=$val; [12756.791792] fi; [12756.791792] exit $rc [12757.388485] Lustre: DEBUG MARKER: dmesg [12757.931395] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test 15: Dryrun mode OI scrub ======================================================== 11:11:22 \(1491563482\) [12758.094245] Lustre: DEBUG MARKER: == sanity-scrub test 15: Dryrun mode OI scrub ======================================================== 11:11:22 (1491563482) [12758.247101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12758.247101] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12758.247101] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12761.048446] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12764.369639] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1 [12764.652539] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12764.914569] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [12765.240370] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12766.340194] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12766.591122] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [12772.968349] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12774.119314] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12774.372552] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12774.615111] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12774.859390] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12774.995273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12775.119931] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12775.380967] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12775.658532] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12775.970768] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12776.219616] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12776.492859] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 [12776.680730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12776.820755] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P1 /mnt/lustre-brpt [12776.945405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12777.070753] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12777.334702] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12777.584150] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12777.830392] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12778.134365] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12778.376529] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 lustre-MDT0000 [12783.200484] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12783.444211] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt [12783.687861] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12783.939472] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12784.080654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12784.204340] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && getfattr -R -d -m '.*' -P . > /tmp/backup_restore.ea && cd - [12784.461571] Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz -C /mnt/lustre-brpt/ . [12784.736957] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12785.064957] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12785.309784] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12785.562237] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat [12785.717111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12785.858447] Lustre: DEBUG MARKER: mount -t ldiskfs /dev/lvm-Role_MDS/P3 /mnt/lustre-brpt [12785.983669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [12786.107817] Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz -C /mnt/lustre-brpt [12786.375990] Lustre: DEBUG MARKER: cd /mnt/lustre-brpt && setfattr --restore=/tmp/backup_restore.ea && cd - [12786.623702] Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS [12786.868228] Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt [12787.180358] Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.ea /tmp/backup_restore.tgz [12787.424169] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 lustre-MDT0002 [12792.347860] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12792.607909] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12792.854592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12793.098197] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12793.253376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12793.262068] LustreError: 28803:0:(osd_scrub.c:2768:osd_scrub_setup()) lustre-MDT0000: UUID has been changed from e8813ad8-1226-980f-433c-8636f51d5541 to 474edf2c-6181-808f-4c98-c40c3ed08c99 [12793.266748] LustreError: 28803:0:(osd_scrub.c:2768:osd_scrub_setup()) Skipped 3 previous similar messages [12793.512293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12793.767715] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12794.263954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12794.264828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12794.429855] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12794.431952] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12794.573379] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12794.825347] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12795.078016] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12795.335194] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12795.587422] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12797.753448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12797.909389] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12799.342718] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12799.603105] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12799.856250] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12800.108583] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o user_xattr,noscrub /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12800.255970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12800.451296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12800.715541] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12801.306709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12801.311204] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12801.559620] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12801.565921] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12801.722216] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12802.015774] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12802.271369] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12802.533101] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12802.787309] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12804.960403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12805.123643] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12806.575587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12806.575587] awk '/^status/ { print $2 }' [12806.842188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12806.842188] awk '/^status/ { print $2 }' [12807.620350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12807.620350] awk '/^status/ { print $2 }' [12807.895389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12807.895389] awk '/^status/ { print $2 }' [12808.696738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12809.043859] Lustre: lustre-MDT0000: hit invalid OI mapping for [0x2000059f7:0x2:0x0] during recovering, that may because auto scrub is disabled on related MDT, and will cause recovery failure. Please enable auto scrub and retry the recovery. [12809.050279] LustreError: 30152:0:(osp_object.c:527:osp_attr_get()) lustre-MDT0000-osp-MDT0002:osp_attr_get update error [0x2000059f7:0x1:0x0]: rc = -78 [12809.056312] LustreError: 30152:0:(osp_object.c:527:osp_attr_get()) Skipped 1 previous similar message [12809.255824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12809.780985] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 5 --dryrun [12810.304730] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 5 --dryrun [12810.827311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12810.827311] awk '/^status/ { print $2 }' [12811.082052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12811.082052] awk '/^status/ { print $2 }' [12811.867232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12811.867232] awk '/^status/ { print $2 }' [12812.121948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12812.121948] awk '/^status/ { print $2 }' [12812.897115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12813.407459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12813.921615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12814.433116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12814.944973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12815.457734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12815.975949] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 10 --dryrun [12816.489971] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 10 --dryrun [12817.002032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12817.002032] awk '/^status/ { print $2 }' [12817.257491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12817.257491] awk '/^status/ { print $2 }' [12818.026496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12818.026496] awk '/^status/ { print $2 }' [12818.282041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12818.282041] awk '/^status/ { print $2 }' [12819.053691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12819.567909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12820.086782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12820.605345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12821.111675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12821.635665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12822.151198] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 15 [12822.656970] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 15 [12823.166333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12823.166333] awk '/^status/ { print $2 }' [12823.420979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12823.420979] awk '/^status/ { print $2 }' [12824.188568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12824.188568] awk '/^status/ { print $2 }' [12824.445117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12824.445117] awk '/^status/ { print $2 }' [12825.214414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12825.729347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12826.243351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12826.752142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12827.268170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12827.775873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12828.289528] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 20 [12828.800934] Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 20 [12829.325803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12829.325803] awk '/^status/ { print $2 }' [12829.579310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub | [12829.579310] awk '/^status/ { print $2 }' [12830.348697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12830.348697] awk '/^status/ { print $2 }' [12830.603931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub | [12830.603931] awk '/^status/ { print $2 }' [12831.375186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12831.886179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12832.404432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12832.916305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12833.430260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.oi_scrub [12833.946109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0002.oi_scrub [12834.515348] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [12835.400882] Lustre: DEBUG MARKER: rc=0; [12835.400882] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [12835.400882] if [[ $? -eq 0 && $val -ne 0 ]]; then [12835.400882] echo $(hostname -s): $val; [12835.400882] rc=$val; [12835.400882] fi; [12835.400882] exit $rc [12835.995004] Lustre: DEBUG MARKER: dmesg [12836.999479] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12837.261378] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [12839.044175] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [12839.044176] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [12839.044177] Lustre: Skipped 2 previous similar messages [12839.049636] Lustre: Skipped 2 previous similar messages [12839.135090] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006314c600 x1564004880897952/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [12839.139480] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [12843.627449] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12850.734595] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12850.984216] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [12857.346470] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12875.798115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [12875.798115] /usr/sbin/lctl lustre_build_version 2>/dev/null || [12875.798115] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [12876.107232] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [12876.695105] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12876.711714] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12876.722676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12876.810400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [12876.916026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12877.057981] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [12877.065636] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12877.091571] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [12877.195823] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [12877.229902] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [12877.591201] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [12877.836134] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12878.109492] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [12878.316730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12879.427379] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [12879.672275] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [12879.924562] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [12880.141907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12889.119142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test complete, duration 1933 sec ===================================================== 11:13:33 \(1491563613\) [12889.282095] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration 1933 sec ===================================================== 11:13:33 (1491563613) [12889.499390] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-hsm ============----- Fri Apr 7 11:13:34 UTC 2017 [12889.659799] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-hsm ============----- Fri Apr 7 11:13:34 UTC 2017 [12889.915131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration -o sec ========================================================= 11:13:34 \(1491563614\) [12890.069500] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration -o sec ========================================================= 11:13:34 (1491563614) [12892.560538] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12892.560538] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12892.560538] if [ $running -ne $mpts ]; then [12892.560538] echo $(hostname) env are INSANE!; [12892.560538] exit 1; [12892.560538] fi [12893.114142] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [12893.114142] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [12893.114142] if [ $running -ne $mpts ]; then [12893.114142] echo $(hostname) env are INSANE!; [12893.114142] exit 1; [12893.114142] fi [12893.680894] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [12893.680894] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [12893.680894] if [ $running -ne $mpts ]; then [12893.680894] echo $(hostname) env are INSANE!; [12893.680894] exit 1; [12893.680894] fi [12894.614671] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [12894.873898] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [12895.119500] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [12895.365506] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [12895.518105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12895.573103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.603361] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [12895.605923] Lustre: Skipped 4 previous similar messages [12895.621787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [12895.637538] Lustre: lustre-MDT0000: new disk, initializing [12895.677884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12895.680474] Lustre: Skipped 3 previous similar messages [12895.846874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12896.098087] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [12896.582170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12896.584115] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12896.745196] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12896.747346] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12896.888040] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12897.138538] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12897.386913] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12897.636078] Lustre: DEBUG MARKER: sync; sync; sync [12902.946332] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [12903.201941] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12904.447219] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [12904.449633] Lustre: Skipped 4 previous similar messages [12904.726142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12905.422592] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12905.582332] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12912.311227] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [12912.573257] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [12912.823287] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [12913.073188] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [12913.229121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12913.276787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12913.291768] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [12913.294440] Lustre: Skipped 4 previous similar messages [12913.302203] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [12913.304706] Lustre: Skipped 1 previous similar message [12913.320390] Lustre: lustre-MDT0002: new disk, initializing [12913.350378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [12913.353234] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [12913.517316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [12913.771521] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [12914.263409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12914.266259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12914.443066] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12914.443655] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12914.589932] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12914.846748] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12915.099138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [12915.350308] Lustre: DEBUG MARKER: sync; sync; sync [12920.814463] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [12921.067882] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [12922.423622] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [12922.426282] Lustre: Skipped 4 previous similar messages [12923.385243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12923.540045] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12932.222110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12932.376203] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12935.194550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [12935.197533] Lustre: Skipped 1 previous similar message [12940.551181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12940.810662] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12948.965928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12949.126705] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12957.634601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12957.800562] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12966.010343] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12966.173597] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12973.537496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000400000400-0x0000000440000400]:4:ost [12973.540766] Lustre: Skipped 3 previous similar messages [12974.331822] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12974.498699] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12982.754212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12982.920121] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12991.161493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12991.333741] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12998.808178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [12999.048669] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13001.659177] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13001.816062] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13002.003221] Lustre: DEBUG MARKER: lctl get_param -n timeout [13002.320081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [13002.479250] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13002.619556] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [13008.023337] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [13008.303634] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13008.820159] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [13009.331198] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13009.331198] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13009.331198] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13009.798575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [13009.959766] Lustre: DEBUG MARKER: excepting tests: [13010.576229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir \/shared_test\/autotest2\/2017-04-07\/062101-70344545792740 [13010.576246] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir \/shared_test\/autotest2\/2017-04-07\/062101-70344545792740 [13010.795320] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing check_logdir /shared_test/autotest2/2017-04-07/062101-70344545792740 [13010.795666] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing check_logdir /shared_test/autotest2/2017-04-07/062101-70344545792740 [13011.792331] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [13011.929385] Lustre: Modifying parameter general.mdt.lustre-MDT0000.hsm_control in log params [13011.932368] Lustre: Skipped 4 previous similar messages [13012.066572] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0001.hsm_control=enabled [13012.321373] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0002.hsm_control=enabled [13012.576459] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0003.hsm_control=enabled [13017.821854] LustreError: 9750:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [13032.836751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13033.091637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13033.859506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13034.114434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13034.889105] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [13035.405057] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [13035.999006] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1: lfs hsm flags root\/non-root access ============================================= 11:16:00 \(1491563760\) [13036.151143] Lustre: DEBUG MARKER: == sanity-hsm test 1: lfs hsm flags root/non-root access ============================================= 11:16:00 (1491563760) [13036.407901] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13037.296666] Lustre: DEBUG MARKER: rc=0; [13037.296666] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13037.296666] if [[ $? -eq 0 && $val -ne 0 ]]; then [13037.296666] echo $(hostname -s): $val; [13037.296666] rc=$val; [13037.296666] fi; [13037.296666] exit $rc [13037.899745] Lustre: DEBUG MARKER: dmesg [13038.426669] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 1a: mmap & cat a HSM released file ================================================ 11:16:03 \(1491563763\) [13038.582459] Lustre: DEBUG MARKER: == sanity-hsm test 1a: mmap [13039.776604] LustreError: 9740:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent 'ba976f76-f68d-3cdb-16f3-355a9f831a1e': rc = -11 [13039.902634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13040.158339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13041.414781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13042.675993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13043.942793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13045.209426] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13046.467587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13047.725872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13048.981728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13050.243839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13051.508066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13052.778543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13057.742890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13057.999521] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13058.768200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13059.023148] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13059.790161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13060.042896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13061.302555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13062.561626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13063.816565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13065.078623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13065.334333] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13066.364691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13066.621369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13067.875089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13068.130508] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13069.154732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13069.408712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13070.180197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13070.434430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13071.760636] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13072.647886] Lustre: DEBUG MARKER: rc=0; [13072.647886] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13072.647886] if [[ $? -eq 0 && $val -ne 0 ]]; then [13072.647886] echo $(hostname -s): $val; [13072.647886] rc=$val; [13072.647886] fi; [13072.647886] exit $rc [13073.255621] Lustre: DEBUG MARKER: dmesg [13073.780640] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 11:16:38 \(1491563798\) [13073.935871] Lustre: DEBUG MARKER: == sanity-hsm test 2: Check file dirtyness when doing setattr ======================================== 11:16:38 (1491563798) [13074.228977] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13075.119192] Lustre: DEBUG MARKER: rc=0; [13075.119192] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13075.119192] if [[ $? -eq 0 && $val -ne 0 ]]; then [13075.119192] echo $(hostname -s): $val; [13075.119192] rc=$val; [13075.119192] fi; [13075.119192] exit $rc [13075.738954] Lustre: DEBUG MARKER: dmesg [13076.262224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 11:16:41 \(1491563801\) [13076.419805] Lustre: DEBUG MARKER: == sanity-hsm test 3: Check file dirtyness when opening for write ==================================== 11:16:41 (1491563801) [13076.724377] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13077.612022] Lustre: DEBUG MARKER: rc=0; [13077.612022] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13077.612022] if [[ $? -eq 0 && $val -ne 0 ]]; then [13077.612022] echo $(hostname -s): $val; [13077.612022] rc=$val; [13077.612022] fi; [13077.612022] exit $rc [13078.205641] Lustre: DEBUG MARKER: dmesg [13078.752269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 4: Useless cancel must not be registered ========================================== 11:16:43 \(1491563803\) [13078.907245] Lustre: DEBUG MARKER: == sanity-hsm test 4: Useless cancel must not be registered ========================================== 11:16:43 (1491563803) [13079.265315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [13079.577851] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13080.454045] Lustre: DEBUG MARKER: rc=0; [13080.454045] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13080.454045] if [[ $? -eq 0 && $val -ne 0 ]]; then [13080.454045] echo $(hostname -s): $val; [13080.454045] rc=$val; [13080.454045] fi; [13080.454045] exit $rc [13081.045071] Lustre: DEBUG MARKER: dmesg [13081.578928] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 8: Test default archive number ==================================================== 11:16:46 \(1491563806\) [13081.744882] Lustre: DEBUG MARKER: == sanity-hsm test 8: Test default archive number ==================================================== 11:16:46 (1491563806) [13082.656506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13082.919669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x3:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13085.743721] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13086.001652] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13086.783918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13087.041105] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13087.809796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13088.064857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13089.319382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13090.573463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13091.828693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13093.083193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13093.337964] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13096.878760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13097.139270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13097.394418] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13098.424970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13098.678939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13099.446838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13099.700725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13101.031560] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13101.923736] Lustre: DEBUG MARKER: rc=0; [13101.923736] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13101.923736] if [[ $? -eq 0 && $val -ne 0 ]]; then [13101.923736] echo $(hostname -s): $val; [13101.923736] rc=$val; [13101.923736] fi; [13101.923736] exit $rc [13102.540783] Lustre: DEBUG MARKER: dmesg [13103.088530] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 11:17:07 \(1491563827\) [13103.247702] Lustre: DEBUG MARKER: == sanity-hsm test 9: Use of explicit archive number, with dedicated copytool ======================== 11:17:07 (1491563827) [13105.939748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13106.196344] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13106.968302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13107.232092] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13107.997567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13108.252326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13109.511749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13110.767425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13112.021251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13113.273642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13114.525859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13114.779449] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13118.323700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13118.577548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13118.832535] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13119.867628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13120.124772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13120.892443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13121.148283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13131.627919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [13131.787755] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [13132.403415] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre [13132.562839] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre [13132.706116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep 84b484b6-8248-a20e-62a5-61e74724d1f5 [13133.228229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep 84b484b6-8248-a20e-62a5-61e74724d1f5 [13133.779351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13134.034576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13136.852118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13137.110104] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13137.878241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13138.133501] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13138.899243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13139.154330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13140.407715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13141.668532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13142.925645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13144.185140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13144.443983] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13145.471527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13145.725104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13145.981367] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13147.018378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13147.272522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13148.045556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13148.296648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13149.635088] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13150.527467] Lustre: DEBUG MARKER: rc=0; [13150.527467] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13150.527467] if [[ $? -eq 0 && $val -ne 0 ]]; then [13150.527467] echo $(hostname -s): $val; [13150.527467] rc=$val; [13150.527467] fi; [13150.527467] exit $rc [13151.121424] Lustre: DEBUG MARKER: dmesg [13151.653973] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 9a: Multiple remote agents ======================================================== 11:17:56 \(1491563876\) [13151.806874] Lustre: DEBUG MARKER: == sanity-hsm test 9a: Multiple remote agents ======================================================== 11:17:56 (1491563876) [13151.993841] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [13152.146080] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_9a Need 3 or more clients, have 2 [13152.338612] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13153.223669] Lustre: DEBUG MARKER: rc=0; [13153.223669] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13153.223669] if [[ $? -eq 0 && $val -ne 0 ]]; then [13153.223669] echo $(hostname -s): $val; [13153.223669] rc=$val; [13153.223669] fi; [13153.223669] exit $rc [13153.816439] Lustre: DEBUG MARKER: dmesg [13154.376991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10a: Archive a file =============================================================== 11:17:59 \(1491563879\) [13154.534459] Lustre: DEBUG MARKER: == sanity-hsm test 10a: Archive a file =============================================================== 11:17:59 (1491563879) [13155.444580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13155.712620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x5:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13159.051614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13159.307414] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13160.078546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13160.336094] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13161.108024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13161.365443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13162.620953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13163.874534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13165.127365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13166.387843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13166.646161] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13167.675042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13167.947969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13168.210992] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13169.235776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13169.490936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13170.261210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13170.515413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13171.832593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13172.751703] Lustre: DEBUG MARKER: rc=0; [13172.751703] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13172.751703] if [[ $? -eq 0 && $val -ne 0 ]]; then [13172.751703] echo $(hostname -s): $val; [13172.751703] rc=$val; [13172.751703] fi; [13172.751703] exit $rc [13173.345251] Lustre: DEBUG MARKER: dmesg [13174.034921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 11:18:18 \(1491563898\) [13174.195076] Lustre: DEBUG MARKER: == sanity-hsm test 10b: Archive of non dirty file must work without doing request ==================== 11:18:18 (1491563898) [13175.090391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13175.349483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13175.611746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0x6:0x0'.*action='ARCHIVE'/ {n++}; END {print n}' [13178.406861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13178.679725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13179.463760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13179.729089] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13180.496724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13180.748774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13182.003494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13183.266573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13184.521194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13185.774717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13186.028155] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13188.310150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13188.564701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13188.816522] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13189.867259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13190.118495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13190.895741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13191.164787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13192.478067] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13193.364442] Lustre: DEBUG MARKER: rc=0; [13193.364442] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13193.364442] if [[ $? -eq 0 && $val -ne 0 ]]; then [13193.364442] echo $(hostname -s): $val; [13193.364442] rc=$val; [13193.364442] fi; [13193.364442] exit $rc [13193.960917] Lustre: DEBUG MARKER: dmesg [13194.479239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10c: Check forbidden archive ====================================================== 11:18:39 \(1491563919\) [13194.639075] Lustre: DEBUG MARKER: == sanity-hsm test 10c: Check forbidden archive ====================================================== 11:18:39 (1491563919) [13198.125038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13198.380834] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13199.159997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13199.416438] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13200.186638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13200.441438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13201.703176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13202.964949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13204.224431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13205.482120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13206.736099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13206.989908] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13209.307115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13209.573477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13209.829298] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13210.860143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13211.116513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13211.886844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13212.145089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13213.465593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13214.351088] Lustre: DEBUG MARKER: rc=0; [13214.351088] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13214.351088] if [[ $? -eq 0 && $val -ne 0 ]]; then [13214.351088] echo $(hostname -s): $val; [13214.351088] rc=$val; [13214.351088] fi; [13214.351088] exit $rc [13214.945881] Lustre: DEBUG MARKER: dmesg [13215.476281] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 11:19:00 \(1491563940\) [13215.672794] Lustre: DEBUG MARKER: == sanity-hsm test 10d: Archive a file on the default archive id ===================================== 11:19:00 (1491563940) [13216.589093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13216.846809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x8:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13217.113865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [13219.909773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13220.166046] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13220.941184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13221.196219] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13221.972366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13222.225128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13223.477582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13224.730697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13225.982570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13227.240588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13227.494601] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13229.813899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13230.065419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13230.319988] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13231.354194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13231.609164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13232.381580] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13232.636070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13233.937544] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13234.822809] Lustre: DEBUG MARKER: rc=0; [13234.822809] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13234.822809] if [[ $? -eq 0 && $val -ne 0 ]]; then [13234.822809] echo $(hostname -s): $val; [13234.822809] rc=$val; [13234.822809] fi; [13234.822809] exit $rc [13235.414343] Lustre: DEBUG MARKER: dmesg [13235.986182] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11a: Import a file ================================================================ 11:19:20 \(1491563960\) [13236.138113] Lustre: DEBUG MARKER: == sanity-hsm test 11a: Import a file ================================================================ 11:19:20 (1491563960) [13237.593116] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13238.476366] Lustre: DEBUG MARKER: rc=0; [13238.476366] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13238.476366] if [[ $? -eq 0 && $val -ne 0 ]]; then [13238.476366] echo $(hostname -s): $val; [13238.476366] rc=$val; [13238.476366] fi; [13238.476366] exit $rc [13239.105584] Lustre: DEBUG MARKER: dmesg [13239.653732] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 11:19:24 \(1491563964\) [13239.808904] Lustre: DEBUG MARKER: == sanity-hsm test 11b: Import a deleted file using its FID ========================================== 11:19:24 (1491563964) [13240.712880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13240.971821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13244.100532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13244.354136] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13245.120561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13245.376060] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13246.143025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13246.398865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13247.661217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13248.920193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13250.185122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13251.446914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13251.698819] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13252.726753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13252.980451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13253.234903] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13254.261846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13254.514911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13255.279912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13255.532981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13256.863857] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13257.737607] Lustre: DEBUG MARKER: rc=0; [13257.737607] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13257.737607] if [[ $? -eq 0 && $val -ne 0 ]]; then [13257.737607] echo $(hostname -s): $val; [13257.737607] rc=$val; [13257.737607] fi; [13257.737607] exit $rc [13258.336825] Lustre: DEBUG MARKER: dmesg [13258.914578] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 11:19:43 \(1491563983\) [13259.080721] Lustre: DEBUG MARKER: == sanity-hsm test 12a: Restore an imported file explicitly ========================================== 11:19:43 (1491563983) [13260.754533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13261.014653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0x3:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13264.058904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13264.314232] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13265.083160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13265.338628] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13266.118996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13266.372261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13267.625499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13268.881076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13270.137458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13271.395832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13271.666718] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13273.981943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13274.241180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13274.494428] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13275.515165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13275.768338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13276.540309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13276.803032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13278.127018] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13279.030061] Lustre: DEBUG MARKER: rc=0; [13279.030061] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13279.030061] if [[ $? -eq 0 && $val -ne 0 ]]; then [13279.030061] echo $(hostname -s): $val; [13279.030061] rc=$val; [13279.030061] fi; [13279.030061] exit $rc [13279.641155] Lustre: DEBUG MARKER: dmesg [13280.185484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 11:20:05 \(1491564005\) [13280.342477] Lustre: DEBUG MARKER: == sanity-hsm test 12b: Restore an imported file implicitly ========================================== 11:20:05 (1491564005) [13284.803566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13285.061336] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13285.835182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13286.095820] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13286.870597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13287.127093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13288.382099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13289.634118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13290.889092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13292.147330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13292.418464] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13294.713976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13294.967446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13295.222425] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13296.257915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13296.521645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13297.295412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13297.566256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13298.893939] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13299.810640] Lustre: DEBUG MARKER: rc=0; [13299.810640] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13299.810640] if [[ $? -eq 0 && $val -ne 0 ]]; then [13299.810640] echo $(hostname -s): $val; [13299.810640] rc=$val; [13299.810640] fi; [13299.810640] exit $rc [13300.407424] Lustre: DEBUG MARKER: dmesg [13300.943162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 11:20:25 \(1491564025\) [13301.099174] Lustre: DEBUG MARKER: == sanity-hsm test 12c: Restore a file with stripe of 2 ============================================== 11:20:25 (1491564025) [13302.286119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13302.544501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13303.800369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13305.064456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13306.325119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13307.585274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13315.582952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13315.840858] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13316.620352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13316.877768] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13317.648047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13317.899908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13319.152980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13320.405861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13321.664893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13322.919120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13323.174927] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13325.465130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13325.738060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13326.008108] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13327.041317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13327.296749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13328.064523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13328.320105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13329.644150] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13330.536948] Lustre: DEBUG MARKER: rc=0; [13330.536948] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13330.536948] if [[ $? -eq 0 && $val -ne 0 ]]; then [13330.536948] echo $(hostname -s): $val; [13330.536948] rc=$val; [13330.536948] fi; [13330.536948] exit $rc [13331.131944] Lustre: DEBUG MARKER: dmesg [13331.689782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 11:20:56 \(1491564056\) [13331.844523] Lustre: DEBUG MARKER: == sanity-hsm test 12d: Restore of a non archived, non released file must work ======================= 11:20:56 (1491564056) [13332.749444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [13333.019477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13333.277802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13333.538906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk -vn=0 '/'0x200000405:0xb:0x0'.*action='RESTORE'/ {n++}; END {print n}' [13336.327524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13336.584132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13337.361593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13337.616935] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13338.392930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13338.646921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13339.900423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13341.155400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13342.414869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13343.671720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13343.927953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13346.212376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13346.467678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13346.722542] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13347.769154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13348.022247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13348.788890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13349.042291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13350.349830] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13351.246923] Lustre: DEBUG MARKER: rc=0; [13351.246923] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13351.246923] if [[ $? -eq 0 && $val -ne 0 ]]; then [13351.246923] echo $(hostname -s): $val; [13351.246923] rc=$val; [13351.246923] fi; [13351.246923] exit $rc [13351.859383] Lustre: DEBUG MARKER: dmesg [13352.407174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12e: Check forbidden restore ====================================================== 11:21:17 \(1491564077\) [13352.562966] Lustre: DEBUG MARKER: == sanity-hsm test 12e: Check forbidden restore ====================================================== 11:21:17 (1491564077) [13353.467545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13353.745509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13356.606899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13356.865330] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13357.653701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13357.907704] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13358.679087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13358.933853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13360.187662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13361.448860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13362.703609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13363.957489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13364.211274] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13366.498858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13366.752889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13367.008386] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13368.040550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13368.294628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13369.058962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13369.311302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13370.636630] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13371.533285] Lustre: DEBUG MARKER: rc=0; [13371.533285] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13371.533285] if [[ $? -eq 0 && $val -ne 0 ]]; then [13371.533285] echo $(hostname -s): $val; [13371.533285] rc=$val; [13371.533285] fi; [13371.533285] exit $rc [13372.143306] Lustre: DEBUG MARKER: dmesg [13372.697557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12f: Restore a released file explicitly =========================================== 11:21:37 \(1491564097\) [13372.854036] Lustre: DEBUG MARKER: == sanity-hsm test 12f: Restore a released file explicitly =========================================== 11:21:37 (1491564097) [13373.807522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13374.067125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13374.339479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13374.597450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13377.411294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13377.669265] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13378.442759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13378.696785] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13379.462920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13379.714561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13380.969562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13382.232093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13383.495614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13384.757836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13385.018531] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13387.305923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13387.558647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13387.814280] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13388.835848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13389.089173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13389.859645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13390.112769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13391.442463] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13392.327102] Lustre: DEBUG MARKER: rc=0; [13392.327102] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13392.327102] if [[ $? -eq 0 && $val -ne 0 ]]; then [13392.327102] echo $(hostname -s): $val; [13392.327102] rc=$val; [13392.327102] fi; [13392.327102] exit $rc [13392.920561] Lustre: DEBUG MARKER: dmesg [13393.465433] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12g: Restore a released file implicitly =========================================== 11:21:58 \(1491564118\) [13393.622539] Lustre: DEBUG MARKER: == sanity-hsm test 12g: Restore a released file implicitly =========================================== 11:21:58 (1491564118) [13394.528478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13394.791996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13395.141357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13395.401722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xe:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13398.201236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13398.472118] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13399.244752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13399.510720] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13400.313842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13400.571352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13401.828930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13403.088941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13404.348857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13405.604278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13405.861251] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13408.160769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13408.415296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13408.669930] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13409.698775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13409.954741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13410.725394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13410.978844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13412.300788] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13413.181487] Lustre: DEBUG MARKER: rc=0; [13413.181487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13413.181487] if [[ $? -eq 0 && $val -ne 0 ]]; then [13413.181487] echo $(hostname -s): $val; [13413.181487] rc=$val; [13413.181487] fi; [13413.181487] exit $rc [13413.780721] Lustre: DEBUG MARKER: dmesg [13414.350305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 11:22:19 \(1491564139\) [13414.500262] Lustre: DEBUG MARKER: == sanity-hsm test 12h: Restore a released file implicitly from a second node ======================== 11:22:19 (1491564139) [13415.409888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13415.667433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13416.266399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13416.524388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0xf:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13419.331859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13419.587770] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13420.362555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13420.634369] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13421.418880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13421.676556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13422.930046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13424.183111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13425.436590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13426.696532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13426.951527] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13429.236105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13429.503482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13429.758514] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13430.805906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13431.060836] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13431.829331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13432.083488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13433.408423] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13434.285810] Lustre: DEBUG MARKER: rc=0; [13434.285810] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13434.285810] if [[ $? -eq 0 && $val -ne 0 ]]; then [13434.285810] echo $(hostname -s): $val; [13434.285810] rc=$val; [13434.285810] fi; [13434.285810] exit $rc [13434.875361] Lustre: DEBUG MARKER: dmesg [13435.406953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12m: Archive\/release\/implicit restore ============================================= 11:22:40 \(1491564160\) [13435.566540] Lustre: DEBUG MARKER: == sanity-hsm test 12m: Archive/release/implicit restore ============================================= 11:22:40 (1491564160) [13436.507533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13436.783624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x10:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13439.632935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13439.893405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13440.666847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13440.922729] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13441.703410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13441.958086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13443.217802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13444.480793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13445.740594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13446.997168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13447.252443] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13449.537087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13449.790267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13450.045385] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13451.079839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13451.352624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13452.138327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13452.395458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13453.739404] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13454.621220] Lustre: DEBUG MARKER: rc=0; [13454.621220] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13454.621220] if [[ $? -eq 0 && $val -ne 0 ]]; then [13454.621220] echo $(hostname -s): $val; [13454.621220] rc=$val; [13454.621220] fi; [13454.621220] exit $rc [13455.223445] Lustre: DEBUG MARKER: dmesg [13455.767084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12n: Import\/implicit restore\/release ============================================== 11:23:00 \(1491564180\) [13455.924816] Lustre: DEBUG MARKER: == sanity-hsm test 12n: Import/implicit restore/release ============================================== 11:23:00 (1491564180) [13460.363898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13460.620472] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13461.419019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13461.671776] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13462.442431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13462.697829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13463.955357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13465.208766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13466.464771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13467.728126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13467.988442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13470.269105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13470.523286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13470.777037] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13471.805272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13472.061383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13472.828360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13473.082031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13474.389771] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13475.263529] Lustre: DEBUG MARKER: rc=0; [13475.263529] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13475.263529] if [[ $? -eq 0 && $val -ne 0 ]]; then [13475.263529] echo $(hostname -s): $val; [13475.263529] rc=$val; [13475.263529] fi; [13475.263529] exit $rc [13475.857088] Lustre: DEBUG MARKER: dmesg [13476.403199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 11:23:21 \(1491564201\) [13476.561476] Lustre: DEBUG MARKER: == sanity-hsm test 12o: Layout-swap failure during Restore leaves file released ====================== 11:23:21 (1491564201) [13477.452248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13477.708630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13477.977027] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x152 [13478.233077] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [13478.755915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [13479.226943] Lustre: *** cfs_fail_loc=152, val=0*** [13479.358070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13479.622470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13479.882125] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [13480.401633] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [13480.916910] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [13481.170484] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [13481.678618] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [13482.189093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13482.440563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13483.206435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13483.461413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13484.227811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [13494.594135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13494.851397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x11:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13497.648194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13497.903261] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13498.707041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13498.964228] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13499.758976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13500.014350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13501.272739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13502.529272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13503.782186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13505.036355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13505.294043] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13506.322154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13506.575195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13506.832117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13507.869867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13508.133738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13508.897732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13509.152024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13510.476079] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13511.363305] Lustre: DEBUG MARKER: rc=0; [13511.363305] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13511.363305] if [[ $? -eq 0 && $val -ne 0 ]]; then [13511.363305] echo $(hostname -s): $val; [13511.363305] rc=$val; [13511.363305] fi; [13511.363305] exit $rc [13511.968170] Lustre: DEBUG MARKER: dmesg [13512.603574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 11:23:57 \(1491564237\) [13512.760259] Lustre: DEBUG MARKER: == sanity-hsm test 12p: implicit restore of a file on copytool mount point =========================== 11:23:57 (1491564237) [13513.699062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13513.954436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x12:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13517.737444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13517.995337] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13518.771592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13519.028504] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13519.802628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13520.059758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13521.312703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13522.568209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13523.827874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13525.082160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13525.343156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13527.633193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13527.887336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13528.142754] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13529.171769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13529.427403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13530.194344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13530.451066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13531.776423] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13532.659612] Lustre: DEBUG MARKER: rc=0; [13532.659612] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13532.659612] if [[ $? -eq 0 && $val -ne 0 ]]; then [13532.659612] echo $(hostname -s): $val; [13532.659612] rc=$val; [13532.659612] fi; [13532.659612] exit $rc [13533.265162] Lustre: DEBUG MARKER: dmesg [13533.811403] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 11:24:18 \(1491564258\) [13533.969227] Lustre: DEBUG MARKER: == sanity-hsm test 12q: file attributes are refreshed after restore ================================== 11:24:18 (1491564258) [13534.110539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [13534.110539] /usr/sbin/lctl lustre_build_version 2>/dev/null || [13534.110539] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [13534.757965] Lustre: lustre-MDT0002: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [13534.757966] Lustre: lustre-MDT0000: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [13534.757967] Lustre: Skipped 135 previous similar messages [13535.384729] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13535.538774] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13536.656976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13536.921943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13538.191631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13539.465741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x13:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13556.935303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13557.191230] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13557.959907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13558.217252] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13558.988319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13559.244819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13560.499593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13561.765507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13563.034255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13564.291336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13564.547341] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13568.083412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13568.340885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13568.598327] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13569.666744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13569.922045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13570.704710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13570.967140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13573.120446] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13574.018670] Lustre: DEBUG MARKER: rc=0; [13574.018670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13574.018670] if [[ $? -eq 0 && $val -ne 0 ]]; then [13574.018670] echo $(hostname -s): $val; [13574.018670] rc=$val; [13574.018670] fi; [13574.018670] exit $rc [13574.618374] Lustre: DEBUG MARKER: dmesg [13575.158471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 13: Recursively import and restore a directory ==================================== 11:24:59 \(1491564299\) [13575.324982] Lustre: DEBUG MARKER: == sanity-hsm test 13: Recursively import and restore a directory ==================================== 11:24:59 (1491564299) [13608.860478] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13609.115905] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13609.888065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13610.141726] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13610.923884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13611.178744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13612.433966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13613.688663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13614.951903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13616.214207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13616.471517] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13618.754278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13619.009129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13619.264786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13620.299691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13620.555068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13621.327595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13621.584662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13622.941667] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13623.825761] Lustre: DEBUG MARKER: rc=0; [13623.825761] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13623.825761] if [[ $? -eq 0 && $val -ne 0 ]]; then [13623.825761] echo $(hostname -s): $val; [13623.825761] rc=$val; [13623.825761] fi; [13623.825761] exit $rc [13624.417919] Lustre: DEBUG MARKER: dmesg [13624.978159] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 11:25:49 \(1491564349\) [13625.137694] Lustre: DEBUG MARKER: == sanity-hsm test 14: Rebind archived file to a new fid ============================================= 11:25:49 (1491564349) [13626.362330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13626.620369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13627.879970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13629.139750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x14:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13634.347048] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13634.606064] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13635.409255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13635.666926] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13636.432469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13636.686857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13637.940111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13639.196161] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13640.451474] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13641.705914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13641.968329] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13642.992624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13643.250505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13643.504276] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13644.544916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13644.800396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13645.573924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13645.827032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13647.153203] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13648.059253] Lustre: DEBUG MARKER: rc=0; [13648.059253] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13648.059253] if [[ $? -eq 0 && $val -ne 0 ]]; then [13648.059253] echo $(hostname -s): $val; [13648.059253] rc=$val; [13648.059253] fi; [13648.059253] exit $rc [13648.648542] Lustre: DEBUG MARKER: dmesg [13649.187146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 15: Rebind a list of files ======================================================== 11:26:14 \(1491564374\) [13649.344682] Lustre: DEBUG MARKER: == sanity-hsm test 15: Rebind a list of files ======================================================== 11:26:14 (1491564374) [13651.901124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [13652.159767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [13653.419081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [13654.676517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [13655.943092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [13669.909361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13670.166287] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13670.945785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13671.203685] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13671.972352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13672.227743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13673.489991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13674.745864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13676.002195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13677.256387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13677.511509] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13678.534943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13678.787068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13679.042089] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13680.073547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13680.331234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13681.103114] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13681.356609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13682.681450] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13683.579384] Lustre: DEBUG MARKER: rc=0; [13683.579384] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13683.579384] if [[ $? -eq 0 && $val -ne 0 ]]; then [13683.579384] echo $(hostname -s): $val; [13683.579384] rc=$val; [13683.579384] fi; [13683.579384] exit $rc [13684.176901] Lustre: DEBUG MARKER: dmesg [13684.718191] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 16: Test CT bandwith control option =============================================== 11:26:49 \(1491564409\) [13684.879315] Lustre: DEBUG MARKER: == sanity-hsm test 16: Test CT bandwith control option =============================================== 11:26:49 (1491564409) [13686.207398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13686.485527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13687.754397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13689.017817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13690.285541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13691.548916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13692.807699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13694.071251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13695.345037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13696.608782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13697.869906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13699.128908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13700.392625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13701.656556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13702.936561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13704.203701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13705.471624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13706.739791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13709.540983] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13709.797700] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13710.566076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13710.822777] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13711.591508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13711.845657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13713.100129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13714.355977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13715.612052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13716.868237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13717.129369] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13719.409307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13719.664567] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13719.924053] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13720.957149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13721.211272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13722.010643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13722.264640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13723.597477] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13724.491214] Lustre: DEBUG MARKER: rc=0; [13724.491214] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13724.491214] if [[ $? -eq 0 && $val -ne 0 ]]; then [13724.491214] echo $(hostname -s): $val; [13724.491214] rc=$val; [13724.491214] fi; [13724.491214] exit $rc [13725.109552] Lustre: DEBUG MARKER: dmesg [13725.651796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 20: Release is not permitted ====================================================== 11:27:30 \(1491564450\) [13725.805658] Lustre: DEBUG MARKER: == sanity-hsm test 20: Release is not permitted ====================================================== 11:27:30 (1491564450) [13726.031398] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13726.924615] Lustre: DEBUG MARKER: rc=0; [13726.924615] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13726.924615] if [[ $? -eq 0 && $val -ne 0 ]]; then [13726.924615] echo $(hostname -s): $val; [13726.924615] rc=$val; [13726.924615] fi; [13726.924615] exit $rc [13727.521615] Lustre: DEBUG MARKER: dmesg [13728.074185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 21: Simple release tests ========================================================== 11:27:32 \(1491564452\) [13728.240571] Lustre: DEBUG MARKER: == sanity-hsm test 21: Simple release tests ========================================================== 11:27:32 (1491564452) [13729.373978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug [13729.702180] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=150 [13730.057378] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1; [13730.359601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13730.616963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13731.888279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13733.163887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13733.490117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x44:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13733.748850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000404:0x44:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13734.085516] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug_mb=4 [13734.659123] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" [13737.482259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13737.740483] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13738.517699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13738.773381] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13739.541517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13739.796284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13741.050420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13742.306720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13743.562199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13743.818066] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13744.844467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13745.099487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13745.352070] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13746.376467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13746.629586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13747.397289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13747.651358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13748.972476] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13749.857302] Lustre: DEBUG MARKER: rc=0; [13749.857302] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13749.857302] if [[ $? -eq 0 && $val -ne 0 ]]; then [13749.857302] echo $(hostname -s): $val; [13749.857302] rc=$val; [13749.857302] fi; [13749.857302] exit $rc [13750.457127] Lustre: DEBUG MARKER: dmesg [13751.021082] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 22: Could not swap a release file ================================================= 11:27:55 \(1491564475\) [13751.171035] Lustre: DEBUG MARKER: == sanity-hsm test 22: Could not swap a release file ================================================= 11:27:55 (1491564475) [13752.317974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13752.576357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13753.836698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13755.097945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13758.143709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13758.401171] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13759.170440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13759.426966] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13760.204343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13760.461179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13761.715352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13762.971109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13764.230959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13764.487297] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13765.516714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13765.775610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13766.030144] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13767.068397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13767.324231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13768.102875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13768.359964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13769.687577] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13770.583224] Lustre: DEBUG MARKER: rc=0; [13770.583224] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13770.583224] if [[ $? -eq 0 && $val -ne 0 ]]; then [13770.583224] echo $(hostname -s): $val; [13770.583224] rc=$val; [13770.583224] fi; [13770.583224] exit $rc [13771.181737] Lustre: DEBUG MARKER: dmesg [13771.734332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 23: Release does not change a\/mtime \(utime\) ======================================= 11:28:16 \(1491564496\) [13771.903209] Lustre: DEBUG MARKER: == sanity-hsm test 23: Release does not change a/mtime (utime) ======================================= 11:28:16 (1491564496) [13773.032340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13773.294309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13774.561049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13775.837407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13778.672030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13778.929290] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13779.698194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13779.962389] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13780.745724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13781.000542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13782.255363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13783.509278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13784.764639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13786.017250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13786.271787] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13787.304799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13787.564769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13787.826389] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13788.852691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13789.108160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13789.883558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13790.148961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13791.498686] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13792.408699] Lustre: DEBUG MARKER: rc=0; [13792.408699] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13792.408699] if [[ $? -eq 0 && $val -ne 0 ]]; then [13792.408699] echo $(hostname -s): $val; [13792.408699] rc=$val; [13792.408699] fi; [13792.408699] exit $rc [13793.003129] Lustre: DEBUG MARKER: dmesg [13793.567531] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24a: Archive, release, and restore does not change a\/mtime \(i\/o\) ================== 11:28:38 \(1491564518\) [13793.725948] Lustre: DEBUG MARKER: == sanity-hsm test 24a: Archive, release, and restore does not change a/mtime (i/o) ================== 11:28:38 (1491564518) [13795.857281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13796.116481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13797.377357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13798.638778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13798.928866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13799.187129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13800.450879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13801.721628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1f:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13804.522273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13804.777420] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13805.554188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13805.813135] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13806.585191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13806.842518] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13808.097332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13809.349308] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13810.604130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13811.858650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13812.115484] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13813.150311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13813.406515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13813.660870] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13814.677096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13814.933961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13815.699443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13815.955544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13817.492742] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13818.403599] Lustre: DEBUG MARKER: rc=0; [13818.403599] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13818.403599] if [[ $? -eq 0 && $val -ne 0 ]]; then [13818.403599] echo $(hostname -s): $val; [13818.403599] rc=$val; [13818.403599] fi; [13818.403599] exit $rc [13819.009748] Lustre: DEBUG MARKER: dmesg [13819.572740] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 11:29:04 \(1491564544\) [13819.731658] Lustre: DEBUG MARKER: == sanity-hsm test 24b: root can archive, release, and restore user files ============================ 11:29:04 (1491564544) [13820.928388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13821.187313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13822.446600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13823.712085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13823.996570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13824.256698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13825.524329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13826.789776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x20:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13831.702996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13831.959473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13832.727381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13832.984283] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13833.759477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13834.032967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13835.289188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13836.542973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13837.799229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13839.056837] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13839.321707] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13840.393697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13840.654702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13840.912291] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13841.935040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13842.190028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13842.966162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13843.220930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13844.547863] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13845.468030] Lustre: DEBUG MARKER: rc=0; [13845.468030] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13845.468030] if [[ $? -eq 0 && $val -ne 0 ]]; then [13845.468030] echo $(hostname -s): $val; [13845.468030] rc=$val; [13845.468030] fi; [13845.468030] exit $rc [13846.067719] Lustre: DEBUG MARKER: dmesg [13846.655709] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24c: check that user,group,other request masks work =============================== 11:29:31 \(1491564571\) [13846.805605] Lustre: DEBUG MARKER: == sanity-hsm test 24c: check that user,group,other request masks work =============================== 11:29:31 (1491564571) [13847.694256] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.user_request_mask [13847.953502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.group_request_mask [13848.209189] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.other_request_mask [13848.748951] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=archive [13849.257619] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.user_request_mask=archive [13850.007546] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=archive [13850.519403] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.group_request_mask=archive [13851.253750] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=archive [13851.768395] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.other_request_mask=archive [13854.855337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13855.112092] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13855.893803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13856.148441] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13856.918641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13857.173107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13858.434312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13859.698788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13860.969784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13862.238394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13862.496165] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13863.530210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13863.804764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13864.064638] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13865.111633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13865.366639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13866.131896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13866.385695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13867.650735] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.user_request_mask=RESTORE [13868.163089] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.user_request_mask=RESTORE [13868.684349] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.group_request_mask=RESTORE [13869.194162] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.group_request_mask=RESTORE [13869.719542] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.other_request_mask=RESTORE [13870.228243] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.other_request_mask=RESTORE [13870.791720] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13871.708344] Lustre: DEBUG MARKER: rc=0; [13871.708344] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13871.708344] if [[ $? -eq 0 && $val -ne 0 ]]; then [13871.708344] echo $(hostname -s): $val; [13871.708344] rc=$val; [13871.708344] fi; [13871.708344] exit $rc [13872.313839] Lustre: DEBUG MARKER: dmesg [13872.857572] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 11:29:57 \(1491564597\) [13873.017659] Lustre: DEBUG MARKER: == sanity-hsm test 24d: check that read-only mounts are respected ==================================== 11:29:57 (1491564597) [13874.377542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13874.533030] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [13875.464914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13875.722964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13876.983283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13878.246992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13878.522016] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13878.779544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13880.037263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13881.295547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x24:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [13886.223851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13886.482452] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13887.251232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13887.510448] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13888.287320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13888.544261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13889.805787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13891.059129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13892.315107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13893.569948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13893.828458] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13894.855676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13895.110542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13895.365762] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13896.388621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13896.643645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13897.421260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13897.676477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13899.839916] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13900.771448] Lustre: DEBUG MARKER: rc=0; [13900.771448] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13900.771448] if [[ $? -eq 0 && $val -ne 0 ]]; then [13900.771448] echo $(hostname -s): $val; [13900.771448] rc=$val; [13900.771448] fi; [13900.771448] exit $rc [13901.389353] Lustre: DEBUG MARKER: dmesg [13901.944482] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 11:30:26 \(1491564626\) [13902.101874] Lustre: DEBUG MARKER: == sanity-hsm test 24e: tar succeeds on HSM released files =========================================== 11:30:26 (1491564626) [13903.204550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13903.462822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13904.727488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13905.992711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x25:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13910.907460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13911.165579] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13911.933432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13912.189591] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13912.967734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13913.241706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13914.500195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13915.763253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13917.019062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13918.273440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13918.527306] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13919.560904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13919.814439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13920.067646] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13921.088829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13921.342942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13922.116109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13922.369975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13923.682730] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13924.567847] Lustre: DEBUG MARKER: rc=0; [13924.567847] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13924.567847] if [[ $? -eq 0 && $val -ne 0 ]]; then [13924.567847] echo $(hostname -s): $val; [13924.567847] rc=$val; [13924.567847] fi; [13924.567847] exit $rc [13925.172093] Lustre: DEBUG MARKER: dmesg [13925.877263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 11:30:50 \(1491564650\) [13926.063339] Lustre: DEBUG MARKER: == sanity-hsm test 24f: root can archive, release, and restore tar files ============================= 11:30:50 (1491564650) [13927.070922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13927.331460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x26:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13930.221773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13930.478651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13931.247234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13931.502102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13932.273486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13932.530142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13933.784546] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13935.039312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13936.300306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13937.563713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13937.837532] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13940.139279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13940.398140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13940.663469] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13941.699823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13941.955323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13942.725341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13942.981082] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13944.303712] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13945.194580] Lustre: DEBUG MARKER: rc=0; [13945.194580] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13945.194580] if [[ $? -eq 0 && $val -ne 0 ]]; then [13945.194580] echo $(hostname -s): $val; [13945.194580] rc=$val; [13945.194580] fi; [13945.194580] exit $rc [13945.794398] Lustre: DEBUG MARKER: dmesg [13946.356566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25a: Restore lost file \(HS_LOST flag\) from import ================================= 11:31:11 \(1491564671\) [13946.513116] Lustre: DEBUG MARKER: == sanity-hsm test 25a: Restore lost file (HS_LOST flag) from import ================================= 11:31:11 (1491564671) [13950.650875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13950.908115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13951.675270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13951.930854] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13952.705439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13952.961351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13954.217128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13955.471046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13956.725574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13957.983119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13958.238717] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13960.531316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13960.785156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13961.042730] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13962.067391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13962.318926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13963.085389] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13963.340614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13964.672038] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13965.554685] Lustre: DEBUG MARKER: rc=0; [13965.554685] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13965.554685] if [[ $? -eq 0 && $val -ne 0 ]]; then [13965.554685] echo $(hostname -s): $val; [13965.554685] rc=$val; [13965.554685] fi; [13965.554685] exit $rc [13966.148659] Lustre: DEBUG MARKER: dmesg [13966.701557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 25b: Restore lost file \(HS_LOST flag\) after release =============================== 11:31:31 \(1491564691\) [13966.854219] Lustre: DEBUG MARKER: == sanity-hsm test 25b: Restore lost file (HS_LOST flag) after release =============================== 11:31:31 (1491564691) [13967.761823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13968.019028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x27:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13970.834861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13971.092863] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [13971.872905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13972.130332] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [13972.902514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13973.154099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13974.412767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13975.667477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13976.927101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13978.185324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13978.440206] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [13981.988965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13982.244467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13982.499937] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [13983.530035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13983.787536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [13984.571354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13984.826139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [13986.153661] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [13987.059348] Lustre: DEBUG MARKER: rc=0; [13987.059348] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [13987.059348] if [[ $? -eq 0 && $val -ne 0 ]]; then [13987.059348] echo $(hostname -s): $val; [13987.059348] rc=$val; [13987.059348] fi; [13987.059348] exit $rc [13987.667082] Lustre: DEBUG MARKER: dmesg [13988.221467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 26: Remove the archive of a valid file ============================================ 11:31:53 \(1491564713\) [13988.375409] Lustre: DEBUG MARKER: == sanity-hsm test 26: Remove the archive of a valid file ============================================ 11:31:53 (1491564713) [13990.704597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13990.964505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13992.222549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13993.486074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13994.755968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13996.020831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13997.281931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13998.541550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [13999.800701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14001.068494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14002.345508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14003.609494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14004.870374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14006.144174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14007.413960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14008.672354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14009.930728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14011.194408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14012.455402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14013.717367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14014.979286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14016.242377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14017.506980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14018.770172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14020.032313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14021.302157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14022.570688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14023.834768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14025.096818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14026.358305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14027.619414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14028.882832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14029.150042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14029.409903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x28:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14032.219922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14032.479185] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14033.270880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14033.528829] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14034.302466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14034.557930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14035.812527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14037.068900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14038.324733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14039.580601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14039.836635] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14043.380504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14043.637296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14043.891946] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14044.913387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14045.169060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14045.944917] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14046.202340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14047.533762] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14048.437922] Lustre: DEBUG MARKER: rc=0; [14048.437922] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14048.437922] if [[ $? -eq 0 && $val -ne 0 ]]; then [14048.437922] echo $(hostname -s): $val; [14048.437922] rc=$val; [14048.437922] fi; [14048.437922] exit $rc [14049.039791] Lustre: DEBUG MARKER: dmesg [14049.601988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27a: Remove the archive of an imported file \(Operation not permitted\) ============= 11:32:54 \(1491564774\) [14049.782455] Lustre: DEBUG MARKER: == sanity-hsm test 27a: Remove the archive of an imported file (Operation not permitted) ============= 11:32:54 (1491564774) [14056.246927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14056.507762] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14057.284194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14057.539508] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14058.304718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14058.561472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14059.823085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14061.078859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14061.335462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14064.902202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14065.169343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14065.426476] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14066.453589] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14066.708509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14067.481299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14067.734648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14069.077739] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14070.016808] Lustre: DEBUG MARKER: rc=0; [14070.016808] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14070.016808] if [[ $? -eq 0 && $val -ne 0 ]]; then [14070.016808] echo $(hostname -s): $val; [14070.016808] rc=$val; [14070.016808] fi; [14070.016808] exit $rc [14070.622332] Lustre: DEBUG MARKER: dmesg [14071.163042] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 27b: Remove the archive of a relased file \(Operation not permitted\) =============== 11:33:15 \(1491564795\) [14071.324529] Lustre: DEBUG MARKER: == sanity-hsm test 27b: Remove the archive of a relased file (Operation not permitted) =============== 11:33:15 (1491564795) [14073.049674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14073.309445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14074.569302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14075.826355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14077.086815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14078.355641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14079.630915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14080.902360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14082.162094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14083.423879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14084.688111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14085.955868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14087.216603] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14088.477793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14089.737870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14091.001862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14092.265046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14093.530981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14094.794295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14096.063253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14097.330788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14098.607028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14099.872182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14101.138753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14102.413064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14103.681131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14104.948477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14106.216604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14107.484526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14108.762582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14110.028818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14111.291930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x29:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14114.126638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14114.384456] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14115.165643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14115.422726] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14116.193823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14116.449333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14117.704860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14118.959280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14120.214920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14121.469875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14121.725470] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14125.261786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14125.520055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14125.779516] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14126.830028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14127.083340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14127.857708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14128.113255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14129.478537] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14130.362884] Lustre: DEBUG MARKER: rc=0; [14130.362884] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14130.362884] if [[ $? -eq 0 && $val -ne 0 ]]; then [14130.362884] echo $(hostname -s): $val; [14130.362884] rc=$val; [14130.362884] fi; [14130.362884] exit $rc [14130.969711] Lustre: DEBUG MARKER: dmesg [14131.504760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 28: Concurrent archive\/file remove ================================================ 11:34:16 \(1491564856\) [14131.663704] Lustre: DEBUG MARKER: == sanity-hsm test 28: Concurrent archive/file remove ================================================ 11:34:16 (1491564856) [14133.632342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14133.891231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14135.155556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14136.419194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14137.684131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14138.951545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14140.218505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14141.489913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14142.750628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14144.010350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14145.270509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14146.529607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14147.790273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14149.054566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14150.315642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14151.574377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14152.836800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14154.099381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14155.365024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14156.631984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14157.894405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14159.161937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14160.428442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14161.688717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14162.951097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14164.210761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14165.473381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14166.736863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14168.004246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14169.266776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14170.527849] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14170.789986] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [14171.303195] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [14171.822304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14172.077379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14172.850358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14173.115057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14173.889855] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14174.400613] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14174.917594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14175.173976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14175.942064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14176.196674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14176.971553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14177.228086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2a:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14180.027669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14180.283953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14181.054455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14181.309096] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14182.081086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14182.338597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14183.593511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14184.847624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14185.103798] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14186.135906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14186.389833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14186.646405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14187.674049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14187.937527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14188.701883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14188.957203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14190.272388] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14191.159828] Lustre: DEBUG MARKER: rc=0; [14191.159828] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14191.159828] if [[ $? -eq 0 && $val -ne 0 ]]; then [14191.159828] echo $(hostname -s): $val; [14191.159828] rc=$val; [14191.159828] fi; [14191.159828] exit $rc [14191.765206] Lustre: DEBUG MARKER: dmesg [14192.300956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 11:35:17 \(1491564917\) [14192.453726] Lustre: DEBUG MARKER: == sanity-hsm test 29a: Tests --mntpath and --archive options ======================================== 11:35:17 (1491564917) [14195.887512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14196.143256] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14196.921195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14197.181473] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14197.951824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14198.209111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14199.462951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14200.719416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14201.977890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14203.231591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14204.490522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14205.747834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14206.002678] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14207.026122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14207.281193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14207.534980] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14208.562063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14208.817547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14209.597452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14209.853052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14211.168867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14212.053769] Lustre: DEBUG MARKER: rc=0; [14212.053769] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14212.053769] if [[ $? -eq 0 && $val -ne 0 ]]; then [14212.053769] echo $(hostname -s): $val; [14212.053769] rc=$val; [14212.053769] fi; [14212.053769] exit $rc [14212.655652] Lustre: DEBUG MARKER: dmesg [14213.199164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29b: Archive\/delete\/remove by FID from the archive. =============================== 11:35:38 \(1491564938\) [14213.353117] Lustre: DEBUG MARKER: == sanity-hsm test 29b: Archive/delete/remove by FID from the archive. =============================== 11:35:38 (1491564938) [14214.494845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14214.753506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14216.017915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14217.279374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14217.544365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14217.802796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2b:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14220.589145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14220.851140] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14221.625465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14221.883069] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14222.662586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14222.917207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14224.171026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14225.431071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14226.684299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14227.939634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14228.197461] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14229.226674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14229.479637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14229.734981] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14230.761598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14231.026206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14231.799891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14232.073791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14233.400272] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14234.304197] Lustre: DEBUG MARKER: rc=0; [14234.304197] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14234.304197] if [[ $? -eq 0 && $val -ne 0 ]]; then [14234.304197] echo $(hostname -s): $val; [14234.304197] rc=$val; [14234.304197] fi; [14234.304197] exit $rc [14234.913212] Lustre: DEBUG MARKER: dmesg [14235.484881] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29c: Archive\/delete\/remove by FID, using a file list. ============================= 11:36:00 \(1491564960\) [14235.653112] Lustre: DEBUG MARKER: == sanity-hsm test 29c: Archive/delete/remove by FID, using a file list. ============================= 11:36:00 (1491564960) [14237.216816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14237.476235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14238.735667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14239.999303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14240.259821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14240.517971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14240.776134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14241.033196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14241.306063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14241.563963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2c:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14241.822601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14242.079865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2d:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14242.341936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14242.600465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2e:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [14245.395244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14245.652921] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14246.440033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14246.700428] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14247.477420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14247.752648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14249.008289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14250.268907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14251.524610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14251.779737] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14252.814241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14253.071105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14253.327766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14254.391684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14254.651691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14255.419568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14255.689118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14257.008518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14257.925177] Lustre: DEBUG MARKER: rc=0; [14257.925177] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14257.925177] if [[ $? -eq 0 && $val -ne 0 ]]; then [14257.925177] echo $(hostname -s): $val; [14257.925177] rc=$val; [14257.925177] fi; [14257.925177] exit $rc [14258.528556] Lustre: DEBUG MARKER: dmesg [14259.100396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 11:36:23 \(1491564983\) [14259.262608] Lustre: DEBUG MARKER: == sanity-hsm test 29d: hsm_remove by FID with archive_id 0 for unlinked file cause ================== 11:36:23 (1491564983) [14259.450389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [14259.614431] Lustre: DEBUG MARKER: SKIP: sanity-hsm test_29d Need 3 or more clients, have 2 [14259.812125] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14260.722019] Lustre: DEBUG MARKER: rc=0; [14260.722019] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14260.722019] if [[ $? -eq 0 && $val -ne 0 ]]; then [14260.722019] echo $(hostname -s): $val; [14260.722019] rc=$val; [14260.722019] fi; [14260.722019] exit $rc [14261.319662] Lustre: DEBUG MARKER: dmesg [14261.894560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30a: Restore at exec \(import case\) ================================================ 11:36:26 \(1491564986\) [14262.050672] Lustre: DEBUG MARKER: == sanity-hsm test 30a: Restore at exec (import case) ================================================ 11:36:26 (1491564986) [14263.668326] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [14264.180968] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [14265.044925] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [14265.560243] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [14268.618139] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14268.875892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14269.656886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14269.923138] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14270.702957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14270.961487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14272.215968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14273.473027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14274.731734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14275.988064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14276.244196] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14277.273954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14277.529246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14277.784899] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14278.818271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14279.071913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14279.844523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14280.099923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14281.418116] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14282.337406] Lustre: DEBUG MARKER: rc=0; [14282.337406] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14282.337406] if [[ $? -eq 0 && $val -ne 0 ]]; then [14282.337406] echo $(hostname -s): $val; [14282.337406] rc=$val; [14282.337406] fi; [14282.337406] exit $rc [14282.938441] Lustre: DEBUG MARKER: dmesg [14283.499770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30b: Restore at exec \(release case\) =============================================== 11:36:48 \(1491565008\) [14283.655928] Lustre: DEBUG MARKER: == sanity-hsm test 30b: Restore at exec (release case) =============================================== 11:36:48 (1491565008) [14284.604509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14284.861218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x2f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14285.133178] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [14285.649965] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [14286.533919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [14287.048279] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [14290.105722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14290.363502] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14291.136103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14291.392419] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14292.162988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14292.417693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14293.671185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14294.927128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14296.181916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14297.436028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14297.694077] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14298.742072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14298.999025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14299.253543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14300.298535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14300.570091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14301.352640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14301.610190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14302.966115] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14303.863195] Lustre: DEBUG MARKER: rc=0; [14303.863195] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14303.863195] if [[ $? -eq 0 && $val -ne 0 ]]; then [14303.863195] echo $(hostname -s): $val; [14303.863195] rc=$val; [14303.863195] fi; [14303.863195] exit $rc [14304.461791] Lustre: DEBUG MARKER: dmesg [14305.026396] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 30c: Update during exec of released file must fail ================================ 11:37:09 \(1491565029\) [14305.192236] Lustre: DEBUG MARKER: == sanity-hsm test 30c: Update during exec of released file must fail ================================ 11:37:09 (1491565029) [14306.115945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14306.373485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x30:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14306.649184] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [14307.175482] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [14318.061884] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [14318.572557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [14321.616058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14321.877892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14322.653507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14322.908934] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14323.674668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14323.930672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14325.190600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14326.446967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14327.702212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14328.955697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14329.211397] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14330.244054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14330.497576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14330.753070] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14331.777307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14332.031529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14332.798116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14333.051671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14334.368405] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14335.276135] Lustre: DEBUG MARKER: rc=0; [14335.276135] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14335.276135] if [[ $? -eq 0 && $val -ne 0 ]]; then [14335.276135] echo $(hostname -s): $val; [14335.276135] rc=$val; [14335.276135] fi; [14335.276135] exit $rc [14335.900462] Lustre: DEBUG MARKER: dmesg [14336.452348] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31a: Import a large file and check size during restore ============================ 11:37:41 \(1491565061\) [14336.613229] Lustre: DEBUG MARKER: == sanity-hsm test 31a: Import a large file and check size during restore ============================ 11:37:41 (1491565061) [14340.907985] Lustre: HSM agent ba976f76-f68d-3cdb-16f3-355a9f831a1e already registered [14340.937364] LustreError: 8011:0:(mdt_coordinator.c:1405:mdt_hsm_update_request_state()) lustre-MDT0000: Progress on [0x200000406:0x82:0x0] for cookie 0x58e779ff does not match request FID [0x200000407:0xd3:0x0] nor data FID [0x200000406:0x81:0x0] [14383.626667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14383.881894] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14384.653200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14384.907909] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14385.677588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14385.930429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14387.184420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14388.437250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14389.696408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14390.956515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14392.209548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14392.463603] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14393.484205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14393.737434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14393.992565] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14395.021301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14395.277170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14396.043778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14396.297957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14397.620542] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14398.512762] Lustre: DEBUG MARKER: rc=0; [14398.512762] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14398.512762] if [[ $? -eq 0 && $val -ne 0 ]]; then [14398.512762] echo $(hostname -s): $val; [14398.512762] rc=$val; [14398.512762] fi; [14398.512762] exit $rc [14399.122426] Lustre: DEBUG MARKER: dmesg [14399.673556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 11:38:44 \(1491565124\) [14399.845256] Lustre: DEBUG MARKER: == sanity-hsm test 31b: Restore a large unaligned file and check size during restore ================= 11:38:44 (1491565124) [14401.604448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14401.862455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14403.120708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14404.377501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14405.638948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14406.908263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14408.168172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14409.428073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14410.685552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14411.959173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14413.220347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14414.486457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14415.762028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14417.047509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14418.314806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14419.584183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14420.858987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14422.128285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14423.393157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14424.653601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14425.913950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14427.175321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14428.437776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14429.707799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14430.979338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14432.243480] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14433.513497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14434.782100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14436.052853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14437.329570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14438.592613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x31:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14481.480148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14481.738765] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14482.523997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14482.782763] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14483.550883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14483.805915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14485.061762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14486.315995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14486.571445] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14487.598507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14487.854122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14488.109452] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14489.140939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14489.394489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14490.163737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14490.418760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14491.747633] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14492.649487] Lustre: DEBUG MARKER: rc=0; [14492.649487] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14492.649487] if [[ $? -eq 0 && $val -ne 0 ]]; then [14492.649487] echo $(hostname -s): $val; [14492.649487] rc=$val; [14492.649487] fi; [14492.649487] exit $rc [14493.261321] Lustre: DEBUG MARKER: dmesg [14493.906095] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 11:40:18 \(1491565218\) [14494.059962] Lustre: DEBUG MARKER: == sanity-hsm test 31c: Restore a large aligned file and check size during restore =================== 11:40:18 (1491565218) [14495.651926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14495.913961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14497.179101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14498.455044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14499.720769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14500.990381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14502.252506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14503.511808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14504.773833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14506.043098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14507.309517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14508.566588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14509.822782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14511.093446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14512.357132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14513.615920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14514.878227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14516.143242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14517.407788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14518.678501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14519.941822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14521.220622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14522.487067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14523.748809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14525.011743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14526.274424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14527.543165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14528.813913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x32:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14571.690765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14571.954959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14572.737858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14572.994119] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14573.764421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14574.023259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14574.279532] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14579.102149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14579.360394] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14579.616373] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14580.652293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14580.908810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14581.706760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14581.961499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14583.303370] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14584.204160] Lustre: DEBUG MARKER: rc=0; [14584.204160] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14584.204160] if [[ $? -eq 0 && $val -ne 0 ]]; then [14584.204160] echo $(hostname -s): $val; [14584.204160] rc=$val; [14584.204160] fi; [14584.204160] exit $rc [14584.818291] Lustre: DEBUG MARKER: dmesg [14585.363610] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 33: Kill a restore waiting process ================================================ 11:41:50 \(1491565310\) [14585.525406] Lustre: DEBUG MARKER: == sanity-hsm test 33: Kill a restore waiting process ================================================ 11:41:50 (1491565310) [14588.687174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14588.947062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14590.210534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14591.478541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14592.748484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14594.007947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14595.274925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14596.542958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14597.804809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14599.067530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14600.338365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14601.599329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14602.860560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14604.120773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14605.379517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14606.638717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14607.901502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14609.166708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14610.445899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14611.709685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14612.979798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14614.255646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14615.520523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14616.782965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14618.044735] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14619.311958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14620.575089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14621.842828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14623.106188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14624.365351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14625.627328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14625.917409] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [14626.437470] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [14626.957607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14627.227757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14628.016883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14628.274152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14629.047921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [14639.306683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [14639.564893] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=100 [14640.081534] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=100 [14640.606606] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14640.865668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14642.145799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14642.408194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14643.670451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14644.934619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14646.201369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14647.472128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14648.739949] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14650.007520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14651.270935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14652.532913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14653.798288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14655.065183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14656.335449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14657.595652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14658.858801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14660.121197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14661.383412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14662.644195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14663.907764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14665.170029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14666.432324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14667.703262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14668.972159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14670.237529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14671.496622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep '0x200000405:0x33:0x0' | egrep 'WAITING|STARTED' [14671.760604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14672.026853] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x33:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [14672.290414] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [14672.816208] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [14675.857203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14676.114130] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14676.891639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14677.150177] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14677.920760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14678.179290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14679.436292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14680.689734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14680.948600] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14681.971287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14682.227213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14682.484492] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14683.549365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14683.806627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14684.578969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14684.833442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14686.160146] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14687.076457] Lustre: DEBUG MARKER: rc=0; [14687.076457] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14687.076457] if [[ $? -eq 0 && $val -ne 0 ]]; then [14687.076457] echo $(hostname -s): $val; [14687.076457] rc=$val; [14687.076457] fi; [14687.076457] exit $rc [14687.681588] Lustre: DEBUG MARKER: dmesg [14688.243725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 34: Remove file during restore ==================================================== 11:43:33 \(1491565413\) [14688.395267] Lustre: DEBUG MARKER: == sanity-hsm test 34: Remove file during restore ==================================================== 11:43:33 (1491565413) [14690.598058] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14690.856197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14692.114927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14693.373821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14694.634272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14695.894768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14697.153789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14698.412762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14699.673476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14700.938318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14702.207090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14703.475501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14704.736439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14706.004290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14707.270203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14708.533755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14709.795688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14711.057345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14712.323785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14713.594373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14714.865783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14716.130222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14717.391046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14718.649847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14719.912303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14721.181934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14722.460066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14723.722654] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14724.982951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14726.251064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14727.515760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14727.804937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14728.065773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14728.334501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14728.593592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14728.855813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14729.116750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14730.379670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14731.645175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14732.914659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14734.177592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14735.445701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14736.707225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14737.972379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14739.233731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14740.495796] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14741.762416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14743.026171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14744.289279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14745.550710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14746.816577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14748.085888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14749.358334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14750.623507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14751.890210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14753.159102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14754.427706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14755.696078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14756.960912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14758.221954] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14759.482387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14760.745469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14762.010586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14763.271794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14764.544153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x34:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14767.379994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14767.640994] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14768.411135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14768.667959] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14769.437090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14769.692632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14770.948674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14772.204615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14773.457922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14774.713644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14774.969848] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14776.001929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14776.258559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14776.515292] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14777.573445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14777.831703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14778.598959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14778.853461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14780.192234] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14781.114115] Lustre: DEBUG MARKER: rc=0; [14781.114115] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14781.114115] if [[ $? -eq 0 && $val -ne 0 ]]; then [14781.114115] echo $(hostname -s): $val; [14781.114115] rc=$val; [14781.114115] fi; [14781.114115] exit $rc [14781.751322] Lustre: DEBUG MARKER: dmesg [14782.322705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 35: Overwrite file during restore ================================================= 11:45:07 \(1491565507\) [14782.481622] Lustre: DEBUG MARKER: == sanity-hsm test 35: Overwrite file during restore ================================================= 11:45:07 (1491565507) [14784.293094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14784.552364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14785.811077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14787.072181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14788.338238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14789.609764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14790.870291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14792.129300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14793.390242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14794.650412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14795.910891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14797.171739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14798.431770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14799.692873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14800.960599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14802.219844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14803.480227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14804.740115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14805.999752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14807.261767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14808.525521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14809.786285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14811.044925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14812.306188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14813.569056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14814.838296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14816.102636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14817.363718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14818.627599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14819.890334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14821.152645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14821.447173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14821.708191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14821.977930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14822.239576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14822.518406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14822.778516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14824.038029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14825.299609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14826.564043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14827.827675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14829.090636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14830.352403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14831.617163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14832.879681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14834.143199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14835.404991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14836.673592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14837.943571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14839.215729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14840.481527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14841.744859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14843.017571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14844.289227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14845.548605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14846.812921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14848.088337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14849.356941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14850.620055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14851.881557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14853.142950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14854.404069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14855.665319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14856.933446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14858.198097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14859.467231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x35:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14862.325834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14862.587842] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14863.368597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14863.624474] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14864.397913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14864.657086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14865.911873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14867.169381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14868.428279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14869.683257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14869.939649] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14870.972755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14871.228319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14871.485115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14872.516881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14872.770384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14873.540576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14873.796365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14875.143690] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14876.036763] Lustre: DEBUG MARKER: rc=0; [14876.036763] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14876.036763] if [[ $? -eq 0 && $val -ne 0 ]]; then [14876.036763] echo $(hostname -s): $val; [14876.036763] rc=$val; [14876.036763] fi; [14876.036763] exit $rc [14876.649125] Lustre: DEBUG MARKER: dmesg [14877.195286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 36: Move file during restore ====================================================== 11:46:42 \(1491565602\) [14877.352449] Lustre: DEBUG MARKER: == sanity-hsm test 36: Move file during restore ====================================================== 11:46:42 (1491565602) [14879.722325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14879.982812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14881.242112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14882.502950] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14883.766904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14885.029470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14886.288461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14887.548978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14888.809243] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14890.076497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14891.341102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14892.599052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14893.861801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14895.128655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14896.390501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14897.656761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14898.921993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14900.186920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14901.451325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14902.713163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14903.978858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14905.242087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14906.504397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14907.764733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14909.028336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14910.297547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14911.565376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14912.825535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14914.084560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14915.350225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14916.619274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14916.925965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14917.189705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14917.459041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14917.721038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14917.988348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14918.249549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14919.512859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14920.775498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14922.037944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14923.298402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14924.559692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14925.821804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14927.086457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14928.347179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14929.608421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14930.870281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14932.132660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14933.393554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14934.655377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14935.932483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14937.196758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14938.475099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14939.738919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14941.004541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14942.276216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14943.544286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14944.803946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14946.066307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14947.332887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14948.605841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14949.869288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14951.134255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14952.395897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14953.655210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14954.929808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x37:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [14957.785399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14958.058704] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14958.840826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14959.096940] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14959.886655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14960.145045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14961.408490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14962.671282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14963.933441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14965.190112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14965.448301] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14966.481583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14966.737178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14966.992576] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14968.020453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14968.276461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14969.049773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14969.302813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14970.658101] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [14971.585893] Lustre: DEBUG MARKER: rc=0; [14971.585893] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [14971.585893] if [[ $? -eq 0 && $val -ne 0 ]]; then [14971.585893] echo $(hostname -s): $val; [14971.585893] rc=$val; [14971.585893] fi; [14971.585893] exit $rc [14972.193064] Lustre: DEBUG MARKER: dmesg [14972.753649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 37: re-archive a dirty file ======================================================= 11:48:17 \(1491565697\) [14972.916014] Lustre: DEBUG MARKER: == sanity-hsm test 37: re-archive a dirty file ======================================================= 11:48:17 (1491565697) [14975.600793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14975.858380] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [14976.626659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14976.882336] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [14977.649181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14977.903652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14979.157347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14980.411942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14981.666233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14982.918629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14984.172306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14985.434700] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14986.688320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14986.942167] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [14987.969744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14988.225242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14988.477922] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [14989.500961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14989.754462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [14990.522602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14990.775538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [14993.021210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14993.296879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14994.569611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14995.828708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [14996.097738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15008.522378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15008.797262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15010.062939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x38:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15012.853544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15013.107704] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15013.883098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15014.140880] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15014.908714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15015.164370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15016.420077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15017.675432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15018.929094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15020.184894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15020.441345] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15021.462408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15021.718471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15021.972967] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15023.006155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15023.261945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15024.049694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15024.301913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15025.631654] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15026.546195] Lustre: DEBUG MARKER: rc=0; [15026.546195] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15026.546195] if [[ $? -eq 0 && $val -ne 0 ]]; then [15026.546195] echo $(hostname -s): $val; [15026.546195] rc=$val; [15026.546195] fi; [15026.546195] exit $rc [15027.171654] Lustre: DEBUG MARKER: dmesg [15027.756569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 40: Parallel archive requests ===================================================== 11:49:12 \(1491565752\) [15027.918039] Lustre: DEBUG MARKER: == sanity-hsm test 40: Parallel archive requests ===================================================== 11:49:12 (1491565752) [15033.406288] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [15033.928234] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [15034.442325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15034.694887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15035.459273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15035.715724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15036.481115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15047.751311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15048.103995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15049.412551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15050.696227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15053.494774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15053.759100] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15054.556601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15054.814122] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15055.582430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15055.837216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15057.090800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15058.350249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15059.616685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15059.890336] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15062.193054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15062.457870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15062.713022] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15063.741701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15063.994942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15064.767558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15065.020193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15066.415472] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15067.308130] Lustre: DEBUG MARKER: rc=0; [15067.308130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15067.308130] if [[ $? -eq 0 && $val -ne 0 ]]; then [15067.308130] echo $(hostname -s): $val; [15067.308130] rc=$val; [15067.308130] fi; [15067.308130] exit $rc [15067.922229] Lustre: DEBUG MARKER: dmesg [15068.481951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 11:49:53 \(1491565793\) [15068.647541] Lustre: DEBUG MARKER: == sanity-hsm test 52: Opened for write file on an evicted client should be set dirty ================ 11:49:53 (1491565793) [15069.548683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15069.803711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000409:0x34:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15070.074611] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 0430c6b8-19e5-0b04-5065-c582ed85269c [15070.203520] Lustre: 25727:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0430c6b8-19e5-0b04-5065-c582ed85269c at adminstrative request [15071.347557] Lustre: lustre-MDT0000: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [15071.349736] Lustre: Skipped 4 previous similar messages [15075.164125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15075.420484] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15076.198987] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15076.456434] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15077.236586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15077.493994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15078.747571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15080.007883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15080.264322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15082.559730] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15082.814155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15083.069777] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15084.089966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15084.341980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15085.116925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15085.373527] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15086.692509] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15087.609582] Lustre: DEBUG MARKER: rc=0; [15087.609582] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15087.609582] if [[ $? -eq 0 && $val -ne 0 ]]; then [15087.609582] echo $(hostname -s): $val; [15087.609582] rc=$val; [15087.609582] fi; [15087.609582] exit $rc [15088.226362] Lustre: DEBUG MARKER: dmesg [15088.771251] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 11:50:13 \(1491565813\) [15088.924951] Lustre: DEBUG MARKER: == sanity-hsm test 53: Opened for read file on an evicted client should not be set dirty ============= 11:50:13 (1491565813) [15089.832687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040b:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15090.088137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040b:0x2:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15090.360797] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.evict_client 0430c6b8-19e5-0b04-5065-c582ed85269c [15090.491285] Lustre: 27417:0:(genops.c:1532:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0430c6b8-19e5-0b04-5065-c582ed85269c at adminstrative request [15095.443456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15095.700321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15096.474324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15096.729581] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15097.505691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15097.758649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15099.013977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15100.268978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15100.524087] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15102.857598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15103.112234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15103.365845] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15104.394889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15104.648359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15105.440746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15105.695822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15107.022364] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15107.949504] Lustre: DEBUG MARKER: rc=0; [15107.949504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15107.949504] if [[ $? -eq 0 && $val -ne 0 ]]; then [15107.949504] echo $(hostname -s): $val; [15107.949504] rc=$val; [15107.949504] fi; [15107.949504] exit $rc [15108.570443] Lustre: DEBUG MARKER: dmesg [15109.145641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 54: Write during an archive cancels it ============================================ 11:50:33 \(1491565833\) [15109.304075] Lustre: DEBUG MARKER: == sanity-hsm test 54: Write during an archive cancels it ============================================ 11:50:33 (1491565833) [15111.521942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15111.784351] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15112.050277] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15112.583600] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15113.116685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15113.374982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15114.635749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15115.896369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15117.156120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15118.415400] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15119.674910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15120.940946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15122.199295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15123.466657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15124.732967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15126.003323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15127.278440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15128.555761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15129.819471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15131.082390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15132.343473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15133.604326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15134.865679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15136.138984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15137.400894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15138.660327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15139.920754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15141.183999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15142.446359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15143.707724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15144.971206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15146.233740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15147.496774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15148.761451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1c9:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15149.029586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15149.546566] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15152.600927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15152.858973] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15153.629259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15153.888651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15154.662365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15154.916823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15156.173948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15157.428078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15158.683587] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15158.939036] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15163.731053] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15163.988958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15164.247519] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15165.272598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15165.526078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15166.303360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15166.558026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15167.885959] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15168.796136] Lustre: DEBUG MARKER: rc=0; [15168.796136] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15168.796136] if [[ $? -eq 0 && $val -ne 0 ]]; then [15168.796136] echo $(hostname -s): $val; [15168.796136] rc=$val; [15168.796136] fi; [15168.796136] exit $rc [15169.417807] Lustre: DEBUG MARKER: dmesg [15169.990502] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 11:51:34 \(1491565894\) [15170.159292] Lustre: DEBUG MARKER: == sanity-hsm test 55: Truncate during an archive cancels it ========================================= 11:51:34 (1491565894) [15171.872517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15172.130465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15172.398458] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15172.918992] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15173.493380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15173.751036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15175.010091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ca:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15175.277644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15175.791622] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15178.837604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15179.090663] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15179.867156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15180.124159] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15180.898833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15181.152598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15182.407858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15183.671946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15184.929651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15185.184490] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15186.209287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15186.463825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15186.719322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15187.741507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15187.996569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15188.768854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15189.024851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15190.350891] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15191.256945] Lustre: DEBUG MARKER: rc=0; [15191.256945] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15191.256945] if [[ $? -eq 0 && $val -ne 0 ]]; then [15191.256945] echo $(hostname -s): $val; [15191.256945] rc=$val; [15191.256945] fi; [15191.256945] exit $rc [15191.867299] Lustre: DEBUG MARKER: dmesg [15192.424982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 56: Setattr during an archive is ok =============================================== 11:51:57 \(1491565917\) [15192.586466] Lustre: DEBUG MARKER: == sanity-hsm test 56: Setattr during an archive is ok =============================================== 11:51:57 (1491565917) [15194.577658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15194.836246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15195.117354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15195.380979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15196.645931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15197.905366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15199.166699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15200.425150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15201.688445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15202.951355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15204.218996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15205.480492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15206.745387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15208.011864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15209.273899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15210.540074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15211.801636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15213.073638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15214.342251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15215.612314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15216.880467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15218.151695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15219.421865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15220.699659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15221.964112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15223.242237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15224.504734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15225.765561] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15227.033947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15228.300859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15229.564359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15230.828123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15232.088526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cb:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15234.905484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15235.164363] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15235.955856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15236.214312] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15236.993254] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15237.247857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15238.503514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15239.765847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15241.027512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15242.298670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15242.560841] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15243.595282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15243.853042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15244.108102] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15245.159847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15245.418804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15246.191253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15246.445871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15247.765579] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15248.669956] Lustre: DEBUG MARKER: rc=0; [15248.669956] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15248.669956] if [[ $? -eq 0 && $val -ne 0 ]]; then [15248.669956] echo $(hostname -s): $val; [15248.669956] rc=$val; [15248.669956] fi; [15248.669956] exit $rc [15249.289175] Lustre: DEBUG MARKER: dmesg [15249.865821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 11:52:54 \(1491565974\) [15250.025746] Lustre: DEBUG MARKER: == sanity-hsm test 57: Archive a file with dirty cache on another node =============================== 11:52:54 (1491565974) [15251.617951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15251.895685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15253.165705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15254.424470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15257.282838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15257.551651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd4:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15260.342944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15260.599870] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15261.372077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15261.625352] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15262.392153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15262.648105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15263.903702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15265.158677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15266.413261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15267.667445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15267.921231] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15268.947653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15269.202310] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15269.457892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15270.484733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15270.738309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15271.512934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15271.765881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15273.112785] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15274.006161] Lustre: DEBUG MARKER: rc=0; [15274.006161] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15274.006161] if [[ $? -eq 0 && $val -ne 0 ]]; then [15274.006161] echo $(hostname -s): $val; [15274.006161] rc=$val; [15274.006161] fi; [15274.006161] exit $rc [15274.623228] Lustre: DEBUG MARKER: dmesg [15275.162698] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 11:53:19 \(1491565999\) [15275.324369] Lustre: DEBUG MARKER: == sanity-hsm test 58: Truncate a released file will trigger restore ================================= 11:53:19 (1491565999) [15276.298710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15276.559150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15276.962488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15277.293811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15277.571261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15277.904015] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1cd:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15278.233838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15278.493249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15278.853939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1ce:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15281.676077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15281.932898] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15282.711366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15282.976985] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15283.751926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15284.008887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15285.262105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15286.515347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15287.771373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15289.027753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15289.284962] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15290.318280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15290.571530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15290.827496] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15291.871466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15292.128692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15292.906529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15293.162003] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15294.482977] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15295.418782] Lustre: DEBUG MARKER: rc=0; [15295.418782] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15295.418782] if [[ $? -eq 0 && $val -ne 0 ]]; then [15295.418782] echo $(hostname -s): $val; [15295.418782] rc=$val; [15295.418782] fi; [15295.418782] exit $rc [15296.032545] Lustre: DEBUG MARKER: dmesg [15296.632094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 11:53:41 \(1491566021\) [15296.789930] Lustre: DEBUG MARKER: == sanity-hsm test 59: Release stripeless file with non-zero size ==================================== 11:53:41 (1491566021) [15296.932669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [15296.932669] /usr/sbin/lctl lustre_build_version 2>/dev/null || [15296.932669] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [15297.983401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15298.239757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0xc:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15301.041999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15301.295141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15302.073762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15302.329166] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15303.101421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15303.358314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15304.611598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15305.865321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15307.120153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15308.375331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15308.631043] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15310.927229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15311.181641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15311.437728] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15312.470819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15312.724748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15313.496286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15313.748585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15315.074613] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15315.979983] Lustre: DEBUG MARKER: rc=0; [15315.979983] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15315.979983] if [[ $? -eq 0 && $val -ne 0 ]]; then [15315.979983] echo $(hostname -s): $val; [15315.979983] rc=$val; [15315.979983] fi; [15315.979983] exit $rc [15316.593724] Lustre: DEBUG MARKER: dmesg [15317.152417] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 60: Changing progress update interval from default ================================ 11:54:01 \(1491566041\) [15317.301673] Lustre: DEBUG MARKER: == sanity-hsm test 60: Changing progress update interval from default ================================ 11:54:01 (1491566041) [15319.989005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15320.245477] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15321.012685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15321.269492] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15322.043067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15322.307391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15323.562051] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15324.816017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15326.069901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15327.322123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15328.582875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15329.839195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15330.096321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15332.383820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15332.643750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15332.897544] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15333.936644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15334.196804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15334.982639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15335.237455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15337.558197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [15337.832557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [15338.611409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [15339.874354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [15341.134866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [15342.397576] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [15343.659050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_requests | awk '/'0x200000405:0x1cf:0x0'.*action=ARCHIVE/ {print $12}' | cut -f2 -d= [15349.385279] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15349.926143] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15352.977920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15353.234043] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15354.016535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15354.276256] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15355.069262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15355.322759] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15356.578625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15357.838021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15359.095191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15359.348580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15362.894469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15363.150553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15363.406808] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15364.441158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15364.701677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15365.475292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15365.731099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15367.067438] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15367.980333] Lustre: DEBUG MARKER: rc=0; [15367.980333] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15367.980333] if [[ $? -eq 0 && $val -ne 0 ]]; then [15367.980333] echo $(hostname -s): $val; [15367.980333] rc=$val; [15367.980333] fi; [15367.980333] exit $rc [15368.590547] Lustre: DEBUG MARKER: dmesg [15369.167748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 70: Copytool logs JSON register\/unregister events to FIFO ========================= 11:54:53 \(1491566093\) [15369.333180] Lustre: DEBUG MARKER: == sanity-hsm test 70: Copytool logs JSON register/unregister events to FIFO ========================= 11:54:53 (1491566093) [15372.007387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15372.265681] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15373.037162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15373.295116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15374.069765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15374.330163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15375.584967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15376.840636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15378.099387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15379.361027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15379.626385] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15383.177092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15383.449238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15383.704778] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15384.727687] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15384.983945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15385.760843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15386.015608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15389.514211] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15390.033879] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15390.545452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [15390.799170] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep -o ^uuid [15393.605745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15393.862937] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15394.636099] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15394.891193] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15395.669516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15395.923441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15397.178057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15398.431999] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15399.686799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15399.941116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15403.514179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15403.768369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15404.023292] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15405.087487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15405.342444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15406.116568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15406.371957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15408.598256] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15409.495410] Lustre: DEBUG MARKER: rc=0; [15409.495410] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15409.495410] if [[ $? -eq 0 && $val -ne 0 ]]; then [15409.495410] echo $(hostname -s): $val; [15409.495410] rc=$val; [15409.495410] fi; [15409.495410] exit $rc [15410.126005] Lustre: DEBUG MARKER: dmesg [15410.749773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 11:55:35 \(1491566135\) [15410.912668] Lustre: DEBUG MARKER: == sanity-hsm test 71: Copytool logs JSON archive events to FIFO ===================================== 11:55:35 (1491566135) [15413.587563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15413.847303] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15414.622813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15414.876939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15415.652673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15415.906967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15417.159855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15418.413233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15419.667557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15420.920409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15421.178117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15424.727775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15424.983316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15425.237875] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15426.265690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15426.522184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15427.293883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15427.564466] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15431.873767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15432.142560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15433.401821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15434.660172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15435.928124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15437.192637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15438.462844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15439.727783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15440.992467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15442.252940] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15443.512712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15444.772220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15446.036681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15447.300145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15448.566312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15449.843417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15451.104501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15452.367661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15453.624778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15454.885794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15456.146596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15457.414776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15458.683080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15459.949868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15461.212026] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15462.494911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15463.758028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15465.018660] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15466.279223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15467.546523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15468.814277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15470.080364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x1d0:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15471.341646] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15471.858605] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15474.906574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15475.162202] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15475.937282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15476.194453] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15476.964169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15477.219077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15478.475966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15479.734167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15480.010575] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15486.067131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15486.340957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15486.612951] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15487.658925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15487.916452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15488.686644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15488.945451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15490.760161] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15491.660226] Lustre: DEBUG MARKER: rc=0; [15491.660226] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15491.660226] if [[ $? -eq 0 && $val -ne 0 ]]; then [15491.660226] echo $(hostname -s): $val; [15491.660226] rc=$val; [15491.660226] fi; [15491.660226] exit $rc [15492.280281] Lustre: DEBUG MARKER: dmesg [15492.819602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 11:56:57 \(1491566217\) [15492.981965] Lustre: DEBUG MARKER: == sanity-hsm test 72: Copytool logs JSON restore events to FIFO ===================================== 11:56:57 (1491566217) [15495.653793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15495.910520] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15496.685818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15496.940321] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15497.703744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15497.957461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15499.215744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15500.484843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15500.742400] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15506.789128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15507.044883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15507.297932] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15508.324506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15508.595914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15509.370332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15509.624093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15516.022521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15516.281948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15517.546126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15518.815278] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15520.077953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15521.344316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15522.610132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15523.884943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15525.146230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15526.405395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15527.664405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15528.925695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15530.191934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15531.451661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd6:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15532.398972] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15532.913907] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15535.951651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15536.207268] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15536.981175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15537.245992] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15538.026893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15538.285912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15539.540716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15540.800812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15542.064446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15542.318939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15547.121593] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15547.395454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15547.655497] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15548.685806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15548.943249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15549.717210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15549.972176] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15551.812276] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15552.718209] Lustre: DEBUG MARKER: rc=0; [15552.718209] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15552.718209] if [[ $? -eq 0 && $val -ne 0 ]]; then [15552.718209] echo $(hostname -s): $val; [15552.718209] rc=$val; [15552.718209] fi; [15552.718209] exit $rc [15553.348499] Lustre: DEBUG MARKER: dmesg [15553.913221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 90: Archive\/restore a file list =================================================== 11:57:58 \(1491566278\) [15554.095350] Lustre: DEBUG MARKER: == sanity-hsm test 90: Archive/restore a file list =================================================== 11:57:58 (1491566278) [15555.822664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [15556.343885] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [15556.862418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15557.119445] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15557.896657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15558.151121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15558.922433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15569.378354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15569.649702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15570.368475] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15570.686291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | egrep 'WAITING|STARTED' [15573.484468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15573.758636] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15574.531288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15574.787220] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15575.558078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15575.813588] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15577.066797] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15578.323677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15579.577465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15580.830343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15581.083957] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15582.118702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15582.374959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15582.632024] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15583.660505] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15583.916049] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15584.691418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15584.944712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15586.286430] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15587.190717] Lustre: DEBUG MARKER: rc=0; [15587.190717] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15587.190717] if [[ $? -eq 0 && $val -ne 0 ]]; then [15587.190717] echo $(hostname -s): $val; [15587.190717] rc=$val; [15587.190717] fi; [15587.190717] exit $rc [15587.823740] Lustre: DEBUG MARKER: dmesg [15588.401213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 100: Set coordinator \/proc tunables =============================================== 11:58:33 \(1491566313\) [15588.571514] Lustre: DEBUG MARKER: == sanity-hsm test 100: Set coordinator /proc tunables =============================================== 11:58:33 (1491566313) [15588.712774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [15588.969195] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=20 [15589.479005] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=20 [15589.990764] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [15590.245134] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=0 [15590.760632] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=0 [15591.272295] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [15591.792653] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=10 [15592.302630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15592.558137] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=20 [15593.076997] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=20 [15593.591355] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.grace_delay [15593.848105] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=0 [15594.355248] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=0 [15594.867629] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.grace_delay=10 [15595.379363] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.grace_delay=10 [15595.890340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [15596.148354] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=7200 [15596.665109] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=7200 [15597.193138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [15597.460104] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=0 [15597.974982] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=0 [15598.492740] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [15599.009411] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=3600 [15599.525163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [15599.782539] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=6 [15600.306050] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=6 [15600.821075] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [15601.078865] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=0 [15601.601677] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=0 [15602.118841] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.max_requests=3 [15602.627903] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.max_requests=3 [15603.141952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [15603.397791] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [15603.915261] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=2 [15604.425125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [15604.681281] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=0 [15605.199766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=0 [15605.723374] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.default_archive_id=1 [15606.235889] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.default_archive_id=1 [15606.811754] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15607.717788] Lustre: DEBUG MARKER: rc=0; [15607.717788] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15607.717788] if [[ $? -eq 0 && $val -ne 0 ]]; then [15607.717788] echo $(hostname -s): $val; [15607.717788] rc=$val; [15607.717788] fi; [15607.717788] exit $rc [15608.349985] Lustre: DEBUG MARKER: dmesg [15608.918341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 102: Verify coordinator control =================================================== 11:58:53 \(1491566333\) [15609.090844] Lustre: DEBUG MARKER: == sanity-hsm test 102: Verify coordinator control =================================================== 11:58:53 (1491566333) [15609.226105] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15609.737042] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15610.250443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15610.506276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15611.276326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15611.530172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15612.307568] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15612.827129] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15613.342868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15613.595658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15614.364915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15614.617514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15615.385830] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15615.896647] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15616.423856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15616.678615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15617.931993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15619.194633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15620.458390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15621.724639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15622.980676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15623.758377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15624.022435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15624.815339] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15625.338481] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15625.860794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15626.117571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15626.892250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15627.145028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15627.979156] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15628.888897] Lustre: DEBUG MARKER: rc=0; [15628.888897] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15628.888897] if [[ $? -eq 0 && $val -ne 0 ]]; then [15628.888897] echo $(hostname -s): $val; [15628.888897] rc=$val; [15628.888897] fi; [15628.888897] exit $rc [15629.494118] Lustre: DEBUG MARKER: dmesg [15630.076810] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 103: Purge all requests =========================================================== 11:59:14 \(1491566354\) [15630.245496] Lustre: DEBUG MARKER: == sanity-hsm test 103: Purge all requests =========================================================== 11:59:14 (1491566354) [15631.417743] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [15631.950108] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [15632.469754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15632.725083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15633.499701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15633.754169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15634.528621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep -v CANCELED | grep -v SUCCEED | grep -v FAILED [15637.326767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15637.582377] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15638.360794] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15638.620777] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15639.402828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15639.658547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15640.912879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15642.166190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15642.422465] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15645.966144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15646.218552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15646.474134] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15647.495923] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15647.749866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15648.525327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15648.781919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15650.111647] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15651.006606] Lustre: DEBUG MARKER: rc=0; [15651.006606] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15651.006606] if [[ $? -eq 0 && $val -ne 0 ]]; then [15651.006606] echo $(hostname -s): $val; [15651.006606] rc=$val; [15651.006606] fi; [15651.006606] exit $rc [15651.626541] Lustre: DEBUG MARKER: dmesg [15652.187454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 104: Copy tool data field ========================================================= 11:59:37 \(1491566377\) [15652.353083] Lustre: DEBUG MARKER: == sanity-hsm test 104: Copy tool data field ========================================================= 11:59:37 (1491566377) [15654.079296] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15654.593435] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15655.108970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15655.364194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15656.154076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15656.408223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15657.184194] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep 0x200000405:0x218:0x0 | cut -f16 -d= [15657.443781] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15657.960067] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15658.473560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15658.729408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15659.495963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15659.756285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15663.115151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15663.388477] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15664.167525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15664.440706] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15665.223057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15665.478808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15666.739723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15668.001523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15668.258116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15669.289280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15669.548706] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15669.801228] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15670.824962] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15671.081219] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15671.862881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15672.117390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15673.445354] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15674.377962] Lustre: DEBUG MARKER: rc=0; [15674.377962] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15674.377962] if [[ $? -eq 0 && $val -ne 0 ]]; then [15674.377962] echo $(hostname -s): $val; [15674.377962] rc=$val; [15674.377962] fi; [15674.377962] exit $rc [15674.987282] Lustre: DEBUG MARKER: dmesg [15675.562302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 105: Restart of coordinator ======================================================= 12:00:00 \(1491566400\) [15675.726562] Lustre: DEBUG MARKER: == sanity-hsm test 105: Restart of coordinator ======================================================= 12:00:00 (1491566400) [15675.861898] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15676.373062] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15676.887361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15677.144133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15677.925936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15678.183362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15679.065038] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [15679.339263] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15679.858116] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15680.388311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15680.641647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15681.896435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15683.151959] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15684.406228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15685.660486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15686.913929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15688.168151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15689.429624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15690.205590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15690.462879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15691.236200] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15691.751759] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15692.272802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15692.527773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15693.291649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15693.545822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15694.321892] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15694.841348] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15695.355733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15695.609103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15696.383417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15696.639978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15697.406363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [15697.664462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15698.178381] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15698.692878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15698.947938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15699.711086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15699.963728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15700.730900] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=purge [15701.241623] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=purge [15701.766652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15702.022095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15702.789663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15703.043812] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15703.877602] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15704.792268] Lustre: DEBUG MARKER: rc=0; [15704.792268] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15704.792268] if [[ $? -eq 0 && $val -ne 0 ]]; then [15704.792268] echo $(hostname -s): $val; [15704.792268] rc=$val; [15704.792268] fi; [15704.792268] exit $rc [15705.401206] Lustre: DEBUG MARKER: dmesg [15705.961779] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 106: Copytool register\/unregister ================================================= 12:00:30 \(1491566430\) [15706.123497] Lustre: DEBUG MARKER: == sanity-hsm test 106: Copytool register/unregister ================================================= 12:00:30 (1491566430) [15707.503985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15707.684535] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15708.325478] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [15708.479268] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [15708.630395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15709.159879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15712.484591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15712.741867] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15713.527778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15713.784524] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15714.552191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15714.806299] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15716.060382] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15717.316967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15718.569581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15718.826500] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15719.861472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15720.118781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15720.378250] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15721.426529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15721.684649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15722.480640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15722.742968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15724.016568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15724.539609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15726.292055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15726.452699] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [15727.075250] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [15727.236708] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [15727.383569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15727.909918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [15730.963316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15731.217929] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15731.999669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15732.259312] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15733.031510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15733.287110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15734.547705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15735.803401] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15737.071471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15738.329085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15739.582621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15739.839541] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15740.874906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15741.139188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15741.396994] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15742.427825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15742.681946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15743.453975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15743.710231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15745.036160] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15745.944074] Lustre: DEBUG MARKER: rc=0; [15745.944074] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15745.944074] if [[ $? -eq 0 && $val -ne 0 ]]; then [15745.944074] echo $(hostname -s): $val; [15745.944074] rc=$val; [15745.944074] fi; [15745.944074] exit $rc [15746.569450] Lustre: DEBUG MARKER: dmesg [15747.117059] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 12:01:11 \(1491566471\) [15747.272412] Lustre: DEBUG MARKER: == sanity-hsm test 107: Copytool re-register after MDS restart ======================================= 12:01:11 (1491566471) [15748.194595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15748.453662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x219:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15748.721236] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [15748.976168] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [15749.106475] Lustre: Failing over lustre-MDT0000 [15749.108275] Lustre: Skipped 7 previous similar messages [15749.192952] Lustre: server umount lustre-MDT0000 complete [15749.194767] Lustre: Skipped 7 previous similar messages [15749.323486] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [15750.232936] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15750.236960] LustreError: Skipped 81 previous similar messages [15750.708223] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [15750.710550] LustreError: Skipped 174539 previous similar messages [15750.712301] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15750.716269] Lustre: Skipped 11 previous similar messages [15757.511039] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491566476/real 1491566476] req@ffff880063291200 x1564004881177968/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491566483 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [15757.517184] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [15757.519408] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15757.523377] LustreError: Skipped 1 previous similar message [15759.644866] Lustre: DEBUG MARKER: hostname [15759.959177] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [15760.213338] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [15760.464742] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [15760.671783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15760.783797] Lustre: MGS: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [15760.786366] Lustre: Skipped 1 previous similar message [15763.525199] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176be72e4 to 0x52ac6ca176c26005 [15763.571775] LustreError: 17715:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [15763.578628] LustreError: 17720:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [15763.581421] LustreError: 17720:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [15763.590124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15763.592759] Lustre: Skipped 7 previous similar messages [15763.744573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [15764.004329] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [15764.112428] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [15764.115163] Lustre: Skipped 9 previous similar messages [15764.510747] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15764.512838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15764.674988] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15764.676650] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [15764.823387] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [15765.081108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15765.339540] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [15765.602167] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [15766.373041] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15766.374984] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15766.592253] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15766.592254] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15768.586556] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [15768.604632] Lustre: lustre-MDT0000: Recovery over after 0:05, of 7 clients 7 recovered and 0 were evicted. [15768.607766] Lustre: Skipped 8 previous similar messages [15768.870737] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [15768.880299] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [15769.082122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [15769.095754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [15770.445228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15770.705352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15773.506831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15773.763189] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15774.541845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15774.797750] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15775.570460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15775.827077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15777.082798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15778.338791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15779.597023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15780.851525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15781.108819] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15782.148800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15782.404077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15782.659507] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15783.694784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15783.953291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15784.730258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15784.987966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15786.319872] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15787.246610] Lustre: DEBUG MARKER: rc=0; [15787.246610] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15787.246610] if [[ $? -eq 0 && $val -ne 0 ]]; then [15787.246610] echo $(hostname -s): $val; [15787.246610] rc=$val; [15787.246610] fi; [15787.246610] exit $rc [15787.865419] Lustre: DEBUG MARKER: dmesg [15788.445897] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 109: Policy display\/change ======================================================== 12:01:53 \(1491566513\) [15788.613296] Lustre: DEBUG MARKER: == sanity-hsm test 109: Policy display/change ======================================================== 12:01:53 (1491566513) [15788.763632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15789.020319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NBR" [15789.276227] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15789.531505] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="+NRA" [15789.786562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15790.042566] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NBR" [15790.300839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15790.556066] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="-NRA" [15790.812288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15791.070410] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="NRA NBR" [15791.325177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15791.580978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15791.835945] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy="wrong" [15791.964554] Lustre: 21543:0:(mdt_coordinator.c:1804:mdt_hsm_policy_seq_write()) lustre-MDT0000: 'wrong' is unknown, supported policies are: [15792.096101] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.policy [15792.353051] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15792.879561] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15793.393608] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [15793.903151] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [15794.487219] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15795.436778] Lustre: DEBUG MARKER: rc=0; [15795.436778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15795.436778] if [[ $? -eq 0 && $val -ne 0 ]]; then [15795.436778] echo $(hostname -s): $val; [15795.436778] rc=$val; [15795.436778] fi; [15795.436778] exit $rc [15796.055310] Lustre: DEBUG MARKER: dmesg [15796.613206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110a: Non blocking restore policy \(import case\) =================================== 12:02:01 \(1491566521\) [15796.795438] Lustre: DEBUG MARKER: == sanity-hsm test 110a: Non blocking restore policy (import case) =================================== 12:02:01 (1491566521) [15798.426578] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [15798.939851] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NBR [15799.461493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15799.719599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd7:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15799.978633] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [15800.494954] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [15803.562554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15803.822564] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15804.594357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15804.852486] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15805.623838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15805.882078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15807.136619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15808.395109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15809.650181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15809.908664] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15813.448200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15813.705096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15813.962215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15814.986511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15815.240753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15816.017226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15816.272356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15817.623378] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15818.534058] Lustre: DEBUG MARKER: rc=0; [15818.534058] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15818.534058] if [[ $? -eq 0 && $val -ne 0 ]]; then [15818.534058] echo $(hostname -s): $val; [15818.534058] rc=$val; [15818.534058] fi; [15818.534058] exit $rc [15819.150459] Lustre: DEBUG MARKER: dmesg [15819.722432] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 110b: Non blocking restore policy \(release case\) ================================== 12:02:24 \(1491566544\) [15819.878559] Lustre: DEBUG MARKER: == sanity-hsm test 110b: Non blocking restore policy (release case) ================================== 12:02:24 (1491566544) [15820.798965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15821.059146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15821.327821] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NBR [15821.852772] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NBR [15822.381441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15822.645322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21b:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15822.905141] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NBR [15823.414791] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NBR [15826.463906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15826.723126] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15827.509713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15827.769115] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15828.545542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15828.820829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15830.090895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15831.356234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15832.614752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15832.868334] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15833.922395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15834.177517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15834.434187] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15835.460041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15835.716081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15836.493333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15836.749881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15838.086644] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15839.012122] Lustre: DEBUG MARKER: rc=0; [15839.012122] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15839.012122] if [[ $? -eq 0 && $val -ne 0 ]]; then [15839.012122] echo $(hostname -s): $val; [15839.012122] rc=$val; [15839.012122] fi; [15839.012122] exit $rc [15839.662107] Lustre: DEBUG MARKER: dmesg [15840.216947] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111a: No retry policy \(import case\), restore will error =========================== 12:02:45 \(1491566565\) [15840.371528] Lustre: DEBUG MARKER: == sanity-hsm test 111a: No retry policy (import case), restore will error =========================== 12:02:45 (1491566565) [15841.988797] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15842.504373] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15843.511073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15843.780263] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd8:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15844.042290] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15844.556849] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15847.620056] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15847.878179] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15848.671225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15848.929963] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15849.719317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15849.974596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15851.231373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15852.492819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15853.748229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15854.005527] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15855.033936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15855.287885] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15855.542205] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15856.568063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15856.823378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15857.593960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15857.849268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15859.177669] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15860.109552] Lustre: DEBUG MARKER: rc=0; [15860.109552] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15860.109552] if [[ $? -eq 0 && $val -ne 0 ]]; then [15860.109552] echo $(hostname -s): $val; [15860.109552] rc=$val; [15860.109552] fi; [15860.109552] exit $rc [15860.721613] Lustre: DEBUG MARKER: dmesg [15861.295425] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 111b: No retry policy \(release case\), restore will error ========================== 12:03:06 \(1491566586\) [15861.460591] Lustre: DEBUG MARKER: == sanity-hsm test 111b: No retry policy (release case), restore will error ========================== 12:03:06 (1491566586) [15862.389197] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=+NRA [15862.903553] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=+NRA [15863.424713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15863.684884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15864.447083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15864.725373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21c:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15864.987186] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm.policy=-NRA [15865.497677] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm.policy=-NRA [15868.567125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15868.826580] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15869.607783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15869.877303] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15870.651541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15870.906787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15872.161350] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15873.418360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15874.673143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15874.926694] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15875.964631] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15876.217496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15876.472766] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15877.528691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15877.783066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15878.549364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15878.804428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15880.145656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15881.062223] Lustre: DEBUG MARKER: rc=0; [15881.062223] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15881.062223] if [[ $? -eq 0 && $val -ne 0 ]]; then [15881.062223] echo $(hostname -s): $val; [15881.062223] rc=$val; [15881.062223] fi; [15881.062223] exit $rc [15881.684223] Lustre: DEBUG MARKER: dmesg [15882.258952] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 112: State of recorded request ==================================================== 12:03:27 \(1491566607\) [15882.419974] Lustre: DEBUG MARKER: == sanity-hsm test 112: State of recorded request ==================================================== 12:03:27 (1491566607) [15883.327645] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15883.838145] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15884.360443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15884.614064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15885.381503] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15885.637699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15886.426586] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15886.944654] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15887.460507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15887.713425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15888.483163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15888.738813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15889.514618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15889.773528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15892.596744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15892.851985] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15893.633180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15893.889903] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15894.658284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15894.909746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15896.164664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15897.418463] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15897.672557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15898.696712] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15898.949474] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15899.205440] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15900.238848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15900.492345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15901.271124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15901.528738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15902.868576] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15903.782395] Lustre: DEBUG MARKER: rc=0; [15903.782395] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15903.782395] if [[ $? -eq 0 && $val -ne 0 ]]; then [15903.782395] echo $(hostname -s): $val; [15903.782395] rc=$val; [15903.782395] fi; [15903.782395] exit $rc [15904.396904] Lustre: DEBUG MARKER: dmesg [15905.021123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 200: Register\/Cancel archive ====================================================== 12:03:49 \(1491566629\) [15905.213502] Lustre: DEBUG MARKER: == sanity-hsm test 200: Register/Cancel archive ====================================================== 12:03:49 (1491566629) [15911.929316] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15912.444051] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15912.955566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15913.208625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15913.978893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15914.234033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15915.010613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15915.266946] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15915.529211] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15916.050876] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15916.562634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15916.818324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15917.587277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15917.842684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15918.613338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15918.873236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15919.129881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [15919.387457] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21e:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [15922.197147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15922.454092] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15923.225195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15923.480588] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15924.259366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15924.533283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15925.804738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15926.067784] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15927.087549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15927.344854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15927.598760] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15928.627208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15928.878782] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15929.646090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15929.900906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15931.220826] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15932.126733] Lustre: DEBUG MARKER: rc=0; [15932.126733] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15932.126733] if [[ $? -eq 0 && $val -ne 0 ]]; then [15932.126733] echo $(hostname -s): $val; [15932.126733] rc=$val; [15932.126733] fi; [15932.126733] exit $rc [15932.740969] Lustre: DEBUG MARKER: dmesg [15933.306674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 201: Register\/Cancel restore ====================================================== 12:04:18 \(1491566658\) [15933.464244] Lustre: DEBUG MARKER: == sanity-hsm test 201: Register/Cancel restore ====================================================== 12:04:18 (1491566658) [15937.396707] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15937.915691] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [15938.436742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15938.692668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15939.473297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15939.729566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15940.521969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15940.786160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15941.057095] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15941.576957] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15942.093410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15942.353196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15943.122080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15943.374771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15944.143040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15944.401453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [15944.662109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [15944.919187] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xd9:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [15947.716181] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15947.971143] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [15948.751863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15949.009386] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [15949.802104] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15950.056136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15951.312368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15951.568799] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [15952.592898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15952.847754] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15953.103592] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [15954.135515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15954.388155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [15955.158074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15955.411813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [15956.743473] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [15957.655120] Lustre: DEBUG MARKER: rc=0; [15957.655120] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [15957.655120] if [[ $? -eq 0 && $val -ne 0 ]]; then [15957.655120] echo $(hostname -s): $val; [15957.655120] rc=$val; [15957.655120] fi; [15957.655120] exit $rc [15958.271686] Lustre: DEBUG MARKER: dmesg [15958.848149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 202: Register\/Cancel remove ======================================================= 12:04:43 \(1491566683\) [15959.007427] Lustre: DEBUG MARKER: == sanity-hsm test 202: Register/Cancel remove ======================================================= 12:04:43 (1491566683) [15960.909160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15961.166542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15962.426296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15963.689293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15964.952902] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15966.214448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15967.480257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15968.740251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15970.000167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15971.265129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15972.531490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15973.792739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15975.052738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15976.310788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15977.571111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15978.829935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15980.092043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15981.359738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15982.626780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15983.900926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15985.169095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15986.429126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15987.688473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15988.949477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15990.215300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15991.479120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15992.745213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15994.018199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15995.286970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15996.550970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15997.812439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15999.077939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [15999.343393] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [15999.868405] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [16000.380383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16000.634383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16001.404461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16001.659547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16002.450575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16002.716762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16002.981601] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16003.509691] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16004.022860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16004.279941] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16005.049169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16005.304282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16006.081781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16006.341553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x21f:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16009.135669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16009.391901] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16010.168530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16010.427088] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16011.208125] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16011.463345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16012.724479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16013.981183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16014.237519] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16015.270865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16015.526890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16015.783607] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16016.815292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16017.071064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16017.841196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16018.096152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16019.432603] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16020.345992] Lustre: DEBUG MARKER: rc=0; [16020.345992] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16020.345992] if [[ $? -eq 0 && $val -ne 0 ]]; then [16020.345992] echo $(hostname -s): $val; [16020.345992] rc=$val; [16020.345992] fi; [16020.345992] exit $rc [16020.961945] Lustre: DEBUG MARKER: dmesg [16021.546192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220: Changelog for archive ======================================================== 12:05:46 \(1491566746\) [16021.711545] Lustre: DEBUG MARKER: == sanity-hsm test 220: Changelog for archive ======================================================== 12:05:46 (1491566746) [16022.620422] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16022.749164] Lustre: lustre-MDD0000: changelog on [16022.877754] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16023.663207] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16023.792467] Lustre: lustre-MDD0002: changelog on [16023.921136] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16024.717282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16024.977069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x220:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16025.246547] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl1 [16025.375860] Lustre: lustre-MDD0000: changelog off [16025.766895] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl1 [16025.896834] Lustre: lustre-MDD0002: changelog off [16028.827237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16029.084470] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16029.858898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16030.113555] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16030.885844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16031.143858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16032.398186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16033.652501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16034.906746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16035.160446] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16036.193332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16036.448133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16036.703715] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16037.735860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16037.989799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16038.770126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16039.042744] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16040.388934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16041.309439] Lustre: DEBUG MARKER: rc=0; [16041.309439] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16041.309439] if [[ $? -eq 0 && $val -ne 0 ]]; then [16041.309439] echo $(hostname -s): $val; [16041.309439] rc=$val; [16041.309439] fi; [16041.309439] exit $rc [16041.918169] Lustre: DEBUG MARKER: dmesg [16042.509488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 220a: Changelog for failed archive ================================================ 12:06:07 \(1491566767\) [16042.666940] Lustre: DEBUG MARKER: == sanity-hsm test 220a: Changelog for failed archive ================================================ 12:06:07 (1491566767) [16043.571923] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16043.704469] Lustre: lustre-MDD0000: changelog on [16043.832835] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16044.621673] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16044.876692] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16046.129941] LustreError: 30207:0:(mdt_coordinator.c:1375:mdt_hsm_update_request_state()) lustre-MDT0000: Cannot find running request for cookie 0x58e780a9 on fid=[0x200000405:0x221:0x0] [16046.261670] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16046.523596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x221:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16046.794909] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl2 [16046.923652] Lustre: lustre-MDD0000: changelog off [16047.323968] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl2 [16050.369137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16050.631919] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16051.408571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16051.663677] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16052.433598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16052.690374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16053.944046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16055.197617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16056.453710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16056.708839] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16057.732834] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16057.987628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16058.244574] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16059.270607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16059.526115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16060.303815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16060.556960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16061.884670] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16062.783797] Lustre: DEBUG MARKER: rc=0; [16062.783797] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16062.783797] if [[ $? -eq 0 && $val -ne 0 ]]; then [16062.783797] echo $(hostname -s): $val; [16062.783797] rc=$val; [16062.783797] fi; [16062.783797] exit $rc [16063.411426] Lustre: DEBUG MARKER: dmesg [16064.008405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 221: Changelog for archive canceled =============================================== 12:06:28 \(1491566788\) [16064.172469] Lustre: DEBUG MARKER: == sanity-hsm test 221: Changelog for archive canceled =============================================== 12:06:28 (1491566788) [16068.187666] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16068.317163] Lustre: lustre-MDD0000: changelog on [16068.318982] Lustre: Skipped 1 previous similar message [16068.448154] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16069.257805] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16069.514468] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16070.305751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16070.566230] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16070.830710] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16071.088364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16071.351645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16071.613621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16072.872398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16074.132277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16075.395868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16076.658217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16077.919370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16079.182192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16080.450501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16081.721363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16082.982666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16084.245143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16085.506196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16086.772740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16088.036542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16089.297088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16090.560276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16091.828228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16093.096074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16094.366795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16095.629380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16096.896693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16098.158737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16099.419905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16100.684316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x222:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16103.505529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16103.766368] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16104.548984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16104.810743] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16105.585915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16105.842926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16107.099253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16108.362073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16109.628086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16109.901525] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16110.955617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16111.211997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16111.469793] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16112.508319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16112.774417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16113.544769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16113.797747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16115.070477] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl3 [16115.202520] Lustre: lustre-MDD0000: changelog off [16115.204862] Lustre: Skipped 1 previous similar message [16115.598148] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl3 [16116.177856] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16117.090712] Lustre: DEBUG MARKER: rc=0; [16117.090712] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16117.090712] if [[ $? -eq 0 && $val -ne 0 ]]; then [16117.090712] echo $(hostname -s): $val; [16117.090712] rc=$val; [16117.090712] fi; [16117.090712] exit $rc [16117.722277] Lustre: DEBUG MARKER: dmesg [16118.293424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222a: Changelog for explicit restore ============================================== 12:07:23 \(1491566843\) [16118.454468] Lustre: DEBUG MARKER: == sanity-hsm test 222a: Changelog for explicit restore ============================================== 12:07:23 (1491566843) [16120.137362] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16120.268178] Lustre: lustre-MDD0000: changelog on [16120.270093] Lustre: Skipped 1 previous similar message [16120.399669] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16121.181113] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16121.439441] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16122.230655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16122.487594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xda:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16125.290437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16125.545493] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16126.319334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16126.574559] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16127.357922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16127.623905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16128.879112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16130.142489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16131.398343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16132.653604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16132.910358] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16133.944449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16134.198540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16134.454062] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16135.493692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16135.749785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16136.543397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16136.796583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16138.078494] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl4 [16138.207408] Lustre: lustre-MDD0000: changelog off [16138.209096] Lustre: Skipped 1 previous similar message [16138.603964] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl4 [16139.188034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16140.095865] Lustre: DEBUG MARKER: rc=0; [16140.095865] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16140.095865] if [[ $? -eq 0 && $val -ne 0 ]]; then [16140.095865] echo $(hostname -s): $val; [16140.095865] rc=$val; [16140.095865] fi; [16140.095865] exit $rc [16140.726122] Lustre: DEBUG MARKER: dmesg [16141.296174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222b: Changelog for implicit restore ============================================== 12:07:46 \(1491566866\) [16141.462188] Lustre: DEBUG MARKER: == sanity-hsm test 222b: Changelog for implicit restore ============================================== 12:07:46 (1491566866) [16142.373479] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16142.503019] Lustre: lustre-MDD0000: changelog on [16142.504705] Lustre: Skipped 1 previous similar message [16142.634521] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16143.420646] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16143.679254] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16144.463233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16144.720633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16145.027320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16145.284287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x223:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16148.091977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16148.348163] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16149.138280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16149.396189] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16150.172577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16150.427984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16151.683331] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16152.939087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16154.193786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16155.449309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16155.706163] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16156.734676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16156.988552] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16157.246545] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16158.280465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16158.536436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16159.308092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16159.562403] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16160.828314] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl5 [16160.957254] Lustre: lustre-MDD0000: changelog off [16160.959034] Lustre: Skipped 1 previous similar message [16161.356305] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl5 [16161.939867] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16162.851323] Lustre: DEBUG MARKER: rc=0; [16162.851323] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16162.851323] if [[ $? -eq 0 && $val -ne 0 ]]; then [16162.851323] echo $(hostname -s): $val; [16162.851323] rc=$val; [16162.851323] fi; [16162.851323] exit $rc [16163.472205] Lustre: DEBUG MARKER: dmesg [16164.061503] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 12:08:08 \(1491566888\) [16164.223308] Lustre: DEBUG MARKER: == sanity-hsm test 222c: Changelog for failed explicit restore ======================================= 12:08:08 (1491566888) [16165.904156] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16166.032756] Lustre: lustre-MDD0000: changelog on [16166.034549] Lustre: Skipped 1 previous similar message [16166.163933] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16166.944619] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16167.202103] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16168.521434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16168.777979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdb:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16171.581758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16171.836389] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16172.607364] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16172.864994] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16173.640988] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16173.894410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16175.155393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16176.410124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16177.663801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16178.927092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16179.194047] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16180.235212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16180.492639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16180.749027] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16181.776573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16182.032185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16182.798919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16183.053078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16184.322462] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl6 [16184.450466] Lustre: lustre-MDD0000: changelog off [16184.452158] Lustre: Skipped 1 previous similar message [16184.853329] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl6 [16185.438660] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16186.375843] Lustre: DEBUG MARKER: rc=0; [16186.375843] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16186.375843] if [[ $? -eq 0 && $val -ne 0 ]]; then [16186.375843] echo $(hostname -s): $val; [16186.375843] rc=$val; [16186.375843] fi; [16186.375843] exit $rc [16186.986688] Lustre: DEBUG MARKER: dmesg [16187.589672] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 12:08:32 \(1491566912\) [16187.749897] Lustre: DEBUG MARKER: == sanity-hsm test 222d: Changelog for failed implicit restore ======================================= 12:08:32 (1491566912) [16188.662331] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16188.923611] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16189.713850] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16189.969917] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16190.759234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16191.018583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16191.786295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16192.046826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x224:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16194.851679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16195.111158] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16195.901816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16196.163209] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16196.943936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16197.202658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16198.459137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16199.712330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16200.966981] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16202.221257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16202.478336] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16203.520857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16203.782896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16204.038931] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16205.085726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16205.356684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16206.165623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16206.421526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16207.702978] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl7 [16208.222161] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl7 [16208.814226] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16209.757841] Lustre: DEBUG MARKER: rc=0; [16209.757841] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16209.757841] if [[ $? -eq 0 && $val -ne 0 ]]; then [16209.757841] echo $(hostname -s): $val; [16209.757841] rc=$val; [16209.757841] fi; [16209.757841] exit $rc [16210.386462] Lustre: DEBUG MARKER: dmesg [16210.954024] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223a: Changelog for restore canceled \(import case\) ================================ 12:08:55 \(1491566935\) [16211.106342] Lustre: DEBUG MARKER: == sanity-hsm test 223a: Changelog for restore canceled (import case) ================================ 12:08:55 (1491566935) [16214.779107] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16214.909044] Lustre: lustre-MDD0000: changelog on [16214.910836] Lustre: Skipped 3 previous similar messages [16215.039171] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16215.822991] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16216.083097] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16217.124789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16217.390468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16217.661701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16217.921166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16218.188185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16218.448427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16219.717871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16220.982668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16222.244577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16223.515472] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16224.776624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16226.039374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16227.309358] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16228.572221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16229.834384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16231.103422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16232.369066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16233.640367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16234.907619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16236.170992] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16237.438802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16238.703998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16239.967798] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16241.231906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16242.495050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16243.762497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16245.026168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16246.289417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16247.550412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000407:0xdc:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16250.373574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16250.630915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16251.410629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16251.668311] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16252.440908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16252.697262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16253.954115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16255.211775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16256.468846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16257.723460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16257.981906] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16259.044325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16259.298615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16259.553304] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16260.605921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16260.877783] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16261.663868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16261.936173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16263.218696] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl8 [16263.352907] Lustre: lustre-MDD0000: changelog off [16263.355183] Lustre: Skipped 3 previous similar messages [16263.752392] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl8 [16264.338239] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16265.259518] Lustre: DEBUG MARKER: rc=0; [16265.259518] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16265.259518] if [[ $? -eq 0 && $val -ne 0 ]]; then [16265.259518] echo $(hostname -s): $val; [16265.259518] rc=$val; [16265.259518] fi; [16265.259518] exit $rc [16265.897383] Lustre: DEBUG MARKER: dmesg [16266.635511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 223b: Changelog for restore canceled \(release case\) =============================== 12:09:51 \(1491566991\) [16266.810642] Lustre: DEBUG MARKER: == sanity-hsm test 223b: Changelog for restore canceled (release case) =============================== 12:09:51 (1491566991) [16268.939395] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16269.197457] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16269.976232] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16270.231857] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16271.016239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16271.278204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16272.538191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16273.803199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16275.065231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16276.327338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16277.597329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16278.859608] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16280.129030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16281.396129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16282.662429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16283.935149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16285.201345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16286.467868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16287.732300] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16288.993669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16290.255890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16291.522697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16292.788737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16294.054464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16295.318126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16296.582203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16297.849857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16299.123341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16300.395689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16301.659251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16302.919066] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16304.179728] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16305.442925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16306.706109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16307.971733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16309.237772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16309.533183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16309.796258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16310.067536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16310.325912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [16310.588893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16310.848096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16312.116699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16313.384701] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16314.647761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16315.911978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16317.181248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16318.453548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16319.732951] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16320.997847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16322.266658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16323.531832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16324.801296] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16326.065779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16327.330643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16328.603160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16329.868714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16331.129967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16332.392659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16333.659761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16334.922596] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16336.189784] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16337.452491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16338.720537] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16340.000226] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x225:0x0'.*action='CANCEL'/ {print $13}' | cut -f2 -d= [16342.825585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16343.086952] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16343.860441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16344.122308] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16344.919399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16345.184953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16346.449978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16347.706359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16348.962446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16350.217548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16350.473184] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16351.510165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16351.765141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16352.019649] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16353.065088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16353.320874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16354.094529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16354.352446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16355.625925] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl9 [16355.756663] Lustre: lustre-MDD0000: changelog off [16355.758967] Lustre: Skipped 1 previous similar message [16356.147493] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl9 [16356.736840] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16357.671557] Lustre: DEBUG MARKER: rc=0; [16357.671557] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16357.671557] if [[ $? -eq 0 && $val -ne 0 ]]; then [16357.671557] echo $(hostname -s): $val; [16357.671557] rc=$val; [16357.671557] fi; [16357.671557] exit $rc [16358.311795] Lustre: DEBUG MARKER: dmesg [16359.012347] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224: Changelog for remove ========================================================= 12:11:23 \(1491567083\) [16359.193060] Lustre: DEBUG MARKER: == sanity-hsm test 224: Changelog for remove ========================================================= 12:11:23 (1491567083) [16360.091867] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16360.222179] Lustre: lustre-MDD0000: changelog on [16360.224032] Lustre: Skipped 3 previous similar messages [16360.354036] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16361.164873] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16361.418688] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16362.202678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16362.466615] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16362.730572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16362.987661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x226:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16365.798538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16366.057564] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16366.842489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16367.102312] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16367.878683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16368.134135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16369.387225] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16370.641367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16371.896650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16373.150456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16373.407380] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16374.441573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16374.696130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16374.951125] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16375.997699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16376.253455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16377.058395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16377.313931] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16378.592300] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl10 [16379.107040] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl10 [16379.689087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16380.598130] Lustre: DEBUG MARKER: rc=0; [16380.598130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16380.598130] if [[ $? -eq 0 && $val -ne 0 ]]; then [16380.598130] echo $(hostname -s): $val; [16380.598130] rc=$val; [16380.598130] fi; [16380.598130] exit $rc [16381.229385] Lustre: DEBUG MARKER: dmesg [16381.811456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 224a: Changelog for failed remove ================================================= 12:11:46 \(1491567106\) [16381.963075] Lustre: DEBUG MARKER: == sanity-hsm test 224a: Changelog for failed remove ================================================= 12:11:46 (1491567106) [16382.910788] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16383.185237] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16384.008960] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16384.267706] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16385.059249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16385.317624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16386.584297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16386.842090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x227:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [16389.641856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16389.900118] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16390.676955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16390.935082] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16391.706482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16391.960107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16393.216618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16394.470266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16395.725288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16396.979847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16397.235712] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16398.268447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16398.522547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16398.779891] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16399.808674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16400.063260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16400.836611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16401.092133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16402.369607] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl11 [16402.888717] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl11 [16403.472094] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16404.425670] Lustre: DEBUG MARKER: rc=0; [16404.425670] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16404.425670] if [[ $? -eq 0 && $val -ne 0 ]]; then [16404.425670] echo $(hostname -s): $val; [16404.425670] rc=$val; [16404.425670] fi; [16404.425670] exit $rc [16405.046112] Lustre: DEBUG MARKER: dmesg [16405.775769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 225: Changelog for remove canceled ================================================ 12:12:10 \(1491567130\) [16405.942353] Lustre: DEBUG MARKER: == sanity-hsm test 225: Changelog for remove canceled ================================================ 12:12:10 (1491567130) [16409.425409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16409.683178] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16410.461369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16410.717840] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16411.495185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16411.750033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16413.009952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16414.270698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16415.540672] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16416.803137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16418.070857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16418.331411] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16419.364407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16419.618747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16419.875192] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16420.907809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16421.160995] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16421.930274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16422.186890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16423.535994] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16424.440868] Lustre: DEBUG MARKER: rc=0; [16424.440868] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16424.440868] if [[ $? -eq 0 && $val -ne 0 ]]; then [16424.440868] echo $(hostname -s): $val; [16424.440868] rc=$val; [16424.440868] fi; [16424.440868] exit $rc [16425.058823] Lustre: DEBUG MARKER: dmesg [16425.639471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 226: changelog for last rm\/mv with exiting archive ================================ 12:12:30 \(1491567150\) [16425.800690] Lustre: DEBUG MARKER: == sanity-hsm test 226: changelog for last rm/mv with exiting archive ================================ 12:12:30 (1491567150) [16426.724774] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16426.979052] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16427.755093] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16428.012308] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16428.798468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16429.057188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x228:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16429.326614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16429.584155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x229:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16432.419886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16432.677306] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16433.455447] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16433.713922] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16434.492418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16434.748551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16436.003085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16437.257979] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16438.512570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16439.768740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16440.023993] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16441.053716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16441.308996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16441.564711] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16442.594756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16442.847897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16443.648021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16443.902245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16445.164297] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl12 [16445.687039] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl12 [16446.273126] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16447.176964] Lustre: DEBUG MARKER: rc=0; [16447.176964] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16447.176964] if [[ $? -eq 0 && $val -ne 0 ]]; then [16447.176964] echo $(hostname -s): $val; [16447.176964] rc=$val; [16447.176964] fi; [16447.176964] exit $rc [16447.810080] Lustre: DEBUG MARKER: dmesg [16448.408894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 12:12:53 \(1491567173\) [16448.587398] Lustre: DEBUG MARKER: == sanity-hsm test 227: changelog when explicit setting of HSM flags ================================= 12:12:53 (1491567173) [16449.487754] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n [16449.745599] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm [16450.522963] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n [16450.778663] Lustre: DEBUG MARKER: lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm [16451.697167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16451.957188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x22f:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16454.784439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16455.040375] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16455.826644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16456.083644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16456.853943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16457.108659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16458.364398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16459.619062] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16460.875739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16462.131019] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16462.388594] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16463.415269] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16463.672381] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16463.926679] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16464.954370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16465.209164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16465.991881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16466.248651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16467.524327] Lustre: DEBUG MARKER: lctl --device lustre-MDT0000 changelog_deregister cl13 [16468.047722] Lustre: DEBUG MARKER: lctl --device lustre-MDT0002 changelog_deregister cl13 [16468.622457] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16469.559281] Lustre: DEBUG MARKER: rc=0; [16469.559281] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16469.559281] if [[ $? -eq 0 && $val -ne 0 ]]; then [16469.559281] echo $(hostname -s): $val; [16469.559281] rc=$val; [16469.559281] fi; [16469.559281] exit $rc [16470.175047] Lustre: DEBUG MARKER: dmesg [16470.743971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 12:13:15 \(1491567195\) [16470.905266] Lustre: DEBUG MARKER: == sanity-hsm test 228: On released file, return extend to FIEMAP. For [cp,tar] --sparse ============= 12:13:15 (1491567195) [16471.913129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16472.172832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16473.440178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x6d:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16478.716751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16478.973539] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16479.762485] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16480.020914] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16480.791924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16481.046880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16482.301835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16483.554743] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16484.811619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16486.073791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16486.331156] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16487.359648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16487.611563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16487.865117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16488.894239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16489.151193] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16489.922871] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16490.178850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16491.502961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16492.432917] Lustre: DEBUG MARKER: rc=0; [16492.432917] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16492.432917] if [[ $? -eq 0 && $val -ne 0 ]]; then [16492.432917] echo $(hostname -s): $val; [16492.432917] rc=$val; [16492.432917] fi; [16492.432917] exit $rc [16493.054487] Lustre: DEBUG MARKER: dmesg [16493.655696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 250: Coordinator max request ====================================================== 12:13:38 \(1491567218\) [16493.813774] Lustre: DEBUG MARKER: == sanity-hsm test 250: Coordinator max request ====================================================== 12:13:38 (1491567218) [16494.701284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.max_requests [16494.959266] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [16495.475787] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [16495.998586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16496.255655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16497.034157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16497.292428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16506.304258] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16506.819356] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16507.339826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16507.597141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16508.368558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16508.639261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16510.421162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16510.683037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16511.946715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16512.207376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16513.469878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16513.730703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16514.990247] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16515.250890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16516.516396] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16516.778128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16518.044152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16518.306117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16519.566857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16519.828583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16521.093452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16521.354437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16522.627043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16522.902498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16524.177633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16524.452036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16525.729184] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16525.998856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16527.263808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16527.526804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16528.799422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16529.061961] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16530.324361] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16530.586909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16531.854991] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16532.118419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16533.389868] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16533.652852] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16534.916869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16535.180153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16536.445883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16536.706149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16537.965757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | grep STARTED | grep -v CANCEL | wc -l [16538.225455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions | grep WAITING | wc -l [16541.025279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16541.283762] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16542.065913] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16542.323273] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16543.106690] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16543.362406] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16544.618693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16545.874665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16547.131599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16547.388950] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16548.421967] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16548.681419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16548.939856] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16549.970356] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16550.222559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16551.010267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16551.282377] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16552.633408] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16553.559163] Lustre: DEBUG MARKER: rc=0; [16553.559163] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16553.559163] if [[ $? -eq 0 && $val -ne 0 ]]; then [16553.559163] echo $(hostname -s): $val; [16553.559163] rc=$val; [16553.559163] fi; [16553.559163] exit $rc [16554.187402] Lustre: DEBUG MARKER: dmesg [16554.760120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 251: Coordinator request timeout ================================================== 12:14:39 \(1491567279\) [16554.950061] Lustre: DEBUG MARKER: == sanity-hsm test 251: Coordinator request timeout ================================================== 12:14:39 (1491567279) [16558.985684] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=disabled [16559.511066] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=disabled [16560.024417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16560.284479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16561.056083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16561.312777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16562.086519] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.active_request_timeout [16562.344710] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=4 [16562.863066] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=4 [16563.377898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [16563.638383] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=2 [16564.156644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=2 [16564.668935] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16565.179880] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16565.692035] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16565.948551] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16566.716628] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16566.971863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16567.819703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16568.078737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16573.342207] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16573.599899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x230:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16573.861781] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.active_request_timeout=3600 [16574.404442] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.active_request_timeout=3600 [16574.916511] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.hsm.loop_period=10 [16575.436059] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.hsm.loop_period=10 [16578.486228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16578.743060] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16579.539916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16579.794869] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16580.572652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16580.827119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16582.084484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16583.338847] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16584.594742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16585.850763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16586.107421] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16588.395115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16588.652288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16588.907833] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16589.941581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16590.196855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16590.977160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16591.238729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16592.577787] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16593.486521] Lustre: DEBUG MARKER: rc=0; [16593.486521] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16593.486521] if [[ $? -eq 0 && $val -ne 0 ]]; then [16593.486521] echo $(hostname -s): $val; [16593.486521] rc=$val; [16593.486521] fi; [16593.486521] exit $rc [16594.106188] Lustre: DEBUG MARKER: dmesg [16594.674894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 300: On disk coordinator state kept between MDT umount\/mount ====================== 12:15:19 \(1491567319\) [16594.833122] Lustre: DEBUG MARKER: == sanity-hsm test 300: On disk coordinator state kept between MDT umount/mount ====================== 12:15:19 (1491567319) [16594.968211] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16595.485932] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16596.008481] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16596.264231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16599.562994] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16599.822117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16600.593557] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0000.hsm_control [16600.724250] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm_control in log params [16600.726125] Lustre: Skipped 3 previous similar messages [16600.858464] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0001.hsm_control [16601.114030] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0002.hsm_control [16601.372132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d mdt.lustre-MDT0003.hsm_control [16601.629096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16601.882302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16602.657027] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16602.910637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16603.704412] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16603.959778] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [16604.089727] Lustre: Failing over lustre-MDT0000 [16604.131854] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [16604.133575] Lustre: Skipped 7 previous similar messages [16604.172387] Lustre: server umount lustre-MDT0000 complete [16604.301035] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16604.355237] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16604.358738] LustreError: Skipped 36 previous similar messages [16608.607167] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [16608.609162] LustreError: Skipped 1 previous similar message [16608.610752] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16608.614321] Lustre: Skipped 1 previous similar message [16614.129394] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16614.133408] LustreError: Skipped 21 previous similar messages [16614.615496] Lustre: DEBUG MARKER: hostname [16614.931042] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16615.186536] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16615.440612] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16615.601488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16615.619319] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16615.623369] LustreError: 5377:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [16615.625706] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [16615.627895] Lustre: Skipped 27 previous similar messages [16615.629860] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c26005 to 0x52ac6ca176c2a4ca [16615.668797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16615.820346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16616.079815] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [16616.585083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16616.587683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16616.746293] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16616.747793] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16616.895898] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16617.152819] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16617.409822] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16617.669476] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16617.803147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [16617.803182] Lustre: lustre-MDT0000: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [16617.803183] Lustre: Skipped 2 previous similar messages [16618.443898] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16618.469243] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16618.673325] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16618.673327] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16620.264972] Lustre: MGS: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [16620.267747] Lustre: Skipped 13 previous similar messages [16620.664291] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [16620.683434] Lustre: lustre-MDT0000: Recovery over after 0:03, of 7 clients 7 recovered and 0 were evicted. [16620.948522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16620.951597] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16621.159020] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16621.176309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16622.512223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16622.769412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16623.566380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16623.825186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16624.601106] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0000.hsm_control=enabled [16624.729910] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm_control in log params [16624.732593] Lustre: Skipped 3 previous similar messages [16624.864087] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0001.hsm_control=enabled [16625.119941] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0002.hsm_control=enabled [16625.377954] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P mdt.lustre-MDT0003.hsm_control=enabled [16630.202675] LustreError: 32455:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [16645.646375] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16645.915031] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16646.710295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16646.968535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16647.754255] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16648.011244] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [16648.142183] Lustre: Failing over lustre-MDT0000 [16648.208191] Lustre: server umount lustre-MDT0000 complete [16648.337545] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16648.838071] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16648.842647] LustreError: Skipped 11 previous similar messages [16650.664201] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [16650.666842] LustreError: Skipped 2 previous similar messages [16650.668827] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16650.673220] Lustre: Skipped 1 previous similar message [16657.664061] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491567376/real 1491567376] req@ffff8800629f5e00 x1564004881256464/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491567383 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [16657.671028] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16657.673418] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16658.654580] Lustre: DEBUG MARKER: hostname [16658.964047] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16659.220281] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16659.472171] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16659.622092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16663.677175] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [16663.679424] Lustre: Skipped 10 previous similar messages [16663.681388] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c2a4ca to 0x52ac6ca176c2aa96 [16663.730406] LustreError: 897:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [16663.736717] LustreError: 902:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [16663.739223] LustreError: 902:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [16663.747602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16663.898297] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16664.048422] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [16664.160861] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [16664.676630] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16664.681392] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16664.840111] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16664.840121] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16664.986593] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16665.244561] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16665.501569] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16665.762261] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16666.538659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16666.538762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16666.759726] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16666.759728] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16668.743513] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [16668.762576] Lustre: lustre-MDT0000: Recovery over after 0:05, of 7 clients 7 recovered and 0 were evicted. [16669.034581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16669.045894] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16669.207894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16669.308284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16670.606560] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16670.864484] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16671.646370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16671.904970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16672.765755] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16673.693157] Lustre: DEBUG MARKER: rc=0; [16673.693157] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16673.693157] if [[ $? -eq 0 && $val -ne 0 ]]; then [16673.693157] echo $(hostname -s): $val; [16673.693157] rc=$val; [16673.693157] fi; [16673.693157] exit $rc [16674.332872] Lustre: DEBUG MARKER: dmesg [16674.910773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 301: HSM tunnable are persistent ================================================== 12:16:39 \(1491567399\) [16675.080870] Lustre: DEBUG MARKER: == sanity-hsm test 301: HSM tunnable are persistent ================================================== 12:16:39 (1491567399) [16675.247624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [16675.505654] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=2 [16675.634422] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [16675.637046] Lustre: Skipped 3 previous similar messages [16675.769108] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0001.hsm.default_archive_id=2 [16676.032478] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0002.hsm.default_archive_id=2 [16676.290593] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0003.hsm.default_archive_id=2 [16676.557886] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16676.814238] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [16676.945451] Lustre: Failing over lustre-MDT0000 [16677.064533] Lustre: server umount lustre-MDT0000 complete [16677.195785] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16678.743254] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [16678.745831] LustreError: Skipped 1 previous similar message [16678.747869] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16678.752519] Lustre: Skipped 1 previous similar message [16686.717042] Lustre: 5377:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491567405/real 1491567405] req@ffff880064788000 x1564004881260032/t0(0) o101->MGC10.9.5.243@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1491567412 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [16686.723816] Lustre: 5377:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16686.726154] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16686.730407] LustreError: 5377:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [16686.730742] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16686.730744] LustreError: Skipped 67 previous similar messages [16687.516719] Lustre: DEBUG MARKER: hostname [16687.833516] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16688.105176] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16688.368236] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16688.528127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16689.227141] Lustre: MGS: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [16689.229749] Lustre: Skipped 27 previous similar messages [16692.730340] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c2aa96 to 0x52ac6ca176c2b0d9 [16692.766862] LustreError: 3567:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [16692.773935] LustreError: 3572:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [16692.794492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16692.984708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16693.068201] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [16693.244876] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [16693.755004] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16693.756237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16693.917612] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16693.919292] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16694.066751] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16694.328585] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16694.583060] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16694.843532] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16695.619532] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16695.647622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16695.815826] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16695.876060] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16697.783619] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [16698.855822] Lustre: lustre-MDT0000: Recovery over after 0:06, of 7 clients 7 recovered and 0 were evicted. [16699.121513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [16699.202292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [16699.380789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [16699.456361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [16700.747619] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [16701.008024] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [16701.137703] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [16701.140699] Lustre: Skipped 3 previous similar messages [16701.273496] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0001.hsm.default_archive_id [16701.531931] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0002.hsm.default_archive_id [16701.788791] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0003.hsm.default_archive_id [16702.118977] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16703.048757] Lustre: DEBUG MARKER: rc=0; [16703.048757] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16703.048757] if [[ $? -eq 0 && $val -ne 0 ]]; then [16703.048757] echo $(hostname -s): $val; [16703.048757] rc=$val; [16703.048757] fi; [16703.048757] exit $rc [16703.691829] Lustre: DEBUG MARKER: dmesg [16704.274349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 12:17:09 \(1491567429\) [16704.437775] Lustre: DEBUG MARKER: == sanity-hsm test 302: HSM tunnable are persistent when CDT is off ================================== 12:17:09 (1491567429) [16704.574976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [16704.833150] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16705.352998] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16705.868093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16706.123459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16707.377717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16708.632844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16709.893800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16711.151223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16712.407122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16713.660517] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16714.436116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16714.691629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16715.468317] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0000.hsm.default_archive_id=3 [16715.597749] Lustre: Setting parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [16715.599917] Lustre: Skipped 3 previous similar messages [16715.730742] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0001.hsm.default_archive_id=3 [16715.986614] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0002.hsm.default_archive_id=3 [16716.243523] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -n mdt.lustre-MDT0003.hsm.default_archive_id=3 [16716.512325] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [16716.767872] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [16716.897615] Lustre: Failing over lustre-MDT0000 [16716.981268] Lustre: server umount lustre-MDT0000 complete [16717.111280] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16717.783204] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [16717.785338] LustreError: Skipped 1 previous similar message [16717.786910] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16717.790394] Lustre: Skipped 1 previous similar message [16727.431438] Lustre: DEBUG MARKER: hostname [16727.739467] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [16727.991709] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [16728.243166] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [16728.401327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16728.417194] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16728.420796] LustreError: 5377:0:(mgc_request.c:608:do_requeue()) failed processing log: -5 [16728.422955] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [16728.424973] Lustre: Skipped 27 previous similar messages [16728.456712] LustreError: 7220:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [16728.623253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16728.888839] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [16729.402164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16729.403915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16729.576257] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16729.588740] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16729.731128] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16729.985804] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16730.239500] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16730.496454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [16731.262633] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16731.321908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16731.576819] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16731.589498] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16733.796773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16733.814583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16733.953926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16734.066072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [16748.262692] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16748.427388] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16750.139333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16750.199278] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16750.399068] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16750.469821] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16752.721931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [16752.745731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [16752.887813] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [16752.936118] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [16754.260908] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [16754.525593] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [16754.848219] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [16762.324573] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16762.330405] LustreError: Skipped 54 previous similar messages [16765.146608] Lustre: DEBUG MARKER: hostname [16765.457927] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [16765.711601] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [16765.967182] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [16766.126698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16766.160708] Lustre: lustre-MDT0000: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [16766.163572] Lustre: Skipped 32 previous similar messages [16766.177170] LustreError: 9283:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [16766.182067] LustreError: 9287:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [16766.197636] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [16766.200720] Lustre: Skipped 1 previous similar message [16766.347148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [16766.603783] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [16767.109913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16767.113151] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16767.275105] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16767.279229] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16767.326171] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 7 clients reconnect [16767.329027] Lustre: Skipped 1 previous similar message [16767.424510] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [16767.683203] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16767.941425] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [16768.201738] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [16768.966515] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [16768.982712] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [16769.169628] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [16769.197877] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [16771.198590] Lustre: lustre-MDT0002: Recovery over after 0:04, of 7 clients 7 recovered and 0 were evicted. [16771.201761] Lustre: Skipped 1 previous similar message [16771.416312] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [16771.508674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [16771.589489] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [16771.687147] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [16785.914888] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16786.093967] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [16787.806682] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [16787.824970] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [16788.003521] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [16788.069352] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [16790.296242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [16790.315472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [16790.500464] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [16790.574693] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [16791.868088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16792.125689] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16792.901422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16793.159507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16793.956236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.default_archive_id [16794.217597] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0000.hsm.default_archive_id [16794.348378] Lustre: Disabling parameter general.mdt.lustre-MDT0000.hsm.default_archive_id in log params [16794.351148] Lustre: Skipped 3 previous similar messages [16794.485991] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0001.hsm.default_archive_id [16794.755026] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0002.hsm.default_archive_id [16795.010700] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P -d -n mdt.lustre-MDT0003.hsm.default_archive_id [16795.330864] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16796.259803] Lustre: DEBUG MARKER: rc=0; [16796.259803] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16796.259803] if [[ $? -eq 0 && $val -ne 0 ]]; then [16796.259803] echo $(hostname -s): $val; [16796.259803] rc=$val; [16796.259803] fi; [16796.259803] exit $rc [16796.897774] Lustre: DEBUG MARKER: dmesg [16797.484365] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 12:18:42 \(1491567522\) [16797.651533] Lustre: DEBUG MARKER: == sanity-hsm test 400: Single request is sent to the right MDT ====================================== 12:18:42 (1491567522) [16799.132875] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16799.396736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16800.661887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16801.921493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x231:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16807.785736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16808.042939] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16808.829725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16809.086145] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16809.866520] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16810.122565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16811.381491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16811.643202] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16815.197298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16815.452128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16816.706971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16816.962322] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16817.992020] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16818.246973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16819.046283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16819.300746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16820.676518] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16821.589861] Lustre: DEBUG MARKER: rc=0; [16821.589861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16821.589861] if [[ $? -eq 0 && $val -ne 0 ]]; then [16821.589861] echo $(hostname -s): $val; [16821.589861] rc=$val; [16821.589861] fi; [16821.589861] exit $rc [16822.208373] Lustre: DEBUG MARKER: dmesg [16822.792145] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 12:19:07 \(1491567547\) [16822.955072] Lustre: DEBUG MARKER: == sanity-hsm test 401: Compound requests split and sent to their respective MDTs ==================== 12:19:07 (1491567547) [16824.343647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16824.603578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16825.863633] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16827.124907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x232:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16830.455748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16830.712971] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16831.490989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16831.746647] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16832.520061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16832.773524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16834.028080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16835.283089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16836.537629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16836.795133] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16837.851746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16838.108229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16838.363830] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16839.400795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16839.655059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16840.427819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16840.683791] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16842.046329] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16842.943200] Lustre: DEBUG MARKER: rc=0; [16842.943200] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16842.943200] if [[ $? -eq 0 && $val -ne 0 ]]; then [16842.943200] echo $(hostname -s): $val; [16842.943200] rc=$val; [16842.943200] fi; [16842.943200] exit $rc [16843.559558] Lustre: DEBUG MARKER: dmesg [16844.158629] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 12:19:28 \(1491567568\) [16844.321829] Lustre: DEBUG MARKER: == sanity-hsm test 402a: Copytool start fails if all MDTs are inactive =============================== 12:19:28 (1491567568) [16847.013228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16847.272753] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16848.060284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16848.321572] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16849.098577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16849.357280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16850.611649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16851.870776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16853.137454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16854.396675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16855.656718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16856.912338] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16857.167763] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16858.197661] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16858.452867] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16858.708925] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16859.749108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16860.004695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16860.780374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16861.035197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16865.277283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep uuid [16865.806327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep uuid [16866.969720] Lustre: lustre-MDT0000: Client 84b484b6-8248-a20e-62a5-61e74724d1f5 (at 10.9.5.238@tcp) reconnecting [16866.971596] Lustre: Skipped 1 previous similar message [16866.972985] Lustre: lustre-MDT0000: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [16866.974827] Lustre: Skipped 22 previous similar messages [16868.959969] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16869.908754] Lustre: DEBUG MARKER: rc=0; [16869.908754] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16869.908754] if [[ $? -eq 0 && $val -ne 0 ]]; then [16869.908754] echo $(hostname -s): $val; [16869.908754] rc=$val; [16869.908754] fi; [16869.908754] exit $rc [16870.525958] Lustre: DEBUG MARKER: dmesg [16873.629386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16873.890658] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16874.670071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16874.928197] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16875.701217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16875.956569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16877.211955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16877.465562] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16878.506568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16878.761676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16879.015644] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16880.049682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16880.305163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16881.078397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16881.333130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16882.700794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 12:20:07 \(1491567607\) [16882.854294] Lustre: DEBUG MARKER: == sanity-hsm test 402b: CDT must retry request upon slow start of CT ================================ 12:20:07 (1491567607) [16883.867110] LustreError: 17747:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent 'ba976f76-f68d-3cdb-16f3-355a9f831a1e': rc = -11 [16883.994554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.loop_period [16893.872566] LustreError: 17747:0:(mdt_hsm_cdt_agent.c:599:mdt_hsm_agent_send()) lustre-MDT0000: cannot send request to agent 'ba976f76-f68d-3cdb-16f3-355a9f831a1e': rc = -11 [16894.255482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16894.517145] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16895.022861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16895.281129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16896.540680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16897.797205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16899.055929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16900.314803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16901.574630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16902.832723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16904.093465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x85:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16906.890932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16907.148666] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16907.934543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16908.192511] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16908.965257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16909.222884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16910.479808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16911.733611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16912.990091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16914.249471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16914.507692] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16915.537153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16915.792908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16916.048566] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16917.074770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16917.330565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16918.105202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16918.362470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16919.710176] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16920.651354] Lustre: DEBUG MARKER: rc=0; [16920.651354] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16920.651354] if [[ $? -eq 0 && $val -ne 0 ]]; then [16920.651354] echo $(hostname -s): $val; [16920.651354] rc=$val; [16920.651354] fi; [16920.651354] exit $rc [16921.278982] Lustre: DEBUG MARKER: dmesg [16921.869410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 12:20:46 \(1491567646\) [16922.025627] Lustre: DEBUG MARKER: == sanity-hsm test 403: Copytool starts with inactive MDT and register on reconnect ================== 12:20:46 (1491567646) [16924.705942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16924.961103] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16925.741528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16926.000614] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16926.776679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16927.035168] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16928.286799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16929.544840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16930.799879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16932.054599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16933.307915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16934.563199] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16934.818711] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16935.858516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16936.114298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16936.370041] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16937.398078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16937.653285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16938.432861] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16938.687881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16941.937130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [16942.093794] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing ps -C lhsmtool_posix -o args= [16942.734125] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid \/mnt\/lustre2 [16942.888576] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing get_client_uuid /mnt/lustre2 [16943.034884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [16944.587482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [16945.116609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.agents | grep ba976f76-f68d-3cdb-16f3-355a9f831a1e [16948.186271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16948.445752] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16949.219444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16949.475438] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16950.250315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16950.506188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16951.760335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16953.014182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16954.267096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16955.532241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16955.790325] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16956.818651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16957.073024] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16957.329252] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16958.364175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16958.618708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16959.399499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16959.655639] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16961.013181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16961.962861] Lustre: DEBUG MARKER: rc=0; [16961.962861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16961.962861] if [[ $? -eq 0 && $val -ne 0 ]]; then [16961.962861] echo $(hostname -s): $val; [16961.962861] rc=$val; [16961.962861] fi; [16961.962861] exit $rc [16962.589563] Lustre: DEBUG MARKER: dmesg [16963.265891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 12:21:28 \(1491567688\) [16963.442854] Lustre: DEBUG MARKER: == sanity-hsm test 404: Inactive MDT does not block requests for active MDTs ========================= 12:21:28 (1491567688) [16965.305402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16965.564650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16966.825845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16968.086110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x233:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16971.622510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16971.879938] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16972.658022] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16972.913460] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16973.687645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16973.943417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16975.197952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16976.451813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16977.706922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16977.962807] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [16978.999705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16979.256017] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16979.512546] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [16980.545180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16980.798956] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16981.573111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16981.827831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16983.167626] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [16984.088456] Lustre: DEBUG MARKER: rc=0; [16984.088456] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [16984.088456] if [[ $? -eq 0 && $val -ne 0 ]]; then [16984.088456] echo $(hostname -s): $val; [16984.088456] rc=$val; [16984.088456] fi; [16984.088456] exit $rc [16984.704796] Lustre: DEBUG MARKER: dmesg [16985.312535] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 405: archive and release under striped directory ================================== 12:21:50 \(1491567710\) [16985.480565] Lustre: DEBUG MARKER: == sanity-hsm test 405: archive and release under striped directory ================================== 12:21:50 (1491567710) [16986.931685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16987.190378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16988.455716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm.actions | awk '/'0x280000bd1:0x1:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16989.244330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x8a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16989.500524] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x20000040c:0x8a:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [16997.135262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16997.391915] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [16998.166061] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [16998.422092] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [16999.203150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [16999.462547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17000.725894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17001.981323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17003.235032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17003.488439] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17004.523080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17004.776713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17005.034343] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17006.056738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17006.312121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17007.085675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17007.339846] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17008.673424] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17009.594913] Lustre: DEBUG MARKER: rc=0; [17009.594913] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17009.594913] if [[ $? -eq 0 && $val -ne 0 ]]; then [17009.594913] echo $(hostname -s): $val; [17009.594913] rc=$val; [17009.594913] fi; [17009.594913] exit $rc [17010.220616] Lustre: DEBUG MARKER: dmesg [17010.795295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 12:22:15 \(1491567735\) [17010.955706] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 12:22:15 (1491567735) [17011.103568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [17011.103568] /usr/sbin/lctl lustre_build_version 2>/dev/null || [17011.103568] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [17012.355610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17012.614996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17013.874121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17015.139460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= [17015.473025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17015.748740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17017.019666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17018.279434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='RESTORE'/ {print $13}' | cut -f2 -d= [17018.546727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [17018.805676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200000405:0x234:0x0'.*action='REMOVE'/ {print $13}' | cut -f2 -d= [17027.940673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17028.199541] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17028.974636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17029.230860] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17030.009897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17030.265658] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17030.520363] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17035.320159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17035.577028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17035.831651] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17036.861803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17037.119340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17037.951886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17038.220033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17039.555419] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17040.479179] Lustre: DEBUG MARKER: rc=0; [17040.479179] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17040.479179] if [[ $? -eq 0 && $val -ne 0 ]]; then [17040.479179] echo $(hostname -s): $val; [17040.479179] rc=$val; [17040.479179] fi; [17040.479179] exit $rc [17041.097489] Lustre: DEBUG MARKER: dmesg [17041.714745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 12:22:46 \(1491567766\) [17041.877605] Lustre: DEBUG MARKER: == sanity-hsm test 500: various LLAPI HSM tests ====================================================== 12:22:46 (1491567766) [17042.023245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [17042.023245] /usr/sbin/lctl lustre_build_version 2>/dev/null || [17042.023245] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [17044.870635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17045.130730] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17045.916180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17046.170880] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17046.955786] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17047.210594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17048.466267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17049.728316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17050.998678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17051.255917] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17056.077916] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17056.344385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17056.600042] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17057.636385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17057.892245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17058.665684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17058.917938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17068.397706] Lustre: HSM agent 0430c6b8-19e5-0b04-5065-c582ed85269c already registered [17068.399426] Lustre: Skipped 1 previous similar message [17070.160541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17071.057697] Lustre: DEBUG MARKER: rc=0; [17071.057697] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17071.057697] if [[ $? -eq 0 && $val -ne 0 ]]; then [17071.057697] echo $(hostname -s): $val; [17071.057697] rc=$val; [17071.057697] fi; [17071.057697] exit $rc [17071.676787] Lustre: DEBUG MARKER: dmesg [17074.751203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17075.006933] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=shutdown [17075.786434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17076.040657] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=shutdown [17076.808443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17077.062378] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17078.316325] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17079.574030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17080.829674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17082.084662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17082.338986] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0000.hsm_control=enabled [17083.363559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17083.616443] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17083.870247] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.lustre-MDT0002.hsm_control=enabled [17084.902958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17085.156460] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm_control [17085.942731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17086.196738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.hsm_control [17087.527617] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-hsm test complete, duration 4198 sec ======================================================= 12:23:32 \(1491567812\) [17087.692297] Lustre: DEBUG MARKER: == sanity-hsm test complete, duration 4198 sec ======================================================= 12:23:32 (1491567812) [17089.433707] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17089.693337] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [17089.826272] Lustre: Failing over lustre-MDT0000 [17089.827646] Lustre: Skipped 1 previous similar message [17089.960346] Lustre: server umount lustre-MDT0000 complete [17089.961822] Lustre: Skipped 1 previous similar message [17089.975214] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17089.978828] LustreError: Skipped 11 previous similar messages [17090.094671] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17091.202324] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [17091.202668] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17091.202670] Lustre: Skipped 6 previous similar messages [17091.209664] LustreError: Skipped 9 previous similar messages [17093.764240] LustreError: 5366:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491567819 with bad export cookie 5957255847842331474 [17093.768027] LustreError: 5366:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 9 previous similar messages [17097.735054] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491567816/real 1491567816] req@ffff8800779e8600 x1564004881306496/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491567823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17097.741070] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [17097.743206] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17100.414362] Lustre: DEBUG MARKER: hostname [17100.727382] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [17100.979707] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [17101.231980] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [17101.393858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17103.747156] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [17103.749557] Lustre: Skipped 3 previous similar messages [17103.751723] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c2b612 to 0x52ac6ca176c44c4a [17103.755945] Lustre: Skipped 1 previous similar message [17103.791388] LustreError: 1330:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [17103.799085] LustreError: 1335:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [17103.814308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17104.002202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17104.262542] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/test [17104.763762] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17104.765676] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17104.935382] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17104.936420] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17105.085321] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17105.341584] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17105.599145] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17105.856478] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [17106.099238] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [17106.626865] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17106.642287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17106.820037] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17106.867269] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [17108.805432] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [17108.813355] LustreError: Skipped 1 previous similar message [17108.836091] Lustre: lustre-MDT0000: Recovery over after 0:03, of 7 clients 7 recovered and 0 were evicted. [17109.084803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [17109.150237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [17109.259800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [17109.375795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [17131.613693] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [17131.874780] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [17133.805288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17133.808208] Lustre: Skipped 3 previous similar messages [17133.903086] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880050b64f00 x1564004881322272/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17133.908926] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 9 previous similar messages [17138.298742] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17145.422094] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [17145.676498] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [17145.805041] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491567864/real 1491567864] req@ffff880050b64c00 x1564004881322528/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491567871 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17145.813343] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [17146.831066] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880050b66d00 x1564004881322992/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17146.838677] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [17152.011508] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [17219.842954] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: ost-pools ============----- Fri Apr 7 12:25:43 UTC 2017 [17220.000732] Lustre: DEBUG MARKER: -----============= acceptance-small: ost-pools ============----- Fri Apr 7 12:25:43 UTC 2017 [17220.263387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration -o sec ========================================================== 12:25:45 \(1491567945\) [17220.413199] Lustre: DEBUG MARKER: == ost-pools test complete, duration -o sec ========================================================== 12:25:45 (1491567945) [17222.911124] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17222.911124] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17222.911124] if [ $running -ne $mpts ]; then [17222.911124] echo $(hostname) env are INSANE!; [17222.911124] exit 1; [17222.911124] fi [17223.466112] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [17223.466112] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [17223.466112] if [ $running -ne $mpts ]; then [17223.466112] echo $(hostname) env are INSANE!; [17223.466112] exit 1; [17223.466112] fi [17224.039124] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [17224.039124] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [17224.039124] if [ $running -ne $mpts ]; then [17224.039124] echo $(hostname) env are INSANE!; [17224.039124] exit 1; [17224.039124] fi [17226.566953] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [17226.821732] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [17227.068874] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [17227.313707] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [17227.469640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17227.548098] LustreError: 3647:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [17227.705374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17227.960114] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [17228.456107] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17228.456109] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17228.633585] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17228.635149] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17228.780453] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17229.037499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17229.293086] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17229.548121] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [17229.803480] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17231.965418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17232.135824] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17232.557138] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491567953/real 1491567953] req@ffff88005ad1aa00 x1564004881323440/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491567958 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [17232.564541] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [17233.560545] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [17233.822626] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [17234.077721] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [17234.334113] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [17234.482566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17234.540854] LustreError: 4945:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [17234.704820] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [17234.961307] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [17235.468603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17235.472644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17235.636390] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17235.646616] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17235.787433] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17236.045306] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17236.303746] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [17236.562108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [17236.816880] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17238.986422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17239.149800] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17242.465557] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17242.631632] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17245.404240] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17245.569025] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17248.394651] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17248.554164] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17251.362282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17251.530578] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17254.389660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17254.554498] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17257.370342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17257.540149] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17260.423526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17260.582072] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17263.466054] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17263.625289] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17265.743406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17265.904443] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [17267.342963] Lustre: DEBUG MARKER: lctl get_param -n timeout [17267.663292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [17267.818633] Lustre: DEBUG MARKER: Using TIMEOUT=20 [17267.954212] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [17268.241983] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [17277.716782] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [17278.007529] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17278.526431] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [17279.105668] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [17279.265773] Lustre: DEBUG MARKER: excepting tests: [17279.456893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 18 23b [17279.621397] Lustre: DEBUG MARKER: skipping tests SLOW=no: 18 23b [17279.972540] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 12:26:44 \(1491568004\) [17280.122129] Lustre: DEBUG MARKER: == ost-pools test 1a: Create a pool with a 1 character pool name ===================================== 12:26:44 (1491568004) [17280.257703] Lustre: DEBUG MARKER: lctl pool_new lustre.p [17288.529421] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [17288.784702] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [17289.561961] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [17289.817567] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [17290.616764] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17290.873871] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.p [17291.128561] Lustre: DEBUG MARKER: lctl pool_destroy lustre.p [17301.394234] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [17301.648956] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.p 2>/dev/null || echo foo [17302.422944] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [17302.678898] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.p 2>/dev/null || echo foo [17303.528555] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17304.427342] Lustre: DEBUG MARKER: rc=0; [17304.427342] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17304.427342] if [[ $? -eq 0 && $val -ne 0 ]]; then [17304.427342] echo $(hostname -s): $val; [17304.427342] rc=$val; [17304.427342] fi; [17304.427342] exit $rc [17305.034019] Lustre: DEBUG MARKER: dmesg [17305.610184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 12:27:10 \(1491568030\) [17305.779468] Lustre: DEBUG MARKER: == ost-pools test 1b: Create a pool with a 10 char pool name ========================================= 12:27:10 (1491568030) [17305.920981] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12 [17316.185204] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [17316.438903] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [17317.224471] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [17317.477758] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [17318.286233] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17318.543122] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool12 [17318.796805] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool12 [17329.064712] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [17329.320490] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool12 2>/dev/null || echo foo [17330.094287] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [17330.348446] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool12 2>/dev/null || echo foo [17331.208145] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17332.078296] Lustre: DEBUG MARKER: rc=0; [17332.078296] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17332.078296] if [[ $? -eq 0 && $val -ne 0 ]]; then [17332.078296] echo $(hostname -s): $val; [17332.078296] rc=$val; [17332.078296] fi; [17332.078296] exit $rc [17332.678189] Lustre: DEBUG MARKER: dmesg [17333.231919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 12:27:38 \(1491568058\) [17333.406835] Lustre: DEBUG MARKER: == ost-pools test 1c: Create a pool with a 15 char pool name ========================================= 12:27:38 (1491568058) [17333.571187] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [17343.851366] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17344.109535] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17344.890319] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17345.148233] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17345.947239] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17346.204372] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [17346.458091] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [17354.721600] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17354.987390] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17355.760477] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17356.014950] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17356.866797] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17357.758989] Lustre: DEBUG MARKER: rc=0; [17357.758989] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17357.758989] if [[ $? -eq 0 && $val -ne 0 ]]; then [17357.758989] echo $(hostname -s): $val; [17357.758989] rc=$val; [17357.758989] fi; [17357.758989] exit $rc [17358.361004] Lustre: DEBUG MARKER: dmesg [17358.919746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1d: Create a pool with a 16 char pool name\; should fail ============================ 12:28:03 \(1491568083\) [17359.083717] Lustre: DEBUG MARKER: == ost-pools test 1d: Create a pool with a 16 char pool name; should fail ============================ 12:28:03 (1491568083) [17359.218342] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool12345678 [17359.524352] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17360.414257] Lustre: DEBUG MARKER: rc=0; [17360.414257] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17360.414257] if [[ $? -eq 0 && $val -ne 0 ]]; then [17360.414257] echo $(hostname -s): $val; [17360.414257] rc=$val; [17360.414257] fi; [17360.414257] exit $rc [17361.014851] Lustre: DEBUG MARKER: dmesg [17361.568264] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1e: Create a pool with a 1000 char pool name\; should fail ========================== 12:28:06 \(1491568086\) [17361.726333] Lustre: DEBUG MARKER: == ost-pools test 1e: Create a pool with a 1000 char pool name; should fail ========================== 12:28:06 (1491568086) [17361.879104] Lustre: DEBUG MARKER: lctl pool_new lustre.testpooloooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo [17362.197941] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17363.125545] Lustre: DEBUG MARKER: rc=0; [17363.125545] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17363.125545] if [[ $? -eq 0 && $val -ne 0 ]]; then [17363.125545] echo $(hostname -s): $val; [17363.125545] rc=$val; [17363.125545] fi; [17363.125545] exit $rc [17363.721246] Lustre: DEBUG MARKER: dmesg [17364.293255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 12:28:09 \(1491568089\) [17364.450116] Lustre: DEBUG MARKER: == ost-pools test 1f: pool_new should fail if fs-name is missing ===================================== 12:28:09 (1491568089) [17364.589142] Lustre: DEBUG MARKER: lctl pool_new .testpool [17364.898588] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17365.776131] Lustre: DEBUG MARKER: rc=0; [17365.776131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17365.776131] if [[ $? -eq 0 && $val -ne 0 ]]; then [17365.776131] echo $(hostname -s): $val; [17365.776131] rc=$val; [17365.776131] fi; [17365.776131] exit $rc [17366.368289] Lustre: DEBUG MARKER: dmesg [17366.967632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 12:28:11 \(1491568091\) [17367.121938] Lustre: DEBUG MARKER: == ost-pools test 1g: pool_new should fail if fs-name is missing ===================================== 12:28:11 (1491568091) [17367.261730] Lustre: DEBUG MARKER: lctl pool_new testpool [17367.570429] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17368.466985] Lustre: DEBUG MARKER: rc=0; [17368.466985] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17368.466985] if [[ $? -eq 0 && $val -ne 0 ]]; then [17368.466985] echo $(hostname -s): $val; [17368.466985] rc=$val; [17368.466985] fi; [17368.466985] exit $rc [17369.060162] Lustre: DEBUG MARKER: dmesg [17369.652516] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 12:28:14 \(1491568094\) [17369.830133] Lustre: DEBUG MARKER: == ost-pools test 1h: pool_new should fail if poolname is missing ==================================== 12:28:14 (1491568094) [17369.963722] Lustre: DEBUG MARKER: lctl pool_new lustre. [17370.268741] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17371.184171] Lustre: DEBUG MARKER: rc=0; [17371.184171] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17371.184171] if [[ $? -eq 0 && $val -ne 0 ]]; then [17371.184171] echo $(hostname -s): $val; [17371.184171] rc=$val; [17371.184171] fi; [17371.184171] exit $rc [17371.777488] Lustre: DEBUG MARKER: dmesg [17372.335375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 12:28:17 \(1491568097\) [17372.502835] Lustre: DEBUG MARKER: == ost-pools test 1i: pool_new should fail if poolname and fs-name are missing ======================= 12:28:17 (1491568097) [17372.658479] Lustre: DEBUG MARKER: lctl pool_new . [17372.969767] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17373.846130] Lustre: DEBUG MARKER: rc=0; [17373.846130] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17373.846130] if [[ $? -eq 0 && $val -ne 0 ]]; then [17373.846130] echo $(hostname -s): $val; [17373.846130] rc=$val; [17373.846130] fi; [17373.846130] exit $rc [17374.443767] Lustre: DEBUG MARKER: dmesg [17375.015776] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 12:28:19 \(1491568099\) [17375.173053] Lustre: DEBUG MARKER: == ost-pools test 1j: pool_new should fail if poolname format is wrong =============================== 12:28:19 (1491568099) [17375.307091] Lustre: DEBUG MARKER: lctl pool_new lustre,testpool [17375.614828] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17376.498761] Lustre: DEBUG MARKER: rc=0; [17376.498761] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17376.498761] if [[ $? -eq 0 && $val -ne 0 ]]; then [17376.498761] echo $(hostname -s): $val; [17376.498761] rc=$val; [17376.498761] fi; [17376.498761] exit $rc [17377.084639] Lustre: DEBUG MARKER: dmesg [17377.644983] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 12:28:22 \(1491568102\) [17377.805774] Lustre: DEBUG MARKER: == ost-pools test 1k: pool_new should fail if poolname format is wrong =============================== 12:28:22 (1491568102) [17377.947024] Lustre: DEBUG MARKER: lctl pool_new lustre/testpool [17378.251922] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17379.174778] Lustre: DEBUG MARKER: rc=0; [17379.174778] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17379.174778] if [[ $? -eq 0 && $val -ne 0 ]]; then [17379.174778] echo $(hostname -s): $val; [17379.174778] rc=$val; [17379.174778] fi; [17379.174778] exit $rc [17379.761321] Lustre: DEBUG MARKER: dmesg [17380.337487] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 12:28:25 \(1491568105\) [17380.487843] Lustre: DEBUG MARKER: == ost-pools test 1m: pool_new did not fail even though testpool2 existed ============================ 12:28:25 (1491568105) [17380.634342] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [17386.901332] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17387.157770] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17389.192546] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17389.448860] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17390.251489] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [17390.506545] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17390.762357] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [17391.019597] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [17401.283871] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17401.537137] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17402.316677] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17402.574894] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17403.420074] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17404.338066] Lustre: DEBUG MARKER: rc=0; [17404.338066] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17404.338066] if [[ $? -eq 0 && $val -ne 0 ]]; then [17404.338066] echo $(hostname -s): $val; [17404.338066] rc=$val; [17404.338066] fi; [17404.338066] exit $rc [17404.940384] Lustre: DEBUG MARKER: dmesg [17405.531302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 12:28:50 \(1491568130\) [17405.695066] Lustre: DEBUG MARKER: == ost-pools test 1n: Pool with a 15 char pool name works well ======================================= 12:28:50 (1491568130) [17405.830713] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1234567 [17414.102417] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17414.364630] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17415.154453] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17415.415739] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17416.227795] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567 OST0000 [17424.494114] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [17424.494114] sort -u | tr '\n' ' ' [17424.750534] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 | [17424.750534] sort -u | tr '\n' ' ' [17426.794424] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 | [17426.794424] sort -u | tr '\n' ' ' [17427.052122] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 | [17427.052122] sort -u | tr '\n' ' ' [17430.071492] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1234567 OST0000 [17440.519497] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17440.778700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1234567 [17441.040922] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1234567 [17449.311248] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17449.567186] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17450.348334] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17450.616898] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1234567 2>/dev/null || echo foo [17451.480667] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17452.389293] Lustre: DEBUG MARKER: rc=0; [17452.389293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17452.389293] if [[ $? -eq 0 && $val -ne 0 ]]; then [17452.389293] echo $(hostname -s): $val; [17452.389293] rc=$val; [17452.389293] fi; [17452.389293] exit $rc [17452.993941] Lustre: DEBUG MARKER: dmesg [17453.548373] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 12:29:38 \(1491568178\) [17453.712386] Lustre: DEBUG MARKER: == ost-pools test 2a: pool_add: non-existant pool testpool =========================================== 12:29:38 (1491568178) [17453.856197] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17454.111190] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [17454.420394] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17455.309433] Lustre: DEBUG MARKER: rc=0; [17455.309433] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17455.309433] if [[ $? -eq 0 && $val -ne 0 ]]; then [17455.309433] echo $(hostname -s): $val; [17455.309433] rc=$val; [17455.309433] fi; [17455.309433] exit $rc [17455.908141] Lustre: DEBUG MARKER: dmesg [17456.468991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 12:29:41 \(1491568181\) [17456.631510] Lustre: DEBUG MARKER: == ost-pools test 2b: pool_add: Invalid pool name ==================================================== 12:29:41 (1491568181) [17456.772142] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1234567890 lustre-OST0000 [17457.083609] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17457.976623] Lustre: DEBUG MARKER: rc=0; [17457.976623] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17457.976623] if [[ $? -eq 0 && $val -ne 0 ]]; then [17457.976623] echo $(hostname -s): $val; [17457.976623] rc=$val; [17457.976623] fi; [17457.976623] exit $rc [17458.572845] Lustre: DEBUG MARKER: dmesg [17459.136470] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2c: pool_add: OST index combinations =============================================== 12:29:43 \(1491568183\) [17459.292807] Lustre: DEBUG MARKER: == ost-pools test 2c: pool_add: OST index combinations =============================================== 12:29:43 (1491568183) [17459.431892] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17465.701809] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17465.962900] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17469.250294] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17469.503465] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17470.338731] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [17480.612253] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [17490.922754] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [17499.187499] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [17508.501185] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [17518.764564] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17529.054792] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000,0001,0002,0003,0004,0005,0006,0007,] [17539.322979] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000,0001,0002,0003,0004,0005,0006,0007,] [17549.618983] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [17555.907662] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0007/1] [17564.196637] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17564.452108] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17564.707530] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17570.970740] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17571.227209] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17574.521926] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17574.777904] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17575.627660] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17576.509419] Lustre: DEBUG MARKER: rc=0; [17576.509419] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17576.509419] if [[ $? -eq 0 && $val -ne 0 ]]; then [17576.509419] echo $(hostname -s): $val; [17576.509419] rc=$val; [17576.509419] fi; [17576.509419] exit $rc [17577.099935] Lustre: DEBUG MARKER: dmesg [17577.659393] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2d: pool_add: OSTs that don\'t exist should be rejected ============================= 12:31:42 \(1491568302\) [17577.826107] Lustre: DEBUG MARKER: == ost-pools test 2d: pool_add: OSTs that don't exist should be rejected ============================= 12:31:42 (1491568302) [17577.965165] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17588.240121] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17588.494661] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17589.293047] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17589.548194] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17590.342879] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0008_UUID [17590.598324] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17590.856272] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17591.111337] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17601.373145] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17601.629560] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17602.397770] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17602.654334] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17603.503115] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17604.392632] Lustre: DEBUG MARKER: rc=0; [17604.392632] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17604.392632] if [[ $? -eq 0 && $val -ne 0 ]]; then [17604.392632] echo $(hostname -s): $val; [17604.392632] rc=$val; [17604.392632] fi; [17604.392632] exit $rc [17604.983873] Lustre: DEBUG MARKER: dmesg [17605.527904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 12:32:10 \(1491568330\) [17605.688453] Lustre: DEBUG MARKER: == ost-pools test 2e: pool_add: OST already in a pool should be rejected ============================= 12:32:10 (1491568330) [17605.819686] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17614.086871] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17614.344502] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17615.116296] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17615.370422] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17616.164559] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [17624.447005] Lustre: DEBUG MARKER: LOCALE=C /usr/sbin/lctl pool_add lustre.testpool lustre-OST0000_UUID 2>&1 [17624.705438] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17624.964298] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17625.220161] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17629.485634] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17639.751416] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17640.008129] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17640.787867] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17641.042905] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17641.894622] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17642.763662] Lustre: DEBUG MARKER: rc=0; [17642.763662] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17642.763662] if [[ $? -eq 0 && $val -ne 0 ]]; then [17642.763662] echo $(hostname -s): $val; [17642.763662] rc=$val; [17642.763662] fi; [17642.763662] exit $rc [17643.357405] Lustre: DEBUG MARKER: dmesg [17643.936665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3a: pool_remove: non-existant pool ================================================= 12:32:48 \(1491568368\) [17644.098327] Lustre: DEBUG MARKER: == ost-pools test 3a: pool_remove: non-existant pool ================================================= 12:32:48 (1491568368) [17644.231484] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [17644.535934] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17645.412070] Lustre: DEBUG MARKER: rc=0; [17645.412070] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17645.412070] if [[ $? -eq 0 && $val -ne 0 ]]; then [17645.412070] echo $(hostname -s): $val; [17645.412070] rc=$val; [17645.412070] fi; [17645.412070] exit $rc [17646.003375] Lustre: DEBUG MARKER: dmesg [17646.573615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 12:32:51 \(1491568371\) [17646.729802] Lustre: DEBUG MARKER: == ost-pools test 3b: pool_remove: non-existant fsname =============================================== 12:32:51 (1491568371) [17646.866185] Lustre: DEBUG MARKER: lctl pool_remove nonexistantfs.testpool OST0000 [17647.175764] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17648.073478] Lustre: DEBUG MARKER: rc=0; [17648.073478] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17648.073478] if [[ $? -eq 0 && $val -ne 0 ]]; then [17648.073478] echo $(hostname -s): $val; [17648.073478] rc=$val; [17648.073478] fi; [17648.073478] exit $rc [17648.667126] Lustre: DEBUG MARKER: dmesg [17649.211934] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 12:32:54 \(1491568374\) [17649.367148] Lustre: DEBUG MARKER: == ost-pools test 3c: pool_remove: Invalid pool name ================================================= 12:32:54 (1491568374) [17649.503643] Lustre: DEBUG MARKER: lctl pool_remove lustre.p1234567891234567890 lustre-OST0000 [17649.805137] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17650.695119] Lustre: DEBUG MARKER: rc=0; [17650.695119] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17650.695119] if [[ $? -eq 0 && $val -ne 0 ]]; then [17650.695119] echo $(hostname -s): $val; [17650.695119] rc=$val; [17650.695119] fi; [17650.695119] exit $rc [17651.292609] Lustre: DEBUG MARKER: dmesg [17651.861047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 3d: pool_remove: OST index combinations ============================================ 12:32:56 \(1491568376\) [17652.014590] Lustre: DEBUG MARKER: == ost-pools test 3d: pool_remove: OST index combinations ============================================ 12:32:56 (1491568376) [17652.154580] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17660.417694] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17660.672968] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17661.445148] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17661.697632] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17662.499073] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [17670.761548] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST0000 [17681.044428] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000 [17691.305955] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000 [17699.587418] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST0000_UUID [17709.853505] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17718.140615] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [17724.412489] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [17724.412489] sort -u | tr '\n' ' ' [17724.671460] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [17724.671460] sort -u | tr '\n' ' ' [17725.446152] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [17725.446152] sort -u | tr '\n' ' ' [17725.702476] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [17725.702476] sort -u | tr '\n' ' ' [17728.525256] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000-0007/1] [17735.835576] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17736.093471] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17736.361868] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17740.627863] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17740.885156] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17741.657334] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17741.911898] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17745.786217] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17746.701585] Lustre: DEBUG MARKER: rc=0; [17746.701585] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17746.701585] if [[ $? -eq 0 && $val -ne 0 ]]; then [17746.701585] echo $(hostname -s): $val; [17746.701585] rc=$val; [17746.701585] fi; [17746.701585] exit $rc [17747.309901] Lustre: DEBUG MARKER: dmesg [17747.864546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 12:34:32 \(1491568472\) [17748.026300] Lustre: DEBUG MARKER: == ost-pools test 4a: pool_destroy: non-existant pool ================================================ 12:34:32 (1491568472) [17748.166709] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17748.468421] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17749.360651] Lustre: DEBUG MARKER: rc=0; [17749.360651] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17749.360651] if [[ $? -eq 0 && $val -ne 0 ]]; then [17749.360651] echo $(hostname -s): $val; [17749.360651] rc=$val; [17749.360651] fi; [17749.360651] exit $rc [17749.968055] Lustre: DEBUG MARKER: dmesg [17750.547890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 12:34:35 \(1491568475\) [17750.709385] Lustre: DEBUG MARKER: == ost-pools test 4b: pool_destroy: non-existant fs-name ============================================= 12:34:35 (1491568475) [17750.844340] Lustre: DEBUG MARKER: lctl pool_destroy nonexistantfs.testpool [17751.152568] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17752.054153] Lustre: DEBUG MARKER: rc=0; [17752.054153] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17752.054153] if [[ $? -eq 0 && $val -ne 0 ]]; then [17752.054153] echo $(hostname -s): $val; [17752.054153] rc=$val; [17752.054153] fi; [17752.054153] exit $rc [17752.647058] Lustre: DEBUG MARKER: dmesg [17753.223320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 12:34:38 \(1491568478\) [17753.384606] Lustre: DEBUG MARKER: == ost-pools test 4c: pool_destroy: non-empty pool =================================================== 12:34:38 (1491568478) [17753.517735] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17761.778930] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17762.032870] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17764.068069] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17764.321365] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17765.126795] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [17773.398249] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [17773.398249] sort -u | tr '\n' ' ' [17773.655104] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [17773.655104] sort -u | tr '\n' ' ' [17774.433226] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [17774.433226] sort -u | tr '\n' ' ' [17774.689660] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [17774.689660] sort -u | tr '\n' ' ' [17775.489820] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17775.745420] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17776.002312] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17776.256890] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17784.517908] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17792.780924] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17793.036744] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17793.817850] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17794.073790] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17794.929674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17795.805107] Lustre: DEBUG MARKER: rc=0; [17795.805107] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17795.805107] if [[ $? -eq 0 && $val -ne 0 ]]; then [17795.805107] echo $(hostname -s): $val; [17795.805107] rc=$val; [17795.805107] fi; [17795.805107] exit $rc [17796.393180] Lustre: DEBUG MARKER: dmesg [17796.967102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5a: lfs pool_list from client ====================================================== 12:35:21 \(1491568521\) [17797.116818] Lustre: DEBUG MARKER: == ost-pools test 5a: lfs pool_list from client ====================================================== 12:35:21 (1491568521) [17797.259192] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17797.515594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17797.770211] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17806.035106] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17806.290871] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17807.075364] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17807.334413] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17808.143799] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [17818.406615] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17818.661646] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17819.438084] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17819.690766] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17820.484871] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [17826.771101] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17827.025930] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17827.284953] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17835.549632] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [17843.813337] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [17850.085221] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [17858.346237] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [17866.609447] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [17874.871986] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [17885.147289] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [17891.422133] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [17899.684343] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17899.940149] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17900.724887] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17900.977300] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17901.787157] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17902.045204] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [17902.301345] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [17904.561037] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17904.813554] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17913.139531] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17913.395500] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17914.240861] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [17915.131047] Lustre: DEBUG MARKER: rc=0; [17915.131047] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [17915.131047] if [[ $? -eq 0 && $val -ne 0 ]]; then [17915.131047] echo $(hostname -s): $val; [17915.131047] rc=$val; [17915.131047] fi; [17915.131047] exit $rc [17915.733825] Lustre: DEBUG MARKER: dmesg [17916.319780] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 5b: lctl pool_list from MDS ======================================================== 12:37:21 \(1491568641\) [17916.486629] Lustre: DEBUG MARKER: == ost-pools test 5b: lctl pool_list from MDS ======================================================== 12:37:21 (1491568641) [17916.629831] Lustre: DEBUG MARKER: lctl pool_list [17916.883108] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17917.140328] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17917.394500] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [17927.658364] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17927.912310] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [17928.684376] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17928.937041] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [17929.735855] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [17935.999374] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17936.253816] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [17938.282505] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17938.540614] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [17940.342447] Lustre: DEBUG MARKER: lctl pool_list lustre [17940.598734] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [17950.868946] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [17951.124632] Lustre: DEBUG MARKER: lctl pool_list nonexistantfs [17951.379153] Lustre: DEBUG MARKER: lctl pool_list lustre.nonexistantpool [17951.635420] Lustre: DEBUG MARKER: lctl pool_list /mnt/lustrenonexistant [17951.891659] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [17952.144947] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [17952.399518] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [17958.665320] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [17968.932241] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [17977.195332] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [17985.456630] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [17993.719770] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [17999.982350] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [18010.245427] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [18020.510096] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [18028.771955] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18029.025512] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18029.803130] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18030.056736] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18030.855715] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18031.111861] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18031.366054] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [18039.630316] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18039.885918] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18040.656450] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18040.910525] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18041.761249] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18042.665948] Lustre: DEBUG MARKER: rc=0; [18042.665948] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18042.665948] if [[ $? -eq 0 && $val -ne 0 ]]; then [18042.665948] echo $(hostname -s): $val; [18042.665948] rc=$val; [18042.665948] fi; [18042.665948] exit $rc [18043.257668] Lustre: DEBUG MARKER: dmesg [18043.826615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 6: getstripe\/setstripe ============================================================= 12:39:28 \(1491568768\) [18043.973052] Lustre: DEBUG MARKER: == ost-pools test 6: getstripe/setstripe ============================================================= 12:39:28 (1491568768) [18044.112204] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [18052.375746] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18052.632630] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18054.671657] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18054.925672] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18055.722767] Lustre: DEBUG MARKER: lctl pool_list lustre [18055.986698] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [18062.258155] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18062.258155] sort -u | tr '\n' ' ' [18062.514203] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18062.514203] sort -u | tr '\n' ' ' [18063.311852] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18063.311852] sort -u | tr '\n' ' ' [18063.567698] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18063.567698] sort -u | tr '\n' ' ' [18064.392935] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [18072.655646] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18072.910915] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18073.683183] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18073.935687] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18074.731378] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [18084.992061] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18084.992061] sort -u | tr '\n' ' ' [18085.247362] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18085.247362] sort -u | tr '\n' ' ' [18086.024227] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18086.024227] sort -u | tr '\n' ' ' [18086.280197] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18086.280197] sort -u | tr '\n' ' ' [18087.128460] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18088.018610] Lustre: DEBUG MARKER: rc=0; [18088.018610] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18088.018610] if [[ $? -eq 0 && $val -ne 0 ]]; then [18088.018610] echo $(hostname -s): $val; [18088.018610] rc=$val; [18088.018610] fi; [18088.018610] exit $rc [18088.613710] Lustre: DEBUG MARKER: dmesg [18089.122954] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18089.379749] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18089.635291] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [18101.899330] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [18112.169217] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [18122.442343] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [18132.708126] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [18142.970848] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [18151.235400] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [18163.496220] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [18169.758489] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [18178.020208] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18178.276433] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18179.046149] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18179.300438] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18180.107690] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18180.365687] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18180.632708] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [18188.901929] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [18197.171063] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18197.427581] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18199.471676] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18199.728254] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18200.685450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7a: create various pool name ======================================================= 12:42:05 \(1491568925\) [18200.847511] Lustre: DEBUG MARKER: == ost-pools test 7a: create various pool name ======================================================= 12:42:05 (1491568925) [18201.111907] Lustre: DEBUG MARKER: lctl pool_new lustre.1 [18207.377387] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 2>/dev/null || echo foo [18207.631402] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 2>/dev/null || echo foo [18208.407726] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 2>/dev/null || echo foo [18208.661467] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 2>/dev/null || echo foo [18209.462252] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.1 lustre-OST[0-1/1] [18219.728894] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 | [18219.728894] sort -u | tr '\n' ' ' [18219.997450] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.1 | [18219.997450] sort -u | tr '\n' ' ' [18220.793785] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 | [18220.793785] sort -u | tr '\n' ' ' [18221.055598] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.1 | [18221.055598] sort -u | tr '\n' ' ' [18221.890615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.1 [18222.146390] Lustre: DEBUG MARKER: lctl pool_remove lustre.1 lustre-OST0000_UUID [18232.408762] Lustre: DEBUG MARKER: lctl pool_remove lustre.1 lustre-OST0001_UUID [18238.669048] Lustre: DEBUG MARKER: lctl pool_destroy lustre.1 [18246.936058] Lustre: DEBUG MARKER: lctl pool_new lustre.133871033 [18257.213855] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871033 2>/dev/null || echo foo [18257.486494] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871033 2>/dev/null || echo foo [18258.270604] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871033 2>/dev/null || echo foo [18258.543400] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871033 2>/dev/null || echo foo [18259.352607] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.133871033 lustre-OST[0-1/1] [18267.617797] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871033 | [18267.617797] sort -u | tr '\n' ' ' [18267.877108] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871033 | [18267.877108] sort -u | tr '\n' ' ' [18268.658763] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871033 | [18268.658763] sort -u | tr '\n' ' ' [18268.915401] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871033 | [18268.915401] sort -u | tr '\n' ' ' [18269.735965] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.133871033 [18269.993305] Lustre: DEBUG MARKER: lctl pool_remove lustre.133871033 lustre-OST0000_UUID [18276.275312] Lustre: DEBUG MARKER: lctl pool_remove lustre.133871033 lustre-OST0001_UUID [18286.538933] Lustre: DEBUG MARKER: lctl pool_destroy lustre.133871033 [18296.803102] Lustre: DEBUG MARKER: lctl pool_new lustre.133871443223129 [18307.066938] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871443223129 2>/dev/null || echo foo [18307.322481] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871443223129 2>/dev/null || echo foo [18308.128607] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871443223129 2>/dev/null || echo foo [18308.385777] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871443223129 2>/dev/null || echo foo [18309.199888] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.133871443223129 lustre-OST[0-1/1] [18317.464492] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871443223129 | [18317.464492] sort -u | tr '\n' ' ' [18317.720232] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.133871443223129 | [18317.720232] sort -u | tr '\n' ' ' [18318.513501] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871443223129 | [18318.513501] sort -u | tr '\n' ' ' [18318.771185] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.133871443223129 | [18318.771185] sort -u | tr '\n' ' ' [18319.612336] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.133871443223129 [18319.869646] Lustre: DEBUG MARKER: lctl pool_remove lustre.133871443223129 lustre-OST0000_UUID [18326.135130] Lustre: DEBUG MARKER: lctl pool_remove lustre.133871443223129 lustre-OST0001_UUID [18336.401864] Lustre: DEBUG MARKER: lctl pool_destroy lustre.133871443223129 [18344.718582] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18345.612842] Lustre: DEBUG MARKER: rc=0; [18345.612842] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18345.612842] if [[ $? -eq 0 && $val -ne 0 ]]; then [18345.612842] echo $(hostname -s): $val; [18345.612842] rc=$val; [18345.612842] fi; [18345.612842] exit $rc [18346.216493] Lustre: DEBUG MARKER: dmesg [18346.721228] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18346.975209] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18347.233297] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18347.569053] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 12:44:32 \(1491569072\) [18347.720317] Lustre: DEBUG MARKER: == ost-pools test 7b: try to create pool name with invalid lengths or names ========================== 12:44:32 (1491569072) [18347.854730] Lustre: DEBUG MARKER: lctl pool_new qwerty [18348.112407] Lustre: DEBUG MARKER: lctl pool_new lustre. [18348.368375] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789^bdef [18348.623499] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abdefg [18348.929841] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18349.817597] Lustre: DEBUG MARKER: rc=0; [18349.817597] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18349.817597] if [[ $? -eq 0 && $val -ne 0 ]]; then [18349.817597] echo $(hostname -s): $val; [18349.817597] rc=$val; [18349.817597] fi; [18349.817597] exit $rc [18350.417576] Lustre: DEBUG MARKER: dmesg [18350.958724] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 12:44:35 \(1491569075\) [18351.120032] Lustre: DEBUG MARKER: == ost-pools test 7c: create a valid pool name and setstripe with a bad one ========================== 12:44:35 (1491569075) [18351.268731] Lustre: DEBUG MARKER: lctl pool_new lustre.0123456789abcde [18363.532670] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [18363.790090] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [18364.570074] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [18364.828494] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde 2>/dev/null || echo foo [18365.646272] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.0123456789abcde lustre-OST[0-1/1] [18375.919441] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [18375.919441] sort -u | tr '\n' ' ' [18376.176440] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.0123456789abcde | [18376.176440] sort -u | tr '\n' ' ' [18376.957731] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde | [18376.957731] sort -u | tr '\n' ' ' [18377.214745] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.0123456789abcde | [18377.214745] sort -u | tr '\n' ' ' [18378.037062] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.0123456789abcde [18378.294523] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0000_UUID [18384.555388] Lustre: DEBUG MARKER: lctl pool_remove lustre.0123456789abcde lustre-OST0001_UUID [18390.820904] Lustre: DEBUG MARKER: lctl pool_destroy lustre.0123456789abcde [18399.142794] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18400.028151] Lustre: DEBUG MARKER: rc=0; [18400.028151] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18400.028151] if [[ $? -eq 0 && $val -ne 0 ]]; then [18400.028151] echo $(hostname -s): $val; [18400.028151] rc=$val; [18400.028151] fi; [18400.028151] exit $rc [18400.629128] Lustre: DEBUG MARKER: dmesg [18401.118671] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18401.452391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 11: OSTs in overlapping\/multiple pools ============================================= 12:45:26 \(1491569126\) [18401.611921] Lustre: DEBUG MARKER: == ost-pools test 11: OSTs in overlapping/multiple pools ============================================= 12:45:26 (1491569126) [18401.753884] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [18408.035885] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18408.301378] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18409.074327] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18409.327540] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18410.128699] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [18420.402076] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18420.664692] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18421.445501] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18421.700965] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18422.497867] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [18430.767155] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [18441.036662] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18441.036662] sort -u | tr '\n' ' ' [18441.296508] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18441.296508] sort -u | tr '\n' ' ' [18442.088967] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18442.088967] sort -u | tr '\n' ' ' [18442.345550] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18442.345550] sort -u | tr '\n' ' ' [18443.272171] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18443.536736] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18443.814909] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18444.077702] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18444.355982] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18444.619023] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18444.898798] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18445.165460] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18445.445348] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18445.710632] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18445.986618] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18446.250078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18446.528647] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18446.792716] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18447.072729] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18447.335578] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18447.622214] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18447.896195] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18448.196600] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18448.461130] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18448.736423] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18449.001634] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18449.282986] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18449.549452] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18449.826835] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18450.093621] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18450.375251] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18450.642424] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18450.920721] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18451.188450] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18451.470514] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18451.736375] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18452.017213] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18452.282239] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18452.563859] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18452.829041] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18453.111829] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18453.389319] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18453.669242] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18453.934507] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18454.210074] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18454.474183] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18454.754391] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18455.019685] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18455.301085] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18455.580179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18455.873487] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18456.142332] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18456.419041] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18456.682507] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18456.962490] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18457.228738] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18457.511470] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18457.777867] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18458.061106] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18458.325845] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18458.604804] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18458.868657] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18459.143942] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18459.408852] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18459.687707] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18459.958446] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18460.238831] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18460.505241] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18460.853768] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18461.125273] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18461.400962] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18461.664665] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18461.944074] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18462.206480] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18462.489570] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18462.756517] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18463.034312] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18463.296877] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18463.572160] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18463.845712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18464.124785] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18464.393582] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18464.687100] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18464.953915] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18465.232372] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18465.492518] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18465.767100] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18466.028226] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18466.302429] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18466.564174] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18466.840535] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18467.102364] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18467.378743] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18467.642973] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18467.919493] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18468.182582] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18468.458862] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18468.723050] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18469.015194] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18469.276606] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18469.552144] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18469.812714] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18470.085517] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18470.347023] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18470.626381] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18470.900582] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18471.183570] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18471.449643] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18471.729142] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18471.993052] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18472.275883] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18472.543364] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18472.821471] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18473.087733] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18473.366169] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18473.631827] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18473.908514] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18474.183761] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18474.461161] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18474.723431] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18475.001044] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18475.267495] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18475.544177] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18475.806509] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18476.082128] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18476.347401] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18476.629915] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18476.892933] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18477.169789] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18477.433717] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18477.713672] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18477.979591] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18478.258035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18478.519855] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18478.799384] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18479.065768] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18479.359122] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18479.623760] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18479.903577] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18480.170481] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18480.455601] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18480.718233] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18480.997894] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18481.259385] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18481.539594] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18481.802100] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18482.083980] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18482.348604] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18482.628941] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18482.893490] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18483.182291] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18483.446864] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18483.727487] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18483.994730] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18484.272213] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18484.534289] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18484.809955] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18485.076184] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18485.356462] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18485.620324] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18485.904263] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18486.177196] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18486.455582] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18486.718091] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18486.993676] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18487.258158] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18487.539530] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18487.803473] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18488.081679] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18488.346934] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18488.627274] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18488.892341] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18489.169106] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18489.430967] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18489.728503] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18489.996774] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18490.275123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18490.537664] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18490.814193] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18491.076332] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18491.351414] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18491.616467] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18491.894484] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18492.160117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18492.438587] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18492.706048] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18492.985081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18493.248637] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18493.524433] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18493.786111] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18494.066274] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18494.329114] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18494.607284] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18494.869983] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18495.148895] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18495.411104] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18495.688936] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18495.948752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18496.225965] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18496.486904] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18496.764874] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18497.028303] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18497.308206] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18497.571900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18497.971340] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18498.235163] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18498.508906] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18498.768807] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18499.043500] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18499.305998] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18499.581356] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18499.842375] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18500.133685] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18500.396034] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18500.676373] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18500.939414] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18501.225035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18501.490157] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18501.768835] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18502.034895] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18502.314736] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18502.581179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18502.862721] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18503.126342] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18503.404977] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18503.667088] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18503.948705] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18504.212213] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18504.488094] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18504.751898] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18505.032513] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18505.305546] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18505.586740] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18505.849253] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18506.127825] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18506.392554] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18506.667929] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18506.932035] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18507.210248] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18507.477073] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18507.755243] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18508.020328] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18508.300796] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18508.566770] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18508.848498] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18509.113078] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18509.392748] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18509.654615] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18509.935443] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18510.198339] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18510.475857] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18510.739923] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18511.020463] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18511.283557] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18511.560619] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18511.823180] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18512.101658] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18512.365767] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18512.645644] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18512.908725] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18513.184380] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18513.443933] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18513.721688] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18513.986847] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18514.266908] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18514.530095] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18514.805852] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18515.069997] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18515.346247] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18515.627382] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18515.903428] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18516.169329] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18516.446910] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18516.710188] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18516.989084] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18517.252461] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18517.529289] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18517.795032] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18518.078326] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18518.341155] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18518.623731] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18518.892895] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18519.174777] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18519.438536] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18519.717634] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18519.985166] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18520.267187] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18520.535030] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18520.829665] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18521.095354] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18521.372969] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18521.635514] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18521.914045] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18522.178610] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18522.457790] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18522.720225] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18522.998732] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18523.262929] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18523.540064] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18523.801413] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18524.080867] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18524.342624] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18524.617480] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18524.878455] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18525.160567] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18525.428334] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18525.708725] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18525.970680] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18526.248670] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18526.508429] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18526.781455] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18527.044212] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18527.321931] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18527.585678] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18527.860922] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18528.123294] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18528.394843] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18528.655399] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18528.928583] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18529.191877] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18529.466360] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18529.727926] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18530.002976] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18530.264482] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18530.546189] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18530.809411] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18531.098879] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18531.365062] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18531.642858] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18531.908753] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18532.187076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18532.453077] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18532.731227] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18532.995889] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18533.271153] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18533.535444] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18533.810884] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18534.073480] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18534.350499] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18534.613975] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18534.891214] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18535.153892] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18535.430154] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18535.694065] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18535.970213] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18536.236073] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18536.521829] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18536.797225] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18537.090378] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18537.365152] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18537.643916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18537.909116] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18538.186249] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18538.455375] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18538.740520] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18539.002920] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18539.281256] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18539.544754] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18539.820834] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18540.084167] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18540.359999] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18540.627451] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18540.901967] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18541.166454] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18541.452229] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18541.717071] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18542.001700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18542.264828] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18542.544706] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18542.809076] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18543.086333] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18543.350473] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18543.628862] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18543.891866] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18544.173712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18544.439287] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18544.719756] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18544.983984] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18545.265699] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18545.528522] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18545.805591] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18546.070169] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18546.353631] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18546.628687] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18546.911525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18547.177896] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18547.459466] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18547.723182] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18548.001291] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18548.267563] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18548.545082] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18548.806625] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18549.082586] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18549.345409] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18549.629417] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18549.903989] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18550.187243] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18550.452285] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18550.730511] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18550.993277] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18551.270824] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18551.533043] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18551.816611] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18552.082879] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18552.537510] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18553.425708] Lustre: DEBUG MARKER: rc=0; [18553.425708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18553.425708] if [[ $? -eq 0 && $val -ne 0 ]]; then [18553.425708] echo $(hostname -s): $val; [18553.425708] rc=$val; [18553.425708] fi; [18553.425708] exit $rc [18554.010238] Lustre: DEBUG MARKER: dmesg [18554.483399] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18554.748494] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18555.000734] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [18565.261340] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [18571.524965] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [18583.796178] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [18594.068217] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [18604.336887] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18604.592529] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18605.379293] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18605.634286] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18606.429226] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18606.690406] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18606.945960] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [18617.213900] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [18623.476687] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [18635.734084] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [18643.994183] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [18648.280464] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18648.538346] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18649.329066] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18649.586664] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18650.475309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 12: OST Pool Membership ============================================================ 12:49:35 \(1491569375\) [18650.636185] Lustre: DEBUG MARKER: == ost-pools test 12: OST Pool Membership ============================================================ 12:49:35 (1491569375) [18650.796179] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [18655.069124] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18655.325661] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18656.103778] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18656.362997] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18659.180951] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [18663.445256] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18663.701627] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18669.499089] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18669.752757] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18670.548161] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [18672.814119] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [18679.077748] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18679.077748] sort -u | tr '\n' ' ' [18679.333591] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18679.333591] sort -u | tr '\n' ' ' [18680.111355] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18680.111355] sort -u | tr '\n' ' ' [18680.368278] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18680.368278] sort -u | tr '\n' ' ' [18681.209116] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18681.472939] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18681.751041] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18682.016286] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18682.295904] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST[0000] [18688.563037] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool [18688.822465] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0000] [18697.091893] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18697.091893] sort -u | tr '\n' ' ' [18697.366883] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18697.366883] sort -u | tr '\n' ' ' [18699.418817] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18699.418817] sort -u | tr '\n' ' ' [18699.695548] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18699.695548] sort -u | tr '\n' ' ' [18700.507599] Lustre: DEBUG MARKER: lctl pool_list lustre.testpool2 [18700.866302] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18701.147808] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18701.433392] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18701.697998] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18702.033343] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18702.932497] Lustre: DEBUG MARKER: rc=0; [18702.932497] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18702.932497] if [[ $? -eq 0 && $val -ne 0 ]]; then [18702.932497] echo $(hostname -s): $val; [18702.932497] rc=$val; [18702.932497] fi; [18702.932497] exit $rc [18703.547947] Lustre: DEBUG MARKER: dmesg [18704.020086] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18704.281952] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18704.538973] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [18710.802387] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [18721.064505] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [18727.328320] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [18737.590900] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18737.848393] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18738.625127] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18738.879271] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18739.680540] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18739.936915] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [18740.193548] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [18746.457620] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [18754.720265] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [18760.988401] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [18771.250329] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [18777.518787] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [18783.793617] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18784.046633] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18784.822488] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18785.077756] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18785.973645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 13: Striping characteristics in a pool ============================================= 12:51:50 \(1491569510\) [18786.126073] Lustre: DEBUG MARKER: == ost-pools test 13: Striping characteristics in a pool ============================================= 12:51:50 (1491569510) [18786.268948] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [18796.535357] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18796.788960] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18797.559530] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18797.815493] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18798.609626] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [18804.882294] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18804.882294] sort -u | tr '\n' ' ' [18805.140761] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [18805.140761] sort -u | tr '\n' ' ' [18808.440529] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18808.440529] sort -u | tr '\n' ' ' [18808.700073] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [18808.700073] sort -u | tr '\n' ' ' [18809.533916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18809.798139] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18810.091514] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18810.360956] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18810.652081] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18810.919103] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18811.210025] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18811.476162] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18811.773123] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18812.040917] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18812.331635] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18812.599903] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18812.891917] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18813.159835] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18813.460445] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18813.730112] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18814.026618] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18814.301044] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18814.597359] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18814.865700] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18815.243559] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18815.508744] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18815.786585] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18816.054072] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18816.336143] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18816.603902] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18816.879410] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18817.151209] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18817.432273] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18817.701163] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18817.980132] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18818.248548] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18818.529649] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18818.796941] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18819.077991] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18819.345977] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18819.637415] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18819.908965] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18820.231727] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18820.501536] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18820.805986] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18821.073079] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18821.349451] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18821.617110] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18821.900593] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18822.168178] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18822.444396] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18822.711896] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18822.993069] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18823.258456] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18823.534157] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18823.798491] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18824.077101] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18824.345412] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18824.641570] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18824.914066] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18825.195956] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18825.462621] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18825.740023] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18826.005592] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18826.307877] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18826.573450] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18826.848496] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18827.115177] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18827.390160] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18827.656971] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18827.973635] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18828.243683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18828.515720] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18828.782876] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18829.058282] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18829.324940] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18829.602366] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18829.869620] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18830.144543] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18830.412146] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18830.686274] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18830.953162] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18831.224918] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18831.490408] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18831.783714] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18832.046067] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18832.318637] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18832.585086] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18832.859378] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18833.124972] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18833.397457] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18833.662161] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18833.934634] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18834.205812] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18834.481056] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18834.745022] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18835.022294] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18835.290290] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18835.563605] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18835.829250] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18836.104621] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18836.373691] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18836.650974] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18836.916667] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18837.300550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18838.190414] Lustre: DEBUG MARKER: rc=0; [18838.190414] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18838.190414] if [[ $? -eq 0 && $val -ne 0 ]]; then [18838.190414] echo $(hostname -s): $val; [18838.190414] rc=$val; [18838.190414] fi; [18838.190414] exit $rc [18838.817818] Lustre: DEBUG MARKER: dmesg [18839.326293] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [18839.581851] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [18839.837136] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [18846.101051] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [18856.360883] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [18866.625304] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [18874.887928] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [18883.150859] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [18889.420667] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [18899.697892] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [18909.963690] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [18920.232270] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18920.486225] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [18921.261731] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18921.526382] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [18922.424406] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 12:54:07 \(1491569647\) [18922.574532] Lustre: DEBUG MARKER: == ost-pools test 14: Round robin and QOS striping within a pool ===================================== 12:54:07 (1491569647) [18922.772553] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_14 OST size 16504586 is larger than 2000000 [18922.934638] Lustre: DEBUG MARKER: SKIP: ost-pools test_14 OST size 16504586 is larger than 2000000 [18923.130523] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [18924.020284] Lustre: DEBUG MARKER: rc=0; [18924.020284] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [18924.020284] if [[ $? -eq 0 && $val -ne 0 ]]; then [18924.020284] echo $(hostname -s): $val; [18924.020284] rc=$val; [18924.020284] fi; [18924.020284] exit $rc [18924.613835] Lustre: DEBUG MARKER: dmesg [18925.199100] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 15: One directory per OST\/pool ===================================================== 12:54:10 \(1491569650\) [18925.368728] Lustre: DEBUG MARKER: == ost-pools test 15: One directory per OST/pool ===================================================== 12:54:10 (1491569650) [18925.505852] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool0 [18935.766881] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [18936.022323] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [18936.795615] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [18937.048936] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [18937.850702] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool0 lustre-OST[0000] [18946.113602] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [18946.113602] sort -u | tr '\n' ' ' [18946.369035] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 | [18946.369035] sort -u | tr '\n' ' ' [18947.148716] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 | [18947.148716] sort -u | tr '\n' ' ' [18947.404413] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 | [18947.404413] sort -u | tr '\n' ' ' [18949.416022] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [18955.687316] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [18955.943118] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [18956.722182] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [18956.973884] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [18957.775842] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 lustre-OST[0001] [18962.037770] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [18962.037770] sort -u | tr '\n' ' ' [18962.292514] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 | [18962.292514] sort -u | tr '\n' ' ' [18963.076697] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 | [18963.076697] sort -u | tr '\n' ' ' [18963.340679] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 | [18963.340679] sort -u | tr '\n' ' ' [18967.432496] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [18977.704809] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18977.959687] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [18978.742183] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18979.014587] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [18979.827372] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0002] [18990.095925] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18990.095925] sort -u | tr '\n' ' ' [18990.353507] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [18990.353507] sort -u | tr '\n' ' ' [18991.133324] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18991.133324] sort -u | tr '\n' ' ' [18991.408389] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [18991.408389] sort -u | tr '\n' ' ' [18993.585518] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool3 [19001.860259] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [19002.119991] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [19002.899450] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [19003.154115] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [19003.984749] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool3 lustre-OST[0003] [19014.260636] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 | [19014.260636] sort -u | tr '\n' ' ' [19014.518870] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 | [19014.518870] sort -u | tr '\n' ' ' [19015.302427] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 | [19015.302427] sort -u | tr '\n' ' ' [19015.578282] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 | [19015.578282] sort -u | tr '\n' ' ' [19017.614783] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool4 [19025.908857] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [19026.165369] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [19026.933669] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [19027.188052] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [19027.989624] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool4 lustre-OST[0004] [19036.266967] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 | [19036.266967] sort -u | tr '\n' ' ' [19036.532100] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 | [19036.532100] sort -u | tr '\n' ' ' [19037.321913] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 | [19037.321913] sort -u | tr '\n' ' ' [19037.578346] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 | [19037.578346] sort -u | tr '\n' ' ' [19039.632896] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool5 [19047.902597] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [19048.162661] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [19048.951830] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [19049.217289] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [19050.015467] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool5 lustre-OST[0005] [19056.279704] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 | [19056.279704] sort -u | tr '\n' ' ' [19056.538594] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 | [19056.538594] sort -u | tr '\n' ' ' [19057.339294] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 | [19057.339294] sort -u | tr '\n' ' ' [19057.596600] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 | [19057.596600] sort -u | tr '\n' ' ' [19059.607987] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool6 [19065.882204] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [19066.138031] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [19066.908995] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [19067.165349] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [19067.997285] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool6 lustre-OST[0006] [19074.263697] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 | [19074.263697] sort -u | tr '\n' ' ' [19074.521042] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 | [19074.521042] sort -u | tr '\n' ' ' [19075.307111] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 | [19075.307111] sort -u | tr '\n' ' ' [19075.566717] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 | [19075.566717] sort -u | tr '\n' ' ' [19077.578364] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool7 [19081.841654] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [19082.096715] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [19085.391598] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [19085.648123] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [19086.467088] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool7 lustre-OST[0007] [19090.737721] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 | [19090.737721] sort -u | tr '\n' ' ' [19090.997576] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 | [19090.997576] sort -u | tr '\n' ' ' [19095.607271] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 | [19095.607271] sort -u | tr '\n' ' ' [19095.863779] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 | [19095.863779] sort -u | tr '\n' ' ' [19097.883806] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19098.779479] Lustre: DEBUG MARKER: rc=0; [19098.779479] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19098.779479] if [[ $? -eq 0 && $val -ne 0 ]]; then [19098.779479] echo $(hostname -s): $val; [19098.779479] rc=$val; [19098.779479] fi; [19098.779479] exit $rc [19099.374783] Lustre: DEBUG MARKER: dmesg [19099.866794] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19100.123332] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool0 [19100.380375] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool0 lustre-OST0000_UUID [19112.647117] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool0 [19120.918538] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [19121.175674] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool0 2>/dev/null || echo foo [19121.946402] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [19122.199712] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool0 2>/dev/null || echo foo [19123.010358] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19123.268458] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [19123.523885] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0001_UUID [19131.837293] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [19138.105196] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [19138.361574] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [19139.147375] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [19139.407171] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [19140.221584] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19140.488550] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19140.747045] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0002_UUID [19145.019218] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [19153.293395] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19153.549786] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19154.326613] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19154.585564] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19155.389572] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19155.647435] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool3 [19155.902829] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool3 lustre-OST0003_UUID [19162.171988] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool3 [19172.440699] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [19172.695897] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool3 2>/dev/null || echo foo [19173.472917] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [19173.727730] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool3 2>/dev/null || echo foo [19174.519154] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19174.775242] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool4 [19175.030388] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool4 lustre-OST0004_UUID [19177.291861] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool4 [19187.552750] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [19187.809669] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool4 2>/dev/null || echo foo [19188.583141] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [19188.838189] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool4 2>/dev/null || echo foo [19189.637536] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19189.894672] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool5 [19190.149403] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool5 lustre-OST0005_UUID [19196.411343] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool5 [19202.671180] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [19202.927408] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool5 2>/dev/null || echo foo [19203.703959] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [19203.960169] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool5 2>/dev/null || echo foo [19204.764462] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19205.021990] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool6 [19205.277965] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool6 lustre-OST0006_UUID [19209.537671] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool6 [19217.801046] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [19218.055482] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool6 2>/dev/null || echo foo [19218.831757] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [19219.085200] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool6 2>/dev/null || echo foo [19219.891192] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19220.147454] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool7 [19220.404332] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool7 lustre-OST0007_UUID [19224.665536] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool7 [19232.932797] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [19233.188987] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool7 2>/dev/null || echo foo [19233.962643] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [19234.217251] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool7 2>/dev/null || echo foo [19235.198914] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 16: Inheritance of pool properties ================================================= 12:59:20 \(1491569960\) [19235.358855] Lustre: DEBUG MARKER: == ost-pools test 16: Inheritance of pool properties ================================================= 12:59:20 (1491569960) [19235.490773] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19239.751786] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19240.009566] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19244.559041] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19244.812714] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19245.613779] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [19253.878764] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19253.878764] sort -u | tr '\n' ' ' [19254.135783] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19254.135783] sort -u | tr '\n' ' ' [19254.910949] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19254.910949] sort -u | tr '\n' ' ' [19255.188066] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19255.188066] sort -u | tr '\n' ' ' [19256.047229] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19256.318424] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19256.600568] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19256.866181] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19257.146026] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19257.413752] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19257.696392] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19257.962608] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19258.243152] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19258.510671] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19258.792447] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19259.055789] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19259.338036] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19259.604880] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19259.887444] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19260.155786] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19260.433755] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19260.698490] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19260.980863] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19261.261125] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19261.645458] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19262.534131] Lustre: DEBUG MARKER: rc=0; [19262.534131] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19262.534131] if [[ $? -eq 0 && $val -ne 0 ]]; then [19262.534131] echo $(hostname -s): $val; [19262.534131] rc=$val; [19262.534131] fi; [19262.534131] exit $rc [19263.134177] Lustre: DEBUG MARKER: dmesg [19263.628033] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19263.886378] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19264.140904] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19272.405502] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [19280.673114] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [19290.942911] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19301.208346] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19307.474212] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19307.743578] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19308.525509] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19308.788908] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19309.664980] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 17: Referencing an empty pool ====================================================== 13:00:34 \(1491570034\) [19309.819761] Lustre: DEBUG MARKER: == ost-pools test 17: Referencing an empty pool ====================================================== 13:00:34 (1491570034) [19309.956617] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19314.229698] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19314.494931] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19316.556451] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19316.810752] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19317.603814] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [19319.880300] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19319.880300] sort -u | tr '\n' ' ' [19320.138385] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19320.138385] sort -u | tr '\n' ' ' [19323.436435] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19323.436435] sort -u | tr '\n' ' ' [19323.693390] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19323.693390] sort -u | tr '\n' ' ' [19326.538382] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19326.804838] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19327.096627] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19327.366804] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19327.666379] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19327.934787] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19328.225286] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19328.491842] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19328.786839] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19329.061725] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19329.374107] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19329.658757] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19329.967836] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19330.241887] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19330.542356] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19330.819024] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19331.114318] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19331.383659] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19331.693032] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19331.965670] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19332.260281] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19332.515693] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19332.770098] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19341.031740] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [19349.296266] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [19357.566690] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [19365.829297] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [19374.092311] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [19380.368681] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19386.630754] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [19392.893154] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19401.159285] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19401.417816] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19402.186771] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19402.440346] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19403.330981] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19404.219138] Lustre: DEBUG MARKER: rc=0; [19404.219138] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19404.219138] if [[ $? -eq 0 && $val -ne 0 ]]; then [19404.219138] echo $(hostname -s): $val; [19404.219138] rc=$val; [19404.219138] fi; [19404.219138] exit $rc [19404.812137] Lustre: DEBUG MARKER: dmesg [19405.374338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_18 skipping SLOW test 18 [19405.545074] Lustre: DEBUG MARKER: SKIP: ost-pools test_18 skipping SLOW test 18 [19405.748932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 19: Pools should not come into play when not specified ============================= 13:02:10 \(1491570130\) [19405.919851] Lustre: DEBUG MARKER: == ost-pools test 19: Pools should not come into play when not specified ============================= 13:02:10 (1491570130) [19406.056112] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19412.336348] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19412.601841] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19414.627038] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19414.885062] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19415.681359] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [19423.951021] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19423.951021] sort -u | tr '\n' ' ' [19424.209974] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19424.209974] sort -u | tr '\n' ' ' [19425.023508] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19425.023508] sort -u | tr '\n' ' ' [19425.287214] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19425.287214] sort -u | tr '\n' ' ' [19426.121111] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19426.386456] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19426.667414] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19426.935963] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19427.219448] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19427.483777] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19427.764257] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19428.029732] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19428.306643] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19428.570466] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19428.848196] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19429.114488] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19429.393051] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19429.657595] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19429.937128] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19430.209053] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19430.496712] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19430.760263] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19431.038967] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19431.305845] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19431.587140] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19431.852200] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19432.129469] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19432.392916] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19432.812678] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19433.703154] Lustre: DEBUG MARKER: rc=0; [19433.703154] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19433.703154] if [[ $? -eq 0 && $val -ne 0 ]]; then [19433.703154] echo $(hostname -s): $val; [19433.703154] rc=$val; [19433.703154] fi; [19433.703154] exit $rc [19434.320532] Lustre: DEBUG MARKER: dmesg [19434.801741] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19435.057074] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19435.316425] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19445.579716] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [19455.843401] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [19466.107441] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19474.369713] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19482.640857] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19482.898637] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19483.675038] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19483.934469] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19484.810056] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 13:03:29 \(1491570209\) [19484.961652] Lustre: DEBUG MARKER: == ost-pools test 20: Different pools in a directory hierarchy. ====================================== 13:03:29 (1491570209) [19485.097426] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19493.359947] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19493.612571] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19494.385062] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19494.641963] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19495.442305] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [19505.717502] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19505.974422] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19506.753091] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19507.024654] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19507.853431] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [19514.129936] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19514.129936] sort -u | tr '\n' ' ' [19514.390203] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19514.390203] sort -u | tr '\n' ' ' [19515.174739] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19515.174739] sort -u | tr '\n' ' ' [19515.464782] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19515.464782] sort -u | tr '\n' ' ' [19517.287408] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 lustre-OST[0001-0007/2] [19527.555123] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [19527.555123] sort -u | tr '\n' ' ' [19527.812965] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [19527.812965] sort -u | tr '\n' ' ' [19528.585745] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [19528.585745] sort -u | tr '\n' ' ' [19528.907965] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [19528.907965] sort -u | tr '\n' ' ' [19529.776087] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19530.092117] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19530.371900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19530.639900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19531.158488] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19532.069254] Lustre: DEBUG MARKER: rc=0; [19532.069254] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19532.069254] if [[ $? -eq 0 && $val -ne 0 ]]; then [19532.069254] echo $(hostname -s): $val; [19532.069254] rc=$val; [19532.069254] fi; [19532.069254] exit $rc [19532.669633] Lustre: DEBUG MARKER: dmesg [19533.175669] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19533.435605] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19533.690402] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19545.957714] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [19554.224360] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [19562.492356] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19572.763824] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19583.033417] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19583.290105] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19584.064892] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19584.322597] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19585.131894] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19585.390070] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19585.659068] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0001_UUID [19593.933149] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0003_UUID [19602.201119] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0005_UUID [19612.465664] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0007_UUID [19622.734916] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [19633.006091] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19633.262224] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19634.034336] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19634.289706] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19635.164795] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 13:06:00 \(1491570360\) [19635.328575] Lustre: DEBUG MARKER: == ost-pools test 21: OST pool with fewer OSTs than stripe count ===================================== 13:06:00 (1491570360) [19635.460515] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19641.726899] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19641.981182] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19642.757180] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19643.012230] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19643.813709] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/2] [19654.085832] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19654.085832] sort -u | tr '\n' ' ' [19654.342965] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19654.342965] sort -u | tr '\n' ' ' [19655.127207] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19655.127207] sort -u | tr '\n' ' ' [19655.388702] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19655.388702] sort -u | tr '\n' ' ' [19656.225328] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19656.494275] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19656.842031] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19657.734293] Lustre: DEBUG MARKER: rc=0; [19657.734293] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19657.734293] if [[ $? -eq 0 && $val -ne 0 ]]; then [19657.734293] echo $(hostname -s): $val; [19657.734293] rc=$val; [19657.734293] fi; [19657.734293] exit $rc [19658.329178] Lustre: DEBUG MARKER: dmesg [19658.833921] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19659.090381] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19659.346186] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19667.611046] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [19679.903588] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [19688.177717] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19696.458241] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19704.721700] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19704.979475] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19705.762742] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19706.019350] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19706.893160] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 13:07:11 \(1491570431\) [19707.068017] Lustre: DEBUG MARKER: == ost-pools test 22: Simultaneous manipulation of a pool ============================================ 13:07:11 (1491570431) [19707.200464] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19711.465539] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19711.722489] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19716.270789] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19716.525358] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19717.318155] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST0000 [19725.589198] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19725.589198] sort -u | tr '\n' ' ' [19725.848132] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19725.848132] sort -u | tr '\n' ' ' [19726.626263] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19726.626263] sort -u | tr '\n' ' ' [19726.883086] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19726.883086] sort -u | tr '\n' ' ' [19727.682093] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [19735.946664] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19736.203069] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19737.021123] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19737.276947] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19738.098912] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000 [19740.360856] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [19740.360856] sort -u | tr '\n' ' ' [19740.616361] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 | [19740.616361] sort -u | tr '\n' ' ' [19741.402317] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [19741.402317] sort -u | tr '\n' ' ' [19741.737982] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 | [19741.737982] sort -u | tr '\n' ' ' [19748.682725] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19748.743693] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19759.073491] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19759.073492] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19765.365096] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19765.369608] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19771.651260] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19771.653060] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19777.942071] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19777.945291] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19784.227613] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19784.230477] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19794.525310] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19794.525311] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19800.813610] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19800.815830] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19809.109454] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19809.109489] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19815.399050] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19815.406404] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19821.706164] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19821.707194] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19827.991813] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19827.995576] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19834.289832] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19834.298978] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19842.589819] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19842.593047] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19848.882159] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19848.885775] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19859.175890] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19859.178836] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19867.479297] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19867.479607] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19877.781003] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19877.781808] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19886.069658] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST[0000-0007/2] [19886.073579] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool OST[0000-0007/1] [19894.359173] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 OST[0002-0007/2] [19894.371713] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool OST[0001-0007/1] [19902.706391] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19903.594469] Lustre: DEBUG MARKER: rc=0; [19903.594469] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19903.594469] if [[ $? -eq 0 && $val -ne 0 ]]; then [19903.594469] echo $(hostname -s): $val; [19903.594469] rc=$val; [19903.594469] fi; [19903.594469] exit $rc [19904.187793] Lustre: DEBUG MARKER: dmesg [19904.666363] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19904.931483] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19905.186486] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19909.448696] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [19915.712172] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19915.966537] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19916.747561] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19917.009244] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19917.801418] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19918.057993] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [19918.313882] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [19922.575996] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [19930.844690] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19931.117623] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [19931.893657] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19932.155362] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [19933.042185] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 23a: OST pools and quota =========================================================== 13:10:57 \(1491570657\) [19933.201976] Lustre: DEBUG MARKER: == ost-pools test 23a: OST pools and quota =========================================================== 13:10:57 (1491570657) [19933.486289] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [19937.766815] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19938.026156] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [19943.825354] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19944.079744] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [19944.881610] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/3] [19953.157486] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19953.157486] sort -u | tr '\n' ' ' [19953.419266] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [19953.419266] sort -u | tr '\n' ' ' [19954.222540] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19954.222540] sort -u | tr '\n' ' ' [19954.481903] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [19954.481903] sort -u | tr '\n' ' ' [19955.319259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [19955.319259] /usr/sbin/lctl lustre_build_version 2>/dev/null || [19955.319259] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [19955.589429] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug [19966.868473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark first dd failed with EDQUOT. [19967.029958] Lustre: DEBUG MARKER: first dd failed with EDQUOT. [19967.253708] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [19968.143861] Lustre: DEBUG MARKER: rc=0; [19968.143861] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [19968.143861] if [[ $? -eq 0 && $val -ne 0 ]]; then [19968.143861] echo $(hostname -s): $val; [19968.143861] rc=$val; [19968.143861] fi; [19968.143861] exit $rc [19968.750463] Lustre: DEBUG MARKER: dmesg [19969.226307] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [19969.485345] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [19969.743208] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [19978.008146] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [19984.280836] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [19992.552383] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20002.826026] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20003.085694] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20003.866557] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20004.127086] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20005.008135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: ost-pools test_23b skipping SLOW test 23b [20005.163636] Lustre: DEBUG MARKER: SKIP: ost-pools test_23b skipping SLOW test 23b [20005.367461] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 13:12:10 \(1491570730\) [20005.534848] Lustre: DEBUG MARKER: == ost-pools test 24: Independence of pool from other setstripe parameters =========================== 13:12:10 (1491570730) [20005.671399] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20005.671399] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20005.671399] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20005.939514] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool [20014.202977] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20014.457877] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20015.253599] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20015.513124] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20016.335427] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool lustre-OST[0000-0007/1] [20026.607693] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20026.607693] sort -u | tr '\n' ' ' [20026.868669] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool | [20026.868669] sort -u | tr '\n' ' ' [20027.657816] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20027.657816] sort -u | tr '\n' ' ' [20027.916185] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool | [20027.916185] sort -u | tr '\n' ' ' [20028.763620] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20029.033528] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20029.337981] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20029.604683] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20029.909198] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20030.178888] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20030.482261] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20030.750471] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20031.050744] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20031.320087] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20031.618998] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20031.887841] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20032.194758] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20032.461223] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20032.765810] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20033.033216] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20033.340324] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20033.608145] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20033.915883] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20034.194051] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20034.526693] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20034.792179] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20035.078275] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20035.355945] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20035.648544] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20035.926992] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20036.212124] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20036.481864] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20036.772281] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20037.041366] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20037.441525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20037.707876] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20037.992431] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20038.260565] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20038.550230] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20038.818349] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20039.110525] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20039.375430] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20039.659737] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20039.925900] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20040.589105] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20041.489349] Lustre: DEBUG MARKER: rc=0; [20041.489349] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20041.489349] if [[ $? -eq 0 && $val -ne 0 ]]; then [20041.489349] echo $(hostname -s): $val; [20041.489349] rc=$val; [20041.489349] fi; [20041.489349] exit $rc [20042.082721] Lustre: DEBUG MARKER: dmesg [20042.575395] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20042.830869] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool [20043.086752] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0000_UUID [20051.357984] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0001_UUID [20057.632758] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0002_UUID [20063.895725] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0003_UUID [20074.158326] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0004_UUID [20080.424347] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0005_UUID [20088.688065] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0006_UUID [20096.951365] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool lustre-OST0007_UUID [20103.215570] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool [20109.478804] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20109.733825] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool 2>/dev/null || echo foo [20111.764777] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20112.019750] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool 2>/dev/null || echo foo [20112.894675] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 25: Create new pool and restart MDS ================================================ 13:13:57 \(1491570837\) [20113.057631] Lustre: DEBUG MARKER: == ost-pools test 25: Create new pool and restart MDS ================================================ 13:13:57 (1491570837) [20113.199475] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool1 [20121.465116] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [20121.718441] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [20122.492850] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [20122.755635] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [20123.562944] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool1 OST0000; sync [20133.907279] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [20134.167634] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [20134.295776] Lustre: Failing over lustre-MDT0000 [20134.408620] Lustre: server umount lustre-MDT0000 complete [20134.409981] Lustre: Skipped 2 previous similar messages [20134.522115] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [20134.525390] LustreError: Skipped 64 previous similar messages [20134.542954] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [20135.573184] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [20135.573212] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20135.573213] Lustre: Skipped 4 previous similar messages [20135.580022] LustreError: Skipped 4 previous similar messages [20144.857045] Lustre: DEBUG MARKER: hostname [20145.160215] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [20145.414452] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [20145.667035] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [20145.825529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20145.842660] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20145.846281] LustreError: Skipped 1 previous similar message [20145.848176] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [20145.850520] Lustre: Skipped 61 previous similar messages [20145.852341] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c451e5 to 0x52ac6ca176c8eb0b [20145.903583] LustreError: 4515:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [20145.907759] LustreError: 4515:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [20145.912302] LustreError: 4520:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [20145.924061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20145.926378] Lustre: Skipped 2 previous similar messages [20146.083550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [20146.342150] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [20146.839950] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20146.846508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20147.004240] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20147.006021] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20147.152895] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20147.409059] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20147.667479] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20147.928522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [20147.995187] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [20148.195279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20148.195279] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20148.195279] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20148.468431] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20148.468431] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20148.468431] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20148.741520] Lustre: DEBUG MARKER: lctl list_param osc.lustre-OST*-osc > /dev/null 2>&1 [20149.000687] Lustre: DEBUG MARKER: lctl get_param -n at_min [20149.264086] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [20149.762152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [20149.764744] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [20149.955715] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [20149.955946] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing wait_import_state FULL osc.lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [20150.153015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20150.155674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20150.314855] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20150.316185] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20150.917835] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20151.870323] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [20152.161461] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [20152.423753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [20159.756855] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [20160.028853] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [20160.365936] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20161.284096] Lustre: DEBUG MARKER: rc=0; [20161.284096] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20161.284096] if [[ $? -eq 0 && $val -ne 0 ]]; then [20161.284096] echo $(hostname -s): $val; [20161.284096] rc=$val; [20161.284096] fi; [20161.284096] exit $rc [20161.898849] Lustre: DEBUG MARKER: dmesg [20162.376369] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20162.653247] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool1 [20162.912996] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool1 lustre-OST0000_UUID [20165.213719] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool1 [20171.480572] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [20171.738980] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool1 2>/dev/null || echo foo [20173.771850] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [20174.029830] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool1 2>/dev/null || echo foo [20175.000963] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 13:14:59 \(1491570899\) [20175.160706] Lustre: DEBUG MARKER: == ost-pools test 26: Choose other OSTs in the pool first in the creation remedy ===================== 13:14:59 (1491570899) [20175.302119] Lustre: DEBUG MARKER: lctl pool_new lustre.testpool2 [20179.566990] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20179.822601] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20181.864235] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20182.121619] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20182.962204] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0000; sync [20192.368329] Lustre: DEBUG MARKER: lctl pool_add lustre.testpool2 OST0002; sync [20199.773241] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x147 [20200.039076] Lustre: DEBUG MARKER: lctl set_param fail_val=0 [20200.177998] Lustre: *** cfs_fail_loc=147, val=0*** [20200.310326] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [20200.569607] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20200.834435] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20201.167831] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20202.081772] Lustre: DEBUG MARKER: rc=0; [20202.081772] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20202.081772] if [[ $? -eq 0 && $val -ne 0 ]]; then [20202.081772] echo $(hostname -s): $val; [20202.081772] rc=$val; [20202.081772] fi; [20202.081772] exit $rc [20202.686629] Lustre: DEBUG MARKER: dmesg [20203.186698] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre [20203.449652] Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.testpool2 [20203.708309] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0000_UUID [20213.973397] Lustre: DEBUG MARKER: lctl pool_remove lustre.testpool2 lustre-OST0002_UUID [20224.245316] Lustre: DEBUG MARKER: lctl pool_destroy lustre.testpool2 [20234.521713] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20234.777724] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.testpool2 2>/dev/null || echo foo [20235.553634] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20235.807542] Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.testpool2 2>/dev/null || echo foo [20236.685136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == ost-pools test complete, duration 3016 sec ======================================================== 13:16:01 \(1491570961\) [20236.843315] Lustre: DEBUG MARKER: == ost-pools test complete, duration 3016 sec ======================================================== 13:16:01 (1491570961) [20237.797940] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [20238.055682] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [20238.186519] Lustre: Failing over lustre-MDT0000 [20238.321035] Lustre: server umount lustre-MDT0000 complete [20238.450178] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [20239.573478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [20239.577355] LustreError: Skipped 30 previous similar messages [20240.917140] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [20240.917232] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20240.917233] Lustre: Skipped 1 previous similar message [20240.924525] LustreError: Skipped 3 previous similar messages [20248.766404] Lustre: DEBUG MARKER: hostname [20249.082117] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [20249.333814] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [20249.351067] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491570966/real 1491570966] req@ffff880050ac9200 x1564004881821760/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491570973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20249.356878] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [20249.358999] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20249.588796] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [20249.748277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20249.773616] Lustre: MGS: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [20249.775980] Lustre: Skipped 25 previous similar messages [20255.362385] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c8eb0b to 0x52ac6ca176c9846a [20255.440719] LustreError: 10007:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [20255.447738] LustreError: 10012:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [20255.460629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20255.646960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [20255.907406] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [20256.404821] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20256.407683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20256.568505] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20256.578229] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20256.720712] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20256.982108] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20257.238297] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20257.499619] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [20258.266409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20258.289937] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20258.453709] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20258.508470] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [20258.534952] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [20260.453857] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [20260.483461] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [20260.750811] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [20260.824665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [20260.909507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [20261.018534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [20273.000128] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [20273.265923] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [20274.774121] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [20274.774122] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [20274.774123] Lustre: Skipped 22 previous similar messages [20274.782621] Lustre: Skipped 2 previous similar messages [20275.535093] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064f91b00 x1564004881842016/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [20275.541036] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [20276.587123] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [20276.590149] Lustre: Skipped 9 previous similar messages [20279.514210] Lustre: server umount lustre-MDT0000 complete [20279.643479] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [20280.453565] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [20280.456503] LustreError: Skipped 2 previous similar messages [20280.458951] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [20280.464167] Lustre: Skipped 3 previous similar messages [20286.792411] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [20287.047535] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [20287.599090] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064cc9e00 x1564004881842736/t0(0) o13->lustre-OST0002-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [20287.604777] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [20287.687033] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491571006/real 1491571006] req@ffff880060810f00 x1564004881842272/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491571013 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20287.694706] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [20287.697376] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20289.471063] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064cc8c00 x1564004881842784/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [20289.476768] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [20291.695055] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880064ccaa00 x1564004881842848/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [20291.700753] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [20291.795865] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [20293.420643] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [20361.747838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-ost-single ============----- Fri Apr 7 13:18:05 UTC 2017 [20361.901608] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-ost-single ============----- Fri Apr 7 13:18:05 UTC 2017 [20362.152341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test complete, duration -o sec ================================================== 13:18:06 \(1491571086\) [20362.301683] Lustre: DEBUG MARKER: == replay-ost-single test complete, duration -o sec ================================================== 13:18:06 (1491571086) [20362.616543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [20362.767326] Lustre: DEBUG MARKER: excepting tests: [20362.947081] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 5 [20363.094048] Lustre: DEBUG MARKER: skipping tests SLOW=no: 5 [20365.457156] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [20365.457156] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [20365.457156] if [ $running -ne $mpts ]; then [20365.457156] echo $(hostname) env are INSANE!; [20365.457156] exit 1; [20365.457156] fi [20366.078425] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [20366.078425] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [20366.078425] if [ $running -ne $mpts ]; then [20366.078425] echo $(hostname) env are INSANE!; [20366.078425] exit 1; [20366.078425] fi [20366.642965] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [20366.642965] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [20366.642965] if [ $running -ne $mpts ]; then [20366.642965] echo $(hostname) env are INSANE!; [20366.642965] exit 1; [20366.642965] fi [20369.147892] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [20369.405832] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [20369.653316] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [20369.898663] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [20370.055645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20370.074716] Lustre: MGS: Connection restored to 954b8a51-0e6a-8dd4-3466-44b6b2d3dca4 (at 0@lo) [20370.077139] Lustre: Skipped 25 previous similar messages [20370.117405] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20370.121972] LustreError: Skipped 67 previous similar messages [20370.144902] LustreError: 12459:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [20370.159618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20370.307130] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [20370.562492] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [20371.071567] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20371.092915] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20371.257658] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20371.268969] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20371.407405] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20371.662312] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20371.932052] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20372.190701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [20372.443191] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20374.721462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20374.878887] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20375.153045] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491571095/real 1491571095] req@ffff880064cc9b00 x1564004881843216/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491571100 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [20376.300453] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [20376.563030] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [20376.815992] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [20377.069551] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [20377.230234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20377.308068] LustreError: 13755:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [20377.503349] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [20377.760442] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [20378.259229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20378.259939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20378.423688] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20378.424435] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20378.572743] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20378.833293] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20379.088189] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [20379.346728] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [20379.602072] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20381.874327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20382.026472] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20385.396594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20385.548693] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20388.426292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20388.578899] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20391.420684] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20391.579542] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20394.438839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20394.597960] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20397.487945] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20397.647982] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20400.531601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20400.691543] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20403.602231] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20403.763778] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20406.658858] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20406.821162] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20408.999999] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20409.173404] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20410.607283] Lustre: DEBUG MARKER: lctl get_param -n timeout [20410.920901] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [20411.087453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [20411.224102] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [20416.626287] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [20416.912263] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20417.429146] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [20418.027690] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 0a: target handle mismatch \(bug 5317\) ====================================== 13:19:02 \(1491571142\) [20418.187464] Lustre: DEBUG MARKER: == replay-ost-single test 0a: target handle mismatch (bug 5317) ====================================== 13:19:02 (1491571142) [20428.878589] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20429.784314] Lustre: DEBUG MARKER: rc=0; [20429.784314] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20429.784314] if [[ $? -eq 0 && $val -ne 0 ]]; then [20429.784314] echo $(hostname -s): $val; [20429.784314] rc=$val; [20429.784314] fi; [20429.784314] exit $rc [20430.390123] Lustre: DEBUG MARKER: dmesg [20430.938158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 0b: empty replay =========================================================== 13:19:15 \(1491571155\) [20431.092978] Lustre: DEBUG MARKER: == replay-ost-single test 0b: empty replay =========================================================== 13:19:15 (1491571155) [20432.415424] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [20432.418419] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [20444.176942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20444.353417] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20445.812959] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20445.863533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20446.051367] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20446.080339] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20468.088729] Lustre: lustre-MDT0002: haven't heard from client 66f88f98-4416-58db-75bf-9c8ad587b109 (at 10.9.5.237@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp ffff88006501b400, cur 1491571193 expire 1491571163 last 1491571143 [20514.577388] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [20514.580476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [20514.770824] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [20514.807961] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 68 sec [20516.244498] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20517.230840] Lustre: DEBUG MARKER: rc=0; [20517.230840] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20517.230840] if [[ $? -eq 0 && $val -ne 0 ]]; then [20517.230840] echo $(hostname -s): $val; [20517.230840] rc=$val; [20517.230840] fi; [20517.230840] exit $rc [20517.837150] Lustre: DEBUG MARKER: dmesg [20518.454241] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 1: touch =================================================================== 13:20:43 \(1491571243\) [20518.620430] Lustre: DEBUG MARKER: == replay-ost-single test 1: touch =================================================================== 13:20:43 (1491571243) [20530.704905] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [20530.707621] Lustre: Skipped 42 previous similar messages [20531.629663] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20531.802298] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20533.235689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20533.238826] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20533.459666] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20533.470996] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20534.734929] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20534.752852] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20534.931404] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20534.980371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20536.339862] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20537.236517] Lustre: DEBUG MARKER: rc=0; [20537.236517] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20537.236517] if [[ $? -eq 0 && $val -ne 0 ]]; then [20537.236517] echo $(hostname -s): $val; [20537.236517] rc=$val; [20537.236517] fi; [20537.236517] exit $rc [20537.843925] Lustre: DEBUG MARKER: dmesg [20538.434372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 2: \|x\| 10 open\(O_CREAT\)s =================================================== 13:21:03 \(1491571263\) [20538.603625] Lustre: DEBUG MARKER: == replay-ost-single test 2: |x| 10 open(O_CREAT)s =================================================== 13:21:03 (1491571263) [20553.688785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20553.852295] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20555.326309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20555.343528] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20555.613675] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20555.737467] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20557.996207] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20558.103921] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20558.237148] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20558.307990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20559.700312] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20560.593400] Lustre: DEBUG MARKER: rc=0; [20560.593400] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20560.593400] if [[ $? -eq 0 && $val -ne 0 ]]; then [20560.593400] echo $(hostname -s): $val; [20560.593400] rc=$val; [20560.593400] fi; [20560.593400] exit $rc [20561.196470] Lustre: DEBUG MARKER: dmesg [20561.786061] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 3: Fail OST during write, with verification ================================ 13:21:26 \(1491571286\) [20561.941696] Lustre: DEBUG MARKER: == replay-ost-single test 3: Fail OST during write, with verification ================================ 13:21:26 (1491571286) [20575.051976] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20575.213941] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20576.651467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20576.651468] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20576.846837] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20576.879604] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20579.126154] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20579.149047] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20579.300303] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20579.350165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20580.741700] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20581.642228] Lustre: DEBUG MARKER: rc=0; [20581.642228] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20581.642228] if [[ $? -eq 0 && $val -ne 0 ]]; then [20581.642228] echo $(hostname -s): $val; [20581.642228] rc=$val; [20581.642228] fi; [20581.642228] exit $rc [20582.271215] Lustre: DEBUG MARKER: dmesg [20582.838603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 4: Fail OST during read, with verification ================================= 13:21:47 \(1491571307\) [20583.012220] Lustre: DEBUG MARKER: == replay-ost-single test 4: Fail OST during read, with verification ================================= 13:21:47 (1491571307) [20585.723863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 10.9.5.244@tcp failed: rc = -107 [20585.726502] LustreError: Skipped 14 previous similar messages [20585.728648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [20585.733320] Lustre: Skipped 7 previous similar messages [20596.617642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20596.777440] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20598.231308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20598.264562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20598.457908] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20598.478621] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20599.663302] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20599.768946] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20599.823878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20599.979598] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 1 sec [20601.319784] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20602.216605] Lustre: DEBUG MARKER: rc=0; [20602.216605] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20602.216605] if [[ $? -eq 0 && $val -ne 0 ]]; then [20602.216605] echo $(hostname -s): $val; [20602.216605] rc=$val; [20602.216605] fi; [20602.216605] exit $rc [20602.825377] Lustre: DEBUG MARKER: dmesg [20603.417513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-ost-single test_5 skipping SLOW test 5 [20603.573225] Lustre: DEBUG MARKER: SKIP: replay-ost-single test_5 skipping SLOW test 5 [20603.773488] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 13:22:08 \(1491571328\) [20603.931710] Lustre: DEBUG MARKER: == replay-ost-single test 6: Fail OST before obd_destroy ============================================= 13:22:08 (1491571328) [20609.164785] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [20609.425799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [20615.308623] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [20615.631172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126889360 after_dd: 126884240 took 0 seconds [20615.799605] Lustre: DEBUG MARKER: before: 126889360 after_dd: 126884240 took 0 seconds [20615.806852] Lustre: *** cfs_fail_loc=119, val=2147483648*** [20615.809157] Lustre: Skipped 4 previous similar messages [20615.811425] LustreError: 12419:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800779f2d00 x1564025649379376/t47244640345(0) o36->c142831b-29df-271e-d6a2-4e361fca03bc@10.9.5.237@tcp:-1/-1 lens 624/456 e 0 to 0 dl 1491571347 ref 1 fl Interpret:/0/0 rc 0/0 [20622.820229] Lustre: lustre-MDT0000: Client c142831b-29df-271e-d6a2-4e361fca03bc (at 10.9.5.237@tcp) reconnecting [20622.823020] Lustre: Skipped 3 previous similar messages [20635.918327] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20636.072153] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20637.509906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20637.522523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20637.732914] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20637.746801] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20639.986290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20640.048683] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20640.194315] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20640.269302] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20642.069435] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [20642.253712] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [20642.471193] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [20642.744234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [20645.326432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20646.667895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126889360 after: 126889360 [20646.852142] Lustre: DEBUG MARKER: before: 126889360 after: 126889360 [20647.049387] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20647.959493] Lustre: DEBUG MARKER: rc=0; [20647.959493] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20647.959493] if [[ $? -eq 0 && $val -ne 0 ]]; then [20647.959493] echo $(hostname -s): $val; [20647.959493] rc=$val; [20647.959493] fi; [20647.959493] exit $rc [20648.568060] Lustre: DEBUG MARKER: dmesg [20649.167025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 13:22:53 \(1491571373\) [20649.340269] Lustre: DEBUG MARKER: == replay-ost-single test 7: Fail OST before obd_destroy ============================================= 13:22:53 (1491571373) [20654.527880] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [20654.788577] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [20659.632221] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126889360 after_dd: 126884240 took 1 seconds [20659.789034] Lustre: DEBUG MARKER: before: 126889360 after_dd: 126884240 took 1 seconds [20674.510450] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20674.672218] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20676.105208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20676.106130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20676.320916] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20676.335708] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20678.600796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20678.600804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20678.804269] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20678.825371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20680.651271] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [20680.813817] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 [20681.046707] Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null [20681.306285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed [20683.871516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20684.141287] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [20684.423664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20685.708597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20686.988570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20688.263174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [20689.634547] Lustre: DEBUG MARKER: /usr/sbin/lctl mark before: 126889360 after: 126889360 [20689.798130] Lustre: DEBUG MARKER: before: 126889360 after: 126889360 [20689.990665] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20690.887085] Lustre: DEBUG MARKER: rc=0; [20690.887085] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20690.887085] if [[ $? -eq 0 && $val -ne 0 ]]; then [20690.887085] echo $(hostname -s): $val; [20690.887085] rc=$val; [20690.887085] fi; [20690.887085] exit $rc [20691.499137] Lustre: DEBUG MARKER: dmesg [20692.065782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error ==================== 13:23:36 \(1491571416\) [20692.229984] Lustre: DEBUG MARKER: == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error ==================== 13:23:36 (1491571416) [20692.365649] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20692.365649] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20692.365649] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20719.746462] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20720.642855] Lustre: DEBUG MARKER: rc=0; [20720.642855] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20720.642855] if [[ $? -eq 0 && $val -ne 0 ]]; then [20720.642855] echo $(hostname -s): $val; [20720.642855] rc=$val; [20720.642855] fi; [20720.642855] exit $rc [20721.243844] Lustre: DEBUG MARKER: dmesg [20721.820977] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8b: Verify redo io: redo io should success after recovery ================== 13:24:06 \(1491571446\) [20721.983657] Lustre: DEBUG MARKER: == replay-ost-single test 8b: Verify redo io: redo io should success after recovery ================== 13:24:06 (1491571446) [20722.117769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20722.117769] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20722.117769] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20755.926055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20756.087394] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20757.532286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20757.532287] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20757.721607] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20757.768223] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20760.016158] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20760.043412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20760.227977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20760.310402] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [20769.986524] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20770.870712] Lustre: DEBUG MARKER: rc=0; [20770.870712] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20770.870712] if [[ $? -eq 0 && $val -ne 0 ]]; then [20770.870712] echo $(hostname -s): $val; [20770.870712] rc=$val; [20770.870712] fi; [20770.870712] exit $rc [20771.473940] Lustre: DEBUG MARKER: dmesg [20772.087485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction ===================== 13:24:56 \(1491571496\) [20772.237435] Lustre: DEBUG MARKER: == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction ===================== 13:24:56 (1491571496) [20772.376388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20772.376388] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20772.376388] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20815.904595] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20816.802708] Lustre: DEBUG MARKER: rc=0; [20816.802708] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20816.802708] if [[ $? -eq 0 && $val -ne 0 ]]; then [20816.802708] echo $(hostname -s): $val; [20816.802708] rc=$val; [20816.802708] fi; [20816.802708] exit $rc [20817.392223] Lustre: DEBUG MARKER: dmesg [20817.979459] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS =================================== 13:25:42 \(1491571542\) [20818.138079] Lustre: DEBUG MARKER: == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS =================================== 13:25:42 (1491571542) [20818.271050] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20818.271050] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20818.271050] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20818.539947] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x187 [20818.671294] Lustre: *** cfs_fail_loc=187, val=0*** [20819.673675] Lustre: *** cfs_fail_loc=187, val=0*** [20821.675669] Lustre: *** cfs_fail_loc=187, val=0*** [20824.677765] Lustre: *** cfs_fail_loc=187, val=0*** [20828.679734] Lustre: *** cfs_fail_loc=187, val=0*** [20838.808722] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [20839.809283] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x187 [20839.942501] Lustre: *** cfs_fail_loc=187, val=0*** [20839.944237] Lustre: Skipped 1 previous similar message [20860.075249] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [20861.132941] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20862.021916] Lustre: DEBUG MARKER: rc=0; [20862.021916] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20862.021916] if [[ $? -eq 0 && $val -ne 0 ]]; then [20862.021916] echo $(hostname -s): $val; [20862.021916] rc=$val; [20862.021916] fi; [20862.021916] exit $rc [20862.620102] Lustre: DEBUG MARKER: dmesg [20863.155906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS ===================== 13:26:27 \(1491571587\) [20863.315689] Lustre: DEBUG MARKER: == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS ===================== 13:26:27 (1491571587) [20863.456476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [20863.456476] /usr/sbin/lctl lustre_build_version 2>/dev/null || [20863.456476] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [20885.054941] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20885.935185] Lustre: DEBUG MARKER: rc=0; [20885.935185] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20885.935185] if [[ $? -eq 0 && $val -ne 0 ]]; then [20885.935185] echo $(hostname -s): $val; [20885.935185] rc=$val; [20885.935185] fi; [20885.935185] exit $rc [20886.528219] Lustre: DEBUG MARKER: dmesg [20889.073130] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 9: Verify that no req deadline happened during recovery ==================== 13:26:53 \(1491571613\) [20889.230393] Lustre: DEBUG MARKER: == replay-ost-single test 9: Verify that no req deadline happened during recovery ==================== 13:26:53 (1491571613) [20896.207576] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [20896.209878] LustreError: Skipped 7 previous similar messages [20896.211643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [20896.215564] Lustre: Skipped 7 previous similar messages [20903.933267] Lustre: lustre-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [20903.935488] Lustre: Skipped 28 previous similar messages [20904.862763] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20905.024820] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20906.459752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20906.462985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20906.660812] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20906.728030] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [20969.195210] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 62 sec [20969.293782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 62 sec [20969.549851] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 62 sec [20969.606635] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 62 sec [20971.605839] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [20972.520081] Lustre: DEBUG MARKER: rc=0; [20972.520081] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [20972.520081] if [[ $? -eq 0 && $val -ne 0 ]]; then [20972.520081] echo $(hostname -s): $val; [20972.520081] rc=$val; [20972.520081] fi; [20972.520081] exit $rc [20973.142788] Lustre: DEBUG MARKER: dmesg [20973.724726] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test 10: conflicting PW & PR locks on a client ================================== 13:28:18 \(1491571698\) [20973.880827] Lustre: DEBUG MARKER: == replay-ost-single test 10: conflicting PW [20990.916536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [20991.085031] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21034.098676] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21035.010418] Lustre: DEBUG MARKER: rc=0; [21035.010418] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21035.010418] if [[ $? -eq 0 && $val -ne 0 ]]; then [21035.010418] echo $(hostname -s): $val; [21035.010418] rc=$val; [21035.010418] fi; [21035.010418] exit $rc [21035.614355] Lustre: DEBUG MARKER: dmesg [21036.175722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-ost-single test complete, duration 675 sec ================================================= 13:29:21 \(1491571761\) [21036.338055] Lustre: DEBUG MARKER: == replay-ost-single test complete, duration 675 sec ================================================= 13:29:21 (1491571761) [21047.183022] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21047.441348] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [21049.199369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [21049.311061] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880065f0c600 x1564004881905232/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [21049.316195] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [21049.997342] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [21049.999862] Lustre: Skipped 8 previous similar messages [21051.455057] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880065f0fc00 x1564004881905296/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [21051.460132] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [21051.945063] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [21051.947435] Lustre: Skipped 3 previous similar messages [21053.677251] Lustre: server umount lustre-MDT0000 complete [21053.679395] Lustre: Skipped 1 previous similar message [21053.807145] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21054.199302] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [21054.204029] LustreError: Skipped 3 previous similar messages [21060.471050] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491571779/real 1491571779] req@ffff880050b66d00 x1564004881905472/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491571786 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21060.478169] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [21060.480915] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21060.951738] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [21061.206938] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [21061.487059] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880065f0db00 x1564004881905984/t0(0) o13->lustre-OST0003-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [21061.492580] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [21061.948867] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [21064.351059] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005fe24600 x1564004881906064/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [21067.020384] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [21067.023072] Lustre: Skipped 7 previous similar messages [21067.449261] Lustre: server umount lustre-MDT0002 complete [21067.576596] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21135.453413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: insanity ============----- Fri Apr 7 13:30:59 UTC 2017 [21135.608792] Lustre: DEBUG MARKER: -----============= acceptance-small: insanity ============----- Fri Apr 7 13:30:59 UTC 2017 [21135.864422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test complete, duration -o sec =========================================================== 13:31:00 \(1491571860\) [21136.028594] Lustre: DEBUG MARKER: == insanity test complete, duration -o sec =========================================================== 13:31:00 (1491571860) [21136.306834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 15 [21136.459868] Lustre: DEBUG MARKER: excepting tests: 15 [21138.932308] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [21138.932308] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [21138.932308] if [ $running -ne $mpts ]; then [21138.932308] echo $(hostname) env are INSANE!; [21138.932308] exit 1; [21138.932308] fi [21139.495312] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [21139.495312] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [21139.495312] if [ $running -ne $mpts ]; then [21139.495312] echo $(hostname) env are INSANE!; [21139.495312] exit 1; [21139.495312] fi [21140.075119] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [21140.075119] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [21140.075119] if [ $running -ne $mpts ]; then [21140.075119] echo $(hostname) env are INSANE!; [21140.075119] exit 1; [21140.075119] fi [21142.568852] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [21142.826266] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [21143.087708] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [21143.351252] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [21143.507795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21143.594892] LustreError: 27264:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [21143.599255] LustreError: 27264:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [21143.612790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21143.615123] Lustre: Skipped 1 previous similar message [21143.760284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21144.012754] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21144.502933] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21144.513854] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21144.671792] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21144.674557] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21144.815087] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21145.066714] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21145.319174] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21145.574522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [21145.825498] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21148.147789] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21148.315712] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21148.606107] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491571869/real 1491571869] req@ffff88005fe27900 x1564004881906448/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491571874 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21148.613779] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [21149.738904] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [21150.004371] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [21150.258389] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [21150.512331] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [21150.661674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21150.753837] LustreError: 28560:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [21150.758802] LustreError: 28564:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [21150.973155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21151.228920] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21151.869521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21151.878725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21152.037452] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21152.037581] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21152.186793] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21152.447925] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21152.703449] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21152.965484] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [21153.222670] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21155.576456] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21155.739620] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21159.143187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21159.301845] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21162.244996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21162.407212] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21165.288834] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21165.463210] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21168.372195] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21168.547105] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21171.427975] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21171.594338] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21174.504992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21174.682930] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21177.686052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21177.848523] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21180.768079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21180.925597] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21183.108248] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21183.260377] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21184.680500] Lustre: DEBUG MARKER: lctl get_param -n timeout [21184.999350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [21185.162558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [21185.302316] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [21190.709910] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [21190.985530] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21191.504362] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21192.187133] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 0: Fail all nodes, independently ==================================================== 13:31:57 \(1491571917\) [21192.344237] Lustre: DEBUG MARKER: == insanity test 0: Fail all nodes, independently ==================================================== 13:31:57 (1491571917) [21192.492271] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21192.746810] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [21192.877276] Lustre: Failing over lustre-MDT0000 [21192.958709] Lustre: server umount lustre-MDT0000 complete [21193.087148] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21203.392269] Lustre: DEBUG MARKER: hostname [21203.695542] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [21203.855053] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491571922/real 1491571922] req@ffff880061e20000 x1564004881915552/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491571929 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21203.863549] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [21203.866471] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21203.956653] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [21204.212425] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [21204.363965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21209.871176] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9a5c6 to 0x52ac6ca176c9aea2 [21209.925058] LustreError: 31270:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [21209.932593] LustreError: 31275:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [21209.944853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21209.947663] Lustre: Skipped 1 previous similar message [21210.119724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21210.394116] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21210.988878] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21210.997025] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21211.242040] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21211.244333] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21211.396629] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21211.650824] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21211.904606] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21212.167025] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [21212.723924] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [21212.916603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21212.920074] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21213.141334] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21213.156050] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21214.939266] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [21214.961837] Lustre: lustre-MDT0000: Recovery over after 0:02, of 5 clients 5 recovered and 0 were evicted. [21215.387259] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21215.533992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21215.604812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21215.714639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21229.975427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21230.143101] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21231.813908] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21231.815916] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21232.039184] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21232.043759] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21234.272510] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21234.416189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21234.498616] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21234.670205] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21235.955272] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [21236.213904] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [21236.345823] Lustre: Failing over lustre-MDT0002 [21236.447462] Lustre: server umount lustre-MDT0002 complete [21236.576064] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21246.873167] Lustre: DEBUG MARKER: hostname [21247.178273] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [21247.436808] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [21247.691616] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [21247.852080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21247.937613] LustreError: 876:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [21247.942490] LustreError: 880:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0002: cannot take the layout locks needed for registered restore: -2 [21247.983856] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [21248.138605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21248.396388] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21248.909759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21248.923467] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21249.062164] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 5 clients reconnect [21249.085471] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21249.087060] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21249.231258] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21249.487419] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21249.740897] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21249.999112] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [21250.753932] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21250.776705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21250.951188] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21251.005694] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21253.854568] Lustre: lustre-MDT0002: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [21254.222386] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [21254.343526] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [21254.408806] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [21254.560949] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 3 sec [21268.820728] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21268.980871] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21270.667899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21270.669252] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21270.861338] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21270.903755] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21273.173076] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21273.327978] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21274.085534] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [21274.232521] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 3 sec [21288.401380] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21288.569146] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21290.012752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [21290.015389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [21290.227079] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [21290.244729] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [21294.505274] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [21294.521817] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [21294.800858] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [21294.802718] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 4 sec [21308.955112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21309.123689] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21310.562519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [21310.563857] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [21310.732216] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [21310.811298] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0001-osc-*.ost_server_uuid [21314.066581] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 3 sec [21314.216910] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 3 sec [21314.948687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [21315.100724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 4 sec [21329.248913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21329.418454] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21330.854868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [21330.872751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [21331.086250] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [21331.086259] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0002-osc-*.ost_server_uuid [21334.346170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [21334.381655] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [21334.570522] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [21334.596545] Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-*.ost_server_uuid in FULL state after 3 sec [21348.791785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21348.956421] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21350.395573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [21350.398757] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [21350.610735] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [21350.637754] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0003-osc-*.ost_server_uuid [21350.841316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 0 sec [21350.920800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 0 sec [21350.991617] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 0 sec [21351.136611] Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-*.ost_server_uuid in FULL state after 0 sec [21365.354585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21365.514547] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21366.953874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [21366.954194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [21367.166041] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [21367.168714] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0004-osc-*.ost_server_uuid [21369.457320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 2 sec [21369.694457] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 2 sec [21370.447092] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 3 sec [21370.599330] Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-*.ost_server_uuid in FULL state after 3 sec [21384.747659] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21384.907659] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21386.355134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [21386.357189] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [21386.525891] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [21386.605215] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0005-osc-*.ost_server_uuid [21388.762953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [21388.878427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [21388.926298] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [21389.063797] Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-*.ost_server_uuid in FULL state after 2 sec [21403.301764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21403.542945] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21404.997031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [21405.000212] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [21405.207416] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [21405.222811] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0006-osc-*.ost_server_uuid [21405.420651] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [21405.509693] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [21405.579547] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [21405.724730] Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-*.ost_server_uuid in FULL state after 0 sec [21419.883127] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21420.055157] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21421.491277] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [21421.533985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [21421.904400] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [21421.919944] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0007-osc-*.ost_server_uuid [21424.139722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [21424.140261] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [21424.339854] Lustre: DEBUG MARKER: osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [21424.414233] Lustre: DEBUG MARKER: osc.lustre-OST0007-osc-*.ost_server_uuid in FULL state after 2 sec [21425.743587] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21426.645720] Lustre: DEBUG MARKER: rc=0; [21426.645720] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21426.645720] if [[ $? -eq 0 && $val -ne 0 ]]; then [21426.645720] echo $(hostname -s): $val; [21426.645720] rc=$val; [21426.645720] fi; [21426.645720] exit $rc [21427.265558] Lustre: DEBUG MARKER: dmesg [21427.861608] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 1: MDS\/MDS failure ================================================================== 13:35:52 \(1491572152\) [21428.020836] Lustre: DEBUG MARKER: == insanity test 1: MDS/MDS failure ================================================================== 13:35:52 (1491572152) [21429.297074] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21429.552470] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [21429.682903] Lustre: Failing over lustre-MDT0000 [21429.747900] Lustre: server umount lustre-MDT0000 complete [21429.878498] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21432.905014] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [21432.910216] LustreError: Skipped 99 previous similar messages [21440.359056] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491572158/real 1491572158] req@ffff880063d83c00 x1564004881942112/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491572165 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21440.366661] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 1 previous similar message [21440.369274] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21507.314986] Lustre: lustre-MDT0002: Connection restored to 41644d15-10ee-51db-e650-5dbbe8359511 (at 10.9.5.239@tcp) [21507.317774] Lustre: Skipped 125 previous similar messages [21535.373039] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491572236/real 1491572236] req@ffff88005ff8d500 x1564004881945264/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491572261 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21535.381104] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [21573.203874] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21574.182843] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21582.526642] Lustre: DEBUG MARKER: hostname [21582.798101] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [21583.063822] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [21583.317475] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [21583.569751] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [21583.721971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21595.373456] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9aea2 to 0x52ac6ca176c9bbde [21595.427365] LustreError: 6292:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [21595.434301] LustreError: 6297:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [21595.457324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21595.633173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21595.853896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [21595.903095] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21596.401039] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21596.405580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21597.114268] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21597.121846] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21597.606796] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21597.863585] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21598.122613] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21598.386434] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [21598.645859] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21600.439431] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [21600.463446] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [21601.905859] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21602.799511] Lustre: DEBUG MARKER: rc=0; [21602.799511] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21602.799511] if [[ $? -eq 0 && $val -ne 0 ]]; then [21602.799511] echo $(hostname -s): $val; [21602.799511] rc=$val; [21602.799511] fi; [21602.799511] exit $rc [21603.402414] Lustre: DEBUG MARKER: dmesg [21603.996774] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 2: Second Failure Mode: MDS\/OST Fri Apr 7 13:38:48 UTC 2017 ======================== 13:38:48 \(1491572328\) [21604.152112] Lustre: DEBUG MARKER: == insanity test 2: Second Failure Mode: MDS/OST Fri Apr 7 13:38:48 UTC 2017 ======================== 13:38:48 (1491572328) [21605.436971] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21605.693276] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [21605.824767] Lustre: Failing over lustre-MDT0000 [21605.930542] Lustre: server umount lustre-MDT0000 complete [21606.061962] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21610.440166] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [21610.443041] LustreError: Skipped 32 previous similar messages [21610.445373] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21610.450294] Lustre: Skipped 32 previous similar messages [21617.452047] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21647.257491] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [21647.527714] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [21647.658766] Lustre: Failing over lustre-MDT0002 [21647.856305] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21787.034877] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21787.199999] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21788.186347] Lustre: DEBUG MARKER: hostname [21788.453658] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [21788.718228] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [21788.969408] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [21789.223360] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [21789.383521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21789.465309] LustreError: 8454:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [21789.481068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21789.634934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21789.905162] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21790.421652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21790.424833] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21790.579824] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21790.593484] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21790.733396] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21790.995862] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21791.265873] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21791.531214] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [21791.786407] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21794.370405] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21794.474110] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491572515/real 1491572515] req@ffff880036001800 x1564004881954784/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491572520 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [21794.482510] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [21794.540598] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21795.964325] Lustre: DEBUG MARKER: hostname [21796.234617] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [21796.499594] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [21796.765272] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [21797.021507] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [21797.171091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21797.252936] LustreError: 9825:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [21797.449054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21797.711288] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21798.255622] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21798.261155] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21798.451195] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21798.452908] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21798.463570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [21798.608371] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21798.865753] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21799.121631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21799.388968] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [21799.645972] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21802.211642] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21802.376706] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21812.093757] Lustre: lustre-MDT0002: Recovery over after 0:13, of 5 clients 5 recovered and 0 were evicted. [21813.583134] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21814.476190] Lustre: DEBUG MARKER: rc=0; [21814.476190] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21814.476190] if [[ $? -eq 0 && $val -ne 0 ]]; then [21814.476190] echo $(hostname -s): $val; [21814.476190] rc=$val; [21814.476190] fi; [21814.476190] exit $rc [21815.082201] Lustre: DEBUG MARKER: dmesg [21815.691174] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 3: Thirdb Failure Mode: MDS\/CLIENT Fri Apr 7 13:42:20 UTC 2017 ===================== 13:42:20 \(1491572540\) [21815.847446] Lustre: DEBUG MARKER: == insanity test 3: Thirdb Failure Mode: MDS/CLIENT Fri Apr 7 13:42:20 UTC 2017 ===================== 13:42:20 (1491572540) [21815.991285] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21816.248586] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [21816.378377] Lustre: Failing over lustre-MDT0000 [21816.611100] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21824.223056] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21826.932039] Lustre: DEBUG MARKER: hostname [21827.248550] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [21827.504932] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [21827.757849] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [21827.916929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21830.227311] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9bfb2 to 0x52ac6ca176c9c745 [21830.270648] LustreError: 11654:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [21830.275431] LustreError: 11654:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [21830.280226] LustreError: 11659:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [21830.472229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21830.728795] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21831.292106] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21831.304147] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21831.473257] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21831.477362] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21831.618458] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21831.877553] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21832.134757] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21832.395341] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [21833.151203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21833.154119] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21833.310210] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21833.396111] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [21835.285861] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [21835.546455] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21835.693045] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21835.704479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21835.860924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [21850.178666] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21850.338638] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21852.041413] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21852.043428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21852.226118] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21852.299480] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [21854.460893] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21854.578899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21854.620362] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21854.792162] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [21856.066928] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [21856.323124] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [21856.677639] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21866.966404] Lustre: DEBUG MARKER: hostname [21867.275968] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [21867.537201] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [21867.795409] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [21867.955491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21868.037673] LustreError: 13680:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [21868.239617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [21868.498638] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [21868.636940] Lustre: lustre-MDT0002: Will be in recovery for at least 1:00, or until 5 clients reconnect [21868.639913] Lustre: Skipped 2 previous similar messages [21868.998660] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21868.998782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21869.227377] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21869.236524] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21869.377624] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [21869.636836] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21869.892121] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [21870.154982] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [21870.916906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21870.916913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21871.114815] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21871.154848] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [21873.078330] Lustre: lustre-MDT0002: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [21873.081380] Lustre: Skipped 2 previous similar messages [21873.356648] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [21873.457836] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [21873.509127] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [21873.646024] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [21887.930783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21888.093172] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21889.780410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21889.785178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21889.998993] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21890.003752] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [21892.262508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21892.279094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21892.475725] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21892.492691] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [21895.012899] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [21895.172709] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [21942.001469] Lustre: lustre-MDT0002: haven't heard from client b192a2ab-05cd-b329-9ae8-d946595bcefe (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff880060e75400, cur 1491572667 expire 1491572637 last 1491572619 [21942.007953] Lustre: Skipped 1 previous similar message [21950.092722] Lustre: lustre-MDT0000: haven't heard from client b192a2ab-05cd-b329-9ae8-d946595bcefe (at 10.9.5.238@tcp) in 54 seconds. I think it's dead, and I am evicting it. exp ffff8800600e9400, cur 1491572675 expire 1491572645 last 1491572621 [21950.099761] LNet: Service thread pid 8414 completed after 44.04s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [21952.798490] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21953.039787] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [21956.496676] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [21957.388529] Lustre: DEBUG MARKER: rc=0; [21957.388529] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [21957.388529] if [[ $? -eq 0 && $val -ne 0 ]]; then [21957.388529] echo $(hostname -s): $val; [21957.388529] rc=$val; [21957.388529] fi; [21957.388529] exit $rc [21957.997330] Lustre: DEBUG MARKER: dmesg [21958.571652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 4: Fourth Failure Mode: OST\/MDS Fri Apr 7 13:44:43 UTC 2017 ======================== 13:44:43 \(1491572683\) [21958.727185] Lustre: DEBUG MARKER: == insanity test 4: Fourth Failure Mode: OST/MDS Fri Apr 7 13:44:43 UTC 2017 ======================== 13:44:43 (1491572683) [21964.714353] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [21964.974665] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [21965.205185] Lustre: server umount lustre-MDT0000 complete [21965.207605] Lustre: Skipped 3 previous similar messages [21965.336300] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [21975.279044] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [22006.517415] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [22006.773180] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [22006.901976] Lustre: Failing over lustre-MDT0002 [22006.904249] Lustre: Skipped 2 previous similar messages [22007.101699] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22125.434673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22125.593180] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22126.557918] Lustre: DEBUG MARKER: hostname [22126.822186] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [22127.091596] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [22127.353445] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [22127.603171] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [22127.761416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22127.780477] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [22127.783987] Lustre: Skipped 123 previous similar messages [22127.866836] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [22127.871645] LustreError: Skipped 748 previous similar messages [22127.897143] LustreError: 16553:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [22127.901523] LustreError: 16553:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [22127.915052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [22127.917435] Lustre: Skipped 3 previous similar messages [22128.067449] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22128.322920] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22128.814263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22128.815846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22128.974049] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22128.978573] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22129.120908] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22129.371340] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22129.623170] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22129.877590] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [22130.130863] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22132.706641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22132.869503] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22132.908126] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491572853/real 1491572853] req@ffff8800600ee100 x1564004881981568/t0(0) o38->lustre-MDT0003-osp-MDT0000@10.9.5.239@tcp:24/4 lens 520/544 e 0 to 1 dl 1491572858 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [22132.916491] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [22134.309093] Lustre: DEBUG MARKER: hostname [22134.590584] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [22134.857891] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [22135.120412] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [22135.375735] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [22135.525368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22135.607458] LustreError: 17922:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [22135.802357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22136.064263] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22136.559863] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22136.559870] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22136.727221] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22136.728779] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22136.804148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [22136.882995] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22137.143195] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22137.402706] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22137.663088] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [22137.918068] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22140.550037] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22140.713117] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22144.637742] Lustre: lustre-MDT0002: Recovery over after 0:08, of 5 clients 5 recovered and 0 were evicted. [22149.630161] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22150.524757] Lustre: DEBUG MARKER: rc=0; [22150.524757] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22150.524757] if [[ $? -eq 0 && $val -ne 0 ]]; then [22150.524757] echo $(hostname -s): $val; [22150.524757] rc=$val; [22150.524757] fi; [22150.524757] exit $rc [22151.185523] Lustre: DEBUG MARKER: dmesg [22151.827603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 5: Fifth Failure Mode: OST\/OST Fri Apr 7 13:47:56 UTC 2017 ========================= 13:47:56 \(1491572876\) [22151.985648] Lustre: DEBUG MARKER: == insanity test 5: Fifth Failure Mode: OST/OST Fri Apr 7 13:47:56 UTC 2017 ========================= 13:47:56 (1491572876) [22187.235391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22187.398837] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22190.587409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22190.751583] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22212.993961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22213.904856] Lustre: DEBUG MARKER: rc=0; [22213.904856] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22213.904856] if [[ $? -eq 0 && $val -ne 0 ]]; then [22213.904856] echo $(hostname -s): $val; [22213.904856] rc=$val; [22213.904856] fi; [22213.904856] exit $rc [22214.512962] Lustre: DEBUG MARKER: dmesg [22215.132620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 6: Sixth Failure Mode: OST\/CLIENT Fri Apr 7 13:48:59 UTC 2017 ====================== 13:48:59 \(1491572939\) [22215.300085] Lustre: DEBUG MARKER: == insanity test 6: Sixth Failure Mode: OST/CLIENT Fri Apr 7 13:48:59 UTC 2017 ====================== 13:48:59 (1491572939) [22220.719424] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [22220.722162] LustreError: Skipped 24 previous similar messages [22220.724376] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [22220.729433] Lustre: Skipped 18 previous similar messages [22232.711203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: , to fail: 1, failed: 0 [22232.882726] Lustre: DEBUG MARKER: Request fail clients: , to fail: 1, failed: 0 [22277.088382] Lustre: lustre-MDT0002: haven't heard from client 464f4fa3-d90b-8648-e117-cdd767d5ad18 (at 10.9.5.238@tcp) in 49 seconds. I think it's dead, and I am evicting it. exp ffff880063272400, cur 1491573002 expire 1491572972 last 1491572953 [22280.645498] Lustre: lustre-MDT0000: haven't heard from client 464f4fa3-d90b-8648-e117-cdd767d5ad18 (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff880064711800, cur 1491573006 expire 1491572976 last 1491572958 [22288.059080] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22288.220090] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22290.441019] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22290.598883] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22361.241882] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22362.157377] Lustre: DEBUG MARKER: rc=0; [22362.157377] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22362.157377] if [[ $? -eq 0 && $val -ne 0 ]]; then [22362.157377] echo $(hostname -s): $val; [22362.157377] rc=$val; [22362.157377] fi; [22362.157377] exit $rc [22362.774998] Lustre: DEBUG MARKER: dmesg [22363.358234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 7: Seventh Failure Mode: CLIENT\/MDS Fri Apr 7 13:51:28 UTC 2017 ==================== 13:51:28 \(1491573088\) [22363.741450] Lustre: DEBUG MARKER: == insanity test 7: Seventh Failure Mode: CLIENT/MDS Fri Apr 7 13:51:28 UTC 2017 ==================== 13:51:28 (1491573088) [22365.333562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [22365.499323] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [22412.088733] Lustre: lustre-MDT0000: haven't heard from client f2ca86a2-0006-7051-fbaf-f9a5576526fb (at 10.9.5.238@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880051d93c00, cur 1491573137 expire 1491573107 last 1491573090 [22475.228402] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [22475.483345] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [22475.612927] Lustre: Failing over lustre-MDT0000 [22475.645508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [22475.839389] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22486.151436] Lustre: DEBUG MARKER: hostname [22486.451677] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [22486.702672] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [22486.955899] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [22487.126299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22487.145402] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [22487.152219] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9d220 to 0x52ac6ca176c9df4e [22487.201436] LustreError: 20802:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [22487.206404] LustreError: 20802:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [22487.211519] LustreError: 20807:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [22487.224487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [22487.227082] Lustre: Skipped 1 previous similar message [22487.390298] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22487.648794] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22488.143476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22488.143508] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22488.310964] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22488.311638] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22488.456193] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22488.710833] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22488.966657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22489.110930] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [22489.113788] Lustre: Skipped 1 previous similar message [22489.225683] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [22489.976399] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22490.024606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22490.197687] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22490.297698] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [22492.216463] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [22492.296931] Lustre: lustre-MDT0000: Recovery over after 0:03, of 4 clients 4 recovered and 0 were evicted. [22492.299919] Lustre: Skipped 1 previous similar message [22493.455473] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [22493.611808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [22507.918885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22508.102616] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22509.771560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [22509.787206] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [22510.000951] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [22510.014125] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [22512.263224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [22512.416678] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [22513.689355] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [22513.946686] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 [22514.306447] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [22524.594336] Lustre: DEBUG MARKER: hostname [22524.899920] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [22525.153959] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [22525.407948] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [22525.569824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [22525.860616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [22526.138234] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [22526.651767] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22526.661203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22526.819125] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22526.822204] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22526.968447] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [22527.231902] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22527.491736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [22527.753607] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [22528.518136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [22528.519804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [22528.733347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [22528.733575] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid [22531.024895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [22531.177393] Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 2 sec [22545.456457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22545.619142] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22547.288842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [22547.302991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [22547.477818] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [22547.524164] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid [22549.743635] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [22549.892564] Lustre: DEBUG MARKER: mdc.lustre-MDT0003-mdc-*.mds_server_uuid in FULL state after 2 sec [22552.711942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22552.886886] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22614.331294] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22615.228303] Lustre: DEBUG MARKER: rc=0; [22615.228303] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22615.228303] if [[ $? -eq 0 && $val -ne 0 ]]; then [22615.228303] echo $(hostname -s): $val; [22615.228303] rc=$val; [22615.228303] fi; [22615.228303] exit $rc [22615.847731] Lustre: DEBUG MARKER: dmesg [22616.442569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 8: Eighth Failure Mode: CLIENT\/OST Fri Apr 7 13:55:41 UTC 2017 ===================== 13:55:41 \(1491573341\) [22616.601707] Lustre: DEBUG MARKER: == insanity test 8: Eighth Failure Mode: CLIENT/OST Fri Apr 7 13:55:41 UTC 2017 ===================== 13:55:41 (1491573341) [22618.169471] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [22618.318777] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [22664.531433] Lustre: lustre-MDT0002: haven't heard from client 320a1bad-b95c-19ad-abd5-6d6bc48dcb9c (at 10.9.5.238@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff880052184800, cur 1491573390 expire 1491573360 last 1491573342 [22664.537461] Lustre: Skipped 1 previous similar message [22674.111252] LNet: Service thread pid 19719 completed after 44.89s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [22753.787084] Lustre: MGS: Connection restored to 72672d26-f8e0-002c-35d9-fe024c372f95 (at 10.9.5.238@tcp) [22753.790151] Lustre: Skipped 109 previous similar messages [22754.459156] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22754.615154] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22757.049938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22757.213363] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22822.526604] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [22823.424232] Lustre: DEBUG MARKER: rc=0; [22823.424232] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [22823.424232] if [[ $? -eq 0 && $val -ne 0 ]]; then [22823.424232] echo $(hostname -s): $val; [22823.424232] rc=$val; [22823.424232] fi; [22823.424232] exit $rc [22824.021778] Lustre: DEBUG MARKER: dmesg [22824.613149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 9: Ninth Failure Mode: CLIENT\/CLIENT Fri Apr 7 13:59:09 UTC 2017 =================== 13:59:09 \(1491573549\) [22824.774811] Lustre: DEBUG MARKER: == insanity test 9: Ninth Failure Mode: CLIENT/CLIENT Fri Apr 7 13:59:09 UTC 2017 =================== 13:59:09 (1491573549) [22826.341446] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [22826.497111] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [22848.381047] Lustre: 16514:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491573563/real 1491573563] req@ffff880065230600 x1564004882049552/t0(0) o104->lustre-MDT0000@10.9.5.238@tcp:15/16 lens 296/224 e 0 to 1 dl 1491573574 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [22848.388612] Lustre: 16514:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [22874.532457] Lustre: lustre-MDT0002: haven't heard from client c7b5ef3b-ae42-4333-a6e0-2c5476c1155e (at 10.9.5.238@tcp) in 49 seconds. I think it's dead, and I am evicting it. exp ffff8800647b8800, cur 1491573600 expire 1491573570 last 1491573551 [22874.538317] Lustre: Skipped 1 previous similar message [22879.112323] LNet: Service thread pid 16514 completed after 41.73s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [22942.039134] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 1 [22942.193047] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 1 [22942.380580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark No clients failed! [22942.535156] Lustre: DEBUG MARKER: No clients failed! [22945.367819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [22945.521304] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23006.967833] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23007.864711] Lustre: DEBUG MARKER: rc=0; [23007.864711] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23007.864711] if [[ $? -eq 0 && $val -ne 0 ]]; then [23007.864711] echo $(hostname -s): $val; [23007.864711] rc=$val; [23007.864711] fi; [23007.864711] exit $rc [23008.468620] Lustre: DEBUG MARKER: dmesg [23009.049845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 10: Tenth Failure Mode: MDT0\/OST\/MDT1 Fri Apr 7 14:02:13 UTC 2017 ================== 14:02:13 \(1491573733\) [23009.210046] Lustre: DEBUG MARKER: == insanity test 10: Tenth Failure Mode: MDT0/OST/MDT1 Fri Apr 7 14:02:13 UTC 2017 ================== 14:02:13 (1491573733) [23009.348848] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23009.614537] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23009.759809] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.238@tcp (stopping) [23009.762403] Lustre: Skipped 1 previous similar message [23009.860319] Lustre: server umount lustre-MDT0000 complete [23009.862523] Lustre: Skipped 3 previous similar messages [23009.998602] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23010.697254] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [23010.697261] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [23010.697262] Lustre: Skipped 10 previous similar messages [23010.697372] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [23010.697373] LustreError: Skipped 64 previous similar messages [23010.714225] LustreError: Skipped 13 previous similar messages [23128.297940] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23128.461166] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23150.304740] Lustre: DEBUG MARKER: hostname [23150.586551] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [23150.851365] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23151.105498] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23151.357354] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23151.515603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23172.543719] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9df4e to 0x52ac6ca176c9f026 [23172.590857] LustreError: 26242:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [23172.595522] LustreError: 26242:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [23172.600375] LustreError: 26247:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [23172.603021] LustreError: 26247:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [23172.627326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [23172.630415] Lustre: Skipped 1 previous similar message [23172.872131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23173.130796] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23173.627544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23173.627546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23173.795254] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23173.798306] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23173.947183] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23174.203126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23174.460917] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23174.722303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23174.977907] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23176.128914] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [23176.131963] Lustre: Skipped 1 previous similar message [23177.595993] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23177.609445] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23177.752873] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23181.693909] Lustre: lustre-MDT0000: Recovery over after 0:06, of 5 clients 5 recovered and 0 were evicted. [23181.698131] Lustre: Skipped 1 previous similar message [23183.191763] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23184.088725] Lustre: DEBUG MARKER: rc=0; [23184.088725] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23184.088725] if [[ $? -eq 0 && $val -ne 0 ]]; then [23184.088725] echo $(hostname -s): $val; [23184.088725] rc=$val; [23184.088725] fi; [23184.088725] exit $rc [23184.695630] Lustre: DEBUG MARKER: dmesg [23185.371224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 11: Eleventh Failure Mode: MDS0\/CLIENT\/MDS1 Fri Apr 7 14:05:10 UTC 2017 ============ 14:05:10 \(1491573910\) [23185.530642] Lustre: DEBUG MARKER: == insanity test 11: Eleventh Failure Mode: MDS0/CLIENT/MDS1 Fri Apr 7 14:05:10 UTC 2017 ============ 14:05:10 (1491573910) [23185.685637] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23185.940881] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23186.072666] Lustre: Failing over lustre-MDT0000 [23186.074930] Lustre: Skipped 2 previous similar messages [23186.279170] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23194.823076] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [23194.827771] LustreError: Skipped 1 previous similar message [23196.586611] Lustre: DEBUG MARKER: hostname [23196.888868] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23197.140956] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23197.392578] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23197.552337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23200.830315] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9f026 to 0x52ac6ca176c9f424 [23201.058511] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23201.321882] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23201.815258] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23201.815750] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23201.983651] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23201.985612] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23202.134204] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23202.391336] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23202.648608] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23202.908605] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23203.667815] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23203.669253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23203.831547] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23203.894283] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23205.885603] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23207.148290] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23207.172131] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23207.315335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23207.417233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23209.894263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [23210.043842] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [23258.799554] Lustre: lustre-MDT0000: haven't heard from client 860fe7f9-ad0f-f3a1-ca41-af9e8e236f22 (at 10.9.5.238@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp ffff880052185400, cur 1491573984 expire 1491573954 last 1491573934 [23258.805504] Lustre: Skipped 1 previous similar message [23261.495181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23261.651932] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23274.804512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23274.965850] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23276.650358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23276.665632] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23276.862397] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23276.894374] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23279.145460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23279.160981] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23279.334549] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23279.424172] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [23283.900181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23284.793085] Lustre: DEBUG MARKER: rc=0; [23284.793085] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23284.793085] if [[ $? -eq 0 && $val -ne 0 ]]; then [23284.793085] echo $(hostname -s): $val; [23284.793085] rc=$val; [23284.793085] fi; [23284.793085] exit $rc [23285.389433] Lustre: DEBUG MARKER: dmesg [23285.973164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 12: Twelve Failure Mode: MDS0,MDS1\/OST0, OST1\/CLIENTS Fri Apr 7 14:06:50 UTC 2017 == 14:06:50 \(1491574010\) [23286.133983] Lustre: DEBUG MARKER: == insanity test 12: Twelve Failure Mode: MDS0,MDS1/OST0, OST1/CLIENTS Fri Apr 7 14:06:50 UTC 2017 == 14:06:50 (1491574010) [23286.312126] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23286.566143] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23286.921106] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23298.141252] Lustre: DEBUG MARKER: hostname [23298.451968] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23298.706130] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23298.958870] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23299.118121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23303.991238] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9f424 to 0x52ac6ca176c9fb94 [23304.206787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23304.466331] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23304.991162] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23304.993378] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23305.154455] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23305.156753] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23305.304339] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23305.564155] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23305.820826] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23306.080651] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23309.049586] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23318.545987] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23318.715240] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23320.421466] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23320.421475] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23320.589572] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23320.687098] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23322.887904] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23322.924235] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23323.108611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23323.186346] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23323.308362] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23323.428623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23323.455838] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23323.655644] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23339.837667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23340.019724] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23353.008350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23353.172730] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23354.077552] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [23354.080618] Lustre: Skipped 109 previous similar messages [23354.635758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23354.635764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23354.838822] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23354.876363] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23355.059652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23355.164716] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23355.214573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23355.363021] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23355.496804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 0 sec [23355.618170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 0 sec [23355.665174] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 0 sec [23355.813912] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 0 sec [23358.289837] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [23358.464244] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [23408.133843] Lustre: lustre-MDT0002: haven't heard from client aec5a3ee-b6d1-7006-080b-b541d182b5d8 (at 10.9.5.238@tcp) in 51 seconds. I think it's dead, and I am evicting it. exp ffff880061e2a000, cur 1491574133 expire 1491574103 last 1491574082 [23408.139892] Lustre: Skipped 1 previous similar message [23411.711855] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23411.872958] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23415.336655] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23416.225825] Lustre: DEBUG MARKER: rc=0; [23416.225825] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23416.225825] if [[ $? -eq 0 && $val -ne 0 ]]; then [23416.225825] echo $(hostname -s): $val; [23416.225825] rc=$val; [23416.225825] fi; [23416.225825] exit $rc [23416.840685] Lustre: DEBUG MARKER: dmesg [23417.423142] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 13: Thirteen Failure Mode: MDS0,MDS1\/CLIENTS\/OST0,OST1 Fri Apr 7 14:09:02 UTC 2017 ====================================================================================================== 14:09:02 \(1491574142\) [23417.591723] Lustre: DEBUG MARKER: == insanity test 13: Thirteen Failure Mode: MDS0,MDS1/CLIENTS/OST0,OST1 Fri Apr 7 14:09:02 UTC 2017 ====================================================================================================== 14:09:02 (1491574142) [23417.740480] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23417.996932] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23418.134695] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [23418.423409] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23429.582663] Lustre: DEBUG MARKER: hostname [23429.895201] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23430.152840] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23430.404801] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23430.555777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23432.049241] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176c9fb94 to 0x52ac6ca176ca0390 [23432.285507] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23432.543738] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23433.039909] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23433.041674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23433.252671] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23433.256742] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23433.400953] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23433.656869] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23433.914151] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23434.173311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23437.118975] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23446.545824] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23446.706725] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23448.419075] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23448.426242] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23448.611255] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23448.662175] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23450.894845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23450.937112] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23451.108917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23451.210735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23451.332938] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23451.481418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23451.481426] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23451.707775] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23454.171627] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [23454.333512] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [23476.220066] Lustre: 16515:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491574190/real 1491574190] req@ffff8800785c7600 x1564004882105584/t0(0) o104->lustre-MDT0000@10.9.5.238@tcp:15/16 lens 296/224 e 0 to 1 dl 1491574201 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [23476.228658] Lustre: 16515:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [23504.761111] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23504.920431] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23522.065961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23522.231723] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23535.222873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23535.431727] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23536.879297] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23536.895599] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23537.054830] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23537.154758] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23537.291440] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23537.403628] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23537.457350] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23537.581749] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23539.811114] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [23539.837637] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [23539.984759] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [23540.064564] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 2 sec [23542.545734] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23543.441688] Lustre: DEBUG MARKER: rc=0; [23543.441688] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23543.441688] if [[ $? -eq 0 && $val -ne 0 ]]; then [23543.441688] echo $(hostname -s): $val; [23543.441688] rc=$val; [23543.441688] fi; [23543.441688] exit $rc [23544.054534] Lustre: DEBUG MARKER: dmesg [23544.661620] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 14: Fourteen Failure Mode: OST0,OST1\/CLIENTS\/MDS0,MDS1 Fri Apr 7 14:11:09 UTC 2017 ====================================================================================================== 14:11:09 \(1491574269\) [23544.836183] Lustre: DEBUG MARKER: == insanity test 14: Fourteen Failure Mode: OST0,OST1/CLIENTS/MDS0,MDS1 Fri Apr 7 14:11:09 UTC 2017 ====================================================================================================== 14:11:09 (1491574269) [23558.768686] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23558.930803] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23572.072132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23572.239694] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23573.697700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23573.759476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23573.932197] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23573.981984] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid,osc.lustre-OST0001-osc-*.ost_server_uuid [23574.136832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23574.252985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23574.290221] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23574.435658] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 0 sec [23575.637402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [23575.663063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [23575.826662] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [23575.892650] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-*.ost_server_uuid in FULL state after 1 sec [23578.377367] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Request fail clients: 2, to fail: 1, failed: 0 [23578.539665] Lustre: DEBUG MARKER: Request fail clients: 2, to fail: 1, failed: 0 [23625.627374] Lustre: lustre-MDT0000: haven't heard from client f6679dea-1c3c-fdc4-4cca-f2e9932bde38 (at 10.9.5.238@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880050b67800, cur 1491574351 expire 1491574321 last 1491574304 [23625.633022] Lustre: Skipped 3 previous similar messages [23628.287484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23628.440026] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23631.861492] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23632.118299] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23632.271122] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007a23b900 x1564004882120336/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23632.277028] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [23632.325022] Lustre: server umount lustre-MDT0000 complete [23632.327585] Lustre: Skipped 3 previous similar messages [23632.456767] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23635.139637] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [23635.145401] LustreError: Skipped 580 previous similar messages [23637.118164] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [23637.118185] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [23637.118186] Lustre: Skipped 25 previous similar messages [23637.129504] LustreError: Skipped 27 previous similar messages [23643.711853] Lustre: DEBUG MARKER: hostname [23644.030158] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23644.286939] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23644.540100] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23644.699646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23649.559436] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176ca0390 to 0x52ac6ca176ca0ddf [23649.779342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23650.036725] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23650.532211] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23650.552994] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23650.713343] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23650.717789] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23650.862495] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23651.120481] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23651.374963] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23651.635853] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23654.614781] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23664.020379] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23664.188937] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23665.892358] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23665.892402] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23666.123803] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23666.162158] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [23668.340700] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23668.475460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23668.557406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23668.815845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [23668.836862] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23669.038282] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23669.054998] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23669.244372] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [23671.697729] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23672.594218] Lustre: DEBUG MARKER: rc=0; [23672.594218] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23672.594218] if [[ $? -eq 0 && $val -ne 0 ]]; then [23672.594218] echo $(hostname -s): $val; [23672.594218] rc=$val; [23672.594218] fi; [23672.594218] exit $rc [23673.211287] Lustre: DEBUG MARKER: dmesg [23673.794040] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: insanity test_15 skipping ALWAYS excluded test 15 [23673.948215] Lustre: DEBUG MARKER: SKIP: insanity test_15 skipping ALWAYS excluded test 15 [23674.158384] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test complete, duration 2538 sec ========================================================= 14:13:18 \(1491574398\) [23674.334223] Lustre: DEBUG MARKER: == insanity test complete, duration 2538 sec ========================================================= 14:13:18 (1491574398) [23674.513192] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23674.770752] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [23675.104366] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23685.413948] Lustre: DEBUG MARKER: hostname [23685.727847] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23685.981443] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23686.233198] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23686.382708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23686.400140] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176ca0ddf to 0x52ac6ca176ca13f8 [23686.450071] LustreError: 6980:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [23686.454650] LustreError: 6980:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 4 previous similar messages [23686.459635] LustreError: 6985:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [23686.462232] LustreError: 6985:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 4 previous similar messages [23686.633322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23686.889641] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [23687.391696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23687.444094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23687.622992] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23687.624458] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23687.770127] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23688.028443] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23688.283924] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23688.543760] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23689.363902] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23689.363913] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23689.525147] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23689.528362] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [23691.467988] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [23692.766320] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23692.867431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23693.025219] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23693.196154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [23705.259711] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [23705.535337] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [23706.467763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [23706.467764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [23706.783092] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880050bdf600 x1564004882132160/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23706.789074] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [23711.910510] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23719.055668] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [23719.312309] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [23720.399062] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800785c6d00 x1564004882132944/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23720.404991] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [23722.479078] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880050bdc300 x1564004882132976/t0(0) o13->lustre-OST0000-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [23725.672343] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [23793.435192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-sec ============----- Fri Apr 7 14:15:17 UTC 2017 [23793.585627] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-sec ============----- Fri Apr 7 14:15:17 UTC 2017 [23793.844391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test complete, duration -o sec ========================================================= 14:15:18 \(1491574518\) [23794.078421] Lustre: DEBUG MARKER: == sanity-sec test complete, duration -o sec ========================================================= 14:15:18 (1491574518) [23794.329337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23794.329337] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23794.329337] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23797.351184] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [23797.351184] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [23797.351184] if [ $running -ne $mpts ]; then [23797.351184] echo $(hostname) env are INSANE!; [23797.351184] exit 1; [23797.351184] fi [23797.925109] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [23797.925109] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [23797.925109] if [ $running -ne $mpts ]; then [23797.925109] echo $(hostname) env are INSANE!; [23797.925109] exit 1; [23797.925109] fi [23798.505550] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [23798.505550] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [23798.505550] if [ $running -ne $mpts ]; then [23798.505550] echo $(hostname) env are INSANE!; [23798.505550] exit 1; [23798.505550] fi [23801.026943] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [23801.284325] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [23801.532516] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [23801.778609] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [23801.933729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23802.024923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [23802.027298] Lustre: Skipped 5 previous similar messages [23802.174408] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23802.433302] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [23802.947782] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23802.951234] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23803.117938] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23803.123154] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23803.263618] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23803.515656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23803.765223] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23804.021553] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [23804.276419] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23806.576322] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23806.744965] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23808.164285] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [23808.427906] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [23808.680760] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [23808.933050] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [23809.082487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [23809.383501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [23809.641734] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/t [23810.143368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23810.145026] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23810.309763] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23810.313548] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23810.457772] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23810.716592] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23810.973746] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [23811.232291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [23811.488546] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23813.841265] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23813.996177] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23817.382561] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23817.546380] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23820.391454] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23820.552251] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23823.410428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23823.583403] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23826.461800] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23826.627517] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23829.514208] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23829.670166] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23832.584224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23832.753153] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23835.680746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23835.853665] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23838.747851] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23838.926341] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23841.090196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23841.254423] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [23842.689138] Lustre: DEBUG MARKER: lctl get_param -n timeout [23843.004079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [23843.159022] Lustre: DEBUG MARKER: Using TIMEOUT=20 [23843.293542] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [23848.701513] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [23848.981512] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23849.494879] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [23850.017466] Lustre: DEBUG MARKER: lctl get_param -N mdt.\*MDT0000 [23850.274291] Lustre: DEBUG MARKER: mkdir -p /etc/lustre [23850.603469] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 2 5 6 [23850.758467] Lustre: DEBUG MARKER: excepting tests: 2 5 6 [23850.949996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 26 [23851.103541] Lustre: DEBUG MARKER: skipping tests SLOW=no: 26 [23851.242776] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [23851.500221] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [23851.755313] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall /usr/sbin/l_getidentity [23852.011603] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [23853.315380] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*2 [23853.577929] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0002.identity_upcall [23853.835212] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0002.identity_upcall /usr/sbin/l_getidentity [23854.091617] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0002/identity_flush=-1 [23855.460687] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 0: uid permission ================================================================= 14:16:20 \(1491574580\) [23855.616685] Lustre: DEBUG MARKER: == sanity-sec test 0: uid permission ================================================================= 14:16:20 (1491574580) [23855.970036] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23856.856898] Lustre: DEBUG MARKER: rc=0; [23856.856898] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23856.856898] if [[ $? -eq 0 && $val -ne 0 ]]; then [23856.856898] echo $(hostname -s): $val; [23856.856898] rc=$val; [23856.856898] fi; [23856.856898] exit $rc [23857.459361] Lustre: DEBUG MARKER: dmesg [23858.237791] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 1: setuid\/gid ===================================================================== 14:16:23 \(1491574583\) [23858.388638] Lustre: DEBUG MARKER: == sanity-sec test 1: setuid/gid ===================================================================== 14:16:23 (1491574583) [23858.593991] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-sec test_1 without GSS support. [23858.748735] Lustre: DEBUG MARKER: SKIP: sanity-sec test_1 without GSS support. [23858.934060] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23859.841244] Lustre: DEBUG MARKER: rc=0; [23859.841244] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23859.841244] if [[ $? -eq 0 && $val -ne 0 ]]; then [23859.841244] echo $(hostname -s): $val; [23859.841244] rc=$val; [23859.841244] fi; [23859.841244] exit $rc [23860.441274] Lustre: DEBUG MARKER: dmesg [23861.027935] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 4: set supplementary group ======================================================== 14:16:25 \(1491574585\) [23861.193602] Lustre: DEBUG MARKER: == sanity-sec test 4: set supplementary group ======================================================== 14:16:25 (1491574585) [23861.333146] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23861.333146] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23861.333146] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23861.630386] Lustre: DEBUG MARKER: echo '* 501 setgrp' > /etc/lustre/perm.conf [23861.883903] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_flush=-1 [23862.149178] Lustre: DEBUG MARKER: rm -f /etc/lustre/perm.conf [23862.403805] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_flush=-1 [23862.708674] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23863.612773] Lustre: DEBUG MARKER: rc=0; [23863.612773] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23863.612773] if [[ $? -eq 0 && $val -ne 0 ]]; then [23863.612773] echo $(hostname -s): $val; [23863.612773] rc=$val; [23863.612773] fi; [23863.612773] exit $rc [23864.208885] Lustre: DEBUG MARKER: dmesg [23864.725113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23864.725113] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23864.725113] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23864.990897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23865.248900] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23865.248900] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23865.248900] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23865.515292] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23865.834871] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 7: nodemap create and delete ====================================================== 14:16:30 \(1491574590\) [23865.993945] Lustre: DEBUG MARKER: == sanity-sec test 7: nodemap create and delete ====================================================== 14:16:30 (1491574590) [23866.141016] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23866.141016] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23866.141016] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23866.412685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23866.412685] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23866.412685] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23866.682618] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23866.940442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23866.940442] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23866.940442] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23867.211326] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23867.465870] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [23867.720446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23867.977315] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [23868.233966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23868.488050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [23868.743391] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23869.001444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [23869.254731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23869.512320] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [23869.768957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23870.023266] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [23870.277857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23870.533379] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [23870.786932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23871.043611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [23871.298898] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23871.555836] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [23871.810129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23872.065269] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [23872.321037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23872.579054] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [23872.837427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23873.091095] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [23873.343124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23873.599905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [23873.853945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23874.109370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [23874.362965] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23874.618190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [23874.874557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23875.129238] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [23875.386055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23875.642904] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [23875.897324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23876.153399] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [23876.405926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23876.659592] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [23876.915974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23877.170404] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [23877.424510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23877.678967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [23877.934071] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23878.193897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [23878.451525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23878.707725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [23878.962040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23879.215385] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [23879.469873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23879.723105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [23879.979799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23880.236511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [23880.489980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23880.743294] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [23880.995558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23881.249542] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [23881.503442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23881.758160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [23882.010332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23882.264587] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [23882.517777] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23882.772050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [23883.027234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23883.280950] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [23883.535113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23883.843990] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23884.747278] Lustre: DEBUG MARKER: rc=0; [23884.747278] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23884.747278] if [[ $? -eq 0 && $val -ne 0 ]]; then [23884.747278] echo $(hostname -s): $val; [23884.747278] rc=$val; [23884.747278] fi; [23884.747278] exit $rc [23885.346598] Lustre: DEBUG MARKER: dmesg [23885.947227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 8: nodemap reject duplicates ====================================================== 14:16:50 \(1491574610\) [23886.121182] Lustre: DEBUG MARKER: == sanity-sec test 8: nodemap reject duplicates ====================================================== 14:16:50 (1491574610) [23886.288439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23886.288439] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23886.288439] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23886.559642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23886.559642] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23886.559642] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23886.830033] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23887.088771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23887.088771] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23887.088771] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23887.354548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23887.607922] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [23887.864722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23888.120026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [23888.374770] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23888.629872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [23888.887347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23889.146159] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [23889.402113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23889.657628] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [23889.911635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23890.168221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [23890.424966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23890.680651] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [23890.935274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23891.193642] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [23891.448963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23891.705190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [23891.961261] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23892.218846] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [23892.474021] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23892.729953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [23892.986333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23893.243112] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [23893.497833] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23893.756025] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [23894.010582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23894.265854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [23894.524123] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23894.779619] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [23895.033854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23895.291668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [23895.545840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23895.802716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23895.802716] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23895.802716] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23896.070095] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23896.324554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23896.324554] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23896.324554] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23896.593226] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23896.848282] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [23897.101819] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [23897.353592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23897.607340] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [23897.864295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23898.120646] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [23898.374306] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23898.630288] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [23898.885685] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23899.141381] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [23899.396925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23899.651988] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [23899.907766] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23900.164700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [23900.421322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23900.676239] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [23900.929462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23901.184916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [23901.440688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23901.693965] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [23901.948148] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23902.203765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [23902.455532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23902.710862] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [23902.967119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23903.224703] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [23903.478402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23903.734574] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [23903.990513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23904.248437] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [23904.504877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23904.760319] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [23905.014650] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23905.325779] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23906.204567] Lustre: DEBUG MARKER: rc=0; [23906.204567] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23906.204567] if [[ $? -eq 0 && $val -ne 0 ]]; then [23906.204567] echo $(hostname -s): $val; [23906.204567] rc=$val; [23906.204567] fi; [23906.204567] exit $rc [23906.788838] Lustre: DEBUG MARKER: dmesg [23907.368876] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 9: nodemap range add ============================================================== 14:17:12 \(1491574632\) [23907.524668] Lustre: DEBUG MARKER: == sanity-sec test 9: nodemap range add ============================================================== 14:17:12 (1491574632) [23907.683341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23907.683341] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23907.683341] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23907.964711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23907.964711] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23907.964711] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23908.238750] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23908.496069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23908.496069] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23908.496069] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23908.761523] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23909.016111] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [23909.269359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23909.525265] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [23909.780248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23910.035140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [23910.288307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23910.540704] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [23910.793513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23911.046310] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [23911.299395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23911.553825] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [23911.808157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23912.061285] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [23912.316880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23912.570720] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [23912.824016] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23913.078620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [23913.330321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23913.583947] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [23913.840651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23914.097357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [23914.354290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23914.609024] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [23914.865373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23915.122462] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [23915.376982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23915.633085] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [23915.888337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23916.145653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [23916.399963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23916.656030] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [23916.912776] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23917.169919] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [23917.424746] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [23917.681494] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [23917.935916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [23918.190671] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [23918.445847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [23918.702640] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [23918.958673] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [23919.211912] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [23919.468223] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [23919.720660] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [23919.973133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [23920.227177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [23920.481076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [23920.736230] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [23920.990210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [23921.246421] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [23921.500445] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [23921.754998] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [23922.009260] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [23922.262897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [23922.518581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [23922.773785] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [23923.029557] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [23923.285822] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [23923.541585] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [23923.795749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [23924.050200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [23924.305509] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [23924.561092] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [23924.815027] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [23925.069792] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [23925.323231] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [23925.579579] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [23925.833182] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [23926.084520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [23926.341148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [23926.596451] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [23926.851079] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [23927.103788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [23927.355664] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [23927.608470] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [23927.862794] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [23928.117994] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [23928.373783] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [23928.631223] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [23928.888803] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [23929.143636] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [23929.399850] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.0.[1-253]@tcp [23929.654705] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.1.[1-253]@tcp [23929.910096] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.2.[1-253]@tcp [23930.166998] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.0.[1-253]@tcp [23930.422455] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.1.[1-253]@tcp [23930.687427] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.2.[1-253]@tcp [23930.958457] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.0.[1-253]@tcp [23931.221397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.1.[1-253]@tcp [23931.486604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.2.[1-253]@tcp [23931.750418] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.0.[1-253]@tcp [23932.022101] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.1.[1-253]@tcp [23932.277434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.2.[1-253]@tcp [23932.531516] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.0.[1-253]@tcp [23932.786384] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.1.[1-253]@tcp [23933.040948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.2.[1-253]@tcp [23933.293467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.0.[1-253]@tcp [23933.546196] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.1.[1-253]@tcp [23933.798896] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.2.[1-253]@tcp [23934.054736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.0.[1-253]@tcp [23934.306863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.1.[1-253]@tcp [23934.558639] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.2.[1-253]@tcp [23934.814738] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.0.[1-253]@tcp [23935.067452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.1.[1-253]@tcp [23935.320990] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.2.[1-253]@tcp [23935.571246] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.0.[1-253]@tcp [23935.823497] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.1.[1-253]@tcp [23936.075516] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.2.[1-253]@tcp [23936.328074] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.0.[1-253]@tcp [23936.581166] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.1.[1-253]@tcp [23936.834597] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.2.[1-253]@tcp [23937.086109] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.0.[1-253]@tcp [23937.338863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.1.[1-253]@tcp [23937.589607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.2.[1-253]@tcp [23937.843376] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.0.[1-253]@tcp [23938.096180] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.1.[1-253]@tcp [23938.348651] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.2.[1-253]@tcp [23938.601146] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.0.[1-253]@tcp [23938.854596] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.1.[1-253]@tcp [23939.108019] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.2.[1-253]@tcp [23939.361383] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.0.[1-253]@tcp [23939.616141] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.1.[1-253]@tcp [23939.869396] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.2.[1-253]@tcp [23940.122928] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.0.[1-253]@tcp [23940.375740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.1.[1-253]@tcp [23940.627132] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.2.[1-253]@tcp [23940.877913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.0.[1-253]@tcp [23941.128611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.1.[1-253]@tcp [23941.381992] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.2.[1-253]@tcp [23941.635417] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [23941.887841] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23942.145858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [23942.399046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23942.651408] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [23942.904901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23943.158771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [23943.409620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23943.665321] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [23943.919581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23944.176047] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [23944.430386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23944.686156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [23944.939555] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23945.195154] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [23945.449266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23945.704762] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [23945.959434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23946.216344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [23946.470691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23946.724461] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [23946.981147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23947.237104] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [23947.490422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23947.745898] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [23948.002360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23948.258447] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [23948.511873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23948.764358] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [23949.018767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23949.275658] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [23949.532892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23949.849166] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [23950.737534] Lustre: DEBUG MARKER: rc=0; [23950.737534] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [23950.737534] if [[ $? -eq 0 && $val -ne 0 ]]; then [23950.737534] echo $(hostname -s): $val; [23950.737534] rc=$val; [23950.737534] fi; [23950.737534] exit $rc [23951.335121] Lustre: DEBUG MARKER: dmesg [23951.910814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 10: nodemap reject duplicate ranges =============================================== 14:17:56 \(1491574676\) [23952.064638] Lustre: DEBUG MARKER: == sanity-sec test 10: nodemap reject duplicate ranges =============================================== 14:17:56 (1491574676) [23952.215586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23952.215586] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23952.215586] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23952.483154] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23952.483154] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23952.483154] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23952.750700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [23953.014939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [23953.014939] /usr/sbin/lctl lustre_build_version 2>/dev/null || [23953.014939] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [23953.282633] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [23953.537320] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [23953.790330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23954.046689] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [23954.300334] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23954.558125] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [23954.814568] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [23955.070942] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [23955.325957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [23955.580456] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [23955.836203] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [23956.091358] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [23956.346122] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [23956.602700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [23956.858109] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [23957.114056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [23957.367412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [23957.623262] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [23957.878677] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [23958.135348] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [23958.391929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [23958.647096] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [23958.904039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [23959.158847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [23959.486479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [23959.754988] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [23960.014455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [23960.272594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [23960.530393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [23960.791370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [23961.047578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [23961.304145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [23961.558877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [23961.815872] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [23962.072022] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [23962.328123] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [23962.582811] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [23962.838668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [23963.095996] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [23963.350659] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [23963.604984] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [23963.859948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [23964.117859] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [23964.374043] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [23964.627738] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [23964.884368] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [23965.138980] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [23965.394906] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [23965.648645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [23965.901709] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [23966.156429] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [23966.411639] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [23966.665713] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [23966.919607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [23967.172698] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [23967.426593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [23967.678810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [23967.932583] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [23968.188375] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [23968.440114] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [23968.692368] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [23968.945705] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [23969.199483] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [23969.454181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [23969.707110] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [23969.959098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [23970.212831] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [23970.466285] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [23970.717215] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [23970.970648] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [23971.224342] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [23971.475561] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [23971.730430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [23971.983350] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [23972.235758] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [23972.491212] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [23972.743361] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [23972.998434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [23973.250466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [23973.501929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [23973.753995] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [23974.010299] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [23974.139532] LustreError: 12715:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_0': rc = -17 [23974.141432] LustreError: 12715:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_0: rc = -12 [23974.270156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [23974.522911] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [23974.651095] LustreError: 12861:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_0': rc = -17 [23974.653422] LustreError: 12861:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 1 previous similar message [23974.655003] LustreError: 12861:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_0: rc = -12 [23974.657443] LustreError: 12861:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 1 previous similar message [23974.785544] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [23975.042893] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [23975.298751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [23975.554654] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [23975.682565] LustreError: 13153:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_2': rc = -17 [23975.685499] LustreError: 13153:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 3 previous similar messages [23975.687392] LustreError: 13153:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_2: rc = -12 [23975.690383] LustreError: 13153:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 3 previous similar messages [23975.819464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [23976.077559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [23976.332239] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [23976.586385] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [23976.840759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [23977.095795] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [23977.350987] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [23977.605352] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [23977.733908] LustreError: 13737:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_4': rc = -17 [23977.737398] LustreError: 13737:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 7 previous similar messages [23977.739576] LustreError: 13737:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_4: rc = -12 [23977.743157] LustreError: 13737:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 7 previous similar messages [23977.872698] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [23978.128605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [23978.382504] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [23978.638969] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [23978.893797] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [23979.150694] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [23979.406861] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [23979.663798] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [23979.920992] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [23980.177521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [23980.434118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [23980.688287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [23980.942319] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [23981.199498] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [23981.455344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [23981.710696] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [23981.839975] LustreError: 14906:0:(nodemap_handler.c:772:nodemap_add_range_helper()) cannot insert nodemap range into '25087_10': rc = -17 [23981.843555] LustreError: 14906:0:(nodemap_handler.c:772:nodemap_add_range_helper()) Skipped 15 previous similar messages [23981.845706] LustreError: 14906:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE042 for 25087_10: rc = -12 [23981.849166] LustreError: 14906:0:(mgs_handler.c:878:mgs_iocontrol_nodemap()) Skipped 15 previous similar messages [23981.977907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [23982.232350] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [23982.486839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [23982.742179] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [23982.997492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [23983.253755] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [23983.509539] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [23983.764790] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [23984.019355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [23984.308899] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [23984.563498] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [23984.818405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [23985.075133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [23985.330253] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [23985.587160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [23985.840172] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [23986.096348] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [23986.350918] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.0.[1-253]@tcp [23986.607353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.1.[1-253]@tcp [23986.861683] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_0 --range 88.0.2.[1-253]@tcp [23987.116517] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.0.[1-253]@tcp [23987.369825] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.1.[1-253]@tcp [23987.623645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_1 --range 88.1.2.[1-253]@tcp [23987.879661] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.0.[1-253]@tcp [23988.136596] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.1.[1-253]@tcp [23988.392867] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_2 --range 88.2.2.[1-253]@tcp [23988.650442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.0.[1-253]@tcp [23988.914423] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.1.[1-253]@tcp [23989.188472] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_3 --range 88.3.2.[1-253]@tcp [23989.445694] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.0.[1-253]@tcp [23989.698911] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.1.[1-253]@tcp [23989.954524] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_4 --range 88.4.2.[1-253]@tcp [23990.209430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.0.[1-253]@tcp [23990.463121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.1.[1-253]@tcp [23990.717734] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_5 --range 88.5.2.[1-253]@tcp [23990.972636] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.0.[1-253]@tcp [23991.227588] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.1.[1-253]@tcp [23991.479706] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_6 --range 88.6.2.[1-253]@tcp [23991.734123] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.0.[1-253]@tcp [23991.988394] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.1.[1-253]@tcp [23992.244604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_7 --range 88.7.2.[1-253]@tcp [23992.499068] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.0.[1-253]@tcp [23992.754075] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.1.[1-253]@tcp [23993.008390] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_8 --range 88.8.2.[1-253]@tcp [23993.263431] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.0.[1-253]@tcp [23993.517977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.1.[1-253]@tcp [23993.772629] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_9 --range 88.9.2.[1-253]@tcp [23994.026115] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.0.[1-253]@tcp [23994.281415] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.1.[1-253]@tcp [23994.536186] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_10 --range 88.10.2.[1-253]@tcp [23994.789926] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.0.[1-253]@tcp [23995.042303] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.1.[1-253]@tcp [23995.294800] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_11 --range 88.11.2.[1-253]@tcp [23995.547665] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.0.[1-253]@tcp [23995.798345] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.1.[1-253]@tcp [23996.050540] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_12 --range 88.12.2.[1-253]@tcp [23996.302405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.0.[1-253]@tcp [23996.554345] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.1.[1-253]@tcp [23996.808472] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_13 --range 88.13.2.[1-253]@tcp [23997.060560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.0.[1-253]@tcp [23997.312703] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.1.[1-253]@tcp [23997.565335] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_14 --range 88.14.2.[1-253]@tcp [23997.817278] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.0.[1-253]@tcp [23998.074706] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.1.[1-253]@tcp [23998.330336] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_range --name 25087_15 --range 88.15.2.[1-253]@tcp [23998.581236] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [23998.833380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [23999.087482] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [23999.342810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [23999.597881] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [23999.850632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24000.102744] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24000.356173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24000.609163] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24000.860876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24001.115606] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24001.700579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24001.957160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24002.211119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24002.464178] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24002.713884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24002.968070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24003.220881] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24003.474353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24003.725977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24003.982459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24004.238385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24004.494441] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24004.749541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24005.006050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24005.260339] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24005.516508] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24005.770767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24006.026731] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24006.280354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24006.535929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24006.789028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24007.098986] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24008.002351] Lustre: DEBUG MARKER: rc=0; [24008.002351] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24008.002351] if [[ $? -eq 0 && $val -ne 0 ]]; then [24008.002351] echo $(hostname -s): $val; [24008.002351] rc=$val; [24008.002351] fi; [24008.002351] exit $rc [24008.596461] Lustre: DEBUG MARKER: dmesg [24009.178412] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 11: nodemap modify ================================================================ 14:18:54 \(1491574734\) [24009.338931] Lustre: DEBUG MARKER: == sanity-sec test 11: nodemap modify ================================================================ 14:18:54 (1491574734) [24009.496425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24009.496425] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24009.496425] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24009.764534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24009.764534] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24009.764534] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24010.030227] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [24010.285887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24010.285887] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24010.285887] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24010.553087] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [24010.805138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [24011.060077] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24011.316427] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [24011.570059] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24011.827248] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [24012.083877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24012.339605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [24012.596414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24012.851348] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [24013.106970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24013.362940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [24013.617682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24013.875457] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [24014.130105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24014.384265] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [24014.642093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24014.897641] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [24015.150440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24015.405509] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [24015.658884] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24015.913357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [24016.168352] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24016.424398] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [24016.677732] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24016.932210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [24017.186955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24017.441794] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [24017.696244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24017.953564] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [24018.207824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24018.464680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [24018.720809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24018.976863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 1 [24019.234100] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 0 [24019.487173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 1 [24019.741410] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 0 [24019.999024] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 1 [24020.254326] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 0 [24020.513113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 1 [24020.769962] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 0 [24021.029548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 1 [24021.296256] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 0 [24021.551886] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 1 [24021.806777] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 0 [24022.061434] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 1 [24022.317061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 0 [24022.571197] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 1 [24022.828315] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 0 [24023.081653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 1 [24023.336464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 0 [24023.589814] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 1 [24023.846328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 0 [24024.104675] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 1 [24024.357036] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 0 [24024.609704] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 1 [24024.866297] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 0 [24025.119893] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 1 [24025.385781] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 0 [24025.653051] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 1 [24025.913902] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 0 [24026.173642] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 1 [24026.427745] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 0 [24026.679122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 1 [24026.931169] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 0 [24027.184127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 1 [24027.436737] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 0 [24027.687882] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 1 [24027.943211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 0 [24028.195866] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 1 [24028.450701] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 0 [24028.703324] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 1 [24028.956835] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 0 [24029.211499] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 1 [24029.467747] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 0 [24029.720194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 1 [24029.973355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 0 [24030.227281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 1 [24030.482165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 0 [24030.736160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 1 [24030.989176] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 0 [24031.241240] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 1 [24031.492585] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 0 [24031.745767] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 1 [24032.009107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 0 [24032.264521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 1 [24032.517045] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 0 [24032.770262] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 1 [24033.022797] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 0 [24033.275452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 1 [24033.528907] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 0 [24033.781813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 1 [24034.036754] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 0 [24034.293200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 1 [24034.547454] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 0 [24034.802121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 1 [24035.057947] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 0 [24035.314683] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [24035.570253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24035.824852] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [24036.081397] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24036.338786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [24036.593877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24036.852551] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24037.105491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24037.361102] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24037.615966] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24037.871863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24038.127208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24038.384991] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24038.639034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24038.893431] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24039.152342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24039.408170] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24039.664266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24039.922052] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24040.178441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24040.433278] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24040.688190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24040.944160] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24041.197140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24041.451026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24041.705581] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24041.960968] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24042.217725] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24042.471841] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24042.726629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24042.984031] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24043.241756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24043.552246] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24044.468167] Lustre: DEBUG MARKER: rc=0; [24044.468167] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24044.468167] if [[ $? -eq 0 && $val -ne 0 ]]; then [24044.468167] echo $(hostname -s): $val; [24044.468167] rc=$val; [24044.468167] fi; [24044.468167] exit $rc [24045.058784] Lustre: DEBUG MARKER: dmesg [24045.643409] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 12: nodemap set squash ids ======================================================== 14:19:30 \(1491574770\) [24045.800152] Lustre: DEBUG MARKER: == sanity-sec test 12: nodemap set squash ids ======================================================== 14:19:30 (1491574770) [24045.960250] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24045.960250] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24045.960250] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24046.228321] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24046.228321] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24046.228321] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24046.494748] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [24046.751912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24046.751912] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24046.751912] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24047.020004] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [24047.275556] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [24047.531667] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24047.788699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [24048.043453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24048.300472] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [24048.555773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24048.811266] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [24049.068686] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24049.326740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [24049.581908] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24049.839344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [24050.100831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24050.360181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [24050.617590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24050.873611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [24051.126905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24051.382464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [24051.636811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24051.893766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [24052.149197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24052.405492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [24052.661301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24052.919138] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [24053.174202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24053.429662] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [24053.684742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24053.945535] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [24054.218691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24054.490151] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [24054.764233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24055.031889] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [24055.289216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24055.546209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24055.546209] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24055.546209] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24055.811970] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_0 --value 88 [24056.068467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24056.068467] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24056.068467] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24056.337251] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_1 --value 88 [24056.592599] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24056.592599] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24056.592599] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24056.859589] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_2 --value 88 [24057.115952] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24057.115952] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24057.115952] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24057.382437] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_3 --value 88 [24057.642200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24057.642200] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24057.642200] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24057.908635] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_4 --value 88 [24058.164202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24058.164202] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24058.164202] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24058.431685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_5 --value 88 [24058.688215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24058.688215] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24058.688215] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24058.957032] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_6 --value 88 [24059.210998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24059.210998] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24059.210998] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24059.478841] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_7 --value 88 [24059.734600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24059.734600] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24059.734600] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24060.002477] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_8 --value 88 [24060.257705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24060.257705] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24060.257705] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24060.526283] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_9 --value 88 [24060.778197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24060.778197] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24060.778197] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24061.055257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_10 --value 88 [24061.311404] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24061.311404] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24061.311404] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24061.579054] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_11 --value 88 [24061.833272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24061.833272] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24061.833272] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24062.101113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_12 --value 88 [24062.355600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24062.355600] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24062.355600] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24062.623697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_13 --value 88 [24062.878410] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24062.878410] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24062.878410] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24063.144380] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_14 --value 88 [24063.401729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24063.401729] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24063.401729] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24063.668581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name 25087_15 --value 88 [24063.921386] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24063.921386] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24063.921386] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24064.192203] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_0 --value 88 [24064.450329] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24064.450329] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24064.450329] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24064.727202] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_1 --value 88 [24064.983153] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24064.983153] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24064.983153] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24065.254021] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_2 --value 88 [24065.509704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24065.509704] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24065.509704] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24065.778075] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_3 --value 88 [24066.033873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24066.033873] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24066.033873] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24066.311699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_4 --value 88 [24066.566497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24066.566497] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24066.566497] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24066.835720] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_5 --value 88 [24067.090609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24067.090609] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24067.090609] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24067.359606] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_6 --value 88 [24067.615106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24067.615106] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24067.615106] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24067.883972] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_7 --value 88 [24068.142030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24068.142030] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24068.142030] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24068.409454] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_8 --value 88 [24068.664529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24068.664529] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24068.664529] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24068.932649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_9 --value 88 [24069.188964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24069.188964] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24069.188964] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24069.457931] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_10 --value 88 [24069.712042] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24069.712042] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24069.712042] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24069.980042] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_11 --value 88 [24070.236645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24070.236645] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24070.236645] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24070.505748] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_12 --value 88 [24070.761214] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24070.761214] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24070.761214] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24071.030805] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_13 --value 88 [24071.286627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24071.286627] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24071.286627] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24071.566031] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_14 --value 88 [24071.820996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24071.820996] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24071.820996] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24072.088733] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name 25087_15 --value 88 [24072.345261] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [24072.599166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24072.853746] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [24073.107806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24073.377913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [24073.632477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24073.886889] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24074.141665] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24074.398635] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24074.657938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24074.915634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24075.172313] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24075.426927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24075.680107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24075.935517] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24076.188129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24076.442938] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24076.696559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24076.953165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24077.207733] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24077.464069] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24077.717565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24077.974892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24078.229848] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24078.485098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24078.736905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24078.992901] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24079.249365] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24079.505117] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24079.761704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24080.019275] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24080.276549] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24080.594484] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24081.481029] Lustre: DEBUG MARKER: rc=0; [24081.481029] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24081.481029] if [[ $? -eq 0 && $val -ne 0 ]]; then [24081.481029] echo $(hostname -s): $val; [24081.481029] rc=$val; [24081.481029] fi; [24081.481029] exit $rc [24082.076986] Lustre: DEBUG MARKER: dmesg [24082.617350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 13: test nids ===================================================================== 14:20:07 \(1491574807\) [24082.771753] Lustre: DEBUG MARKER: == sanity-sec test 13: test nids ===================================================================== 14:20:07 (1491574807) [24082.929295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24082.929295] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24082.929295] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24083.196501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24083.196501] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24083.196501] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24083.461629] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [24083.717893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24083.717893] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24083.717893] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24083.981521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [24084.236913] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [24084.491644] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24084.744558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [24084.997324] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24085.254183] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [24085.511212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24085.767466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [24086.021694] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24086.276956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [24086.529805] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24086.784614] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [24087.037521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24087.293275] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [24087.545905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24087.801803] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [24088.055653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24088.316523] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [24088.570266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24088.826439] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [24089.082464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24089.335444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [24089.589643] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24089.844171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [24090.097888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24090.352697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [24090.604473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24090.856370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [24091.109748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24091.363474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [24091.615392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24091.868249] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [24092.120585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24092.374124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [24092.627986] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [24092.882107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [24093.134967] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [24093.386635] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [24093.638933] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [24093.891043] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [24094.146563] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [24094.399696] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [24094.654318] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [24094.907771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [24095.163507] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [24095.418338] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [24095.670511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [24095.924611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [24096.178421] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [24096.432534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [24096.687551] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [24096.940318] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [24097.194110] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [24097.447627] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [24097.699756] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [24097.952082] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [24098.205895] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [24098.461582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [24098.716298] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [24098.969283] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [24099.223855] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [24099.477225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [24099.731429] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [24099.985898] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [24100.239674] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [24100.493525] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [24100.748141] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [24101.004165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [24101.257464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [24101.511553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [24101.763585] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [24102.015080] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [24102.267152] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [24102.521281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [24102.773098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [24103.026923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [24103.283679] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [24103.538526] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [24103.794487] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [24104.047548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [24104.300610] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [24104.555520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.1 [24104.807581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.10 [24105.063555] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.64 [24105.319728] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.128 [24105.572631] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.200 [24105.824922] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.250 [24106.078245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.1 [24106.330421] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.10 [24106.581951] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.64 [24106.834392] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.128 [24107.086299] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.200 [24107.340786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.250 [24107.593355] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.1 [24107.848520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.10 [24108.105079] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.64 [24108.357689] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.128 [24108.609487] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.200 [24108.865395] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.250 [24109.118371] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.1 [24109.371776] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.10 [24109.625699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.64 [24109.878317] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.128 [24110.132102] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.200 [24110.386863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.250 [24110.640661] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.1 [24110.894978] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.10 [24111.150214] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.64 [24111.403689] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.128 [24111.657185] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.200 [24111.910284] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.250 [24112.165072] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.1 [24112.421430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.10 [24112.678194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.64 [24112.935663] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.128 [24113.188519] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.200 [24113.442726] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.250 [24113.696584] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.1 [24113.948732] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.10 [24114.202489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.64 [24114.456553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.128 [24114.709692] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.200 [24114.962200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.250 [24115.218534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.1 [24115.471811] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.10 [24115.724311] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.64 [24115.975366] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.128 [24116.227119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.200 [24116.478604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.250 [24116.730562] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.1 [24116.982383] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.10 [24117.235196] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.64 [24117.486296] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.128 [24117.738701] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.200 [24117.994259] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.250 [24118.246776] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.1 [24118.500210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.10 [24118.751869] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.64 [24119.003351] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.128 [24119.256518] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.200 [24119.507164] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.250 [24119.757475] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.1 [24120.010977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.10 [24120.263388] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.64 [24120.516492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.128 [24120.767443] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.200 [24121.018616] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.250 [24121.270420] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.1 [24121.523200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.10 [24121.774729] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.64 [24122.027743] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.128 [24122.279622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.200 [24122.530884] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.250 [24122.783418] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.1 [24123.035180] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.10 [24123.286686] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.64 [24123.537330] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.128 [24123.787905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.200 [24124.040856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.250 [24124.295240] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.1 [24124.550317] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.10 [24124.803371] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.64 [24125.056275] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.128 [24125.307796] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.200 [24125.563144] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.250 [24125.819691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.1 [24126.073534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.10 [24126.325748] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.64 [24126.579605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.128 [24126.832594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.200 [24127.085330] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.250 [24127.340219] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.1 [24127.593700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.10 [24127.845722] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.64 [24128.098484] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.128 [24128.353522] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.200 [24128.604908] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.250 [24128.856617] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.1 [24129.109270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.10 [24129.364478] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.64 [24129.620682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.128 [24129.874071] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.200 [24130.126892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.250 [24130.381678] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.1 [24130.638106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.10 [24130.893411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.64 [24131.145341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.128 [24131.397506] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.200 [24131.649481] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.250 [24131.901387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.1 [24132.154818] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.10 [24132.406971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.64 [24132.700165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.128 [24132.952301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.200 [24133.205133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.250 [24133.458923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.1 [24133.715071] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.10 [24133.969107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.64 [24134.222690] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.128 [24134.477612] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.200 [24134.729618] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.250 [24134.983370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.1 [24135.237111] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.10 [24135.489662] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.64 [24135.742939] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.128 [24135.994750] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.200 [24136.247511] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.250 [24136.501929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.1 [24136.753598] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.10 [24137.005965] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.64 [24137.260312] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.128 [24137.512491] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.200 [24137.764949] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.250 [24138.020066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.1 [24138.272927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.10 [24138.527053] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.64 [24138.778887] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.128 [24139.032295] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.200 [24139.284686] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.250 [24139.537309] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.1 [24139.789829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.10 [24140.044681] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.64 [24140.297553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.128 [24140.548529] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.200 [24140.803122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.250 [24141.056047] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.1 [24141.309459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.10 [24141.564788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.64 [24141.819305] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.128 [24142.073144] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.200 [24142.327772] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.250 [24142.582285] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.1 [24142.837460] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.10 [24143.089293] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.64 [24143.345449] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.128 [24143.597980] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.200 [24143.850310] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.250 [24144.105291] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.1 [24144.358695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.10 [24144.610701] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.64 [24144.865220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.128 [24145.118597] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.200 [24145.372916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.250 [24145.624539] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.1 [24145.876190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.10 [24146.129660] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.64 [24146.383788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.128 [24146.636193] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.200 [24146.887127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.250 [24147.139605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.1 [24147.389568] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.10 [24147.643480] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.64 [24147.894191] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.128 [24148.144178] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.200 [24148.399468] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.250 [24148.650335] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.1 [24148.901407] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.10 [24149.152659] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.64 [24149.404888] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.128 [24149.656105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.200 [24149.910965] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.250 [24150.163512] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.1 [24150.418309] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.10 [24150.666841] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.64 [24150.920308] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.128 [24151.172897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.200 [24151.423596] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.250 [24151.673896] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.1 [24151.926102] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.10 [24152.177459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.64 [24152.428344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.128 [24152.680462] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.200 [24152.932971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.250 [24153.184443] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.1 [24153.438065] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.10 [24153.688510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.64 [24153.939643] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.128 [24154.191140] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.200 [24154.445366] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.250 [24154.700620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.1 [24154.952788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.10 [24155.207284] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.64 [24155.463053] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.128 [24155.718691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.200 [24155.972509] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.250 [24156.227290] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.1 [24156.479854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.10 [24156.731181] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.64 [24156.983879] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.128 [24157.237749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.200 [24157.490956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.250 [24157.744525] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.1 [24157.996672] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.10 [24158.250348] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.64 [24158.503513] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.128 [24158.755430] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.200 [24159.029173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.250 [24159.287695] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.1 [24159.542938] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.10 [24159.796751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.64 [24160.050180] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.128 [24160.303705] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.200 [24160.558085] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.250 [24160.812439] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.1 [24161.065763] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.10 [24161.320343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.64 [24161.573856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.128 [24161.827909] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.200 [24162.084456] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.250 [24162.340784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.1 [24162.594485] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.10 [24162.846691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.64 [24163.101598] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.128 [24163.356281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.200 [24163.610876] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.250 [24163.864541] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.1 [24164.116893] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.10 [24164.368840] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.64 [24164.620143] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.128 [24164.872894] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.200 [24165.126197] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.250 [24165.379435] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.1 [24165.631460] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.10 [24165.883635] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.64 [24166.137351] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.128 [24166.388106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.200 [24166.639373] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.250 [24166.893749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.1 [24167.146076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.10 [24167.398548] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.64 [24167.652209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.128 [24167.905542] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.200 [24168.157884] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.250 [24168.411566] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.1 [24168.662812] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.10 [24168.915972] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.64 [24169.169888] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.128 [24169.423799] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.200 [24169.679378] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.250 [24169.934691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.1 [24170.187765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.10 [24170.439973] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.64 [24170.692226] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.128 [24170.945023] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.200 [24171.197192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.250 [24171.452622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.1 [24171.705767] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.10 [24171.959990] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.64 [24172.212502] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.128 [24172.467702] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.200 [24172.719191] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.250 [24172.970562] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.1 [24173.224491] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.10 [24173.477112] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.64 [24173.731956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.128 [24173.986353] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.200 [24174.238463] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.250 [24174.493725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.1 [24174.745083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.10 [24175.000440] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.64 [24175.249993] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.128 [24175.500340] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.200 [24175.752948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.250 [24176.005349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.1 [24176.257608] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.10 [24176.510699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.64 [24176.763210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.128 [24177.014745] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.200 [24177.266905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.250 [24177.517605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [24177.771652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24178.026419] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [24178.279657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24178.532930] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [24178.785459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24179.039467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24179.292648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24179.546091] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24179.798028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24180.053949] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24180.304341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24180.557571] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24180.809110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24181.060862] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24181.312510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24181.565187] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24181.817405] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24182.073223] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24182.323702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24182.577062] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24182.830149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24183.082144] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24183.336178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24183.590072] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24183.842722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24184.096126] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24184.350818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24184.605902] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24184.858746] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24185.113272] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24185.365843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24185.674050] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24186.557411] Lustre: DEBUG MARKER: rc=0; [24186.557411] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24186.557411] if [[ $? -eq 0 && $val -ne 0 ]]; then [24186.557411] echo $(hostname -s): $val; [24186.557411] rc=$val; [24186.557411] fi; [24186.557411] exit $rc [24187.165241] Lustre: DEBUG MARKER: dmesg [24187.843460] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 14: test default nodemap nid lookup =============================================== 14:21:52 \(1491574912\) [24187.995540] Lustre: DEBUG MARKER: == sanity-sec test 14: test default nodemap nid lookup =============================================== 14:21:52 (1491574912) [24188.143117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24188.143117] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24188.143117] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24188.409188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24188.409188] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24188.409188] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24188.676816] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [24188.931716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24188.931716] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24188.931716] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24189.198895] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [24189.455115] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [24189.712464] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24189.969172] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [24190.222277] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24190.476457] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [24190.732209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24190.986055] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [24191.240758] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24191.497122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [24191.751110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24192.005706] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [24192.262142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24192.516449] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [24192.773387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24193.029073] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [24193.284323] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24193.539953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [24193.796901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24194.050724] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [24194.304739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24194.562732] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [24194.817609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24195.072940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [24195.327586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24195.583082] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [24195.835228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24196.088240] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [24196.340479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24196.595128] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [24196.850972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24197.104625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [24197.360996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24197.615294] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.1 [24197.868519] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.10 [24198.125066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.64 [24198.381336] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.128 [24198.635412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.200 [24198.889229] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.0.250 [24199.142530] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.1 [24199.396088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.10 [24199.649320] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.64 [24199.906101] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.128 [24200.161956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.200 [24200.413964] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.1.250 [24200.666341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.1 [24200.918592] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.10 [24201.172566] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.64 [24201.425527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.128 [24201.679419] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.200 [24201.932519] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.0.2.250 [24202.186582] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.1 [24202.438089] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.10 [24202.690717] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.64 [24202.944990] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.128 [24203.198081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.200 [24203.450341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.0.250 [24203.703481] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.1 [24203.955645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.10 [24204.209446] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.64 [24204.465131] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.128 [24204.719600] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.200 [24204.973858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.1.250 [24205.227657] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.1 [24205.481806] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.10 [24205.734697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.64 [24205.986884] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.128 [24206.240489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.200 [24206.492671] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.1.2.250 [24206.746124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.1 [24206.998225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.10 [24207.250128] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.64 [24207.502815] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.128 [24207.754786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.200 [24208.008682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.0.250 [24208.261474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.1 [24208.512156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.10 [24208.762948] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.64 [24209.014268] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.128 [24209.264578] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.200 [24209.516809] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.1.250 [24209.768891] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.1 [24210.022265] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.10 [24210.275645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.64 [24210.527230] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.128 [24210.781192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.200 [24211.034149] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.2.2.250 [24211.287855] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.1 [24211.538535] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.10 [24211.789904] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.64 [24212.042849] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.128 [24212.295106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.200 [24212.547751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.0.250 [24212.799391] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.1 [24213.052177] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.10 [24213.305365] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.64 [24213.556378] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.128 [24213.809638] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.200 [24214.062449] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.1.250 [24214.315057] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.1 [24214.567212] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.10 [24214.820182] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.64 [24215.078104] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.128 [24215.332091] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.200 [24215.586049] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.3.2.250 [24215.840303] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.1 [24216.093301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.10 [24216.344916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.64 [24216.597028] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.128 [24216.850505] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.200 [24217.105932] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.0.250 [24217.357653] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.1 [24217.610501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.10 [24217.862345] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.64 [24218.115670] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.128 [24218.370452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.200 [24218.624241] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.1.250 [24218.878977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.1 [24219.131632] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.10 [24219.385364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.64 [24219.640085] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.128 [24219.893633] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.200 [24220.146212] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.4.2.250 [24220.398589] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.1 [24220.651359] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.10 [24220.903870] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.64 [24221.156693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.128 [24221.411446] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.200 [24221.664740] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.0.250 [24221.917415] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.1 [24222.170412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.10 [24222.423625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.64 [24222.680200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.128 [24222.933850] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.200 [24223.187605] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.1.250 [24223.442234] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.1 [24223.694346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.10 [24223.948700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.64 [24224.201243] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.128 [24224.454698] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.200 [24224.713480] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.5.2.250 [24224.970692] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.1 [24225.231179] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.10 [24225.483154] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.64 [24225.735107] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.128 [24225.988412] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.200 [24226.241152] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.0.250 [24226.493210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.1 [24226.747129] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.10 [24227.000668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.64 [24227.253364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.128 [24227.504134] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.200 [24227.757699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.1.250 [24228.015216] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.1 [24228.270607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.10 [24228.523301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.64 [24228.776941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.128 [24229.029929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.200 [24229.283019] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.6.2.250 [24229.536439] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.1 [24229.787257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.10 [24230.040723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.64 [24230.293411] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.128 [24230.547168] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.200 [24230.798593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.0.250 [24231.051649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.1 [24231.303337] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.10 [24231.555393] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.64 [24231.806301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.128 [24232.057085] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.200 [24232.308801] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.1.250 [24232.560492] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.1 [24232.815092] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.10 [24233.068164] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.64 [24233.321780] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.128 [24233.573480] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.200 [24233.826864] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.7.2.250 [24234.079466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.1 [24234.332621] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.10 [24234.585847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.64 [24234.838349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.128 [24235.090473] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.200 [24235.341496] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.0.250 [24235.591998] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.1 [24235.844964] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.10 [24236.097221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.64 [24236.349286] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.128 [24236.600776] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.200 [24236.852053] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.1.250 [24237.102139] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.1 [24237.354192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.10 [24237.605523] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.64 [24237.857474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.128 [24238.110444] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.200 [24238.364788] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.8.2.250 [24238.616470] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.1 [24238.868757] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.10 [24239.121774] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.64 [24239.373947] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.128 [24239.627503] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.200 [24239.881186] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.0.250 [24240.132051] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.1 [24240.384424] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.10 [24240.635365] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.64 [24240.885858] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.128 [24241.137999] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.200 [24241.388806] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.1.250 [24241.642599] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.1 [24241.895098] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.10 [24242.149077] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.64 [24242.400494] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.128 [24242.651037] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.200 [24242.904270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.9.2.250 [24243.156116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.1 [24243.407926] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.10 [24243.658176] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.64 [24243.913960] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.128 [24244.164156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.200 [24244.417550] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.0.250 [24244.671404] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.1 [24244.927618] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.10 [24245.180307] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.64 [24245.431810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.128 [24245.683156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.200 [24245.937112] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.1.250 [24246.192709] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.1 [24246.445496] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.10 [24246.696700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.64 [24246.949822] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.128 [24247.202116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.200 [24247.456466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.10.2.250 [24247.709055] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.1 [24247.966529] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.10 [24248.219674] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.64 [24248.470897] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.128 [24248.722834] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.200 [24248.977817] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.0.250 [24249.232342] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.1 [24249.488244] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.10 [24249.742347] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.64 [24249.998419] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.128 [24250.250964] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.200 [24250.503854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.1.250 [24250.756941] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.1 [24251.010505] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.10 [24251.264796] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.64 [24251.519335] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.128 [24251.773760] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.200 [24252.026581] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.11.2.250 [24252.279873] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.1 [24252.536043] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.10 [24252.789446] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.64 [24253.042810] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.128 [24253.301749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.200 [24253.554680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.0.250 [24253.807745] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.1 [24254.061757] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.10 [24254.315207] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.64 [24254.567879] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.128 [24254.820730] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.200 [24255.074558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.1.250 [24255.329184] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.1 [24255.580626] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.10 [24255.834720] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.64 [24256.086697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.128 [24256.342081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.200 [24256.595099] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.12.2.250 [24256.853874] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.1 [24257.106573] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.10 [24257.361593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.64 [24257.614481] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.128 [24257.866225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.200 [24258.120669] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.0.250 [24258.372323] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.1 [24258.626612] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.10 [24258.880338] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.64 [24259.135100] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.128 [24259.387452] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.200 [24259.642281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.1.250 [24259.895476] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.1 [24260.153648] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.10 [24260.407928] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.64 [24260.660527] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.128 [24260.913528] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.200 [24261.166682] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.13.2.250 [24261.418935] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.1 [24261.669373] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.10 [24261.922738] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.64 [24262.176133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.128 [24262.428240] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.200 [24262.681330] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.0.250 [24262.936630] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.1 [24263.187576] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.10 [24263.440432] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.64 [24263.690320] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.128 [24263.942624] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.200 [24264.195593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.1.250 [24264.450558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.1 [24264.702464] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.10 [24264.954905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.64 [24265.209058] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.128 [24265.462030] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.200 [24265.717775] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.14.2.250 [24265.968493] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.1 [24266.223425] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.10 [24266.478520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.64 [24266.729844] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.128 [24266.982323] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.200 [24267.235397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.0.250 [24267.487518] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.1 [24267.741921] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.10 [24267.992687] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.64 [24268.245674] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.128 [24268.502122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.200 [24268.753939] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.1.250 [24269.007513] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.1 [24269.259761] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.10 [24269.512488] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.64 [24269.764940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.128 [24270.017602] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.200 [24270.271490] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_nid 88.15.2.250 [24270.523237] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [24270.775085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24271.027144] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [24271.279169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24271.532815] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [24271.785530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24272.037514] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24272.288536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24272.540445] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24272.794788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24273.048281] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24273.302044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24273.555148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24273.806332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24274.060510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24274.329030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24274.592758] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24274.847286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24275.102545] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24275.356479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24275.611495] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24275.864993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24276.119203] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24276.371448] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24276.625814] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24276.878156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24277.132697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24277.386604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24277.640084] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24277.894585] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24278.149621] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24278.404416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24278.716960] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24279.598504] Lustre: DEBUG MARKER: rc=0; [24279.598504] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24279.598504] if [[ $? -eq 0 && $val -ne 0 ]]; then [24279.598504] echo $(hostname -s): $val; [24279.598504] rc=$val; [24279.598504] fi; [24279.598504] exit $rc [24280.198399] Lustre: DEBUG MARKER: dmesg [24280.820644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 15: test id mapping =============================================================== 14:23:25 \(1491575005\) [24280.985543] Lustre: DEBUG MARKER: == sanity-sec test 15: test id mapping =============================================================== 14:23:25 (1491575005) [24281.132177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24281.132177] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24281.132177] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24281.397938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24281.397938] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24281.397938] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24281.663703] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_uid --name default --value 99 [24281.917985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24281.917985] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24281.917985] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24282.185413] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --property squash_gid --name default --value 99 [24282.437611] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_0 [24282.691429] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24282.949609] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_1 [24283.205369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24283.464886] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_2 [24283.718344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24283.974066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_3 [24284.227081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24284.483590] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_4 [24284.737416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24284.993891] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_5 [24285.249417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24285.503489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_6 [24285.757810] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24286.013340] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_7 [24286.268693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24286.522036] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_8 [24286.775495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24287.031936] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_9 [24287.288790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24287.542594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_10 [24287.795435] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24288.052521] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_11 [24288.307118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24288.560648] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_12 [24288.813692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24289.070352] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_13 [24289.413417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24289.668467] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_14 [24289.922204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24290.178440] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add 25087_15 [24290.431164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24290.686156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.0.[1-253]@tcp [24290.941641] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.1.[1-253]@tcp [24291.199343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_0 --range 88.0.2.[1-253]@tcp [24291.458260] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.0.[1-253]@tcp [24291.711069] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.1.[1-253]@tcp [24291.965864] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_1 --range 88.1.2.[1-253]@tcp [24292.222733] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.0.[1-253]@tcp [24292.476841] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.1.[1-253]@tcp [24292.730393] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_2 --range 88.2.2.[1-253]@tcp [24292.983405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.0.[1-253]@tcp [24293.236956] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.1.[1-253]@tcp [24293.491662] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_3 --range 88.3.2.[1-253]@tcp [24293.745071] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.0.[1-253]@tcp [24293.999825] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.1.[1-253]@tcp [24294.252953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_4 --range 88.4.2.[1-253]@tcp [24294.506459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.0.[1-253]@tcp [24294.759725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.1.[1-253]@tcp [24295.013427] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_5 --range 88.5.2.[1-253]@tcp [24295.266246] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.0.[1-253]@tcp [24295.521877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.1.[1-253]@tcp [24295.774911] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_6 --range 88.6.2.[1-253]@tcp [24296.027361] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.0.[1-253]@tcp [24296.281839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.1.[1-253]@tcp [24296.534944] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_7 --range 88.7.2.[1-253]@tcp [24296.800551] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.0.[1-253]@tcp [24297.068423] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.1.[1-253]@tcp [24297.324048] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_8 --range 88.8.2.[1-253]@tcp [24297.576152] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.0.[1-253]@tcp [24297.829051] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.1.[1-253]@tcp [24298.081696] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_9 --range 88.9.2.[1-253]@tcp [24298.337728] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.0.[1-253]@tcp [24298.590361] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.1.[1-253]@tcp [24298.845645] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_10 --range 88.10.2.[1-253]@tcp [24299.098018] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.0.[1-253]@tcp [24299.348669] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.1.[1-253]@tcp [24299.602892] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_11 --range 88.11.2.[1-253]@tcp [24299.856920] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.0.[1-253]@tcp [24300.109855] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.1.[1-253]@tcp [24300.363170] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_12 --range 88.12.2.[1-253]@tcp [24300.616074] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.0.[1-253]@tcp [24300.869195] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.1.[1-253]@tcp [24301.121703] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_13 --range 88.13.2.[1-253]@tcp [24301.372466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.0.[1-253]@tcp [24301.623349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.1.[1-253]@tcp [24301.877477] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_14 --range 88.14.2.[1-253]@tcp [24302.127751] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.0.[1-253]@tcp [24302.378095] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.1.[1-253]@tcp [24302.629550] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name 25087_15 --range 88.15.2.[1-253]@tcp [24302.882847] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [24303.137249] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [24303.387753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 1 [24303.640081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 1 [24303.892680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 1 [24304.144725] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 1 [24304.398061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 1 [24304.650879] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 1 [24304.906351] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 1 [24305.161894] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 1 [24305.415891] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 1 [24305.670784] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 1 [24305.922769] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 1 [24306.178937] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 1 [24306.433700] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 1 [24306.689080] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 1 [24306.941668] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 1 [24307.195233] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 1 [24307.446244] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 1 [24307.700916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 1 [24307.953245] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 1 [24308.204930] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 1 [24308.457692] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 1 [24308.709538] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 1 [24308.963756] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 1 [24309.217438] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 1 [24309.471377] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 1 [24309.727129] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 1 [24309.981839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 1 [24310.236688] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 1 [24310.488313] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 1 [24310.741149] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 1 [24310.994363] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 1 [24311.247530] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 1 [24311.500331] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.0.100@tcp --idtype uid --id 0 [24311.754289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.1.100@tcp --idtype uid --id 0 [24312.004584] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.2.100@tcp --idtype uid --id 0 [24312.256346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property admin --value 0 [24312.511502] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property admin --value 0 [24312.765415] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property admin --value 0 [24313.020604] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property admin --value 0 [24313.273769] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property admin --value 0 [24313.527874] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property admin --value 0 [24313.783257] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property admin --value 0 [24314.036442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property admin --value 0 [24314.289117] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property admin --value 0 [24314.543972] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property admin --value 0 [24314.799349] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property admin --value 0 [24315.054113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property admin --value 0 [24315.308413] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property admin --value 0 [24315.561610] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property admin --value 0 [24315.812551] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property admin --value 0 [24316.063749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property admin --value 0 [24316.317380] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.0.100@tcp --idtype uid --id 0 [24316.570437] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.1.100@tcp --idtype uid --id 0 [24316.822833] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 88.0.2.100@tcp --idtype uid --id 0 [24317.074206] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_0 --property trusted --value 0 [24317.326929] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_1 --property trusted --value 0 [24317.581556] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_2 --property trusted --value 0 [24317.836724] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_3 --property trusted --value 0 [24318.090490] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_4 --property trusted --value 0 [24318.343368] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_5 --property trusted --value 0 [24318.597063] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_6 --property trusted --value 0 [24318.850596] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_7 --property trusted --value 0 [24319.105803] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_8 --property trusted --value 0 [24319.358343] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_9 --property trusted --value 0 [24319.610925] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_10 --property trusted --value 0 [24319.862389] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_11 --property trusted --value 0 [24320.115853] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_12 --property trusted --value 0 [24320.367915] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_13 --property trusted --value 0 [24320.621105] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_14 --property trusted --value 0 [24320.873787] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name 25087_15 --property trusted --value 0 [24321.126839] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_0 [24321.380369] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_0.id [24321.634477] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_1 [24321.886598] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_1.id [24322.140634] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_2 [24322.396202] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_2.id [24322.652190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_3 [24322.905274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_3.id [24323.161455] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_4 [24323.415280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_4.id [24323.669151] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_5 [24323.924169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_5.id [24324.178318] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_6 [24324.432100] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_6.id [24324.685071] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_7 [24324.938894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_7.id [24325.195333] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_8 [24325.447752] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_8.id [24325.702395] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_9 [24325.955151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_9.id [24326.208082] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_10 [24326.461165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_10.id [24326.714791] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_11 [24326.967102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_11.id [24327.218662] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_12 [24327.470135] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_12.id [24327.722727] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_13 [24327.976558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_13.id [24328.229853] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_14 [24328.482344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_14.id [24328.735383] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del 25087_15 [24328.986817] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.25087_15.id [24329.301184] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24330.193512] Lustre: DEBUG MARKER: rc=0; [24330.193512] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24330.193512] if [[ $? -eq 0 && $val -ne 0 ]]; then [24330.193512] echo $(hostname -s): $val; [24330.193512] rc=$val; [24330.193512] fi; [24330.193512] exit $rc [24330.789964] Lustre: DEBUG MARKER: dmesg [24331.390556] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 16: test nodemap all_off fileops ================================================== 14:24:16 \(1491575056\) [24331.551467] Lustre: DEBUG MARKER: == sanity-sec test 16: test nodemap all_off fileops ================================================== 14:24:16 (1491575056) [24331.703370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24331.703370] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24331.703370] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24331.977515] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [24332.124234] Lustre: 16027:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [24332.272622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [24332.523804] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [24332.775865] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [24333.029374] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [24333.282622] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [24333.535341] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [24333.788411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24334.042268] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24334.314346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.idmap [24335.118258] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24336.204124] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [24336.458165] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [24336.713490] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [24336.968512] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [24337.221272] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [24337.473293] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [24337.728793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24337.984934] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24338.257304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.idmap [24346.823174] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24347.634086] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [24347.891646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24348.151140] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24348.419648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [24356.968303] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24359.339990] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24360.177620] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [24360.435975] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [24360.689259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24360.942574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24361.194385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24361.447819] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24361.700106] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24361.953160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24362.207814] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24362.502697] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24362.756566] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24363.020850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24363.273948] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24363.529586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [24363.781856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [24364.035120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24365.814921] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24366.757927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24367.159786] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24367.452301] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24368.734416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24370.012067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24371.296428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24372.585854] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24373.873787] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24375.160079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24377.021574] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24378.092110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24378.354978] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24378.636824] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24379.911450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24381.187344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24382.460614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24383.742183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24385.028973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24386.777305] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24387.783036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24388.053677] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24388.336126] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24389.613800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24390.891523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24392.168344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24393.457395] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24394.732197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24396.011509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24397.291571] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24398.575602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24398.876455] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [24399.133288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24399.390425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24399.644699] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24399.898889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24402.748716] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24404.261535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24404.528746] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24404.808691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24406.090164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24407.370479] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24408.653046] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24409.930005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24411.203828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24413.876307] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24415.291392] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24415.559338] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24415.852138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24417.143424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24418.435167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24419.728482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24421.016906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24422.299745] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24423.588859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24426.227428] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24427.644141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24427.907692] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24428.195192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24429.481930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24430.767260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24432.056729] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24433.340413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24434.630866] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24435.920037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24436.474910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24436.729697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24436.983327] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24437.237685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24437.493144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24437.748132] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24438.098356] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24438.351658] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24438.614213] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24438.871191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24439.124685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24439.392613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24439.649724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [24439.908155] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [24440.163716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24441.899504] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24442.843775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24451.545491] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24452.522072] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24462.513732] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24463.463030] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24470.749127] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [24471.012336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24471.271785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24471.527868] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24471.785592] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24474.646349] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24476.153821] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24488.357402] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24489.791467] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24503.301147] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24504.708906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24514.788348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24515.055719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24515.310039] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24515.569631] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24515.824221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24516.078272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24516.418629] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24516.672920] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24516.935761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24517.189971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24517.444626] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24517.702844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24517.957590] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [24518.213426] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [24518.468305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24520.262840] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24521.195621] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24521.456733] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24521.742522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24523.023955] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24524.307939] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24525.583142] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24526.859155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24528.137734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24529.421473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24530.706459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24532.479339] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24533.470221] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24533.735071] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24534.028129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24535.305172] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24536.585110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24537.868228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24539.151140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24540.436972] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24542.189710] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24543.146929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24543.408455] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24543.693473] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24544.967870] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24546.248191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24547.526140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24548.807091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24550.093432] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24551.371963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24551.672856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [24551.927969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24552.181865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24552.436586] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24552.690344] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24555.583308] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24557.054292] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24557.320253] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24557.614799] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24558.889383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24560.164679] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24561.445385] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24562.727119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24564.007052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24565.289156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24567.947657] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24569.362720] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24569.641725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24569.940488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24571.230509] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24572.516839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24573.790374] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24575.071311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24576.362412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24577.639763] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24578.918843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24580.201978] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24581.479802] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24584.175383] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24585.642163] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24585.908436] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24586.194335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24587.485492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24588.767379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24590.043451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24591.322179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24592.604502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24593.886094] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24595.165065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24595.844273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24596.107973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24596.364579] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24596.620505] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24596.875882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24597.185938] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24597.504070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24597.763205] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24598.031569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24598.286773] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24598.543925] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24598.800422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24599.054512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [24599.309514] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [24599.562115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24601.341174] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24602.336089] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24611.116883] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24612.068098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24622.068957] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24623.053785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24630.343905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [24630.599280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24630.853703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24631.105881] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24631.360175] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24634.292160] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24635.719267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24647.958799] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24649.402210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24662.926185] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24664.367674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24675.758066] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [24676.023270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [24676.279979] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [24676.536119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [24676.791889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24677.045970] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [24677.299529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24677.557124] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24677.832905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [24678.639406] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24679.537180] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24680.434977] Lustre: DEBUG MARKER: rc=0; [24680.434977] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24680.434977] if [[ $? -eq 0 && $val -ne 0 ]]; then [24680.434977] echo $(hostname -s): $val; [24680.434977] rc=$val; [24680.434977] fi; [24680.434977] exit $rc [24681.037429] Lustre: DEBUG MARKER: dmesg [24681.640760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 17: test nodemap trusted_noadmin fileops ========================================== 14:30:06 \(1491575406\) [24681.819660] Lustre: DEBUG MARKER: == sanity-sec test 17: test nodemap trusted_noadmin fileops ========================================== 14:30:06 (1491575406) [24681.966888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24681.966888] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24681.966888] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24682.243217] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [24682.395398] Lustre: 3493:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [24682.399080] Lustre: 3493:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [24682.545602] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [24682.800277] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [24683.055900] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [24683.310463] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [24683.606195] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [24683.862167] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [24684.131266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24684.653211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [24684.910986] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [24685.166493] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [24685.423210] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [24685.682069] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [24685.940023] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [24686.194286] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24686.576023] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [24686.830880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24687.087927] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24687.360023] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [24691.263635] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24692.075469] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [24692.330723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [24692.586118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24692.841382] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24693.110343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [24700.119668] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24700.926477] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24701.181878] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24701.434741] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24701.688449] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24701.945492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24702.200376] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24702.474067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [24703.281636] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24704.098200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24704.357971] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24704.631311] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [24705.436899] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24706.251354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24706.507696] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24706.762113] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24707.020194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24707.276844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24707.531496] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24707.802614] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24713.270511] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24714.086996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24714.349697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24714.626456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24715.445159] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24716.326176] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24716.582388] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24716.844856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24717.101917] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24717.424737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24724.415150] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24725.233084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24725.494629] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24725.780600] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24726.605690] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24727.418564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [24727.672300] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [24727.931516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24728.189068] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24728.461926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [24733.917780] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24736.286832] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24738.581224] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24739.575935] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24739.837804] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24740.120864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24741.405717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24742.691239] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24743.974624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24745.259934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24746.556273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24747.838305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24749.117550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24750.391444] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24751.675738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24753.421143] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24754.364108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24754.630815] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24754.903655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24756.188379] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24757.469703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24758.743740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24760.017195] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24761.295043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24763.096724] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24764.037583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24764.297095] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24764.580488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24765.861850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24767.139851] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24768.414727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24769.694529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24770.977107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24771.281932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24771.534876] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24771.790108] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24772.045287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24772.299078] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24772.554162] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24772.825498] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24781.352726] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24782.161279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24782.417294] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24782.686718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24783.486711] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24784.366736] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24784.623116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24784.886469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24785.142679] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24785.415103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24790.866748] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24793.225562] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24794.035695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24794.291144] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24794.560756] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24795.361217] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24797.637544] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24798.581150] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24807.386285] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24808.375693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24818.417985] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24819.410143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24831.809158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24832.075607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24832.331287] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24832.592539] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24832.855180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24833.111667] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24833.384856] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24841.916410] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24842.727751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24842.984139] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24843.256237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24844.054451] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24844.938392] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24845.195574] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24845.460976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24845.719606] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24845.991235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24851.444412] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24853.812694] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24854.620779] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24854.875640] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24855.146366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24855.943730] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24858.208542] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24859.151141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24859.431784] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24859.720180] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24860.998626] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24862.292260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24863.575863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24864.853209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24866.131755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24867.888441] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24868.877108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24869.140874] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24869.424118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24870.700453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24871.989054] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24873.268418] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24874.560336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24875.841860] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24877.123108] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24878.395877] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24879.682303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24881.484218] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24882.474785] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24882.739605] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [24883.012436] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24884.302456] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24885.587384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24886.861096] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24888.134218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24889.411039] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24890.692169] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24891.966047] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24892.290818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24892.544244] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24892.799346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24893.053723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24893.306036] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24893.560172] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24893.828838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24902.370996] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24903.188259] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24903.444893] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24903.714314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24904.529726] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24905.433173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24905.688419] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24905.952926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24906.211359] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24906.481930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24915.011253] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24915.839288] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24916.094122] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24916.366597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24917.172073] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24919.440115] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24920.480086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24931.809199] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24932.744476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24942.779073] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [24943.704267] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [24950.969408] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [24951.227961] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [24951.482641] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [24951.738709] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [24951.991907] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24952.247338] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24952.516909] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [24959.496048] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24960.328978] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [24960.602251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24960.863297] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24961.137315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [24968.128790] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24968.997227] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [24969.892424] Lustre: DEBUG MARKER: rc=0; [24969.892424] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [24969.892424] if [[ $? -eq 0 && $val -ne 0 ]]; then [24969.892424] echo $(hostname -s): $val; [24969.892424] rc=$val; [24969.892424] fi; [24969.892424] exit $rc [24970.494165] Lustre: DEBUG MARKER: dmesg [24971.073576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 18: test nodemap mapped_noadmin fileops =========================================== 14:34:55 \(1491575695\) [24971.237132] Lustre: DEBUG MARKER: == sanity-sec test 18: test nodemap mapped_noadmin fileops =========================================== 14:34:55 (1491575695) [24971.385205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [24971.385205] /usr/sbin/lctl lustre_build_version 2>/dev/null || [24971.385205] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [24971.665660] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [24971.809737] Lustre: 20413:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [24971.813387] Lustre: 20413:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [24971.954361] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [24972.209797] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [24972.463608] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [24972.719173] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [24972.974643] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [24973.228583] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [24973.483562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24974.001748] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [24974.256524] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [24974.512242] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [24974.765824] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [24975.019750] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [24975.276303] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [24975.532471] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24975.787868] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [24976.042958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24976.303451] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24976.572320] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [24977.381218] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24978.198600] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [24978.456831] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [24978.711454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24978.966876] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24979.238891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [24986.228553] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24987.039853] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24987.295483] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24987.549400] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24987.806227] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24988.068289] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24988.326069] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24988.610840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [24989.424436] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24990.239634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24990.496870] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24990.767309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [24991.570923] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24992.375074] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [24992.629713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [24992.929602] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [24993.191803] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [24993.506086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24993.762681] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24994.036080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [24996.378677] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24997.193947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [24997.450228] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24997.721521] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [24998.565746] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [24999.415560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [24999.671147] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [24999.935554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25000.190996] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25000.475231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25005.921512] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25008.269844] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25009.124894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25009.387886] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25009.657147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25010.470517] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25011.301718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [25011.555087] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [25011.812134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25012.176275] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25012.446601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25022.534434] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25024.796872] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25025.736102] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25026.010700] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25026.304335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25027.595591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25028.883874] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25030.173548] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25031.461124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25032.737739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25034.017905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25035.295182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25036.577440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25038.353246] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25039.375495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25039.644906] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25039.939452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25041.223666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25042.516218] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25043.807604] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25045.084106] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25046.362914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25047.639328] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25048.912899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25050.218317] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25051.492095] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25053.306490] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25054.254811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25054.520187] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25054.804544] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25056.091371] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25057.380222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25058.665271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25059.952835] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25061.242160] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25061.552111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25061.806751] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25062.061456] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25062.317882] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25062.571634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25062.829671] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25063.100559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25073.187554] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25074.001240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25074.256211] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25074.527433] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25075.339498] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25076.242664] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25076.501133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25076.766957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25077.026272] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25077.300857] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25087.400207] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25088.236446] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25088.497695] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25088.782903] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25089.585222] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25091.867321] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25092.816718] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25102.938909] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25103.930191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25112.758457] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25113.697090] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25122.301337] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25122.556642] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25122.813185] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25123.067813] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25123.322532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25123.580108] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25123.856451] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25133.948117] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25134.763998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25135.036259] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25135.306980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25136.117843] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25137.023289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25137.287346] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25137.550910] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25137.806738] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25138.082152] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25145.089083] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25145.916617] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25146.174074] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25146.446314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25147.252172] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25149.539905] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25150.530862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25150.798117] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25151.086257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25152.364761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25153.640905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25154.919212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25156.196312] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25157.475318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25158.754105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25160.028984] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25161.306671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25163.073361] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25164.012788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25164.273064] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25164.546156] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25165.820270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25167.096251] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25168.377452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25169.664268] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25170.952943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25172.739572] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25173.672121] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25173.933209] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25174.217741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25175.497173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25176.774977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25178.049173] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25179.344322] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25180.629110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25181.910781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25182.219348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25182.475928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25182.732088] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25182.988338] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25183.244164] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25183.500199] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25183.772845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25193.873398] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25194.710862] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25194.967773] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25195.237332] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25196.034566] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25196.928790] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25197.183677] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25197.451762] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25197.711846] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25197.985638] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25203.441825] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25205.825304] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25206.629601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25206.886277] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25207.154765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25207.982153] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25210.246149] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25211.275287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25223.879580] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25224.891512] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25237.536329] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25238.491828] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25247.067680] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [25247.330254] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [25247.593766] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [25247.847466] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [25248.103469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25248.360829] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25248.630932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25254.092637] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25256.464620] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25257.280743] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [25257.535709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25257.791362] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25258.065869] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25266.648287] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25267.537306] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25268.434145] Lustre: DEBUG MARKER: rc=0; [25268.434145] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25268.434145] if [[ $? -eq 0 && $val -ne 0 ]]; then [25268.434145] echo $(hostname -s): $val; [25268.434145] rc=$val; [25268.434145] fi; [25268.434145] exit $rc [25269.028919] Lustre: DEBUG MARKER: dmesg [25269.601267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 19: test nodemap trusted_admin fileops ============================================ 14:39:54 \(1491575994\) [25269.782089] Lustre: DEBUG MARKER: == sanity-sec test 19: test nodemap trusted_admin fileops ============================================ 14:39:54 (1491575994) [25269.955290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [25269.955290] /usr/sbin/lctl lustre_build_version 2>/dev/null || [25269.955290] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [25270.238485] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [25270.390066] Lustre: 5006:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [25270.393684] Lustre: 5006:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [25270.540520] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [25270.806253] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [25271.062194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [25271.318745] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [25271.576278] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [25271.831145] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [25272.088212] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25272.607081] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [25272.879178] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [25273.152771] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [25273.424752] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [25273.683911] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [25273.944056] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [25274.198675] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25274.456423] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [25274.711502] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25274.966598] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25275.237682] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25279.144061] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25279.950696] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [25280.208501] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [25280.463070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25280.718962] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25281.002724] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25291.090149] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25291.903083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25292.160131] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25292.416647] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25292.675845] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25292.932977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25293.190774] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25293.461656] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25294.271358] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25295.082370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25295.337410] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25295.607878] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [25296.424336] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25297.233129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25297.487969] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25297.743991] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25297.999200] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25298.254390] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25298.508938] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25298.779423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25299.585150] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25301.952031] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25302.765901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25303.022476] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25303.295886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25304.102995] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25304.971191] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25305.230769] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25305.496341] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25305.752993] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25306.025772] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25306.851977] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25307.666557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25307.924162] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25308.195232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25309.007093] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25309.834741] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [25310.106357] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [25310.364501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25310.622062] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25310.894774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25314.791437] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25317.049720] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25318.071586] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25318.333461] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25318.616510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25319.901257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25321.191166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25322.480790] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25323.764091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25325.040335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25326.324632] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25328.199676] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25329.165166] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25329.426911] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25329.711883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25330.994895] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25332.274343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25333.547476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25334.827960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25336.119360] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25337.407905] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25339.298856] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25340.245515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25340.507775] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25340.790595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25342.071742] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25343.345566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25344.619695] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25345.896652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25347.174119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25348.452635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25349.728088] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25350.047383] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25350.300671] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25350.556594] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25350.811331] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25351.065554] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25351.320266] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25351.588640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25352.405659] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25353.213831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25353.469128] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25353.739858] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25354.543698] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25355.427994] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25355.680707] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25355.943958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25356.200353] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25356.471224] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25357.275146] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25358.088559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25358.345578] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25358.620041] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25359.422372] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25361.752722] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25362.765412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25374.129083] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25375.110968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25383.839870] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25384.800208] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25392.071753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25392.332497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25392.585870] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25392.840711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25393.093376] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25393.351157] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25393.620879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25394.423223] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25395.243279] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25395.515825] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25395.799522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25396.618586] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25397.513883] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25397.785884] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25398.061450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25398.321072] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25398.593138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25399.400233] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25400.229084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25400.497669] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25400.771816] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25401.579229] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25403.887111] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25404.923353] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25405.183725] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25405.458522] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25406.742274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25408.023610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25409.298220] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25410.592491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25411.878294] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25413.666381] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25414.611034] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25414.890499] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25415.211529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25416.500800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25417.776162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25419.062177] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25420.349245] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25421.628233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25422.909808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25424.672464] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25425.612891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25425.888559] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25426.172276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25427.450964] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25428.724080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25429.996769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25431.286542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25432.560713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25433.839416] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25435.114384] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25436.406393] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25437.690453] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25438.016657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25438.278662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25438.535270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25438.789950] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25439.045458] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25439.304370] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25439.576348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25440.402945] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25441.237060] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25441.503742] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25441.788368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25442.605108] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25443.537968] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25443.792133] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25444.056932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25444.313207] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25444.586209] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25445.393489] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25446.217398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25446.473187] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25446.746398] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25447.548517] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25449.845218] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25450.799897] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25463.445954] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25464.444055] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25474.524248] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25475.464702] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25487.851339] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [25488.106959] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [25488.360528] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [25488.615987] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [25488.870723] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25489.126258] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25489.402739] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25496.399176] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25498.764075] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25499.582940] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [25499.839625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25500.097419] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25500.369025] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25505.825068] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25508.189661] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25510.564219] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25511.434711] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25512.307151] Lustre: DEBUG MARKER: rc=0; [25512.307151] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25512.307151] if [[ $? -eq 0 && $val -ne 0 ]]; then [25512.307151] echo $(hostname -s): $val; [25512.307151] rc=$val; [25512.307151] fi; [25512.307151] exit $rc [25512.907194] Lustre: DEBUG MARKER: dmesg [25513.534521] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 20: test nodemap mapped_admin fileops ============================================= 14:43:58 \(1491576238\) [25513.696435] Lustre: DEBUG MARKER: == sanity-sec test 20: test nodemap mapped_admin fileops ============================================= 14:43:58 (1491576238) [25513.846136] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [25513.846136] /usr/sbin/lctl lustre_build_version 2>/dev/null || [25513.846136] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [25514.125744] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [25514.277950] Lustre: 21997:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [25514.281493] Lustre: 21997:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [25514.436364] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [25514.693174] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [25514.950083] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [25515.205340] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [25515.459693] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [25515.713192] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [25515.967087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25516.504431] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [25516.769515] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [25517.026927] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [25517.281433] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [25517.548852] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [25517.812843] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [25518.067287] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25518.321783] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [25518.578558] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25518.836298] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25519.112342] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25524.559102] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25525.374309] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [25525.632528] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [25525.888501] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25526.143153] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25526.414749] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25533.406707] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25534.250749] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25534.507877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25534.763171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25535.019531] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25535.276737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25535.531612] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25535.802719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25536.601052] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25537.419562] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25537.692332] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25537.978333] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [25538.785077] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25539.626684] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25539.883709] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25540.147998] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25540.405139] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25540.658865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25540.915261] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25541.186004] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25542.010489] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25542.827564] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25543.084102] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25543.358655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25547.269743] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25548.192954] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25548.461692] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25548.726370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25548.981874] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25549.251996] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25550.057531] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25550.900510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25551.157410] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25551.425890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25555.326245] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25556.154260] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [25556.410678] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [25556.678200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25556.940041] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25557.214411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25565.741917] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25568.051304] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25569.067774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25569.331098] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25569.612582] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25570.895291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25572.175844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25573.452708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25574.731886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25576.011921] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25577.289345] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25579.156291] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25580.178894] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25580.456617] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25580.749093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25582.029960] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25583.308832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25584.586079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25585.867808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25587.145652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25588.429295] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25589.711005] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25590.994968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25592.834600] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25593.952872] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25594.230132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25594.527906] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25595.811985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25597.094534] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25598.375771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25599.662926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25600.944420] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25602.222309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25602.522747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25602.779683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25603.034723] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25603.289061] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25603.545618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25603.800890] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25604.074258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25604.907697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25605.746237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25606.002104] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25606.271514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25610.160351] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25611.043319] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25611.299946] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25611.562891] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25611.817476] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25612.091953] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25612.919319] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25613.723233] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25613.977833] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25614.247131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25619.692105] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25621.999088] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25622.972427] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25634.273603] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25635.238831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25643.970908] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25644.924257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25656.050697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25656.308936] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25656.564997] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25656.820733] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25657.076107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25657.334568] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25657.605264] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25658.418535] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25659.245528] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25659.506467] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25659.789727] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25662.135292] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25664.512867] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25665.373450] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25665.629609] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25665.894359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25666.149462] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25666.424845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25667.227539] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25668.068676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25668.326646] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25668.599514] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25674.044697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25676.403976] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25677.354838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25677.617328] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25677.889217] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25679.215526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25680.499526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25681.790131] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25683.070842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25684.349210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25685.625926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25686.899284] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25688.197937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25689.976271] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25691.201494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25691.479565] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25691.778809] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25693.067998] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25694.345266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25695.622179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25696.905134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25698.192133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25699.981106] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25700.947896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25701.210316] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25701.495705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25702.774850] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25704.057367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25705.331980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25706.624540] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25707.904622] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25709.184157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25710.465911] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25711.743124] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25712.053924] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25712.309553] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25712.564593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25712.820993] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25713.078692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25713.334027] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25713.607116] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25714.432969] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25715.240612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25715.501422] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25715.771174] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25724.302934] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25725.210681] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25725.465571] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [25725.731290] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25725.987394] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25726.257740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25727.057877] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25727.876281] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25728.152657] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25728.421414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25735.431175] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25737.775258] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25738.780629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25748.843635] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25749.816063] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25759.881078] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25760.946918] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25772.077987] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [25772.335759] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [25772.589905] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [25772.846119] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [25773.101488] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25773.357475] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25773.632200] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25782.183749] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25783.012405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [25783.280980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25783.539564] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25783.815494] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25790.818498] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25791.700290] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [25792.599464] Lustre: DEBUG MARKER: rc=0; [25792.599464] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [25792.599464] if [[ $? -eq 0 && $val -ne 0 ]]; then [25792.599464] echo $(hostname -s): $val; [25792.599464] rc=$val; [25792.599464] fi; [25792.599464] exit $rc [25793.210602] Lustre: DEBUG MARKER: dmesg [25793.803255] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 21: test nodemap mapped_trusted_noadmin fileops =================================== 14:48:38 \(1491576518\) [25793.958533] Lustre: DEBUG MARKER: == sanity-sec test 21: test nodemap mapped_trusted_noadmin fileops =================================== 14:48:38 (1491576518) [25794.108238] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [25794.108238] /usr/sbin/lctl lustre_build_version 2>/dev/null || [25794.108238] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [25794.384686] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [25794.533973] Lustre: 6451:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [25794.537652] Lustre: 6451:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [25794.679510] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [25794.945977] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [25795.201953] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [25795.459421] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [25795.715225] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [25795.970049] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [25796.229439] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25796.750957] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [25797.008220] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [25797.265957] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [25797.525705] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [25797.781644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [25798.038238] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [25798.292843] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25798.546553] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [25798.800159] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25799.058324] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25799.333556] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [25804.804896] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25805.617389] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [25805.890400] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [25806.148113] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25806.403636] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25806.678246] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [25815.259904] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25816.081792] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25816.338764] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25816.593442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [25816.849502] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [25817.105343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25817.361242] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25817.634678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [25818.448967] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25819.258997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25819.515736] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25819.768051] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25820.022711] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25820.276806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25820.532677] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25820.802680] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25824.699982] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25825.526407] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25825.785449] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25826.060157] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25826.869369] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25827.742738] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25828.000471] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25828.266330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25828.521508] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25828.794411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25838.888287] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25839.703305] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25839.962432] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25840.233455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25841.044253] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25841.860258] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [25842.127306] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [25842.381970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25842.637154] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25842.907864] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25849.898966] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25852.226379] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25853.157803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25853.419944] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25853.705216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25854.980257] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25856.263499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25857.556711] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25858.839591] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25860.116032] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25861.395699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25862.679266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25864.427539] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25865.414465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25865.680765] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25865.971468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25867.253973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25868.534780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25869.821235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25871.108489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25872.388068] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25874.119993] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25875.064840] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25875.344789] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25875.646073] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25876.931137] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25878.208822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25879.482236] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25880.760767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25882.034327] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25883.313737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25883.616170] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [25883.871778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25884.128677] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25884.401493] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25892.943617] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25893.755535] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25894.011774] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25894.322583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25894.578124] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25894.852248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25903.409615] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25906.461113] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25907.893462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25908.158500] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25908.481347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25909.757750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25911.033533] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25912.315708] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25913.594045] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25914.871919] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25916.148422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25917.424989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25920.237463] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25921.684414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25921.952245] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25922.241262] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25923.528091] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25924.848242] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25926.139629] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25927.416336] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25928.698248] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25929.976215] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25931.257610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25932.548422] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25935.248736] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25936.706223] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25936.970975] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [25937.244920] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25938.519620] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25939.795165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25941.075637] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25942.355018] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25943.633880] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25944.924765] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25946.202714] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25947.486206] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25948.776111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25949.322566] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [25949.590496] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [25949.846748] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25950.103673] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25950.359605] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25950.615242] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25950.886275] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25951.689414] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25952.507716] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25952.765835] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25953.041655] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25953.849619] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25954.747750] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [25955.005344] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [25955.266182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25955.519962] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25955.789889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25956.593296] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25957.413387] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25957.673322] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25957.948899] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [25958.752290] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25959.572435] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [25959.827774] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25960.082756] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25960.353510] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [25968.902461] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25969.708651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [25969.966949] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [25970.223513] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [25970.479618] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25970.749889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [25976.197508] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [25978.472693] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25979.440461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25989.436187] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [25990.383265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [25999.174953] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26000.113697] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26008.670391] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26008.927414] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26009.184901] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26009.461216] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [26014.908042] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26015.734110] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26015.992877] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26016.248204] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26016.503874] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26016.774465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26022.253929] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26025.264305] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26026.737076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26040.242669] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26041.655826] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26055.178095] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26056.642863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26068.006441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26068.266283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26068.521559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26068.780033] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26069.032922] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26069.290535] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26069.564663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26070.398899] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26071.214896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26071.470374] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26071.741652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26072.565682] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26073.499406] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26073.795804] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26074.059105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26074.314795] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26074.585648] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26075.393649] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26076.208692] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26076.463153] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26076.734651] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26077.536749] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26078.366235] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26078.621819] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26078.876598] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26079.150118] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26086.134514] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26088.499459] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26089.316461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26089.572328] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26089.828452] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26090.085870] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26090.358808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [26097.357663] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26099.737169] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26102.128326] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26103.120363] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26103.389577] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26103.684578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26104.960052] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26106.240270] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26107.525182] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26108.815771] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26110.102425] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26111.387490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26112.677532] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26113.977309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26115.748432] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26116.759926] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26117.018953] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26117.307547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26118.628179] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26119.908186] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26121.188669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26122.468968] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26123.743688] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26125.018734] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26126.295829] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26128.074367] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26129.054105] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26129.317034] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26129.606454] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26130.888133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26132.166842] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26133.447450] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26134.739183] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26136.018340] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26137.294757] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26138.580611] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26138.879570] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26139.135309] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26139.392170] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26139.665191] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [26145.104546] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26147.479499] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26148.298033] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26148.567764] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26148.831266] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26149.091082] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26149.374149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26156.415801] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26158.811145] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26161.769404] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26163.301750] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26163.563901] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26163.850713] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26165.125896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26166.404314] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26167.678343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26168.953652] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26170.233937] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26171.517800] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26172.851947] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26175.555818] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26177.078117] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26177.340524] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26177.624707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26178.943092] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26180.217982] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26181.490462] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26182.764845] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26184.093563] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26185.372477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26186.650930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26187.929659] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26190.618386] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26192.071573] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26192.336858] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26192.630731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26193.918542] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26195.205873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26196.483630] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26197.765579] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26199.043666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26199.586162] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26199.843065] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26200.096863] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26200.352079] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26200.613538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26200.871092] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26201.141402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26201.965928] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26202.790547] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26203.046875] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26203.318531] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26204.128405] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26204.985076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26205.241308] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26205.505616] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26205.761205] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26206.032859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26206.838863] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26207.649657] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26207.907470] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26208.177705] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26209.008870] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26209.823196] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26210.078715] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26210.333553] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26210.603726] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26217.664403] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26218.482859] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26218.740438] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26219.000912] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26219.257517] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26219.580889] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [26228.127878] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26230.444835] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26231.391285] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26242.684496] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26243.760873] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26255.080780] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26256.064627] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26263.356264] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26263.612839] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26263.869678] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26264.140645] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.admin_nodemap [26272.738270] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26273.546769] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26273.801954] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26274.060043] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26274.316908] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26274.589107] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26281.582251] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26283.955219] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26286.932085] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26288.487271] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26300.689362] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26302.144359] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26315.684445] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26317.157492] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26323.377458] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [26323.632231] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [26323.887372] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [26324.143050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [26324.400499] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26324.659485] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26324.935098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26333.509243] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26334.333765] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26334.588165] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26334.846519] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26335.121572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26343.692521] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26344.565451] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26345.459785] Lustre: DEBUG MARKER: rc=0; [26345.459785] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26345.459785] if [[ $? -eq 0 && $val -ne 0 ]]; then [26345.459785] echo $(hostname -s): $val; [26345.459785] rc=$val; [26345.459785] fi; [26345.459785] exit $rc [26346.126166] Lustre: DEBUG MARKER: dmesg [26346.742783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 22: test nodemap mapped_trusted_admin fileops ===================================== 14:57:51 \(1491577071\) [26346.893040] Lustre: DEBUG MARKER: == sanity-sec test 22: test nodemap mapped_trusted_admin fileops ===================================== 14:57:51 (1491577071) [26347.046178] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26347.046178] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26347.046178] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26347.326080] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [26347.471917] Lustre: 2449:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [26347.475545] Lustre: 2449:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [26347.614829] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [26347.867174] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [26348.122854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [26348.378387] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [26348.633370] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [26348.886612] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [26349.141506] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26349.664503] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [26349.922534] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [26350.178742] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [26350.436199] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [26350.691856] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [26350.948481] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [26351.204538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26351.460402] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [26351.716543] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26351.971155] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26352.242595] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26359.230943] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26360.063568] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [26360.330506] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [26360.583970] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26360.840620] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26361.113428] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26369.648908] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26370.477070] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26370.751476] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26371.006485] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26371.263118] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [26371.518804] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26371.779383] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26372.070487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [26372.884526] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26373.699222] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26373.957602] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26374.211715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26374.466917] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26374.721914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26374.979210] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26375.250971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26380.712193] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26381.562482] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26381.833759] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26382.112190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26382.927978] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26383.795266] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26384.050731] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26384.313119] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26384.568479] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26384.838430] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26385.644373] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26386.471070] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26386.743257] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26387.013943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26387.839687] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26388.664664] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26390.401806] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26391.362974] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26391.628375] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26391.909276] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26393.200134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26394.479057] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26395.760086] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26397.042892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26398.316610] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26400.065756] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26401.022303] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26401.282046] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26401.564699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26402.840067] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26404.122252] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26405.401634] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26406.690098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26407.965304] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26409.244468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26411.121253] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26412.060928] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26412.326951] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26412.610205] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26413.883155] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26415.163111] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26416.442569] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26417.726767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26419.014368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26420.305815] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26421.597491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26421.910388] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26424.838897] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26426.333536] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26426.598039] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26426.879731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26428.154811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26429.428413] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26430.709097] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26431.986028] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26433.265367] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26434.545347] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26437.247931] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26438.692421] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26438.954773] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26439.244437] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26440.523625] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26441.807636] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26443.095158] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26444.384538] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26445.668761] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26446.943465] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26449.760195] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26451.202760] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26451.466475] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26451.752707] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26453.025185] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26454.299293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26455.572669] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26456.845120] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26458.122037] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26459.405235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26459.967495] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26460.235647] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26460.496442] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26460.752130] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26461.006515] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26461.263120] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26461.547526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26462.359158] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26463.170822] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26463.426732] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26463.700887] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26464.512600] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26465.401504] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26465.656122] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26465.918147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26466.172520] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26466.440930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26467.240684] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26468.062409] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26468.321974] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26468.600302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26469.415434] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26470.243525] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26471.990844] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26472.926196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26485.494343] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26486.477646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26495.185640] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26496.139103] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26503.392093] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26506.278651] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26507.817767] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26521.325934] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26522.802440] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26536.342956] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26537.816283] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26547.896825] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26548.156274] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26548.408558] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26548.662795] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26548.917330] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26549.173970] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26549.442698] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26550.240882] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26551.051228] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26551.305158] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26551.575886] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26552.382214] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26553.243514] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26553.498354] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26553.762768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26554.023700] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26554.297855] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26555.102182] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26555.921319] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26556.193929] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26556.464545] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26557.267846] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26558.089423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26559.926836] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26560.909691] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26561.172525] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26561.454583] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26562.729237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26564.004237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26565.293265] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26566.576915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26567.861469] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26569.140958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26571.000874] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26571.975635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26572.304925] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26572.666925] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26573.964087] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26575.253477] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26576.537196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26577.829578] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26579.110240] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26580.864291] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26581.857748] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26582.121558] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26582.395594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26583.673737] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26584.949461] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26586.227232] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26587.516442] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26588.799781] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26589.113863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26592.009540] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26593.432326] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26593.699622] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26593.971699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26595.252468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26596.531529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26597.810210] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26599.089190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26600.371832] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26601.658795] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26602.949806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26604.253487] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26606.948539] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26608.386747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26608.650323] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26608.940904] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26610.227778] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26611.504489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26612.781879] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26614.058483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26615.331411] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26616.610346] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26617.896085] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26619.184976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26621.924068] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26623.371291] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26623.638132] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 [26623.923523] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26625.210476] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26626.493753] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26627.770980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26629.052373] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26630.331143] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26631.607229] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26632.885129] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26634.160171] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26634.716490] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26634.975803] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26635.231830] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26635.489666] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26635.742079] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26635.995622] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26636.275149] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26637.088802] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26637.929307] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26638.183885] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26638.453141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26639.251740] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26640.101557] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26640.355753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26640.621133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26640.877075] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26641.149699] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26641.949643] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26642.772272] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26643.029337] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26643.300612] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26644.102697] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26644.914747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c1.admin_nodemap [26646.633796] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26647.626069] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26660.242122] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26661.193993] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26671.222759] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26672.235134] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26679.491362] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26682.453707] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26683.913302] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26696.132952] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26697.549128] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26712.301484] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync=1 [26713.781188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_* [26723.878906] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [26724.140270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [26724.395100] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [26724.710459] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [26724.967945] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26725.226049] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26725.501646] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26729.407716] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26731.777293] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26734.158615] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26734.983324] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26735.240957] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26735.496672] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26735.770167] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26744.314211] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26746.695225] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26747.569748] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26748.501790] Lustre: DEBUG MARKER: rc=0; [26748.501790] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26748.501790] if [[ $? -eq 0 && $val -ne 0 ]]; then [26748.501790] echo $(hostname -s): $val; [26748.501790] rc=$val; [26748.501790] fi; [26748.501790] exit $rc [26749.114363] Lustre: DEBUG MARKER: dmesg [26749.744580] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 23: test mapped ACLs ============================================================== 15:04:34 \(1491577474\) [26749.906790] Lustre: DEBUG MARKER: == sanity-sec test 23: test mapped ACLs ============================================================== 15:04:34 (1491577474) [26750.053693] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26750.053693] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26750.053693] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26750.334650] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [26750.481087] Lustre: 24572:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [26750.484666] Lustre: 24572:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [26750.627489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [26750.885392] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [26751.140190] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [26751.398427] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [26751.654569] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [26751.909311] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [26752.165624] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26752.689916] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [26752.943610] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [26753.199685] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [26753.453753] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [26753.708465] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [26753.963918] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [26754.218196] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26754.473996] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [26754.727963] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26754.984257] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26755.255844] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26759.136338] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26759.955489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [26760.213593] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [26760.468673] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26760.727826] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26761.056112] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26766.484453] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26768.897430] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26769.756649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26770.017546] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26770.272536] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 0 [26770.527649] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 0 [26770.829768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26771.087612] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26771.441971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [26772.280543] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26773.126930] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26773.385541] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26773.639121] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26773.893390] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26774.150455] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26774.446907] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26774.848197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26777.197919] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26778.085775] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26778.344751] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26778.618989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26779.423107] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26780.266726] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26780.523896] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26780.778141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26781.061452] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26781.337738] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26782.156238] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26788.656343] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26788.910140] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26789.167284] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26789.422116] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26789.675942] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26789.934214] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26790.204192] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26791.024873] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26791.831831] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26792.092856] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26792.379357] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26793.188978] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26794.054782] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26794.312635] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26794.566793] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26794.820910] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26795.090080] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26795.891712] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26802.224255] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26802.478504] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26802.734625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26802.993194] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26803.248424] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26803.503729] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26803.772934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26804.580652] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26805.399253] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26805.655971] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26805.927557] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26806.739768] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26807.646094] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26807.903559] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26808.159282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26808.415614] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26808.688354] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26809.507426] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26815.896719] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26816.160927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26816.414168] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26816.669027] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26816.925380] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26817.182584] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26817.457083] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26818.273853] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26819.093666] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26819.351278] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26819.621197] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26820.427397] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26821.334971] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26821.608827] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26821.880293] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26822.149611] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26822.423516] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26823.234311] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26829.578326] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26829.833517] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26830.091915] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26830.350712] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26830.624958] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26831.438107] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26833.806643] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26836.188399] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26838.547539] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26839.360808] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26839.616641] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26839.871150] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26840.127598] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26840.383098] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26840.638607] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26840.909190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26846.390986] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26847.231029] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26847.488313] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26847.763623] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26848.571369] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26849.441730] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26849.699932] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26849.955975] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26850.214072] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26850.487526] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26857.468646] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26859.847276] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26866.206468] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [26866.462676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [26866.715854] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26866.971525] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26867.228081] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26867.484729] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26867.755640] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [26876.278877] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26877.096863] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26877.354224] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26877.624683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26878.432397] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26879.311923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [26879.566373] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [26879.825539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26880.081447] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26880.351368] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [26888.914946] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26895.296183] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [26895.570607] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [26895.837474] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [26896.093270] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [26896.347574] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26896.605190] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26896.876888] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26905.400643] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26906.239938] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26906.497914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26906.753940] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26907.023976] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26915.569776] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26916.518158] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26917.429375] Lustre: DEBUG MARKER: rc=0; [26917.429375] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26917.429375] if [[ $? -eq 0 && $val -ne 0 ]]; then [26917.429375] echo $(hostname -s): $val; [26917.429375] rc=$val; [26917.429375] fi; [26917.429375] exit $rc [26918.037886] Lustre: DEBUG MARKER: dmesg [26918.661829] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 24: check nodemap proc files for LBUGs and Oopses ================================= 15:07:23 \(1491577643\) [26918.829647] Lustre: DEBUG MARKER: == sanity-sec test 24: check nodemap proc files for LBUGs and Oopses ================================= 15:07:23 (1491577643) [26918.970835] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [26919.115168] Lustre: 4783:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [26919.118640] Lustre: 4783:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 1 previous similar message [26919.259773] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [26919.517706] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [26919.773103] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [26920.030356] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [26920.285109] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [26920.539005] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [26920.793663] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26921.312780] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [26921.567171] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [26921.820794] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [26922.077348] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [26922.334674] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [26922.587644] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [26922.843609] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26923.098632] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [26923.353348] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26923.610553] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26923.884249] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26927.787200] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26928.606071] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [26928.862460] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [26929.116597] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26929.374989] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26929.659508] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26938.197390] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26939.049722] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -R nodemap [26939.342316] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [26939.614498] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [26939.888741] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [26940.157885] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [26940.412613] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26940.669759] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26940.939929] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26946.389624] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26947.197866] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [26947.454366] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26947.711256] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26947.984927] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26958.082987] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26958.955034] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [26959.866077] Lustre: DEBUG MARKER: rc=0; [26959.866077] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [26959.866077] if [[ $? -eq 0 && $val -ne 0 ]]; then [26959.866077] echo $(hostname -s): $val; [26959.866077] rc=$val; [26959.866077] fi; [26959.866077] exit $rc [26960.458348] Lustre: DEBUG MARKER: dmesg [26961.024197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 25: test save and reload nodemap config =========================================== 15:08:05 \(1491577685\) [26961.195164] Lustre: DEBUG MARKER: == sanity-sec test 25: test save and reload nodemap config =========================================== 15:08:05 (1491577685) [26961.353747] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [26961.353747] /usr/sbin/lctl lustre_build_version 2>/dev/null || [26961.353747] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [26962.033506] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [26962.326560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [26962.592726] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [26962.855136] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [26963.113044] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [26963.369743] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [26963.625089] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [26963.881883] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26964.410162] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [26964.665975] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [26964.923208] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [26965.178732] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [26965.432269] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [26965.689691] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [26965.944419] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26966.200289] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [26966.457417] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26966.715132] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26966.988676] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [26972.456816] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26973.277410] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [26973.535557] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [26973.789315] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26974.046650] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26974.318241] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [26982.863917] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26983.674992] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [26983.934560] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [26984.189663] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property admin --value 1 [26984.444221] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c1 --property trusted --value 1 [26984.698530] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26984.955609] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26985.226973] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c1.trusted_nodemap [26993.752381] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26994.576843] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add test26 [26994.832143] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.test26.fileset=/c0dir [26994.963417] Lustre: Setting parameter general.nodemap.test26.fileset in log params [26994.965210] Lustre: Skipped 3 previous similar messages [26995.095683] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [26995.351925] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [26995.622459] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test26.id [27002.649508] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27003.459015] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [27003.713405] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [27006.348163] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [27006.348163] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [27006.348163] if [ $running -ne $mpts ]; then [27006.348163] echo $(hostname) env are INSANE!; [27006.348163] exit 1; [27006.348163] fi [27006.913287] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [27006.913287] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [27006.913287] if [ $running -ne $mpts ]; then [27006.913287] echo $(hostname) env are INSANE!; [27006.913287] exit 1; [27006.913287] fi [27007.497692] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [27007.497692] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [27007.497692] if [ $running -ne $mpts ]; then [27007.497692] echo $(hostname) env are INSANE!; [27007.497692] exit 1; [27007.497692] fi [27008.427262] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [27008.705721] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [27008.963918] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [27009.221581] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [27009.484377] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27009.741104] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*0 [27009.996887] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0000.identity_upcall [27010.252930] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0000.identity_upcall NONE [27010.506782] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0000/identity_flush=-1 [27013.118628] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [27013.383747] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [27013.641518] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [27013.896577] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [27014.155641] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27014.413077] Lustre: DEBUG MARKER: lctl get_param -N mdt.*MDT*2 [27014.669969] Lustre: DEBUG MARKER: lctl get_param -n mdt.lustre-MDT0002.identity_upcall [27014.927352] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre-MDT0002.identity_upcall NONE [27015.181748] Lustre: DEBUG MARKER: lctl set_param -n mdt/lustre-MDT0002/identity_flush=-1 [27026.223914] Lustre: MGS: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [27026.225857] Lustre: Skipped 145 previous similar messages [27027.540201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [27027.692558] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [27029.117347] Lustre: DEBUG MARKER: lctl get_param -n timeout [27029.443719] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [27029.601901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [27029.741580] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [27035.169922] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [27035.461179] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27035.979301] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27036.888742] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [27037.149232] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_info [27037.405512] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del test26 [27037.665087] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27037.920156] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27038.175706] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27038.432026] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27038.686040] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27038.942179] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27039.216813] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27044.666743] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27045.477980] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27045.735190] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27045.992178] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27046.265115] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27054.825572] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27056.799953] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [27056.960024] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [27057.163627] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27058.077378] Lustre: DEBUG MARKER: rc=0; [27058.077378] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27058.077378] if [[ $? -eq 0 && $val -ne 0 ]]; then [27058.077378] echo $(hostname -s): $val; [27058.077378] rc=$val; [27058.077378] fi; [27058.077378] exit $rc [27058.691093] Lustre: DEBUG MARKER: dmesg [27059.244641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanity-sec test_26 skipping SLOW test 26 [27059.407494] Lustre: DEBUG MARKER: SKIP: sanity-sec test_26 skipping SLOW test 26 [27059.615658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test 27: test fileset in nodemap ======================================================= 15:09:44 \(1491577784\) [27059.785708] Lustre: DEBUG MARKER: == sanity-sec test 27: test fileset in nodemap ======================================================= 15:09:44 (1491577784) [27059.933533] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE [27060.080529] Lustre: 14997:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" [27060.084507] Lustre: 14997:0:(mdt_lproc.c:318:mdt_identity_upcall_seq_write()) Skipped 5 previous similar messages [27060.228211] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0 [27060.484258] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.9.5.237@tcp [27060.740840] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000 [27060.998458] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000 [27061.255722] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002 [27061.512625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002 [27061.769989] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27062.299146] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1 [27062.557135] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.9.5.238@tcp [27062.812625] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001 [27063.074848] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001 [27063.335677] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002 [27063.604301] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002 [27063.876865] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27064.141844] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 [27064.405565] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27064.664752] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27064.937914] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27067.284927] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27068.114786] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 [27068.370148] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 [27068.624489] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27068.882466] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27069.156470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27076.200132] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27077.027076] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27077.284412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27077.539209] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27077.796923] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27078.052601] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27078.308833] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27078.582402] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27085.574467] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27086.385335] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27086.644325] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27086.914801] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27087.732162] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27088.589050] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27088.844865] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27089.108668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27089.364565] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27089.634890] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27098.166164] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27099.004818] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27099.261302] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27099.537806] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27100.348475] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27101.180696] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.c0.fileset=/c0dir [27101.311096] Lustre: Setting parameter general.nodemap.c0.fileset in log params [27101.442273] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27101.697025] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27108.695385] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27108.981370] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [27109.803394] Lustre: MGS: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [27109.805239] Lustre: Skipped 11 previous similar messages [27110.349076] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name c0 --fileset '' [27110.604237] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [27110.857696] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.c0.fileset='' [27110.987610] Lustre: Disabling parameter general.nodemap.c0.fileset in log params [27111.119500] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27111.376171] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27111.647144] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset [27117.090343] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27119.454855] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27120.509231] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap [27120.765151] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap [27121.023636] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1 [27121.281179] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 1 [27121.602896] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27121.869398] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27122.143980] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27122.954933] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27125.320226] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27126.149971] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27126.407563] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27126.677372] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27127.480230] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27128.326715] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 0 [27128.583643] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0 [27128.846986] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27129.104224] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27129.376788] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.admin_nodemap [27131.731134] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27134.102408] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27134.923235] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27135.182434] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27135.456740] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap [27136.258602] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27137.082609] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c0 [27137.340463] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del c1 [27137.594540] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0 [27137.852117] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0 [27138.109234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27138.366545] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27138.640470] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap [27145.654057] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27146.477489] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27146.751497] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active [27147.025464] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27147.314044] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active [27151.292647] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27153.672932] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep tcp | cut -f 1 -d '@' [27154.568656] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [27155.463641] Lustre: DEBUG MARKER: rc=0; [27155.463641] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [27155.463641] if [[ $? -eq 0 && $val -ne 0 ]]; then [27155.463641] echo $(hostname -s): $val; [27155.463641] rc=$val; [27155.463641] fi; [27155.463641] exit $rc [27156.059035] Lustre: DEBUG MARKER: dmesg [27156.633285] Lustre: DEBUG MARKER: /usr/sbin/lctl mark cleanup: ====================================================== [27156.789773] Lustre: DEBUG MARKER: cleanup: ====================================================== [27156.922397] Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0 [27167.866106] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [27168.125067] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [27168.924769] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [27168.926537] Lustre: Skipped 23 previous similar messages [27169.071246] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079ded800 x1564004882432096/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [27169.075244] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [27169.279227] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [27169.279309] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [27169.279311] Lustre: Skipped 7 previous similar messages [27169.286422] LustreError: Skipped 7 previous similar messages [27169.615097] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800600ecc00 x1564004882432288/t0(0) o13->lustre-OST0002-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [27169.619447] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [27172.479062] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079def300 x1564004882432384/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [27172.839201] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [27172.841396] Lustre: Skipped 11 previous similar messages [27174.359786] Lustre: server umount lustre-MDT0000 complete [27174.361657] Lustre: Skipped 3 previous similar messages [27174.492566] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [27178.922280] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [27178.926660] LustreError: Skipped 97 previous similar messages [27181.666052] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [27181.919926] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [27182.495088] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880079dee400 x1564004882432960/t0(0) o13->lustre-OST0005-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [27182.829920] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [27182.832394] Lustre: Skipped 10 previous similar messages [27185.639049] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491577904/real 1491577904] req@ffff880079deea00 x1564004882432640/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491577911 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [27185.646017] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 34 previous similar messages [27185.648508] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [27185.652922] LustreError: Skipped 5 previous similar messages [27188.287584] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [27264.647752] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-sec test complete, duration 3470 sec ======================================================= 15:13:08 \(1491577988\) [27264.798966] Lustre: DEBUG MARKER: == sanity-sec test complete, duration 3470 sec ======================================================= 15:13:08 (1491577988) [27265.011783] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: large-scale ============----- Fri Apr 7 15:13:09 UTC 2017 [27265.164143] Lustre: DEBUG MARKER: -----============= acceptance-small: large-scale ============----- Fri Apr 7 15:13:09 UTC 2017 [27265.416146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test complete, duration -o sec ======================================================== 15:13:10 \(1491577990\) [27265.567895] Lustre: DEBUG MARKER: == large-scale test complete, duration -o sec ======================================================== 15:13:10 (1491577990) [27265.873869] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [27266.027098] Lustre: DEBUG MARKER: excepting tests: [27268.417412] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [27268.417412] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [27268.417412] if [ $running -ne $mpts ]; then [27268.417412] echo $(hostname) env are INSANE!; [27268.417412] exit 1; [27268.417412] fi [27268.988054] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [27268.988054] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [27268.988054] if [ $running -ne $mpts ]; then [27268.988054] echo $(hostname) env are INSANE!; [27268.988054] exit 1; [27268.988054] fi [27269.578510] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [27269.578510] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [27269.578510] if [ $running -ne $mpts ]; then [27269.578510] echo $(hostname) env are INSANE!; [27269.578510] exit 1; [27269.578510] fi [27272.096787] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [27272.350689] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [27272.595310] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [27272.840516] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [27272.984724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27273.002509] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [27273.004831] Lustre: Skipped 11 previous similar messages [27273.040506] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [27273.045253] LustreError: Skipped 12 previous similar messages [27273.066119] LustreError: 23976:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [27273.070440] LustreError: 23976:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 2 previous similar messages [27273.085685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [27273.088055] Lustre: Skipped 1 previous similar message [27273.242934] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [27273.514265] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27274.080649] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27274.086989] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27274.335463] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27274.335464] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27274.489662] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27274.757548] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27275.028049] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27275.300246] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [27275.569501] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27277.929965] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27278.078247] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491577998/real 1491577998] req@ffff88005fa26100 x1564004882433456/t0(0) o8->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:28/4 lens 520/544 e 0 to 1 dl 1491578003 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [27278.094305] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27279.507511] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [27279.771582] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [27280.027087] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [27280.279565] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [27280.438039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27280.651638] LustreError: 25274:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [27280.654485] LustreError: 25274:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [27281.044944] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [27281.304212] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27281.815573] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27281.823229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27281.981986] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27281.993309] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27282.133273] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27282.389555] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27282.644772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27282.906182] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [27283.164043] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27285.727900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27285.904612] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27289.286961] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27289.448846] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27292.400667] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27292.561834] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27295.581227] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27295.745205] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27298.756845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27298.920096] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27301.929623] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27302.089129] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27305.095294] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27305.257653] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27308.369718] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27308.533312] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27311.566311] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27311.725201] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27313.951305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27314.118219] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27315.550497] Lustre: DEBUG MARKER: lctl get_param -n timeout [27315.876566] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [27316.038541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [27316.173501] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [27321.578101] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [27321.860254] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27322.372673] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27323.169431] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test 3a: recovery time, 2 clients ===================================================== 15:14:08 \(1491578048\) [27323.328705] Lustre: DEBUG MARKER: == large-scale test 3a: recovery time, 2 clients ===================================================== 15:14:08 (1491578048) [27388.797992] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 : Starting failover on mds1 [27388.971565] Lustre: DEBUG MARKER: 1 : Starting failover on mds1 [27389.139114] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [27389.434613] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [27389.583454] Lustre: Failing over lustre-MDT0000 [27389.585958] Lustre: Skipped 4 previous similar messages [27390.077508] Lustre: server umount lustre-MDT0000 complete [27390.080024] Lustre: Skipped 1 previous similar message [27390.209672] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [27390.776552] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [27390.779503] LustreError: Skipped 1 previous similar message [27390.781850] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [27390.786819] Lustre: Skipped 2 previous similar messages [27397.967059] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [27400.518378] Lustre: DEBUG MARKER: hostname [27400.828675] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [27401.082970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [27401.333467] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [27401.491887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27403.971587] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176cba678 to 0x52ac6ca176e26f54 [27404.062923] LustreError: 28067:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [27404.067826] LustreError: 28067:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [27404.072926] LustreError: 28072:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [27404.117229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [27404.119908] Lustre: Skipped 1 previous similar message [27404.313932] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [27404.574741] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27404.668168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [27404.670758] Lustre: Skipped 5 previous similar messages [27405.080199] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27405.082895] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27405.245672] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27405.249522] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27405.390204] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27405.648699] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27405.904631] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27406.167154] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [27406.426032] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27406.927873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27406.937746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27407.097099] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27407.097250] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27409.083652] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [27409.112794] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [27409.116026] Lustre: Skipped 5 previous similar messages [27412.273201] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [27615.887868] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 : Starting failover on mds1 [27616.060624] Lustre: DEBUG MARKER: 2 : Starting failover on mds1 [27616.225023] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [27616.524276] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [27616.677409] Lustre: Failing over lustre-MDT0000 [27617.207679] Lustre: server umount lustre-MDT0000 complete [27617.336566] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [27617.672793] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [27617.678342] LustreError: Skipped 50 previous similar messages [27619.082358] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [27619.085259] LustreError: Skipped 1 previous similar message [27619.087650] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [27619.092734] Lustre: Skipped 1 previous similar message [27627.335048] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491578344/real 1491578344] req@ffff880068f2db00 x1564004883300240/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491578351 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [27627.343129] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [27627.345932] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [27627.649993] Lustre: DEBUG MARKER: hostname [27627.953990] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [27628.207989] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [27628.461157] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [27628.619571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27629.105908] Lustre: MGS: Connection restored to 10.9.5.239@tcp (at 10.9.5.239@tcp) [27629.108528] Lustre: Skipped 60 previous similar messages [27633.350496] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca176e26f54 to 0x52ac6ca17722c0ea [27633.447913] LustreError: 30051:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [27633.455464] LustreError: 30056:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [27633.489093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [27633.658789] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [27633.918151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27634.106168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [27634.423407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27634.425593] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27634.586173] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27634.586435] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27634.731557] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27634.989091] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27635.244990] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27635.504227] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [27635.763586] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27636.265706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27636.271196] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27636.433749] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27636.436024] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27638.464464] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [27638.482324] Lustre: lustre-MDT0000: Recovery over after 0:05, of 5 clients 5 recovered and 0 were evicted. [27641.603219] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [27839.708768] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 : Starting failover on mds1 [27839.882840] Lustre: DEBUG MARKER: 3 : Starting failover on mds1 [27840.034419] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [27840.316346] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [27840.466770] Lustre: Failing over lustre-MDT0000 [27841.104817] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [27851.419593] Lustre: DEBUG MARKER: hostname [27851.731710] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [27851.987790] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [27852.240831] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [27852.429977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [27857.263598] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca17722c0ea to 0x52ac6ca17762dc01 [27857.556483] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [27857.817256] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27858.316229] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27858.320734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27858.478328] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27858.481927] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [27858.622949] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [27858.697360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [27858.891751] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27859.157219] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [27859.422141] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [27859.683975] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [27860.188569] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27860.191650] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27860.351760] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27860.354628] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [27862.365229] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [27862.394157] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [27865.535912] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [28037.816910] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 1 : Starting failover on mds1 [28038.085725] Lustre: DEBUG MARKER: 1 : Starting failover on mds1 [28038.254942] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [28038.606896] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [28039.608285] Lustre: server umount lustre-MDT0000 complete [28039.610741] Lustre: Skipped 1 previous similar message [28039.740212] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [28041.419534] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [28041.425172] LustreError: Skipped 82 previous similar messages [28042.365473] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [28042.365609] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [28042.365611] Lustre: Skipped 3 previous similar messages [28042.376422] LustreError: Skipped 4 previous similar messages [28049.365038] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491578768/real 1491578768] req@ffff8800600ecc00 x1564004885285904/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491578775 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [28049.373148] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [28049.375971] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [28049.380912] LustreError: Skipped 1 previous similar message [28050.050666] Lustre: DEBUG MARKER: hostname [28050.354180] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [28050.607898] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [28050.862783] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [28051.022259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [28055.383425] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca17762dc01 to 0x52ac6ca177a6e92c [28055.492297] LustreError: 1627:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [28055.497596] LustreError: 1627:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 1 previous similar message [28055.503223] LustreError: 1632:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [28055.506148] LustreError: 1632:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 1 previous similar message [28055.538494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [28055.541238] Lustre: Skipped 1 previous similar message [28055.711703] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [28055.975383] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28056.484512] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28056.485536] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28056.649434] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28056.652813] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28056.796672] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28057.053851] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28057.312180] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28057.572062] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [28057.830418] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28058.339773] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28058.344178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28058.512323] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28058.512354] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28060.516481] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [28063.688888] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [28189.015338] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 2 : Starting failover on mds1 [28189.228815] Lustre: DEBUG MARKER: 2 : Starting failover on mds1 [28189.441136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [28189.752479] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [28189.933721] Lustre: Failing over lustre-MDT0000 [28189.936173] Lustre: Skipped 1 previous similar message [28190.515915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [28190.518787] Lustre: Skipped 1 previous similar message [28190.858551] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [28201.174492] Lustre: DEBUG MARKER: hostname [28201.482362] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [28201.743725] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [28202.004918] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [28202.162928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [28202.186502] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca177a6e92c to 0x52ac6ca177e71473 [28202.481282] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [28202.741364] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28202.845549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect [28202.848297] Lustre: Skipped 1 previous similar message [28203.248772] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28203.322796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28203.494831] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28203.521540] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28203.667478] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28203.928436] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28204.185381] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28204.446525] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [28204.707116] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28205.206565] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28205.207058] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28205.371888] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28205.373991] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28207.291593] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [28207.320658] Lustre: lustre-MDT0000: Recovery over after 0:04, of 5 clients 5 recovered and 0 were evicted. [28207.324147] Lustre: Skipped 1 previous similar message [28210.603760] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [28335.850570] Lustre: DEBUG MARKER: /usr/sbin/lctl mark 3 : Starting failover on mds1 [28336.099620] Lustre: DEBUG MARKER: 3 : Starting failover on mds1 [28336.271912] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [28336.620073] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [28337.293114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [28337.295789] Lustre: Skipped 1 previous similar message [28337.787706] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [28348.112090] Lustre: DEBUG MARKER: hostname [28348.416697] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [28348.670732] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [28348.924252] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [28349.085826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [28354.456356] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [28354.458916] Lustre: Skipped 103 previous similar messages [28354.461536] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca177e71473 to 0x52ac6ca17826f31c [28354.749539] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [28355.058273] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28355.574410] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28355.581268] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28355.746508] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28355.746508] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [28355.892286] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28356.148350] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28356.404454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28356.665755] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [28356.926250] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28357.426879] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28357.427170] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28357.594248] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28357.594973] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 200 [28359.566240] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [28362.805350] Lustre: DEBUG MARKER: lctl get_param -n *.lustre-MDT0000.recovery_status [28493.366247] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [28494.294742] Lustre: DEBUG MARKER: rc=0; [28494.294742] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [28494.294742] if [[ $? -eq 0 && $val -ne 0 ]]; then [28494.294742] echo $(hostname -s): $val; [28494.294742] rc=$val; [28494.294742] fi; [28494.294742] exit $rc [28494.913533] Lustre: DEBUG MARKER: dmesg [28495.551217] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == large-scale test complete, duration 1230 sec ====================================================== 15:33:40 \(1491579220\) [28495.712045] Lustre: DEBUG MARKER: == large-scale test complete, duration 1230 sec ====================================================== 15:33:40 (1491579220) [28526.879178] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [28527.135607] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [28527.710053] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [28527.713400] Lustre: Skipped 2 previous similar messages [28530.031117] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880047d7cf00 x1564004888039216/t0(0) o13->lustre-OST0007-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [28530.036732] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [28532.706060] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.244@tcp (stopping) [28532.708983] Lustre: Skipped 6 previous similar messages [28533.513275] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [28540.677918] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [28540.931510] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [28542.143131] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880045e21200 x1564004888039936/t0(0) o13->lustre-OST0006-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [28542.148506] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [28542.710208] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [28542.713444] Lustre: Skipped 12 previous similar messages [28545.567086] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880045e44300 x1564004888040048/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [28545.572657] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [28547.319131] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [28567.938661] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: mds-survey ============----- Fri Apr 7 15:34:52 UTC 2017 [28568.089122] Lustre: DEBUG MARKER: -----============= acceptance-small: mds-survey ============----- Fri Apr 7 15:34:52 UTC 2017 [28568.343063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test complete, duration -o sec ========================================================= 15:34:53 \(1491579293\) [28568.496346] Lustre: DEBUG MARKER: == mds-survey test complete, duration -o sec ========================================================= 15:34:53 (1491579293) [28568.720755] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [28568.720755] /usr/sbin/lctl lustre_build_version 2>/dev/null || [28568.720755] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [28569.036522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: [28569.190811] Lustre: DEBUG MARKER: excepting tests: [28571.571156] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [28571.571156] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [28571.571156] if [ $running -ne $mpts ]; then [28571.571156] echo $(hostname) env are INSANE!; [28571.571156] exit 1; [28571.571156] fi [28572.143332] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [28572.143332] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [28572.143332] if [ $running -ne $mpts ]; then [28572.143332] echo $(hostname) env are INSANE!; [28572.143332] exit 1; [28572.143332] fi [28572.714407] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [28572.714407] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [28572.714407] if [ $running -ne $mpts ]; then [28572.714407] echo $(hostname) env are INSANE!; [28572.714407] exit 1; [28572.714407] fi [28575.225343] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [28575.481944] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [28575.729392] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [28575.975089] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [28576.142138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [28576.453412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [28576.711763] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28577.228400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28577.230421] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28577.386542] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28577.391256] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28577.532717] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28577.784679] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28578.038293] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28578.294528] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [28578.548282] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28581.120848] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28581.294329] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28582.722282] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [28583.001549] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [28583.256721] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [28583.510496] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [28583.669220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [28583.996572] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [28584.255585] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [28584.856193] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28584.863601] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28585.054400] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28585.056985] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28585.201067] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28585.460685] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28585.715609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [28585.976423] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [28586.236200] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28588.561658] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28588.731257] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28592.224920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28592.383346] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28595.225702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28595.386544] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28598.345202] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28598.499353] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28601.389304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28601.548411] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28604.444457] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28604.607462] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28607.502284] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28607.671137] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28610.576136] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28610.745986] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28613.674562] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28613.850908] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28616.056022] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28616.206487] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28617.654747] Lustre: DEBUG MARKER: lctl get_param -n timeout [28617.971233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [28618.133551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [28618.266857] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [28623.671020] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [28623.975591] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28624.490781] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [28625.773819] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 1: Metadata survey with zero-stripe =============================================== 15:35:50 \(1491579350\) [28625.932208] Lustre: DEBUG MARKER: == mds-survey test 1: Metadata survey with zero-stripe =============================================== 15:35:50 (1491579350) [28626.352217] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [28627.434217] Lustre: Echo OBD driver; http://www.lustre.org/ [28631.745936] LustreError: 12556:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [28631.749052] LustreError: 12556:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [28632.060296] LustreError: 12622:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [28632.377136] LustreError: 12688:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0002-tests: rc = -2 [28632.380224] LustreError: 12688:0:(echo_client.c:1662:echo_md_lookup()) Skipped 1 previous similar message [28632.383256] LustreError: 12688:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0002-tests: rc = -2 [28632.386461] LustreError: 12688:0:(echo_client.c:1861:echo_md_destroy_internal()) Skipped 1 previous similar message [28632.711181] LustreError: 12754:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0002-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [28632.717071] LustreError: 12754:0:(lod_object.c:1370:lod_verify_md_striping()) Skipped 1 previous similar message [28801.783181] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [28802.670916] Lustre: DEBUG MARKER: rc=0; [28802.670916] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [28802.670916] if [[ $? -eq 0 && $val -ne 0 ]]; then [28802.670916] echo $(hostname -s): $val; [28802.670916] rc=$val; [28802.670916] fi; [28802.670916] exit $rc [28803.281559] Lustre: DEBUG MARKER: dmesg [28804.061152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================== 15:38:48 \(1491579528\) [28804.231174] Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 ========================================== 15:38:48 (1491579528) [28804.645165] Lustre: DEBUG MARKER: /usr/sbin/lctl dl [28806.275503] Lustre: Echo OBD driver; http://www.lustre.org/ [28810.447469] LustreError: 15118:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0000-tests: rc = -2 [28810.450087] LustreError: 15118:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0000-tests: rc = -2 [28810.760293] LustreError: 15184:0:(lod_object.c:1370:lod_verify_md_striping()) lustre-MDT0000-mdtlov: invalid lmv_user_md: magic = bd30bd0, stripe_offset = 0, stripe_count = 0: rc = -22 [28810.765017] LustreError: 15184:0:(lod_object.c:1370:lod_verify_md_striping()) Skipped 1 previous similar message [28811.077707] LustreError: 15250:0:(echo_client.c:1662:echo_md_lookup()) lookup MDT0002-tests: rc = -2 [28811.080253] LustreError: 15250:0:(echo_client.c:1662:echo_md_lookup()) Skipped 1 previous similar message [28811.082693] LustreError: 15250:0:(echo_client.c:1861:echo_md_destroy_internal()) Can't find child MDT0002-tests: rc = -2 [28811.085176] LustreError: 15250:0:(echo_client.c:1861:echo_md_destroy_internal()) Skipped 1 previous similar message [28996.028084] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [28997.360226] Lustre: DEBUG MARKER: rc=0; [28997.360226] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [28997.360226] if [[ $? -eq 0 && $val -ne 0 ]]; then [28997.360226] echo $(hostname -s): $val; [28997.360226] rc=$val; [28997.360226] fi; [28997.360226] exit $rc [28997.971550] Lustre: DEBUG MARKER: dmesg [28998.705169] Lustre: MGS: Connection restored to 45163ea6-bba1-e809-9282-56182aedefc5 (at 10.9.5.237@tcp) [28998.707660] Lustre: Skipped 60 previous similar messages [28999.647197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [28999.806435] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [29000.025738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == mds-survey test complete, duration 431 sec ======================================================== 15:42:04 \(1491579724\) [29000.192116] Lustre: DEBUG MARKER: == mds-survey test complete, duration 431 sec ======================================================== 15:42:04 (1491579724) [29011.064329] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29011.370676] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [29012.652648] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [29012.655111] Lustre: Skipped 3 previous similar messages [29013.487063] LustreError: 5314:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004443c300 x1564004894509184/t0(0) o13->lustre-OST0005-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29013.772274] LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [29013.772310] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29013.772311] Lustre: Skipped 8 previous similar messages [29013.781506] LustreError: Skipped 9 previous similar messages [29018.097707] Lustre: server umount lustre-MDT0000 complete [29018.100023] Lustre: Skipped 4 previous similar messages [29018.229708] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29018.772204] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [29018.777175] LustreError: Skipped 143 previous similar messages [29025.400301] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [29025.650940] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [29026.071060] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491579744/real 1491579744] req@ffff880046b0ad00 x1564004894509600/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491579751 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29026.078439] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 30 previous similar messages [29026.081015] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [29026.085469] LustreError: Skipped 3 previous similar messages [29027.279094] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800444eed00 x1564004894510048/t0(0) o13->lustre-OST0003-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [29027.284504] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [29032.009509] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29100.812173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: replay-dual ============----- Fri Apr 7 15:43:44 UTC 2017 [29100.972593] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-dual ============----- Fri Apr 7 15:43:44 UTC 2017 [29101.221827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test complete, duration -o sec ======================================================== 15:43:46 \(1491579826\) [29101.380174] Lustre: DEBUG MARKER: == replay-dual test complete, duration -o sec ======================================================== 15:43:46 (1491579826) [29101.719267] Lustre: DEBUG MARKER: /usr/sbin/lctl mark excepting tests: 14b 15c 26 21b [29101.869020] Lustre: DEBUG MARKER: excepting tests: 14b 15c 26 21b [29102.082363] Lustre: DEBUG MARKER: /usr/sbin/lctl mark skipping tests SLOW=no: 21b [29102.232072] Lustre: DEBUG MARKER: skipping tests SLOW=no: 21b [29104.616230] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29104.616230] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29104.616230] if [ $running -ne $mpts ]; then [29104.616230] echo $(hostname) env are INSANE!; [29104.616230] exit 1; [29104.616230] fi [29105.174224] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [29105.174224] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [29105.174224] if [ $running -ne $mpts ]; then [29105.174224] echo $(hostname) env are INSANE!; [29105.174224] exit 1; [29105.174224] fi [29105.746159] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [29105.746159] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [29105.746159] if [ $running -ne $mpts ]; then [29105.746159] echo $(hostname) env are INSANE!; [29105.746159] exit 1; [29105.746159] fi [29108.258489] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [29108.514560] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29108.760817] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29109.005123] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29109.160290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29109.240084] LustreError: 18536:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [29109.244489] LustreError: 18536:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 4 previous similar messages [29109.259286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [29109.261641] Lustre: Skipped 4 previous similar messages [29109.408412] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29109.660364] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29110.156094] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29110.162511] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29110.321858] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29110.321859] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29110.462818] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29110.716601] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29110.970285] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29111.229396] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29111.482167] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29113.796184] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29113.957094] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29115.372518] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [29115.635447] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [29115.888635] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [29116.140996] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [29116.298749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29116.534867] LustreError: 19835:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0000: Coordinator already started [29116.537530] LustreError: 19835:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 3 previous similar messages [29116.911188] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29117.179393] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29117.671758] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29117.676015] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29117.837721] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29117.839667] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29117.981713] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29118.237215] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29118.491070] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29118.749125] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [29119.004260] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29121.573333] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29121.740216] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29125.151606] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29125.317515] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29128.961926] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29129.126525] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29133.655699] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29134.041575] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29137.087577] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29137.249398] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29140.865124] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29141.031775] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29145.091046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29145.253374] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29148.944777] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29149.178945] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29153.440046] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29153.601147] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29155.782594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29155.938892] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29158.613587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29158.775842] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29158.940353] Lustre: DEBUG MARKER: lctl get_param -n timeout [29159.258730] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [29159.407520] Lustre: DEBUG MARKER: Using TIMEOUT=20 [29159.555815] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [29164.965215] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true [29165.250176] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29165.765261] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29166.370808] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 0a: expired recovery with lost client ============================================ 15:44:51 \(1491579891\) [29166.529605] Lustre: DEBUG MARKER: == replay-dual test 0a: expired recovery with lost client ============================================ 15:44:51 (1491579891) [29166.675516] Lustre: DEBUG MARKER: sync; sync; sync [29167.576560] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29167.835525] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29167.967004] LustreError: 22281:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29167.969895] LustreError: 22281:0:(osd_handler.c:2171:osd_ro()) Skipped 1 previous similar message [29167.980487] Turning device dm-0 (0xfc00000) read-only [29168.111442] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29168.239749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29168.490879] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29168.747096] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29168.878086] Lustre: Failing over lustre-MDT0000 [29168.880378] Lustre: Skipped 1 previous similar message [29168.941799] Removing read-only on unknown block (0xfc00000) [29169.076744] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29179.385770] Lustre: DEBUG MARKER: hostname [29179.697233] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29179.958315] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29180.215129] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29180.375309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29186.392472] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785c6dfd to 0x52ac6ca1785c7fdf [29186.765133] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29187.022531] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29187.530335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29187.536615] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29187.695448] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29187.695843] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29187.839849] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29188.094943] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29188.352732] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29188.613453] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29188.674168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [29188.676834] Lustre: Skipped 1 previous similar message [29191.488961] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29254.879052] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [29254.881437] Lustre: lustre-MDT0000: disconnecting 1 stale clients [29254.901576] Lustre: 22931:0:(ldlm_lib.c:2550:target_recovery_thread()) too long recovery - read logs [29254.904356] Lustre: lustre-MDT0000: Recovery over after 1:06, of 7 clients 6 recovered and 1 was evicted. [29254.904362] LustreError: dumping log to /tmp/lustre-log.1491579980.22931 [29254.910691] Lustre: Skipped 1 previous similar message [29266.828535] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29267.727398] Lustre: DEBUG MARKER: rc=0; [29267.727398] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29267.727398] if [[ $? -eq 0 && $val -ne 0 ]]; then [29267.727398] echo $(hostname -s): $val; [29267.727398] rc=$val; [29267.727398] fi; [29267.727398] exit $rc [29268.343614] Lustre: DEBUG MARKER: dmesg [29268.972673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 0b: lost client during waiting for next transno ================================== 15:46:33 \(1491579993\) [29269.134984] Lustre: DEBUG MARKER: == replay-dual test 0b: lost client during waiting for next transno ================================== 15:46:33 (1491579993) [29269.268952] Lustre: DEBUG MARKER: sync; sync; sync [29270.167700] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29270.434944] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29270.610847] LustreError: 24268:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29270.620721] Turning device dm-0 (0xfc00000) read-only [29270.754472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29270.884032] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29271.079677] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29271.334838] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29271.488769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [29271.491123] Lustre: Skipped 20 previous similar messages [29271.521765] Removing read-only on unknown block (0xfc00000) [29271.657458] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29281.976694] Lustre: DEBUG MARKER: hostname [29282.278278] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29282.531354] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29282.784835] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29282.943853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29282.967833] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785c7fdf to 0x52ac6ca1785c970f [29283.281985] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29283.542476] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29284.044739] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29284.046900] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29284.204309] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29284.211021] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29284.352288] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29284.608189] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29284.866364] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29285.127005] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29288.078769] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29295.413553] Lustre: lustre-MDT0000: Denying connection for new client e7de780c-896d-3ab4-7f63-d1ece1811b84(at 10.9.5.237@tcp), waiting for 7 known clients (5 recovered, 1 in progress, and 0 evicted) to recover in 0:54 [29295.418716] Lustre: Skipped 1 previous similar message [29305.413363] Lustre: lustre-MDT0002: haven't heard from client 7ab27461-038a-ee78-9db7-71fb7ca462f1 (at 10.9.5.237@tcp) in 52 seconds. I think it's dead, and I am evicting it. exp ffff88004eb71000, cur 1491580031 expire 1491580001 last 1491579979 [29305.418840] Lustre: Skipped 1 previous similar message [29315.413189] Lustre: lustre-MDT0000: Denying connection for new client e7de780c-896d-3ab4-7f63-d1ece1811b84(at 10.9.5.237@tcp), waiting for 7 known clients (5 recovered, 1 in progress, and 0 evicted) to recover in 0:34 [29315.418495] Lustre: Skipped 3 previous similar messages Console [trevis-41vm7] log opened at 2017-04-07 15:47:49 UTC. Console [trevis-41vm7] connected to at 04-07 15:47. [29349.599058] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [29349.601549] Lustre: lustre-MDT0000: disconnecting 1 stale clients [29350.412969] Lustre: lustre-MDT0000: Denying connection for new client e7de780c-896d-3ab4-7f63-d1ece1811b84(at 10.9.5.237@tcp), waiting for 7 known clients (5 recovered, 1 in progress, and 1 evicted) to recover in 0:29 [29350.418366] Lustre: Skipped 6 previous similar messages [29379.679040] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [29379.681570] Lustre: lustre-MDT0000: disconnecting 1 stale clients [29380.751457] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29381.657382] Lustre: DEBUG MARKER: rc=0; [29381.657382] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29381.657382] if [[ $? -eq 0 && $val -ne 0 ]]; then [29381.657382] echo $(hostname -s): $val; [29381.657382] rc=$val; [29381.657382] fi; [29381.657382] exit $rc [29382.269591] Lustre: DEBUG MARKER: dmesg [29382.895602] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 1: \|X\| simple create ============================================================= 15:48:27 \(1491580107\) [29383.058172] Lustre: DEBUG MARKER: == replay-dual test 1: |X| simple create ============================================================= 15:48:27 (1491580107) [29383.216833] Lustre: DEBUG MARKER: sync; sync; sync [29384.061182] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29384.318747] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29384.459364] LustreError: 26252:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29384.499116] Turning device dm-0 (0xfc00000) read-only [29384.629760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29384.757944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29384.908980] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29385.165593] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29385.352751] Removing read-only on unknown block (0xfc00000) [29385.487250] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29395.808851] Lustre: DEBUG MARKER: hostname [29396.119026] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29396.380770] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29396.651256] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29396.810906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29401.103459] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785c970f to 0x52ac6ca1785c9f27 [29401.410570] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29401.668323] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29402.166818] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29402.166827] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29402.328761] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29402.331023] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29402.471550] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29402.730023] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29402.983544] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29403.243111] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29404.036132] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29404.050770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29404.194648] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29404.247618] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29406.217604] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29406.482942] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29406.515055] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29406.664980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29406.775505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29408.118371] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29409.006392] Lustre: DEBUG MARKER: rc=0; [29409.006392] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29409.006392] if [[ $? -eq 0 && $val -ne 0 ]]; then [29409.006392] echo $(hostname -s): $val; [29409.006392] rc=$val; [29409.006392] fi; [29409.006392] exit $rc [29409.656353] Lustre: DEBUG MARKER: dmesg [29410.337295] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 2: \|X\| mkdir adir ================================================================ 15:48:55 \(1491580135\) [29410.493885] Lustre: DEBUG MARKER: == replay-dual test 2: |X| mkdir adir ================================================================ 15:48:55 (1491580135) [29410.631630] Lustre: DEBUG MARKER: sync; sync; sync [29411.541734] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29411.800210] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29411.948768] Turning device dm-0 (0xfc00000) read-only [29412.078349] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29412.207109] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29412.352556] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29412.609235] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29412.798793] Removing read-only on unknown block (0xfc00000) [29412.933987] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29423.248169] Lustre: DEBUG MARKER: hostname [29423.559255] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29423.816812] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29424.071707] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29424.230657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29429.176681] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29429.436779] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29429.939634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29429.943262] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29430.105242] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29430.120871] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29430.257960] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29430.513563] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29430.777660] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29431.042739] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29431.819164] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29431.824484] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29432.013836] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29432.069157] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29434.252746] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29434.353280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29434.486852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29434.570429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29435.926731] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29436.844230] Lustre: DEBUG MARKER: rc=0; [29436.844230] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29436.844230] if [[ $? -eq 0 && $val -ne 0 ]]; then [29436.844230] echo $(hostname -s): $val; [29436.844230] rc=$val; [29436.844230] fi; [29436.844230] exit $rc [29437.462540] Lustre: DEBUG MARKER: dmesg [29438.105544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 3: \|X\| mkdir adir, mkdir adir\/bdir =============================================== 15:49:22 \(1491580162\) [29438.273058] Lustre: DEBUG MARKER: == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir =============================================== 15:49:22 (1491580162) [29438.408213] Lustre: DEBUG MARKER: sync; sync; sync [29439.282636] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29439.540848] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29439.684389] LustreError: 30781:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29439.686979] LustreError: 30781:0:(osd_handler.c:2171:osd_ro()) Skipped 1 previous similar message [29439.741147] Turning device dm-0 (0xfc00000) read-only [29439.869117] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29439.998753] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29440.145479] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29440.401178] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29440.592881] Removing read-only on unknown block (0xfc00000) [29440.727344] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29451.043509] Lustre: DEBUG MARKER: hostname [29451.348219] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29451.605271] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29451.858740] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29452.020230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29456.823406] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785ca6c8 to 0x52ac6ca1785cadeb [29456.827606] Lustre: Skipped 1 previous similar message [29457.119823] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29457.379367] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29457.875883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29457.879489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29458.040789] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29458.041627] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29458.182731] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29458.439263] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29458.695394] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29458.952031] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29459.732426] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29459.736394] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29459.962347] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29459.990348] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29461.922207] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29461.927317] LustreError: Skipped 1 previous similar message [29462.224574] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29462.267486] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29462.405287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29462.492559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29463.871465] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29464.781392] Lustre: DEBUG MARKER: rc=0; [29464.781392] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29464.781392] if [[ $? -eq 0 && $val -ne 0 ]]; then [29464.781392] echo $(hostname -s): $val; [29464.781392] rc=$val; [29464.781392] fi; [29464.781392] exit $rc Console [trevis-41vm7] log opened at 2017-04-07 15:49:52 UTC. Console [trevis-41vm7] connected to at 04-07 15:49. [29467.207772] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29467.465619] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29467.614027] Turning device dm-0 (0xfc00000) read-only [29467.743702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29467.873390] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29468.022922] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29468.279105] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29468.468061] Removing read-only on unknown block (0xfc00000) [29468.604870] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29478.923430] Lustre: DEBUG MARKER: hostname [29479.238198] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29479.496290] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29479.751373] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29479.910829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29485.542780] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29485.801175] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29486.303986] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29486.305764] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29486.474962] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29486.474973] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29486.618120] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29486.875583] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29487.139362] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29487.401170] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29488.158407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29488.158422] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29488.436839] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29488.558718] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29490.679554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29490.791088] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29490.833337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29491.017911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29492.363087] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29493.253613] Lustre: DEBUG MARKER: rc=0; [29493.253613] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29493.253613] if [[ $? -eq 0 && $val -ne 0 ]]; then [29493.253613] echo $(hostname -s): $val; [29493.253613] rc=$val; [29493.253613] fi; [29493.253613] exit $rc [29493.864100] Lustre: DEBUG MARKER: dmesg [29494.531149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 5: open, unlink \|X\| close ======================================================== 15:50:19 \(1491580219\) [29494.684804] Lustre: DEBUG MARKER: == replay-dual test 5: open, unlink |X| close ======================================================== 15:50:19 (1491580219) [29494.829079] Lustre: DEBUG MARKER: sync; sync; sync [29495.711347] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29495.967632] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29496.105589] Turning device dm-0 (0xfc00000) read-only [29496.236102] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29496.365627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29496.512486] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29496.767711] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29496.953775] Removing read-only on unknown block (0xfc00000) [29497.088933] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29507.407457] Lustre: DEBUG MARKER: hostname [29507.708821] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29507.962254] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29508.214130] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29508.376253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29508.681607] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29508.941256] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29509.438032] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29509.441449] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29509.603541] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29509.605136] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29509.746750] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29510.005440] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29510.262273] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29510.523471] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29511.363701] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29511.390856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29511.630798] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29511.632566] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29513.940377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29513.940391] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29514.143469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29514.144296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29515.539742] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29516.452491] Lustre: DEBUG MARKER: rc=0; [29516.452491] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29516.452491] if [[ $? -eq 0 && $val -ne 0 ]]; then [29516.452491] echo $(hostname -s): $val; [29516.452491] rc=$val; [29516.452491] fi; [29516.452491] exit $rc [29517.049379] Lustre: DEBUG MARKER: dmesg [29517.641920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 6: open1, open2, unlink \|X\| close1 [fail mds1] close2 ============================ 15:50:42 \(1491580242\) [29517.804896] Lustre: DEBUG MARKER: == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ============================ 15:50:42 (1491580242) [29517.955778] Lustre: DEBUG MARKER: sync; sync; sync [29518.831524] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29519.089481] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29519.235096] LustreError: 5205:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29519.237695] LustreError: 5205:0:(osd_handler.c:2171:osd_ro()) Skipped 2 previous similar messages [29519.277128] Turning device dm-0 (0xfc00000) read-only [29519.407377] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29519.537887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29519.681950] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29519.938206] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29520.160820] Removing read-only on unknown block (0xfc00000) [29520.298813] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29530.619428] Lustre: DEBUG MARKER: hostname [29530.934516] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29531.190649] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29531.442103] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29531.589682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29536.968443] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785cbe37 to 0x52ac6ca1785cc5d1 [29536.972591] Lustre: Skipped 2 previous similar messages [29537.330977] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29537.589984] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29538.094361] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29538.094842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29538.258521] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29538.270168] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29538.407972] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29538.676944] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29538.953522] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29539.224699] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29539.995996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29539.996003] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29540.199827] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29540.241357] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29542.057183] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29542.062493] LustreError: Skipped 2 previous similar messages [29542.422533] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29542.532974] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29542.573352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29542.742651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29544.065541] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29544.967154] Lustre: DEBUG MARKER: rc=0; [29544.967154] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29544.967154] if [[ $? -eq 0 && $val -ne 0 ]]; then [29544.967154] echo $(hostname -s): $val; [29544.967154] rc=$val; [29544.967154] fi; [29544.967154] exit $rc [29545.578058] Lustre: DEBUG MARKER: dmesg [29546.147213] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 8: replay of resent request ====================================================== 15:51:10 \(1491580270\) [29546.309017] Lustre: DEBUG MARKER: == replay-dual test 8: replay of resent request ====================================================== 15:51:10 (1491580270) [29546.452372] Lustre: DEBUG MARKER: sync; sync; sync [29547.360868] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29547.636160] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29547.794080] Turning device dm-0 (0xfc00000) read-only [29547.926689] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29548.055417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29548.188389] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x119 [29548.321698] Lustre: *** cfs_fail_loc=119, val=2147483648*** [29548.323886] LustreError: 20957:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880063fa2400 x1564034812909424/t180388626442(0) o36->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 624/448 e 0 to 0 dl 1491580279 ref 1 fl Interpret:/0/0 rc 0/0 [29556.329249] Lustre: lustre-MDT0000: Client e7de780c-896d-3ab4-7f63-d1ece1811b84 (at 10.9.5.237@tcp) reconnecting [29556.460146] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [29556.726248] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29556.986648] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29557.173003] Removing read-only on unknown block (0xfc00000) [29557.307489] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29567.618216] Lustre: DEBUG MARKER: hostname [29567.930374] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29568.183891] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29568.437426] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29568.587013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29574.751064] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29575.014252] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29575.514224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29575.518178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29575.679746] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29575.679795] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29575.823230] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29576.081150] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29576.338639] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29576.599941] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29577.363448] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29577.363886] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29577.539450] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29577.620969] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29579.855988] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29579.881128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29580.029140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29580.135510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [29581.478220] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29582.374323] Lustre: DEBUG MARKER: rc=0; [29582.374323] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29582.374323] if [[ $? -eq 0 && $val -ne 0 ]]; then [29582.374323] echo $(hostname -s): $val; [29582.374323] rc=$val; [29582.374323] fi; [29582.374323] exit $rc [29582.984421] Lustre: DEBUG MARKER: dmesg [29583.615398] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 9: resending a replayed create =================================================== 15:51:48 \(1491580308\) [29583.774822] Lustre: DEBUG MARKER: == replay-dual test 9: resending a replayed create =================================================== 15:51:48 (1491580308) [29583.910950] Lustre: DEBUG MARKER: sync; sync; sync [29584.839175] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29585.097697] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29585.246943] Turning device dm-0 (0xfc00000) read-only [29585.374542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29585.503437] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29585.639485] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [29585.901999] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29586.158586] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29586.345453] Removing read-only on unknown block (0xfc00000) [29586.478677] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29596.788953] Lustre: DEBUG MARKER: hostname [29597.102157] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29597.372405] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29597.642143] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29597.813890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29601.282127] Lustre: MGS: Connection restored to 10.9.5.244@tcp (at 10.9.5.244@tcp) [29601.284370] Lustre: Skipped 305 previous similar messages [29602.834594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29603.092635] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29603.586845] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29603.590153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29603.757206] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29603.759124] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29603.900374] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29604.161385] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29604.417037] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29604.677702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29605.454149] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29605.464634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29605.676950] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29605.696513] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29607.617154] Lustre: *** cfs_fail_loc=119, val=2147483648*** [29607.619502] LustreError: 10622:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff88005f975500 x1564034812912848/t184683593738(184683593738) o36->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 632/448 e 1 to 0 dl 1491580345 ref 1 fl Complete:/4/0 rc 0/0 [29621.637668] Lustre: lustre-MDT0000: Recovery already passed deadline 0:47. If you do not want to wait more, please abort the recovery by force. [29621.642366] Lustre: Skipped 3 previous similar messages [29622.014708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [29622.038269] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [29622.176319] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [29622.281339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [29623.571456] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [29623.890642] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29624.791311] Lustre: DEBUG MARKER: rc=0; [29624.791311] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29624.791311] if [[ $? -eq 0 && $val -ne 0 ]]; then [29624.791311] echo $(hostname -s): $val; [29624.791311] rc=$val; [29624.791311] fi; [29624.791311] exit $rc [29625.397237] Lustre: DEBUG MARKER: dmesg [29626.011685] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 10: resending a replayed unlink ================================================== 15:52:30 \(1491580350\) [29626.169692] Lustre: DEBUG MARKER: == replay-dual test 10: resending a replayed unlink ================================================== 15:52:30 (1491580350) [29626.312045] Lustre: DEBUG MARKER: sync; sync; sync [29627.182182] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29627.441494] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29627.589282] Turning device dm-0 (0xfc00000) read-only [29627.719996] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29627.850582] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29627.988457] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000119 [29628.254353] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29628.512774] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29628.699951] Removing read-only on unknown block (0xfc00000) [29628.707252] Lustre: server umount lustre-MDT0000 complete [29628.709306] Lustre: Skipped 11 previous similar messages [29628.838901] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29631.296903] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [29631.301787] LustreError: Skipped 462 previous similar messages [29632.601365] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [29632.601444] Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [29632.601445] Lustre: Skipped 22 previous similar messages [29632.610121] LustreError: Skipped 26 previous similar messages [29639.152144] Lustre: DEBUG MARKER: hostname [29639.453004] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29639.496052] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491580358/real 1491580358] req@ffff880064b38c00 x1564004894700848/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491580365 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [29639.502545] Lustre: 5315:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 42 previous similar messages [29639.504828] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [29639.508773] LustreError: Skipped 10 previous similar messages [29639.710379] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29639.965388] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29640.123659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29645.803668] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29646.062367] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29646.558194] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29646.568427] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29646.726813] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29646.729521] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29646.872090] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29647.129136] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29647.386454] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29647.646252] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29648.410342] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29648.410368] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29648.588646] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29648.650555] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29650.630461] Lustre: *** cfs_fail_loc=119, val=2147483648*** [29650.632957] LustreError: 13035:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880062734600 x1564034812916752/t188978561036(188978561036) o36->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 624/456 e 1 to 0 dl 1491580388 ref 1 fl Complete:/4/0 rc 0/0 [29663.651758] Lustre: lustre-MDT0000: Recovery already passed deadline 0:48. If you do not want to wait more, please abort the recovery by force. [29663.939956] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec [29663.955583] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec [29664.178152] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec [29664.184571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec [29665.502342] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [29665.831632] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29666.735777] Lustre: DEBUG MARKER: rc=0; [29666.735777] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29666.735777] if [[ $? -eq 0 && $val -ne 0 ]]; then [29666.735777] echo $(hostname -s): $val; [29666.735777] rc=$val; [29666.735777] fi; [29666.735777] exit $rc [29667.351220] Lustre: DEBUG MARKER: dmesg [29667.924552] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 11: both clients timeout during replay =========================================== 15:53:12 \(1491580392\) [29668.079716] Lustre: DEBUG MARKER: == replay-dual test 11: both clients timeout during replay =========================================== 15:53:12 (1491580392) [29668.215765] Lustre: DEBUG MARKER: sync; sync; sync [29669.112219] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29669.368205] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29669.510082] LustreError: 14723:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [29669.512655] LustreError: 14723:0:(osd_handler.c:2171:osd_ro()) Skipped 3 previous similar messages [29669.558842] Turning device dm-0 (0xfc00000) read-only [29669.687554] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29669.817755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29669.963169] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x0119 [29670.233556] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29670.491498] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29670.679611] Removing read-only on unknown block (0xfc00000) [29670.813707] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29681.128342] Lustre: DEBUG MARKER: hostname [29681.440225] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29681.692801] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29681.949150] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29682.097294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29682.118220] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785cd9e3 to 0x52ac6ca1785ce129 [29682.122228] Lustre: Skipped 3 previous similar messages [29682.418316] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29682.677405] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29683.171066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29683.172803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29683.330132] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29683.330254] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29683.473709] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29683.728995] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29683.983229] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29684.246118] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29685.013324] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29685.015850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29685.223170] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29685.239639] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29687.220620] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [29687.225632] LustreError: Skipped 3 previous similar messages [29687.368607] Lustre: *** cfs_fail_loc=119, val=2147483648*** [29687.370901] LustreError: 15446:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880066782400 x1564034812920672/t193273528330(193273528330) o36->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 632/448 e 0 to 0 dl 1491580419 ref 1 fl Complete:/4/0 rc 0/0 [29694.393591] Lustre: lustre-MDT0000: Recovery already passed deadline 0:54. If you do not want to wait more, please abort the recovery by force. [29695.496804] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [29695.600122] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [29695.653103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [29695.796745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [29695.938137] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 [29696.267609] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29697.182319] Lustre: DEBUG MARKER: rc=0; [29697.182319] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29697.182319] if [[ $? -eq 0 && $val -ne 0 ]]; then [29697.182319] echo $(hostname -s): $val; [29697.182319] rc=$val; [29697.182319] fi; [29697.182319] exit $rc [29697.791347] Lustre: DEBUG MARKER: dmesg [29698.399424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 12: open resend timeout ========================================================== 15:53:43 \(1491580423\) [29698.558408] Lustre: DEBUG MARKER: == replay-dual test 12: open resend timeout ========================================================== 15:53:43 (1491580423) [29698.699755] Lustre: DEBUG MARKER: sync; sync; sync [29699.610158] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29699.872775] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29700.011384] Turning device dm-0 (0xfc00000) read-only [29700.141237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29700.271365] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29700.412956] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000302 [29700.676135] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29700.932149] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29701.118755] Removing read-only on unknown block (0xfc00000) [29701.251557] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29711.561603] Lustre: DEBUG MARKER: hostname [29711.868153] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29712.137687] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29712.399967] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29712.557467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29714.732280] LustreError: 17840:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [29714.736608] LustreError: 17840:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 13 previous similar messages [29714.777804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [29714.780047] Lustre: Skipped 13 previous similar messages [29714.984141] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29715.240252] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29715.736247] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29715.752725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29715.910575] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29715.912154] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29716.056542] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29716.315349] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29716.572291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29716.834772] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29717.092784] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [29720.101961] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29721.002749] Lustre: DEBUG MARKER: rc=0; [29721.002749] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29721.002749] if [[ $? -eq 0 && $val -ne 0 ]]; then [29721.002749] echo $(hostname -s): $val; [29721.002749] rc=$val; [29721.002749] fi; [29721.002749] exit $rc [29721.615670] Lustre: DEBUG MARKER: dmesg [29722.264230] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 13: close resend timeout ========================================================= 15:54:07 \(1491580447\) [29722.429193] Lustre: DEBUG MARKER: == replay-dual test 13: close resend timeout ========================================================= 15:54:07 (1491580447) [29722.575689] Lustre: DEBUG MARKER: sync; sync; sync [29723.466109] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29723.723665] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29723.859551] Turning device dm-0 (0xfc00000) read-only [29723.987985] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29724.119762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29724.260253] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000115 [29724.541625] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29724.817121] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29725.026512] Removing read-only on unknown block (0xfc00000) [29725.160924] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29735.490278] Lustre: DEBUG MARKER: hostname [29735.798379] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29736.051505] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29736.303260] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29736.463414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29736.578596] LustreError: 20023:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [29736.581127] LustreError: 20023:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 13 previous similar messages [29736.777147] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29737.034838] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29737.531939] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29737.532846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29737.696042] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29737.696044] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29737.838906] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29738.095681] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29738.352514] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29738.616311] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29738.875536] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [29741.945850] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29742.841408] Lustre: DEBUG MARKER: rc=0; [29742.841408] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29742.841408] if [[ $? -eq 0 && $val -ne 0 ]]; then [29742.841408] echo $(hostname -s): $val; [29742.841408] rc=$val; [29742.841408] fi; [29742.841408] exit $rc [29743.458295] Lustre: DEBUG MARKER: dmesg [29744.043612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b [29744.195700] Lustre: DEBUG MARKER: SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b [29744.397856] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 15:54:29 \(1491580469\) [29744.561557] Lustre: DEBUG MARKER: == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes =========== 15:54:29 (1491580469) [29744.697133] Lustre: DEBUG MARKER: sync; sync; sync [29745.588778] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29745.844817] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29745.992311] Turning device dm-0 (0xfc00000) read-only [29746.123575] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29746.255698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29746.493564] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29746.750622] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29746.937052] Removing read-only on unknown block (0xfc00000) [29747.071551] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29757.387749] Lustre: DEBUG MARKER: hostname [29757.692807] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29757.946163] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29758.196900] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29758.357626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29758.663892] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29758.925154] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29759.432086] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29759.433647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29759.592739] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29759.595918] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29759.739860] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29759.995042] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29760.252501] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29760.515318] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29761.279998] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29761.296375] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29761.514552] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29761.514553] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29825.247035] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [29825.249542] Lustre: lustre-MDT0000: disconnecting 1 stale clients [29826.027070] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29826.027542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29826.193161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29826.287535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29827.806985] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29828.693856] Lustre: DEBUG MARKER: rc=0; [29828.693856] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29828.693856] if [[ $? -eq 0 && $val -ne 0 ]]; then [29828.693856] echo $(hostname -s): $val; [29828.693856] rc=$val; [29828.693856] fi; [29828.693856] exit $rc [29829.295573] Lustre: DEBUG MARKER: dmesg [29829.868436] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c [29830.026793] Lustre: DEBUG MARKER: SKIP: replay-dual test_15c skipping ALWAYS excluded test 15c [29830.228049] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 16: fail MDS during recovery \(3571\) ============================================== 15:55:55 \(1491580555\) [29830.385602] Lustre: DEBUG MARKER: == replay-dual test 16: fail MDS during recovery (3571) ============================================== 15:55:55 (1491580555) [29830.523264] Lustre: DEBUG MARKER: sync; sync; sync [29831.479042] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [29831.739287] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [29831.886527] Turning device dm-0 (0xfc00000) read-only [29832.016023] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [29832.145693] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [29832.373073] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29832.627256] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29832.756958] Lustre: Failing over lustre-MDT0000 [29832.759064] Lustre: Skipped 14 previous similar messages [29832.817067] Removing read-only on unknown block (0xfc00000) [29832.950927] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29843.328894] Lustre: DEBUG MARKER: hostname [29843.638419] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29843.894862] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29844.149341] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29844.309038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29847.789486] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29848.049151] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29848.081801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [29848.085637] Lustre: Skipped 14 previous similar messages [29848.548931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29848.551982] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29848.711349] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29848.712545] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29848.854456] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29849.111859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29849.366512] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29849.625928] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29869.901289] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [29870.155782] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [29870.298125] LustreError: 25468:0:(ldlm_lib.c:2601:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [29870.300687] Lustre: 24510:0:(ldlm_lib.c:2054:target_recovery_overseer()) recovery is aborted, evict exports in recovery [29870.303234] Lustre: 24510:0:(ldlm_lib.c:2054:target_recovery_overseer()) Skipped 2 previous similar messages [29870.305703] Lustre: 24510:0:(ldlm_lib.c:1625:abort_req_replay_queue()) @@@ aborted: req@ffff8800450c8600 x1564034812935472/t0(210453397538) o101->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 824/0 e 2 to 0 dl 1491580603 ref 1 fl Complete:/4/ffffffff rc 0/-1 [29870.312932] Lustre: lustre-MDT0000: Recovery over after 0:22, of 7 clients 0 recovered and 7 were evicted. [29870.315451] Lustre: Skipped 14 previous similar messages [29870.321213] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (stopping) [29870.323605] Lustre: Skipped 1 previous similar message [29870.326625] LustreError: 24476:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880066780300 x1564004894737360/t0(0) o5->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:28/4 lens 432/432 e 0 to 0 dl 0 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 [29870.326760] LustreError: 24474:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [29870.326762] LustreError: 24474:0:(osp_precreate.c:917:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [29870.339125] LustreError: 24476:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 14 previous similar messages [29871.548133] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [29881.864346] Lustre: DEBUG MARKER: hostname [29882.165021] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [29882.419740] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [29882.672681] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [29882.830438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [29886.182575] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [29886.442479] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [29886.941098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29886.945424] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29887.114423] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29887.116257] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29887.259580] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [29887.518589] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29887.778122] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [29888.039430] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [29888.787788] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29888.823943] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29888.993807] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29889.052885] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [29952.735086] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [29952.737890] Lustre: lustre-MDT0000: disconnecting 1 stale clients [29953.476906] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29953.572605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29953.630457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29953.769131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [29955.310682] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [29956.221495] Lustre: DEBUG MARKER: rc=0; [29956.221495] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [29956.221495] if [[ $? -eq 0 && $val -ne 0 ]]; then [29956.221495] echo $(hostname -s): $val; [29956.221495] rc=$val; [29956.221495] fi; [29956.221495] exit $rc [29956.825556] Lustre: DEBUG MARKER: dmesg [29957.434476] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 17: fail OST during recovery \(3571\) ============================================== 15:58:02 \(1491580682\) [29957.588460] Lustre: DEBUG MARKER: == replay-dual test 17: fail OST during recovery (3571) ============================================== 15:58:02 (1491580682) [29973.563529] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [29973.808954] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30008.256389] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30008.424879] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30009.876144] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30009.877765] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30010.041060] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30010.146521] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30074.996773] LustreError: 5313:0:(client.c:2981:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@ffff880045398300 x1564004894747104/t120259084302(120259084302) o6->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:28/4 lens 664/400 e 8 to 0 dl 1491580809 ref 2 fl Interpret:R/4/0 rc 0/0 [30075.530332] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 65 sec [30075.634828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 65 sec [30075.680518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 65 sec [30075.856340] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 65 sec [30077.374124] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30078.306735] Lustre: DEBUG MARKER: rc=0; [30078.306735] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30078.306735] if [[ $? -eq 0 && $val -ne 0 ]]; then [30078.306735] echo $(hostname -s): $val; [30078.306735] rc=$val; [30078.306735] fi; [30078.306735] exit $rc [30078.908120] Lustre: DEBUG MARKER: dmesg [30079.589919] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export \(3822\) ======================== 16:00:04 \(1491580804\) [30079.759963] Lustre: DEBUG MARKER: == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ======================== 16:00:04 (1491580804) [30079.899973] Lustre: DEBUG MARKER: lctl get_param debug [30080.158206] Lustre: DEBUG MARKER: lctl set_param debug=+dlmtrace [30080.430199] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x8000030b [30081.591584] LustreError: 18496:0:(fail.c:129:__cfs_fail_timeout_set()) cfs_fail_timeout id 30b sleeping for 40000ms [30121.594026] LustreError: 18496:0:(fail.c:133:__cfs_fail_timeout_set()) cfs_fail_timeout id 30b awake [30121.745443] Lustre: DEBUG MARKER: lctl debug_kernel [30122.392518] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30122.712745] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30123.600345] Lustre: DEBUG MARKER: rc=0; [30123.600345] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30123.600345] if [[ $? -eq 0 && $val -ne 0 ]]; then [30123.600345] echo $(hostname -s): $val; [30123.600345] rc=$val; [30123.600345] fi; [30123.600345] exit $rc [30124.204482] Lustre: DEBUG MARKER: dmesg [30124.798172] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 19: resend of open request ======================================================= 16:00:49 \(1491580849\) [30124.957823] Lustre: DEBUG MARKER: == replay-dual test 19: resend of open request ======================================================= 16:00:49 (1491580849) [30125.095387] Lustre: DEBUG MARKER: sync; sync; sync [30125.944692] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30126.200683] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30126.332791] LustreError: 28915:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [30126.335075] LustreError: 28915:0:(osd_handler.c:2171:osd_ro()) Skipped 4 previous similar messages [30126.345689] Turning device dm-0 (0xfc00000) read-only [30126.473882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30126.602788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30126.781926] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x30c [30127.091455] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30127.393315] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30127.648426] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30127.834743] Removing read-only on unknown block (0xfc00000) [30127.968023] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30138.283119] Lustre: DEBUG MARKER: hostname [30138.596265] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30138.849656] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30139.100953] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30139.259359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30143.927634] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785d1134 to 0x52ac6ca1785d2ba5 [30143.931523] Lustre: Skipped 5 previous similar messages [30144.212717] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30144.472939] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30144.972382] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30144.973444] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30145.145985] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30145.148817] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30145.288919] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30145.547187] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30145.804102] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30146.063019] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30146.834304] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30146.853029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30147.013757] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30147.117730] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30149.030628] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30149.035647] LustreError: Skipped 5 previous similar messages [30149.409152] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30149.580136] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30150.239178] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30150.411849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30151.760301] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30152.653497] Lustre: DEBUG MARKER: rc=0; [30152.653497] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30152.653497] if [[ $? -eq 0 && $val -ne 0 ]]; then [30152.653497] echo $(hostname -s): $val; [30152.653497] rc=$val; [30152.653497] fi; [30152.653497] exit $rc [30153.266596] Lustre: DEBUG MARKER: dmesg [30153.896225] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 20: recovery time is not increasing ============================================== 16:01:18 \(1491580878\) [30154.054085] Lustre: DEBUG MARKER: == replay-dual test 20: recovery time is not increasing ============================================== 16:01:18 (1491580878) [30154.202129] Lustre: DEBUG MARKER: sync; sync; sync [30155.086667] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30155.344364] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30155.482422] Turning device dm-0 (0xfc00000) read-only [30155.612201] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30155.739729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30155.926717] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30156.199830] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30156.391240] Removing read-only on unknown block (0xfc00000) [30156.533688] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30166.868193] Lustre: DEBUG MARKER: hostname [30167.173248] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30167.425863] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30167.682615] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30167.844845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Console [trevis-41vm7] log opened at 2017-04-07 16:01:37 UTC. Console [trevis-41vm7] connected to at 04-07 16:01. [30172.645618] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30172.904686] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30173.409665] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30173.409794] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30173.570088] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30173.570100] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30173.713228] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30173.980225] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30174.243531] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30174.505432] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30175.275664] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30175.282838] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30175.514132] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30175.516150] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30240.223037] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [30240.225454] Lustre: lustre-MDT0000: disconnecting 1 stale clients [30240.548593] Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.9.5.243@tcp (at 0@lo) [30240.551198] Lustre: Skipped 285 previous similar messages [30241.012889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [30241.036128] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [30241.232831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [30241.260225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 65 sec [30242.712125] Lustre: DEBUG MARKER: sync; sync; sync [30243.578944] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30243.835356] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30243.982653] Turning device dm-0 (0xfc00000) read-only [30244.114171] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30244.243583] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30244.438948] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30244.698070] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30244.885523] Removing read-only on unknown block (0xfc00000) [30244.892911] Lustre: server umount lustre-MDT0000 complete [30244.895090] Lustre: Skipped 8 previous similar messages [30245.024452] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30247.357654] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [30247.362667] LustreError: Skipped 349 previous similar messages [30247.427321] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [30247.429992] LustreError: Skipped 24 previous similar messages [30247.432207] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [30247.436875] Lustre: Skipped 18 previous similar messages [30255.335572] Lustre: DEBUG MARKER: hostname [30255.638724] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30255.891919] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30256.145270] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30256.303997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30256.325560] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [30256.329922] LustreError: Skipped 8 previous similar messages [30256.621234] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30256.898775] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30257.424052] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30257.425237] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30257.590446] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30257.590447] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30257.731924] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30257.989142] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30258.246052] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30258.504933] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30259.266949] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30259.266951] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30259.448560] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30259.511832] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30322.911037] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [30322.913583] Lustre: lustre-MDT0000: disconnecting 1 stale clients [30323.909586] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [30324.004813] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [30324.062289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [30324.206327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 64 sec [30325.672780] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30326.591969] Lustre: DEBUG MARKER: rc=0; [30326.591969] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30326.591969] if [[ $? -eq 0 && $val -ne 0 ]]; then [30326.591969] echo $(hostname -s): $val; [30326.591969] rc=$val; [30326.591969] fi; [30326.591969] exit $rc [30327.194870] Lustre: DEBUG MARKER: dmesg [30327.959524] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 21a: commit on sharing =========================================================== 16:04:12 \(1491581052\) [30328.121099] Lustre: DEBUG MARKER: == replay-dual test 21a: commit on sharing =========================================================== 16:04:12 (1491581052) [30328.271731] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.*.commit_on_sharing | [30328.271731] while read s; do echo mds1 $s; done [30328.533537] Lustre: DEBUG MARKER: lctl set_param mdt.*.commit_on_sharing=1 [30328.850546] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30329.107707] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30329.293816] Turning device dm-0 (0xfc00000) read-only [30329.421670] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30329.549902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30329.724282] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30329.980240] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30330.166374] Removing read-only on unknown block (0xfc00000) [30330.301466] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30339.351036] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491581057/real 1491581057] req@ffff8800603ecf00 x1564004894785600/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491581064 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [30339.357918] Lustre: 5314:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [30340.624376] Lustre: DEBUG MARKER: hostname [30340.937446] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30341.189933] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30341.443934] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30341.602092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30345.446191] LustreError: 3917:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed for registered restore: -2 [30345.450568] LustreError: 3917:0:(mdt_coordinator.c:883:mdt_hsm_cdt_start()) Skipped 7 previous similar messages [30345.455097] LustreError: 3922:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) lustre-MDT0002: Coordinator already started [30345.457582] LustreError: 3922:0:(mdt_coordinator.c:856:mdt_hsm_cdt_start()) Skipped 6 previous similar messages [30345.498805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [30345.501046] Lustre: Skipped 7 previous similar messages [30345.701653] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30345.959307] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30346.517805] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30346.526485] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30346.681497] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30346.683393] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30346.826859] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30347.084743] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30347.345462] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30347.607303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30413.535036] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [30413.537511] Lustre: lustre-MDT0000: disconnecting 1 stale clients [30413.823761] Lustre: DEBUG MARKER: lctl set_param mdt.*.commit_on_sharing=0 [30414.102319] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0000.commit_on_sharing 0 [30414.359543] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.lustre-MDT0002.commit_on_sharing 0 [30414.686096] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30415.581295] Lustre: DEBUG MARKER: rc=0; [30415.581295] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30415.581295] if [[ $? -eq 0 && $val -ne 0 ]]; then [30415.581295] echo $(hostname -s): $val; [30415.581295] rc=$val; [30415.581295] fi; [30415.581295] exit $rc [30416.185441] Lustre: DEBUG MARKER: dmesg [30416.791407] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-dual test_21b skipping ALWAYS excluded test 21b [30416.973189] Lustre: DEBUG MARKER: SKIP: replay-dual test_21b skipping ALWAYS excluded test 21b [30417.188538] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1\/dir ============= 16:05:42 \(1491581142\) [30417.344840] Lustre: DEBUG MARKER: == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply [30431.043801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30431.216454] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30432.897671] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30432.940120] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30433.086310] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30433.180488] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30435.362931] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30435.435093] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30435.543945] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30435.679298] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30436.954998] Lustre: DEBUG MARKER: sync; sync; sync [30437.839582] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30438.097259] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30438.245516] Turning device dm-0 (0xfc00000) read-only [30438.377769] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30438.510230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30439.188559] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30439.445309] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30439.575018] Lustre: Failing over lustre-MDT0000 [30439.576975] Lustre: Skipped 5 previous similar messages [30439.634305] Removing read-only on unknown block (0xfc00000) [30439.767710] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30450.078602] Lustre: DEBUG MARKER: hostname [30450.394538] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30450.646927] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30450.901123] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30451.059972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30451.367280] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30451.628475] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30452.127731] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30452.133801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30452.288822] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30452.289790] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30452.431741] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30452.692230] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30452.948137] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30453.104274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 7 clients reconnect [30453.106982] Lustre: Skipped 5 previous similar messages [30453.212931] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30453.974735] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30453.977733] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30454.188523] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30454.237976] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30457.484645] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30457.583305] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30457.776117] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30457.889984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [30459.408327] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30460.350054] Lustre: DEBUG MARKER: rc=0; [30460.350054] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30460.350054] if [[ $? -eq 0 && $val -ne 0 ]]; then [30460.350054] echo $(hostname -s): $val; [30460.350054] rc=$val; [30460.350054] fi; [30460.350054] exit $rc [30460.967864] Lustre: DEBUG MARKER: dmesg [30461.578560] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0\/M1, c2 mkdir d1\/dir =========== 16:06:26 \(1491581186\) [30461.748262] Lustre: DEBUG MARKER: == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply [30462.173678] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30462.431397] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30462.592226] LustreError: 18484:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491581188 with bad export cookie 5957255847869238102 [30462.597818] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [30462.600316] Lustre: Skipped 3 previous similar messages [30462.789222] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30473.979116] Lustre: DEBUG MARKER: hostname [30474.298435] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30474.553681] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30474.806883] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30474.954722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30479.751997] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30480.019567] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30480.529089] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30480.531263] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30480.698262] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30480.712889] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30480.853116] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30481.109133] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30481.365998] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30481.625859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30494.228173] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30494.391828] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30496.124543] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30496.124545] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30496.311594] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30496.382094] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30498.295903] Lustre: lustre-MDT0000: Recovery over after 0:18, of 7 clients 7 recovered and 0 were evicted. [30498.298786] Lustre: Skipped 6 previous similar messages [30498.565498] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30498.661303] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30498.725770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30498.869939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30498.965770] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30499.102882] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30499.143901] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30499.305068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30500.597513] Lustre: DEBUG MARKER: sync; sync; sync [30501.455537] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30501.715293] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30501.866755] Turning device dm-0 (0xfc00000) read-only [30501.997135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30502.130116] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30502.812098] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30503.073512] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30503.275362] Removing read-only on unknown block (0xfc00000) [30503.417897] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30513.744728] Lustre: DEBUG MARKER: hostname [30514.053826] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30514.307646] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30514.561345] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30514.711181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30518.054318] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30518.313449] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30518.815249] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30518.829647] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30518.996280] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30519.003310] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30519.145246] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30519.404701] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30519.667082] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30519.930735] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30520.694519] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30520.694522] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30520.919928] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30520.919937] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30523.150652] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30523.212339] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30523.302560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30523.437077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30524.861239] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30525.760072] Lustre: DEBUG MARKER: rc=0; [30525.760072] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30525.760072] if [[ $? -eq 0 && $val -ne 0 ]]; then [30525.760072] echo $(hostname -s): $val; [30525.760072] rc=$val; [30525.760072] fi; [30525.760072] exit $rc [30526.414493] Lustre: DEBUG MARKER: dmesg [30527.018979] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1\/dir ============= 16:07:31 \(1491581251\) [30527.182296] Lustre: DEBUG MARKER: == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update [30527.330771] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [30527.501425] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [30527.503631] LustreError: 19436:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800799a5050 x1564005430505472/t244813135882(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491581263 ref 1 fl Interpret:/0/0 rc 0/0 [30527.601511] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30527.866785] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30528.123049] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30528.469384] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30529.775388] LustreError: 18483:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491581255 with bad export cookie 5957255847869241511 [30529.780171] LustreError: 18483:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [30538.790577] Lustre: DEBUG MARKER: hostname [30539.119335] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30539.372810] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30539.625425] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30539.775308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30540.080635] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30540.343792] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30540.853612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30540.856494] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30541.021795] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30541.025413] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30541.166995] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30541.424721] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30541.683245] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30541.946291] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30542.722544] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30542.738828] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30542.953887] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30542.966909] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30545.222711] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30545.225316] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30545.379175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30545.484640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30546.772322] Lustre: DEBUG MARKER: sync; sync; sync [30547.681092] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30547.941090] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30548.079681] Turning device dm-0 (0xfc00000) read-only [30548.211612] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30548.341886] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30549.028466] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30549.286982] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30549.472493] Removing read-only on unknown block (0xfc00000) [30549.605536] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30559.913138] Lustre: DEBUG MARKER: hostname [30560.219111] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30560.472197] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30560.725181] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30560.885633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30561.198838] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30561.457799] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30561.962644] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30561.973491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30562.136478] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30562.140391] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30562.280514] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30562.536998] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30562.792939] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30563.054192] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30563.819123] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30563.847563] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30564.022779] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30564.078735] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30566.265079] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30566.433674] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30566.481740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30566.606615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30567.989694] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30568.901710] Lustre: DEBUG MARKER: rc=0; [30568.901710] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30568.901710] if [[ $? -eq 0 && $val -ne 0 ]]; then [30568.901710] echo $(hostname -s): $val; [30568.901710] rc=$val; [30568.901710] fi; [30568.901710] exit $rc [30569.510645] Lustre: DEBUG MARKER: dmesg [30570.130418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0\/M1,c2 mkdir d1\/dir =========== 16:08:14 \(1491581294\) [30570.293850] Lustre: DEBUG MARKER: == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update [30570.436786] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [30570.587357] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [30570.589561] LustreError: 19436:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800798dc450 x1564005430509952/t253403070474(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 2088/4320 e 0 to 0 dl 1491581306 ref 1 fl Interpret:/0/0 rc 0/0 [30570.705416] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30570.971353] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30571.227192] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30571.578664] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30572.252179] LustreError: 18483:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491581297 with bad export cookie 5957255847869244647 [30588.267457] Lustre: DEBUG MARKER: hostname [30588.592030] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30588.853156] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30589.112488] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30589.270310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30589.938662] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30590.196893] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30590.691786] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30590.694803] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30590.859444] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30590.861235] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30591.002651] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30591.272796] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30591.532858] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30591.795806] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30604.442885] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30604.606198] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30606.309044] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30606.334481] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30606.527823] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30606.573783] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30608.766587] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30608.870350] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30608.919549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30609.101018] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30609.125308] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30609.313720] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30609.380907] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30609.570171] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30610.842773] Lustre: DEBUG MARKER: sync; sync; sync [30611.728317] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30611.984611] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30612.133844] Turning device dm-0 (0xfc00000) read-only [30612.263443] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30612.392510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30613.080157] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30613.335268] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30613.523151] Removing read-only on unknown block (0xfc00000) [30613.657205] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30623.987443] Lustre: DEBUG MARKER: hostname [30624.302300] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30624.554326] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30624.806851] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30624.969211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30628.388678] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30628.647183] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30629.157335] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30629.157760] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30629.321780] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30629.324487] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30629.466545] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30629.722138] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30629.978303] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30630.242120] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30631.032584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30631.034428] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30631.216071] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30631.261596] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30633.472369] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30633.573605] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30633.698141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30633.792543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30635.229550] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30636.157263] Lustre: DEBUG MARKER: rc=0; [30636.157263] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30636.157263] if [[ $? -eq 0 && $val -ne 0 ]]; then [30636.157263] echo $(hostname -s): $val; [30636.157263] rc=$val; [30636.157263] fi; [30636.157263] exit $rc [30636.768900] Lustre: DEBUG MARKER: dmesg [30637.391489] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ======================= 16:09:22 \(1491581362\) [30637.559601] Lustre: DEBUG MARKER: == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ======================= 16:09:22 (1491581362) [30651.697920] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30651.948435] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30653.686445] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30653.727187] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30653.896759] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30654.038233] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30656.189181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30656.295341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30656.344583] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30656.486492] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [30657.785427] Lustre: DEBUG MARKER: sync; sync; sync [30658.669191] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30658.926984] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30659.075977] Turning device dm-0 (0xfc00000) read-only [30659.206646] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30659.337248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30660.011680] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30660.270073] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30660.456420] Removing read-only on unknown block (0xfc00000) [30660.590336] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30670.912296] Lustre: DEBUG MARKER: hostname [30671.223377] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30671.477126] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30671.730520] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30671.892316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30671.917324] Lustre: Evicted from MGS (at 10.9.5.243@tcp) after server handle changed from 0x52ac6ca1785d7270 to 0x52ac6ca1785d7b76 [30671.921569] Lustre: Skipped 10 previous similar messages [30672.212138] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30672.469843] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30672.966781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30672.966864] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30673.129547] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30673.133561] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30673.274321] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30673.530609] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30673.788495] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30674.050534] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30674.798641] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30674.801043] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30675.028818] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30675.028859] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30677.020413] LustreError: 167-0: lustre-MDT0000-lwp-MDT0002: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [30677.025540] LustreError: Skipped 10 previous similar messages [30677.313891] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30677.381298] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30677.555087] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30677.609113] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30679.034571] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30679.938720] Lustre: DEBUG MARKER: rc=0; [30679.938720] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30679.938720] if [[ $? -eq 0 && $val -ne 0 ]]; then [30679.938720] echo $(hostname -s): $val; [30679.938720] rc=$val; [30679.938720] fi; [30679.938720] exit $rc [30680.553339] Lustre: DEBUG MARKER: dmesg [30681.187253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0\/M1, c2 mkdir d1 ====================== 16:10:06 \(1491581406\) [30681.346692] Lustre: DEBUG MARKER: == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ====================== 16:10:06 (1491581406) [30682.066223] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30682.324727] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30682.713766] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30683.390958] LustreError: 18484:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.9.5.239@tcp arrived at 1491581409 with bad export cookie 5957255847869250520 [30683.395786] LustreError: 18484:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) Skipped 1 previous similar message [30693.909126] Lustre: DEBUG MARKER: hostname [30694.220136] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30694.473751] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30694.726439] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30694.887728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30700.287768] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30700.547233] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30701.065884] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30701.072063] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30701.231994] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30701.232230] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30701.377170] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30701.634502] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30701.890489] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30702.153051] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30714.788781] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30714.962174] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30716.776292] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30716.804336] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30716.967891] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30716.967893] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30719.209233] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30719.344238] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30719.365419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30719.555559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30719.641029] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30719.781702] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30719.820018] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30720.004912] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30721.279642] Lustre: DEBUG MARKER: sync; sync; sync [30722.152314] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30722.408949] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30722.556972] Turning device dm-0 (0xfc00000) read-only [30722.686356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30722.817596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30723.472684] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30723.748232] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30723.903092] LustreError: 5315:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880069060c00 x1564004894846832/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [30723.953563] Removing read-only on unknown block (0xfc00000) [30724.092629] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30734.407413] Lustre: DEBUG MARKER: hostname [30734.727165] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30734.980634] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30735.234663] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30735.384498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30738.579434] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30738.839317] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30739.396598] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30739.406491] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30739.562135] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30739.573101] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30739.712396] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30739.969055] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30740.225511] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30740.489917] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30741.339401] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30741.353372] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30741.604739] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30741.613367] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30743.854203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30743.892411] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30744.063379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30744.099450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30745.525042] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30746.450298] Lustre: DEBUG MARKER: rc=0; [30746.450298] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30746.450298] if [[ $? -eq 0 && $val -ne 0 ]]; then [30746.450298] echo $(hostname -s): $val; [30746.450298] rc=$val; [30746.450298] fi; [30746.450298] exit $rc [30747.104085] Lustre: DEBUG MARKER: dmesg [30747.755280] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ================== 16:11:12 \(1491581472\) [30747.949787] Lustre: DEBUG MARKER: == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ================== 16:11:12 (1491581472) [30748.140139] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [30748.308921] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [30748.311161] LustreError: 19436:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800600f8450 x1564005430533840/t274877906962(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1056/4320 e 0 to 0 dl 1491581483 ref 1 fl Interpret:/0/0 rc 0/0 [30748.407979] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30748.670980] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30748.940612] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30749.289080] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30759.605589] Lustre: DEBUG MARKER: hostname [30759.908869] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30760.161702] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30760.415995] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30760.565128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30760.876559] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30761.136934] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30761.632341] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30761.634228] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30761.797099] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30761.797950] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30761.941741] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30762.199089] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30762.455769] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30762.715789] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30763.473181] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30763.541839] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30763.752703] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30763.776180] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30766.029705] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30766.054889] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30766.197908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30766.294574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30767.610845] Lustre: DEBUG MARKER: sync; sync; sync [30768.516761] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30768.772910] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30768.905294] LustreError: 29404:0:(osd_handler.c:2171:osd_ro()) *** setting lustre-MDT0000 read-only *** [30768.907963] LustreError: 29404:0:(osd_handler.c:2171:osd_ro()) Skipped 9 previous similar messages [30768.918671] Turning device dm-0 (0xfc00000) read-only [30769.047513] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30769.176054] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30769.858337] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30770.125935] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30770.315806] Removing read-only on unknown block (0xfc00000) [30770.454364] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30780.777263] Lustre: DEBUG MARKER: hostname [30781.089167] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30781.342657] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30781.597680] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30781.757322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30782.077674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30782.337846] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30782.841239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30782.841842] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30783.008348] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30783.008349] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30783.152554] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30783.412005] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30783.668219] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30783.930493] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30784.694253] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30784.761387] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30784.946526] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30784.975645] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30787.330603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30787.332098] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30787.602858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30787.637741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30789.046075] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30789.953675] Lustre: DEBUG MARKER: rc=0; [30789.953675] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30789.953675] if [[ $? -eq 0 && $val -ne 0 ]]; then [30789.953675] echo $(hostname -s): $val; [30789.953675] rc=$val; [30789.953675] fi; [30789.953675] exit $rc [30790.563900] Lustre: DEBUG MARKER: dmesg [30791.176066] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0\/M1, c2 mkdir d1 =============== 16:11:55 \(1491581515\) [30791.338404] Lustre: DEBUG MARKER: == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 =============== 16:11:55 (1491581515) [30791.500473] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [30791.667455] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [30791.669702] LustreError: 18503:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff8800799a4c50 x1564005430538240/t283467841554(0) o1000->lustre-MDT0001-mdtlov_UUID@10.9.5.239@tcp:-1/-1 lens 1488/4320 e 0 to 0 dl 1491581527 ref 1 fl Interpret:/0/0 rc 0/0 [30791.761903] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30792.028279] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30792.285357] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30792.651763] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30809.260670] Lustre: DEBUG MARKER: hostname [30809.567805] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30809.833828] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30810.104096] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30810.274866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30810.327087] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.237@tcp (not set up) [30810.329554] Lustre: Skipped 5 previous similar messages [30810.592491] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30810.859386] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30811.377673] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30811.380083] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30811.544842] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30811.545421] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30811.688682] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30811.944736] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30812.201875] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30812.462499] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30825.147696] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30825.314179] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30827.093814] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30827.098873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30827.326723] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30827.347463] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [30829.551634] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30829.553734] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30829.749104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30829.792351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [30829.974062] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30830.045523] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30830.147568] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30830.259230] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [30831.577407] Lustre: DEBUG MARKER: sync; sync; sync [30832.464685] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno [30832.722484] Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly [30832.871743] Turning device dm-0 (0xfc00000) read-only [30832.999710] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 [30833.129697] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [30833.773831] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30834.030938] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30834.218518] Removing read-only on unknown block (0xfc00000) [30834.354838] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [30844.666792] Lustre: DEBUG MARKER: hostname [30844.969312] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [30845.227746] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [30845.480410] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [30845.639643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [30845.661110] Lustre: MGS: Connection restored to 10.9.5.243@tcp (at 0@lo) [30845.663385] Lustre: Skipped 445 previous similar messages [30845.960674] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [30846.218824] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u [30846.722475] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30846.740603] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30846.907622] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30846.913712] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30847.054329] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [30847.312567] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30847.568095] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [30847.827182] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [30848.627186] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30848.684108] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30848.890233] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30848.984443] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [30853.121725] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [30853.226356] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [30853.275331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [30853.424162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [30854.798612] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30855.703121] Lustre: DEBUG MARKER: rc=0; [30855.703121] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30855.703121] if [[ $? -eq 0 && $val -ne 0 ]]; then [30855.703121] echo $(hostname -s): $val; [30855.703121] rc=$val; [30855.703121] fi; [30855.703121] exit $rc [30856.323536] Lustre: DEBUG MARKER: dmesg [30856.922224] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 24: reconstruct on non-existing object =========================================== 16:13:01 \(1491581581\) [30857.080018] Lustre: DEBUG MARKER: == replay-dual test 24: reconstruct on non-existing object =========================================== 16:13:01 (1491581581) [30857.218084] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null || [30857.218084] /usr/sbin/lctl lustre_build_version 2>/dev/null || [30857.218084] /usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2 [30857.500462] Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x119 [30857.631219] Lustre: *** cfs_fail_loc=119, val=2147483648*** [30857.633482] LustreError: 20957:0:(ldlm_lib.c:2916:target_send_reply_msg()) @@@ dropping reply req@ffff880065431200 x1564034813018064/t292057776144(0) o36->e7de780c-896d-3ab4-7f63-d1ece1811b84@10.9.5.237@tcp:-1/-1 lens 608/456 e 0 to 0 dl 1491581589 ref 1 fl Interpret:/0/0 rc 0/0 [30858.756287] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30859.040157] Lustre: DEBUG MARKER: lctl set_param -n osd*.*MDT*.force_sync 1 [30864.630894] Lustre: lustre-MDT0000: Client e7de780c-896d-3ab4-7f63-d1ece1811b84 (at 10.9.5.237@tcp) reconnecting [30864.818590] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30865.721080] Lustre: DEBUG MARKER: rc=0; [30865.721080] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30865.721080] if [[ $? -eq 0 && $val -ne 0 ]]; then [30865.721080] echo $(hostname -s): $val; [30865.721080] rc=$val; [30865.721080] fi; [30865.721080] exit $rc [30866.330643] Lustre: DEBUG MARKER: dmesg [30866.900239] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 25: replay\|resend ================================================================ 16:13:11 \(1491581591\) [30867.060897] Lustre: DEBUG MARKER: == replay-dual test 25: replay|resend ================================================================ 16:13:11 (1491581591) [30867.264917] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x304 [30871.423613] LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 10.9.5.244@tcp failed: rc = -107 [30871.426178] LustreError: Skipped 48 previous similar messages [30871.428209] Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [30871.432547] Lustre: Skipped 39 previous similar messages [30880.860867] Lustre: *** cfs_fail_loc=304, val=0*** [30880.863283] Lustre: Skipped 7 previous similar messages [30881.768418] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30881.933766] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30883.375748] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30883.391266] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30883.578420] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30883.613204] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30886.879751] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [30886.905472] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [30887.064431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [30887.167123] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 3 sec [30887.631919] Lustre: MGS: Received new LWP connection from 10.9.5.237@tcp, removing former export from same NID [30888.458563] Lustre: DEBUG MARKER: lctl set_param fail_loc=0 [30888.487065] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [30888.491777] LustreError: Skipped 15 previous similar messages [30888.494891] LustreError: 4433:0:(ldlm_resource.c:1095:ldlm_resource_complain()) MGC10.9.5.243@tcp: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff8800632a46c0) refcount nonzero (1) after lock cleanup; forcing cleanup. [30888.500094] LustreError: 4433:0:(ldlm_resource.c:1677:ldlm_resource_dump()) --- Resource: [0x65727473756c:0x2:0x0].0x0 (ffff8800632a46c0) refcount = 2 [30888.504766] LustreError: 4433:0:(ldlm_resource.c:1698:ldlm_resource_dump()) Waiting locks: [30888.507315] LustreError: 4433:0:(ldlm_resource.c:1700:ldlm_resource_dump()) ### ### ns: MGC10.9.5.243@tcp lock: ffff880066b95e00/0x52ac6ca1785dae4b lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x52ac6ca1785dae52 expref: -99 pid: 18495 timeout: 0 lvb_type: 0 [30888.818593] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30889.708527] Lustre: DEBUG MARKER: rc=0; [30889.708527] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30889.708527] if [[ $? -eq 0 && $val -ne 0 ]]; then [30889.708527] echo $(hostname -s): $val; [30889.708527] rc=$val; [30889.708527] fi; [30889.708527] exit $rc [30890.323532] Lustre: DEBUG MARKER: dmesg [30890.923474] Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: replay-dual test_26 skipping ALWAYS excluded test 26 [30891.128453] Lustre: DEBUG MARKER: SKIP: replay-dual test_26 skipping ALWAYS excluded test 26 [30891.333203] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test 28: lock replay should be ordered: waiting after granted ========================= 16:13:36 \(1491581616\) [30891.488247] Lustre: DEBUG MARKER: == replay-dual test 28: lock replay should be ordered: waiting after granted ========================= 16:13:36 (1491581616) [30909.263072] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30909.427118] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug -1 all 4 [30910.868576] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30910.868835] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30911.107215] Lustre: DEBUG MARKER: trevis-41vm1.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30911.107238] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing wait_import_state_mount FULL osc.lustre-OST0000-osc-*.ost_server_uuid [30913.461832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [30913.465366] Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [30913.621448] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [30913.627087] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-*.ost_server_uuid in FULL state after 2 sec [30917.026084] Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null [30917.930386] Lustre: DEBUG MARKER: rc=0; [30917.930386] val=$(/usr/sbin/lctl get_param -n catastrophe 2>&1); [30917.930386] if [[ $? -eq 0 && $val -ne 0 ]]; then [30917.930386] echo $(hostname -s): $val; [30917.930386] rc=$val; [30917.930386] fi; [30917.930386] exit $rc [30918.553536] Lustre: DEBUG MARKER: dmesg [30919.191957] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-dual test complete, duration 1818 sec ====================================================== 16:14:04 \(1491581644\) [30919.362367] Lustre: DEBUG MARKER: == replay-dual test complete, duration 1818 sec ====================================================== 16:14:04 (1491581644) [30919.541718] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [30919.801079] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [30922.820322] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [30922.825265] LustreError: Skipped 633 previous similar messages [30951.679057] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491581666/real 1491581666] req@ffff88005aeb4c00 x1564004894879040/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491581677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [30951.686385] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 18 previous similar messages [31080.113067] INFO: task umount:5655 blocked for more than 120 seconds. [31080.115467] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31080.117972] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31080.122732] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31080.125417] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31080.128128] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31080.131232] Call Trace: [31080.133477] [] schedule+0x29/0x70 [31080.135788] [] schedule_timeout+0x239/0x2d0 [31080.138175] [] ? discard_slab+0x39/0x50 [31080.140434] [] ? __slab_free+0x2ca/0x2f0 [31080.142666] [] wait_for_completion+0x116/0x170 [31080.144949] [] ? wake_up_state+0x20/0x20 [31080.147186] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31080.149404] [] mgs_put_fsdb+0x53/0x770 [mgs] [31080.151605] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31080.153781] [] mgs_device_fini+0x97/0x5b0 [mgs] [31080.156160] [] class_cleanup+0x7f4/0xd80 [obdclass] [31080.158283] [] class_process_config+0x1f84/0x2c30 [obdclass] [31080.160320] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31080.162461] [] class_manual_cleanup+0xef/0x810 [obdclass] [31080.164589] [] server_put_super+0xb20/0xcd0 [obdclass] [31080.166656] [] generic_shutdown_super+0x72/0xf0 [31080.168640] [] kill_anon_super+0x12/0x20 [31080.170648] [] lustre_kill_super+0x32/0x50 [obdclass] [31080.172546] [] deactivate_locked_super+0x49/0x60 [31080.174424] [] deactivate_super+0x46/0x60 [31080.176221] [] mntput_no_expire+0xc5/0x120 [31080.178010] [] SyS_umount+0xa0/0x3b0 [31080.179759] [] system_call_fastpath+0x16/0x1b [31180.863059] LustreError: 1925:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1491581606, 300s ago); not entering recovery in server code, just going back to sleep ns: MGS lock: ffff880049241600/0x52ac6ca1785dae2f lrc: 3/0,1 mode: --/EX res: [0x65727473756c:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 1925 timeout: 0 lvb_type: 0 [31180.871882] LustreError: dumping log to /tmp/lustre-log.1491581906.1925 [31188.522046] LustreError: 18495:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1491581614, 300s ago), entering recovery for MGS@10.9.5.243@tcp ns: MGC10.9.5.243@tcp lock: ffff880066b93c00/0x52ac6ca1785daec2 lrc: 4/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 1 type: PLN flags: 0x1000000000000 nid: local remote: 0x52ac6ca1785daec9 expref: -99 pid: 18495 timeout: 0 lvb_type: 0 [31200.181057] INFO: task umount:5655 blocked for more than 120 seconds. [31200.183109] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31200.185209] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31200.187279] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31200.189440] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31200.191481] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31200.193496] Call Trace: [31200.195017] [] schedule+0x29/0x70 [31200.196767] [] schedule_timeout+0x239/0x2d0 [31200.198558] [] ? discard_slab+0x39/0x50 [31200.200297] [] ? __slab_free+0x2ca/0x2f0 [31200.202021] [] wait_for_completion+0x116/0x170 [31200.203746] [] ? wake_up_state+0x20/0x20 [31200.205500] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31200.207297] [] mgs_put_fsdb+0x53/0x770 [mgs] [31200.209043] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31200.210850] [] mgs_device_fini+0x97/0x5b0 [mgs] [31200.212643] [] class_cleanup+0x7f4/0xd80 [obdclass] [31200.214458] [] class_process_config+0x1f84/0x2c30 [obdclass] [31200.216340] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31200.218188] [] class_manual_cleanup+0xef/0x810 [obdclass] [31200.220061] [] server_put_super+0xb20/0xcd0 [obdclass] [31200.221909] [] generic_shutdown_super+0x72/0xf0 [31200.223703] [] kill_anon_super+0x12/0x20 [31200.225460] [] lustre_kill_super+0x32/0x50 [obdclass] [31200.227299] [] deactivate_locked_super+0x49/0x60 [31200.229108] [] deactivate_super+0x46/0x60 [31200.230861] [] mntput_no_expire+0xc5/0x120 [31200.232607] [] SyS_umount+0xa0/0x3b0 [31200.234296] [] system_call_fastpath+0x16/0x1b [31320.236049] INFO: task umount:5655 blocked for more than 120 seconds. [31320.237959] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31320.239907] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31320.241875] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31320.243830] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31320.245807] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31320.247797] Call Trace: [31320.249291] [] schedule+0x29/0x70 [31320.251043] [] schedule_timeout+0x239/0x2d0 [31320.252808] [] ? discard_slab+0x39/0x50 [31320.254509] [] ? __slab_free+0x2ca/0x2f0 [31320.256168] [] wait_for_completion+0x116/0x170 [31320.257843] [] ? wake_up_state+0x20/0x20 [31320.259466] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31320.261107] [] mgs_put_fsdb+0x53/0x770 [mgs] [31320.262692] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31320.264299] [] mgs_device_fini+0x97/0x5b0 [mgs] [31320.265872] [] class_cleanup+0x7f4/0xd80 [obdclass] [31320.267434] [] class_process_config+0x1f84/0x2c30 [obdclass] [31320.269054] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31320.270604] [] class_manual_cleanup+0xef/0x810 [obdclass] [31320.272195] [] server_put_super+0xb20/0xcd0 [obdclass] [31320.273757] [] generic_shutdown_super+0x72/0xf0 [31320.275279] [] kill_anon_super+0x12/0x20 [31320.276747] [] lustre_kill_super+0x32/0x50 [obdclass] [31320.278311] [] deactivate_locked_super+0x49/0x60 [31320.279840] [] deactivate_super+0x46/0x60 [31320.281317] [] mntput_no_expire+0xc5/0x120 [31320.282802] [] SyS_umount+0xa0/0x3b0 [31320.284235] [] system_call_fastpath+0x16/0x1b [31440.285063] INFO: task umount:5655 blocked for more than 120 seconds. [31440.286672] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31440.288364] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31440.290501] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31440.292258] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31440.293971] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31440.295902] Call Trace: [31440.297164] [] schedule+0x29/0x70 [31440.298903] [] schedule_timeout+0x239/0x2d0 [31440.300441] [] ? discard_slab+0x39/0x50 [31440.302138] [] ? __slab_free+0x2ca/0x2f0 [31440.303744] [] wait_for_completion+0x116/0x170 [31440.305448] [] ? wake_up_state+0x20/0x20 [31440.307083] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31440.308996] [] mgs_put_fsdb+0x53/0x770 [mgs] [31440.310898] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31440.312620] [] mgs_device_fini+0x97/0x5b0 [mgs] [31440.314202] [] class_cleanup+0x7f4/0xd80 [obdclass] [31440.315763] [] class_process_config+0x1f84/0x2c30 [obdclass] [31440.317373] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31440.318934] [] class_manual_cleanup+0xef/0x810 [obdclass] [31440.320518] [] server_put_super+0xb20/0xcd0 [obdclass] [31440.322086] [] generic_shutdown_super+0x72/0xf0 [31440.323599] [] kill_anon_super+0x12/0x20 [31440.325075] [] lustre_kill_super+0x32/0x50 [obdclass] [31440.326625] [] deactivate_locked_super+0x49/0x60 [31440.328157] [] deactivate_super+0x46/0x60 [31440.329635] [] mntput_no_expire+0xc5/0x120 [31440.331124] [] SyS_umount+0xa0/0x3b0 [31440.332560] [] system_call_fastpath+0x16/0x1b [31524.529845] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [31524.533775] LustreError: Skipped 2147 previous similar messages [31560.334059] INFO: task umount:5655 blocked for more than 120 seconds. [31560.335753] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31560.337544] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31560.339263] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31560.341033] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31560.343144] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31560.345033] Call Trace: [31560.346528] [] schedule+0x29/0x70 [31560.348191] [] schedule_timeout+0x239/0x2d0 [31560.349908] [] ? discard_slab+0x39/0x50 [31560.351655] [] ? __slab_free+0x2ca/0x2f0 [31560.353345] [] wait_for_completion+0x116/0x170 [31560.355108] [] ? wake_up_state+0x20/0x20 [31560.356802] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31560.358565] [] mgs_put_fsdb+0x53/0x770 [mgs] [31560.360284] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31560.362065] [] mgs_device_fini+0x97/0x5b0 [mgs] [31560.363812] [] class_cleanup+0x7f4/0xd80 [obdclass] [31560.365559] [] class_process_config+0x1f84/0x2c30 [obdclass] [31560.367376] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31560.369225] [] class_manual_cleanup+0xef/0x810 [obdclass] [31560.371377] [] server_put_super+0xb20/0xcd0 [obdclass] [31560.373166] [] generic_shutdown_super+0x72/0xf0 [31560.374884] [] kill_anon_super+0x12/0x20 [31560.376573] [] lustre_kill_super+0x32/0x50 [obdclass] [31560.378357] [] deactivate_locked_super+0x49/0x60 [31560.380095] [] deactivate_super+0x46/0x60 [31560.381758] [] mntput_no_expire+0xc5/0x120 [31560.383447] [] SyS_umount+0xa0/0x3b0 [31560.385097] [] system_call_fastpath+0x16/0x1b [31565.679062] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491582266/real 1491582266] req@ffff88004592d200 x1564004894902432/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491582291 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [31565.684864] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [31680.386068] INFO: task umount:5655 blocked for more than 120 seconds. [31680.387885] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31680.389770] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31680.392123] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31680.394049] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31680.396096] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31680.398118] Call Trace: [31680.399761] [] schedule+0x29/0x70 [31680.401614] [] schedule_timeout+0x239/0x2d0 [31680.403492] [] ? discard_slab+0x39/0x50 [31680.405342] [] ? __slab_free+0x2ca/0x2f0 [31680.407152] [] wait_for_completion+0x116/0x170 [31680.408959] [] ? wake_up_state+0x20/0x20 [31680.410716] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31680.412541] [] mgs_put_fsdb+0x53/0x770 [mgs] [31680.414314] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31680.416121] [] mgs_device_fini+0x97/0x5b0 [mgs] [31680.417899] [] class_cleanup+0x7f4/0xd80 [obdclass] [31680.419753] [] class_process_config+0x1f84/0x2c30 [obdclass] [31680.421636] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31680.423476] [] class_manual_cleanup+0xef/0x810 [obdclass] [31680.425358] [] server_put_super+0xb20/0xcd0 [obdclass] [31680.427202] [] generic_shutdown_super+0x72/0xf0 [31680.429015] [] kill_anon_super+0x12/0x20 [31680.430763] [] lustre_kill_super+0x32/0x50 [obdclass] [31680.432599] [] deactivate_locked_super+0x49/0x60 [31680.434410] [] deactivate_super+0x46/0x60 [31680.436170] [] mntput_no_expire+0xc5/0x120 [31680.437969] [] SyS_umount+0xa0/0x3b0 [31680.439667] [] system_call_fastpath+0x16/0x1b [31800.441045] INFO: task umount:5655 blocked for more than 120 seconds. [31800.442822] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31800.444650] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31800.446943] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31800.448861] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31800.450862] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31800.452868] Call Trace: [31800.454269] [] schedule+0x29/0x70 [31800.456204] [] schedule_timeout+0x239/0x2d0 [31800.457981] [] ? discard_slab+0x39/0x50 [31800.459733] [] ? __slab_free+0x2ca/0x2f0 [31800.461492] [] wait_for_completion+0x116/0x170 [31800.463306] [] ? wake_up_state+0x20/0x20 [31800.465082] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31800.466861] [] mgs_put_fsdb+0x53/0x770 [mgs] [31800.468584] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31800.470339] [] mgs_device_fini+0x97/0x5b0 [mgs] [31800.472054] [] class_cleanup+0x7f4/0xd80 [obdclass] [31800.473787] [] class_process_config+0x1f84/0x2c30 [obdclass] [31800.475536] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31800.477235] [] class_manual_cleanup+0xef/0x810 [obdclass] [31800.478954] [] server_put_super+0xb20/0xcd0 [obdclass] [31800.480627] [] generic_shutdown_super+0x72/0xf0 [31800.482277] [] kill_anon_super+0x12/0x20 [31800.483867] [] lustre_kill_super+0x32/0x50 [obdclass] [31800.485537] [] deactivate_locked_super+0x49/0x60 [31800.487198] [] deactivate_super+0x46/0x60 [31800.488781] [] mntput_no_expire+0xc5/0x120 [31800.490436] [] SyS_umount+0xa0/0x3b0 [31800.491997] [] system_call_fastpath+0x16/0x1b [31920.493075] INFO: task umount:5655 blocked for more than 120 seconds. [31920.494699] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [31920.496391] umount D ffff88005e63c400 0 5655 5654 0x00000080 [31920.498089] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [31920.499796] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [31920.501533] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [31920.503248] Call Trace: [31920.504470] [] schedule+0x29/0x70 [31920.505964] [] schedule_timeout+0x239/0x2d0 [31920.507476] [] ? discard_slab+0x39/0x50 [31920.508968] [] ? __slab_free+0x2ca/0x2f0 [31920.510447] [] wait_for_completion+0x116/0x170 [31920.511974] [] ? wake_up_state+0x20/0x20 [31920.513454] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [31920.514987] [] mgs_put_fsdb+0x53/0x770 [mgs] [31920.516498] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [31920.518067] [] mgs_device_fini+0x97/0x5b0 [mgs] [31920.519627] [] class_cleanup+0x7f4/0xd80 [obdclass] [31920.521203] [] class_process_config+0x1f84/0x2c30 [obdclass] [31920.522812] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [31920.524369] [] class_manual_cleanup+0xef/0x810 [obdclass] [31920.525964] [] server_put_super+0xb20/0xcd0 [obdclass] [31920.527525] [] generic_shutdown_super+0x72/0xf0 [31920.529046] [] kill_anon_super+0x12/0x20 [31920.530519] [] lustre_kill_super+0x32/0x50 [obdclass] [31920.532084] [] deactivate_locked_super+0x49/0x60 [31920.533607] [] deactivate_super+0x46/0x60 [31920.535088] [] mntput_no_expire+0xc5/0x120 [31920.536563] [] SyS_umount+0xa0/0x3b0 [31920.538004] [] system_call_fastpath+0x16/0x1b [32040.539046] INFO: task umount:5655 blocked for more than 120 seconds. [32040.540652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32040.542367] umount D ffff88005e63c400 0 5655 5654 0x00000080 [32040.544525] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [32040.546293] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [32040.548013] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [32040.549920] Call Trace: [32040.551359] [] schedule+0x29/0x70 [32040.552992] [] schedule_timeout+0x239/0x2d0 [32040.554649] [] ? discard_slab+0x39/0x50 [32040.556381] [] ? __slab_free+0x2ca/0x2f0 [32040.558004] [] wait_for_completion+0x116/0x170 [32040.559645] [] ? wake_up_state+0x20/0x20 [32040.561158] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [32040.562883] [] mgs_put_fsdb+0x53/0x770 [mgs] [32040.564529] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [32040.566235] [] mgs_device_fini+0x97/0x5b0 [mgs] [32040.567933] [] class_cleanup+0x7f4/0xd80 [obdclass] [32040.569677] [] class_process_config+0x1f84/0x2c30 [obdclass] [32040.571443] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [32040.573133] [] class_manual_cleanup+0xef/0x810 [obdclass] [32040.574854] [] server_put_super+0xb20/0xcd0 [obdclass] [32040.576532] [] generic_shutdown_super+0x72/0xf0 [32040.578193] [] kill_anon_super+0x12/0x20 [32040.579776] [] lustre_kill_super+0x32/0x50 [obdclass] [32040.581450] [] deactivate_locked_super+0x49/0x60 [32040.583116] [] deactivate_super+0x46/0x60 [32040.584693] [] mntput_no_expire+0xc5/0x120 [32040.586350] [] SyS_umount+0xa0/0x3b0 [32040.587912] [] system_call_fastpath+0x16/0x1b [32124.659673] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.237@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [32124.663441] LustreError: Skipped 2146 previous similar messages [32160.589061] INFO: task umount:5655 blocked for more than 120 seconds. [32160.590748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32160.592504] umount D ffff88005e63c400 0 5655 5654 0x00000080 [32160.594289] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [32160.596076] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [32160.597853] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [32160.599640] Call Trace: [32160.600933] [] schedule+0x29/0x70 [32160.602497] [] schedule_timeout+0x239/0x2d0 [32160.604093] [] ? discard_slab+0x39/0x50 [32160.605660] [] ? __slab_free+0x2ca/0x2f0 [32160.607218] [] wait_for_completion+0x116/0x170 [32160.608827] [] ? wake_up_state+0x20/0x20 [32160.610409] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [32160.612028] [] mgs_put_fsdb+0x53/0x770 [mgs] [32160.613608] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [32160.615277] [] mgs_device_fini+0x97/0x5b0 [mgs] [32160.616884] [] class_cleanup+0x7f4/0xd80 [obdclass] [32160.618494] [] class_process_config+0x1f84/0x2c30 [obdclass] [32160.620173] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [32160.621811] [] class_manual_cleanup+0xef/0x810 [obdclass] [32160.623480] [] server_put_super+0xb20/0xcd0 [obdclass] [32160.625112] [] generic_shutdown_super+0x72/0xf0 [32160.626700] [] kill_anon_super+0x12/0x20 [32160.628277] [] lustre_kill_super+0x32/0x50 [obdclass] [32160.629924] [] deactivate_locked_super+0x49/0x60 [32160.631540] [] deactivate_super+0x46/0x60 [32160.633110] [] mntput_no_expire+0xc5/0x120 [32160.634675] [] SyS_umount+0xa0/0x3b0 [32160.636212] [] system_call_fastpath+0x16/0x1b [32190.679052] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491582891/real 1491582891] req@ffff880063f53600 x1564004894926768/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491582916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [32190.684802] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [32725.129765] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.238@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [32725.133987] LustreError: Skipped 2144 previous similar messages [32815.679175] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491583516/real 1491583516] req@ffff880065433c00 x1564004894951072/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491583541 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [32815.685448] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [33325.308062] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [33325.312965] LustreError: Skipped 2151 previous similar messages [33445.679222] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491584146/real 1491584146] req@ffff8800638f4300 x1564004894975568/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491584171 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [33445.686154] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [33925.309052] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [33925.313811] LustreError: Skipped 2144 previous similar messages [34046.679046] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491584746/real 1491584746] req@ffff880065f9d200 x1564004894998896/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491584771 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [34046.686114] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [34525.311122] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [34525.316961] LustreError: Skipped 2146 previous similar messages [34579.099812] SysRq : Show State [34579.101971] task PC stack pid father [34579.104282] systemd S 0000000000000000 0 1 0 0x00000000 [34579.106931] ffff88007c7bbdb8 0000000000000082 ffff88007c7b0000 ffff88007c7bbfd8 [34579.109610] ffff88007c7bbfd8 ffff88007c7bbfd8 ffff88007c7b0000 0000000000000000 [34579.112210] 0000000000000000 ffff880036042120 ffff88007c7b0000 0000000000000000 [34579.114652] Call Trace: [34579.116883] [] schedule+0x29/0x70 [34579.119213] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34579.121700] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34579.124242] [] schedule_hrtimeout_range+0x13/0x20 [34579.126654] [] ep_poll+0x23e/0x360 [34579.128989] [] ? __fput+0x183/0x260 [34579.131364] [] ? wake_up_state+0x20/0x20 [34579.133728] [] SyS_epoll_wait+0xed/0x120 [34579.136149] [] system_call_fastpath+0x16/0x1b [34579.138524] kthreadd S ffff88007c7b0fb0 0 2 0 0x00000000 [34579.140994] ffff88007c027e70 0000000000000046 ffff88007c7b0fb0 ffff88007c027fd8 [34579.143620] ffff88007c027fd8 ffff88007c027fd8 ffff88007c7b0fb0 ffffffff819f9d00 [34579.146242] ffff88005aeb7980 ffff88007c7b0fb0 ffff88007c7b0fb0 ffff88007c7b0fb0 [34579.148667] Call Trace: [34579.150849] [] schedule+0x29/0x70 [34579.153107] [] kthreadd+0x2e6/0x2f0 [34579.155387] [] ? kthread_create_on_cpu+0x60/0x60 [34579.157685] [] ret_from_fork+0x58/0x90 [34579.159946] [] ? kthread_create_on_cpu+0x60/0x60 [34579.162188] ksoftirqd/0 S 0000000000000000 0 3 2 0x00000000 [34579.164384] ffff88007c037e80 0000000000000046 ffff88007c7b1f60 ffff88007c037fd8 [34579.166763] ffff88007c037fd8 ffff88007c037fd8 ffff88007c7b1f60 ffff88007c7b1f60 [34579.169090] ffff88007d002380 ffffffff819ed0c0 0000000000000000 0000000000000000 [34579.171261] Call Trace: [34579.173165] [] schedule+0x29/0x70 [34579.175137] [] smpboot_thread_fn+0xdd/0x180 [34579.177212] [] ? lg_double_unlock+0x90/0x90 [34579.179221] [] kthread+0xcf/0xe0 [34579.181102] [] ? kthread_create_on_node+0x140/0x140 [34579.183078] [] ret_from_fork+0x58/0x90 [34579.184945] [] ? kthread_create_on_node+0x140/0x140 [34579.186856] kworker/0:0H S ffff88007fc16840 0 5 2 0x00000000 [34579.188763] ffff88007c03fe50 0000000000000046 ffff88007c7b3ec0 ffff88007c03ffd8 [34579.190813] ffff88007c03ffd8 ffff88007c03ffd8 ffff88007c7b3ec0 ffff88007fc16860 [34579.192838] ffff88007d03f9b0 ffff88007c7b3ec0 ffff88007d03f980 ffff88007fc16840 [34579.194851] Call Trace: [34579.196200] [] schedule+0x29/0x70 [34579.198054] [] worker_thread+0x1d9/0x410 [34579.199703] [] ? rescuer_thread+0x460/0x460 [34579.201356] [] kthread+0xcf/0xe0 [34579.202914] [] ? kthread_create_on_node+0x140/0x140 [34579.204601] [] ret_from_fork+0x58/0x90 [34579.206207] [] ? kthread_create_on_node+0x140/0x140 [34579.207895] migration/0 S 0000000000000000 0 7 2 0x00000000 [34579.209570] ffff88007c06be80 0000000000000046 ffff88007c7b5e20 ffff88007c06bfd8 [34579.211436] ffff88007c06bfd8 ffff88007c06bfd8 ffff88007c7b5e20 ffff88007c7b5e20 [34579.213314] ffff88007d002390 ffffffff81a04f00 0000000000000000 0000000000000000 [34579.215007] Call Trace: [34579.216450] [] schedule+0x29/0x70 [34579.218030] [] smpboot_thread_fn+0xdd/0x180 [34579.219654] [] ? lg_double_unlock+0x90/0x90 [34579.221305] [] kthread+0xcf/0xe0 [34579.222845] [] ? kthread_create_on_node+0x140/0x140 [34579.224543] [] ret_from_fork+0x58/0x90 [34579.226169] [] ? kthread_create_on_node+0x140/0x140 [34579.227860] rcu_bh S 0000000000000000 0 8 2 0x00000000 [34579.229599] ffff88007c06fe10 0000000000000046 ffff88007c7b6dd0 ffff88007c06ffd8 [34579.231564] ffff88007c06ffd8 ffff88007c06ffd8 ffff88007c7b6dd0 ffffffff81a05d40 [34579.233415] ffff88007c7b6dd0 ffffffff81a1a3a0 0000000000000000 0000000000000000 [34579.235309] Call Trace: [34579.236563] [] schedule+0x29/0x70 [34579.238230] [] rcu_gp_kthread+0xb2/0x7e0 [34579.239834] [] ? __schedule+0x3b0/0x990 [34579.241457] [] ? wake_up_atomic_t+0x30/0x30 [34579.243102] [] ? rcu_gp_fqs+0xb0/0xb0 [34579.244689] [] kthread+0xcf/0xe0 [34579.246301] [] ? kthread_create_on_node+0x140/0x140 [34579.247979] [] ret_from_fork+0x58/0x90 [34579.249558] [] ? kthread_create_on_node+0x140/0x140 [34579.251266] rcu_sched S 000000000000f888 0 9 2 0x00000000 [34579.252915] ffff88007c07bd60 0000000000000046 ffff88007c070000 ffff88007c07bfd8 [34579.254783] ffff88007c07bfd8 ffff88007c07bfd8 ffff88007c070000 ffff88007c0dc000 [34579.256618] ffff88007c07bd90 00000001020b0f5c ffff88007c0dc000 000000000000f888 [34579.258462] Call Trace: [34579.259697] [] schedule+0x29/0x70 [34579.261416] [] schedule_timeout+0x175/0x2d0 [34579.263128] [] ? internal_add_timer+0x70/0x70 [34579.264768] [] ? prepare_to_wait+0x56/0x90 [34579.266412] [] rcu_gp_kthread+0x349/0x7e0 [34579.268006] [] ? wake_up_atomic_t+0x30/0x30 [34579.269591] [] ? rcu_gp_fqs+0xb0/0xb0 [34579.271061] [] kthread+0xcf/0xe0 [34579.272666] [] ? kthread_create_on_node+0x140/0x140 [34579.274330] [] ret_from_fork+0x58/0x90 [34579.275881] [] ? kthread_create_on_node+0x140/0x140 [34579.277540] watchdog/0 S 0000000000000000 0 10 2 0x00000000 [34579.279184] ffff88007c07fe80 0000000000000046 ffff88007c070fb0 ffff88007c07ffd8 [34579.281167] ffff88007c07ffd8 ffff88007c07ffd8 ffff88007c070fb0 ffff88007c070fb0 [34579.282843] ffff88007d002890 ffffffff81a057e0 0000000000000000 0000000000000000 [34579.284680] Call Trace: [34579.285897] [] schedule+0x29/0x70 [34579.287593] [] smpboot_thread_fn+0xdd/0x180 [34579.289238] [] ? lg_double_unlock+0x90/0x90 [34579.290832] [] kthread+0xcf/0xe0 [34579.292367] [] ? kthread_create_on_node+0x140/0x140 [34579.294025] [] ret_from_fork+0x58/0x90 [34579.295658] [] ? kthread_create_on_node+0x140/0x140 [34579.297201] watchdog/1 S 0000000000000000 0 11 2 0x00000000 [34579.298801] ffff88007c0c7e80 0000000000000046 ffff88007c072f10 ffff88007c0c7fd8 [34579.300485] ffff88007c0c7fd8 ffff88007c0c7fd8 ffff88007c072f10 ffff88007c072f10 [34579.302173] ffff88007d0028a0 ffffffff81a057e0 0000000000000000 0000000000000000 [34579.303823] Call Trace: [34579.305064] [] schedule+0x29/0x70 [34579.306544] [] smpboot_thread_fn+0xdd/0x180 [34579.308060] [] ? lg_double_unlock+0x90/0x90 [34579.309553] [] kthread+0xcf/0xe0 [34579.310962] [] ? kthread_create_on_node+0x140/0x140 [34579.312501] [] ret_from_fork+0x58/0x90 [34579.313944] [] ? kthread_create_on_node+0x140/0x140 [34579.315475] migration/1 S 0000000000000000 0 12 2 0x00000000 [34579.317077] ffff88007c0cbe80 0000000000000046 ffff88007c073ec0 ffff88007c0cbfd8 [34579.318762] ffff88007c0cbfd8 ffff88007c0cbfd8 ffff88007c073ec0 ffff88007c073ec0 [34579.320444] ffff88007d0028b0 ffffffff81a04f00 0000000000000000 0000000000000000 [34579.322117] Call Trace: [34579.323334] [] schedule+0x29/0x70 [34579.324784] [] smpboot_thread_fn+0xdd/0x180 [34579.326298] [] ? lg_double_unlock+0x90/0x90 [34579.327803] [] kthread+0xcf/0xe0 [34579.329209] [] ? kthread_create_on_node+0x140/0x140 [34579.330764] [] ret_from_fork+0x58/0x90 [34579.332237] [] ? kthread_create_on_node+0x140/0x140 [34579.333775] ksoftirqd/1 S 0000000000000000 0 13 2 0x00000000 [34579.335379] ffff88007c0cfe80 0000000000000046 ffff88007c074e70 ffff88007c0cffd8 [34579.337056] ffff88007c0cffd8 ffff88007c0cffd8 ffff88007c074e70 ffff88007c074e70 [34579.338726] ffff88007d0028c0 ffffffff819ed0c0 0000000000000000 0000000000000000 [34579.340424] Call Trace: [34579.341633] [] schedule+0x29/0x70 [34579.343128] [] smpboot_thread_fn+0xdd/0x180 [34579.344648] [] ? lg_double_unlock+0x90/0x90 [34579.346179] [] kthread+0xcf/0xe0 [34579.347615] [] ? kthread_create_on_node+0x140/0x140 [34579.349207] [] ret_from_fork+0x58/0x90 [34579.350695] [] ? kthread_create_on_node+0x140/0x140 [34579.352267] kworker/1:0H S ffff88007fd16840 0 15 2 0x00000000 [34579.353885] ffff88007c0dbe50 0000000000000046 ffff88007c076dd0 ffff88007c0dbfd8 [34579.355612] ffff88007c0dbfd8 ffff88007c0dbfd8 ffff88007c076dd0 ffff88007fd16860 [34579.357342] ffff88007c780030 ffff88007c076dd0 ffff88007c780000 ffff88007fd16840 [34579.359024] Call Trace: [34579.360231] [] schedule+0x29/0x70 [34579.361701] [] worker_thread+0x1d9/0x410 [34579.363204] [] ? rescuer_thread+0x460/0x460 [34579.364718] [] kthread+0xcf/0xe0 [34579.366170] [] ? kthread_create_on_node+0x140/0x140 [34579.367740] [] ret_from_fork+0x58/0x90 [34579.369217] [] ? kthread_create_on_node+0x140/0x140 [34579.370774] kdevtmpfs S 0000000000000000 0 17 2 0x00000000 [34579.372405] ffff88007c147e70 0000000000000046 ffff88007c128fb0 ffff88007c147fd8 [34579.374139] ffff88007c147fd8 ffff88007c147fd8 ffff88007c128fb0 0000000000000000 [34579.375830] 0000000000000000 ffff88007c128fb0 0000000000000000 0000000000000000 [34579.377561] Call Trace: [34579.378778] [] schedule+0x29/0x70 [34579.380268] [] devtmpfsd+0x168/0x180 [34579.381751] [] ? handle_create.isra.2+0x220/0x220 [34579.383316] [] kthread+0xcf/0xe0 [34579.384751] [] ? kthread_create_on_node+0x140/0x140 [34579.386331] [] ret_from_fork+0x58/0x90 [34579.387813] [] ? kthread_create_on_node+0x140/0x140 [34579.389384] netns S ffff88007c780300 0 18 2 0x00000000 [34579.391014] ffff88007c187e08 0000000000000046 ffff88007c129f60 ffff88007c187fd8 [34579.392723] ffff88007c187fd8 ffff88007c187fd8 ffff88007c129f60 ffff88007c129f60 [34579.394471] ffff88007c780300 ffff88007c0ff080 ffff88007c780330 ffff88007c780300 [34579.396173] Call Trace: [34579.397376] [] schedule+0x29/0x70 [34579.398863] [] rescuer_thread+0x359/0x460 [34579.400363] [] ? __schedule+0x3b0/0x990 [34579.401854] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.403429] [] kthread+0xcf/0xe0 [34579.404878] [] ? kthread_create_on_node+0x140/0x140 [34579.406450] [] ret_from_fork+0x58/0x90 [34579.407923] [] ? kthread_create_on_node+0x140/0x140 [34579.409477] khungtaskd S 000000000000f888 0 19 2 0x00000000 [34579.411115] ffff88007c1dfd90 0000000000000046 ffff88007c12af10 ffff88007c1dffd8 [34579.412833] ffff88007c1dffd8 ffff88007c1dffd8 ffff88007c12af10 ffffffff81e380c0 [34579.414543] ffff88007c1dfdc0 00000001020c9b5e ffffffff81e380c0 000000000000f888 [34579.416259] Call Trace: [34579.417482] [] schedule+0x29/0x70 [34579.418984] [] schedule_timeout+0x175/0x2d0 [34579.420509] [] ? internal_add_timer+0x70/0x70 [34579.422060] [] schedule_timeout_interruptible+0x1e/0x20 [34579.423677] [] watchdog+0x4c/0x2c0 [34579.425140] [] ? reset_hung_task_detector+0x20/0x20 [34579.426712] [] kthread+0xcf/0xe0 [34579.428147] [] ? kthread_create_on_node+0x140/0x140 [34579.429730] [] ret_from_fork+0x58/0x90 [34579.431238] [] ? kthread_create_on_node+0x140/0x140 [34579.432811] writeback S ffff88007c1c0280 0 20 2 0x00000000 [34579.434453] ffff88007c1e3e08 0000000000000046 ffff88007c12bec0 ffff88007c1e3fd8 [34579.436163] ffff88007c1e3fd8 ffff88007c1e3fd8 ffff88007c12bec0 ffff88007c12bec0 [34579.437856] ffff88007c1c0280 ffff88007c1da880 ffff88007c1c02b0 ffff88007c1c0280 [34579.439578] Call Trace: [34579.440808] [] schedule+0x29/0x70 [34579.442311] [] rescuer_thread+0x359/0x460 [34579.443839] [] ? __schedule+0x3b0/0x990 [34579.445334] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.446914] [] kthread+0xcf/0xe0 [34579.448368] [] ? kthread_create_on_node+0x140/0x140 [34579.449957] [] ret_from_fork+0x58/0x90 [34579.451434] [] ? kthread_create_on_node+0x140/0x140 [34579.453010] kintegrityd S ffff88007c1c0380 0 21 2 0x00000000 [34579.454637] ffff88007c1ebe08 0000000000000046 ffff88007c12ce70 ffff88007c1ebfd8 [34579.456376] ffff88007c1ebfd8 ffff88007c1ebfd8 ffff88007c12ce70 ffff88007c12ce70 [34579.458082] ffff88007c1c0380 ffff88007c1daa80 ffff88007c1c03b0 ffff88007c1c0380 [34579.459762] Call Trace: [34579.461025] [] schedule+0x29/0x70 [34579.462491] [] rescuer_thread+0x359/0x460 [34579.464002] [] ? __schedule+0x3b0/0x990 [34579.465487] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.467034] [] kthread+0xcf/0xe0 [34579.468503] [] ? kthread_create_on_node+0x140/0x140 [34579.470070] [] ret_from_fork+0x58/0x90 [34579.471552] [] ? kthread_create_on_node+0x140/0x140 [34579.473127] bioset S ffff88007c1c0500 0 22 2 0x00000000 [34579.474762] ffff88007c1efe08 0000000000000046 ffff88007c12de20 ffff88007c1effd8 [34579.476497] ffff88007c1effd8 ffff88007c1effd8 ffff88007c12de20 ffff88007c12de20 [34579.478218] ffff88007c1c0500 ffff88007c1dac80 ffff88007c1c0530 ffff88007c1c0500 [34579.479901] Call Trace: [34579.481123] [] schedule+0x29/0x70 [34579.482593] [] rescuer_thread+0x359/0x460 [34579.484100] [] ? __schedule+0x3b0/0x990 [34579.485580] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.487156] [] kthread+0xcf/0xe0 [34579.488592] [] ? kthread_create_on_node+0x140/0x140 [34579.490152] [] ret_from_fork+0x58/0x90 [34579.491614] [] ? kthread_create_on_node+0x140/0x140 [34579.493176] kblockd S ffff88007c1c0600 0 23 2 0x00000000 [34579.494820] ffff88007c1fbe08 0000000000000046 ffff88007c12edd0 ffff88007c1fbfd8 [34579.496547] ffff88007c1fbfd8 ffff88007c1fbfd8 ffff88007c12edd0 ffff88007c12edd0 [34579.498243] ffff88007c1c0600 ffff88007c1db080 ffff88007c1c0630 ffff88007c1c0600 [34579.499917] Call Trace: [34579.501162] [] schedule+0x29/0x70 [34579.502602] [] rescuer_thread+0x359/0x460 [34579.504110] [] ? __schedule+0x3b0/0x990 [34579.505606] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.507181] [] kthread+0xcf/0xe0 [34579.508593] [] ? kthread_create_on_node+0x140/0x140 [34579.510150] [] ret_from_fork+0x58/0x90 [34579.511636] [] ? kthread_create_on_node+0x140/0x140 [34579.513196] md S ffff88007c243a80 0 24 2 0x00000000 [34579.514804] ffff88007c293e08 0000000000000046 ffff88007c128000 ffff88007c293fd8 [34579.516518] ffff88007c293fd8 ffff88007c293fd8 ffff88007c128000 ffff88007c128000 [34579.518237] ffff88007c243a80 ffff88007c28e880 ffff88007c243ab0 ffff88007c243a80 [34579.519905] Call Trace: [34579.521150] [] schedule+0x29/0x70 [34579.522599] [] rescuer_thread+0x359/0x460 [34579.524115] [] ? __schedule+0x3b0/0x990 [34579.525589] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.527158] [] kthread+0xcf/0xe0 [34579.528572] [] ? kthread_create_on_node+0x140/0x140 [34579.530143] [] ret_from_fork+0x58/0x90 [34579.531633] [] ? kthread_create_on_node+0x140/0x140 [34579.533205] kswapd0 S 0000000000000000 0 31 2 0x00000000 [34579.534817] ffff88007b83fe08 0000000000000046 ffff88007c369f60 ffff88007b83ffd8 [34579.536529] ffff88007b83ffd8 ffff88007b83ffd8 ffff88007c369f60 ffff88007ffd6000 [34579.538241] ffff88007fffc4e8 0000000000000000 ffff88007b83fe80 0000000000000000 [34579.539916] Call Trace: [34579.541191] [] schedule+0x29/0x70 [34579.542644] [] kswapd+0x394/0x450 [34579.544095] [] ? wake_up_atomic_t+0x30/0x30 [34579.545617] [] ? balance_pgdat+0x5e0/0x5e0 [34579.547118] [] kthread+0xcf/0xe0 [34579.548526] [] ? kthread_create_on_node+0x140/0x140 [34579.550088] [] ret_from_fork+0x58/0x90 [34579.551557] [] ? kthread_create_on_node+0x140/0x140 [34579.553110] ksmd S 0000000000000000 0 32 2 0x00000000 [34579.554716] ffff88007b833e50 0000000000000046 ffff88007c36de20 ffff88007b833fd8 [34579.556420] ffff88007b833fd8 ffff88007b833fd8 ffff88007c36de20 ffff88007c36de20 [34579.558097] ffff88007c36de20 ffff88007b833e88 ffff88007c36de20 0000000000000000 [34579.559755] Call Trace: [34579.561002] [] schedule+0x29/0x70 [34579.562438] [] ksm_scan_thread+0x16a/0x240 [34579.563916] [] ? wake_up_atomic_t+0x30/0x30 [34579.565445] [] ? ksm_do_scan+0x11e0/0x11e0 [34579.566937] [] kthread+0xcf/0xe0 [34579.568350] [] ? kthread_create_on_node+0x140/0x140 [34579.569909] [] ret_from_fork+0x58/0x90 [34579.571364] [] ? kthread_create_on_node+0x140/0x140 [34579.572923] khugepaged S 000000000000f888 0 33 2 0x00000000 [34579.574530] ffff880077637d80 0000000000000046 ffff88007c36ce70 ffff880077637fd8 [34579.576236] ffff880077637fd8 ffff880077637fd8 ffff88007c36ce70 ffffffff81e380c0 [34579.577911] ffff880077637db0 00000001020b1db7 ffffffff81e380c0 000000000000f888 [34579.579633] Call Trace: [34579.580843] [] schedule+0x29/0x70 [34579.582345] [] schedule_timeout+0x175/0x2d0 [34579.583857] [] ? internal_add_timer+0x70/0x70 [34579.585382] [] ? prepare_to_wait+0x56/0x90 [34579.586883] [] khugepaged+0x1e3/0x480 [34579.588333] [] ? wake_up_atomic_t+0x30/0x30 [34579.589825] [] ? khugepaged_scan_mm_slot+0xc70/0xc70 [34579.591389] [] kthread+0xcf/0xe0 [34579.592792] [] ? kthread_create_on_node+0x140/0x140 [34579.594350] [] ret_from_fork+0x58/0x90 [34579.595811] [] ? kthread_create_on_node+0x140/0x140 [34579.597360] fsnotify_mark S ffff88007763be00 0 34 2 0x00000000 [34579.598973] ffff88007763be28 0000000000000046 ffff88007c36edd0 ffff88007763bfd8 [34579.600683] ffff88007763bfd8 ffff88007763bfd8 ffff88007c36edd0 ffff88007763be58 [34579.602390] ffff88007763be58 ffff88007c36edd0 ffff880035e061e0 ffff88007763be00 [34579.604059] Call Trace: [34579.605250] [] schedule+0x29/0x70 [34579.606694] [] fsnotify_mark_destroy+0x141/0x180 [34579.608225] [] ? wake_up_atomic_t+0x30/0x30 [34579.609728] [] ? fsnotify_put_mark+0x40/0x40 [34579.611239] [] kthread+0xcf/0xe0 [34579.612638] [] ? kthread_create_on_node+0x140/0x140 [34579.614181] [] ret_from_fork+0x58/0x90 [34579.615628] [] ? kthread_create_on_node+0x140/0x140 [34579.617182] crypto S ffff88007c152500 0 35 2 0x00000000 [34579.618790] ffff880077647e08 0000000000000046 ffff88007c36bec0 ffff880077647fd8 [34579.620510] ffff880077647fd8 ffff880077647fd8 ffff88007c36bec0 ffff88007c36bec0 [34579.622220] ffff88007c152500 ffff88007b837680 ffff88007c152530 ffff88007c152500 [34579.623885] Call Trace: [34579.625139] [] schedule+0x29/0x70 [34579.626571] [] rescuer_thread+0x359/0x460 [34579.628066] [] ? __schedule+0x3b0/0x990 [34579.629532] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.631107] [] kthread+0xcf/0xe0 [34579.632531] [] ? kthread_create_on_node+0x140/0x140 [34579.634095] [] ret_from_fork+0x58/0x90 [34579.635560] [] ? kthread_create_on_node+0x140/0x140 [34579.637123] kthrotld S ffff88007c152680 0 43 2 0x00000000 [34579.638751] ffff88007766fe08 0000000000000046 ffff88007762de20 ffff88007766ffd8 [34579.640483] ffff88007766ffd8 ffff88007766ffd8 ffff88007762de20 ffff88007762de20 [34579.642218] ffff88007c152680 ffff88007b837a80 ffff88007c1526b0 ffff88007c152680 [34579.643905] Call Trace: [34579.645172] [] schedule+0x29/0x70 [34579.646610] [] rescuer_thread+0x359/0x460 [34579.648114] [] ? __schedule+0x3b0/0x990 [34579.649594] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.651163] [] kthread+0xcf/0xe0 [34579.652576] [] ? kthread_create_on_node+0x140/0x140 [34579.654130] [] ret_from_fork+0x58/0x90 [34579.655589] [] ? kthread_create_on_node+0x140/0x140 [34579.657152] kmpath_rdacd S ffff88007c152b00 0 45 2 0x00000000 [34579.658767] ffff8800776afe08 0000000000000046 ffff88007762ce70 ffff8800776affd8 [34579.660501] ffff8800776affd8 ffff8800776affd8 ffff88007762ce70 ffff88007762ce70 [34579.662227] ffff88007c152b00 ffff8800776aa280 ffff88007c152b30 ffff88007c152b00 [34579.663911] Call Trace: [34579.665128] [] schedule+0x29/0x70 [34579.666589] [] rescuer_thread+0x359/0x460 [34579.668104] [] ? __schedule+0x3b0/0x990 [34579.669668] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.671240] [] kthread+0xcf/0xe0 [34579.672673] [] ? kthread_create_on_node+0x140/0x140 [34579.674238] [] ret_from_fork+0x58/0x90 [34579.675719] [] ? kthread_create_on_node+0x140/0x140 [34579.677288] kpsmoused S ffff88007769d100 0 46 2 0x00000000 [34579.678909] ffff8800776bfe08 0000000000000046 ffff88007762bec0 ffff8800776bffd8 [34579.680638] ffff8800776bffd8 ffff8800776bffd8 ffff88007762bec0 ffff88007762bec0 [34579.682365] ffff88007769d100 ffff8800776aa880 ffff88007769d130 ffff88007769d100 [34579.684051] Call Trace: [34579.685266] [] schedule+0x29/0x70 [34579.686750] [] rescuer_thread+0x359/0x460 [34579.688258] [] ? __schedule+0x3b0/0x990 [34579.689749] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.691313] [] kthread+0xcf/0xe0 [34579.692738] [] ? kthread_create_on_node+0x140/0x140 [34579.694316] [] ret_from_fork+0x58/0x90 [34579.695794] [] ? kthread_create_on_node+0x140/0x140 [34579.697365] ipv6_addrconf S ffff88007769da80 0 48 2 0x00000000 [34579.698996] ffff8800776f7e08 0000000000000046 ffff880077629f60 ffff8800776f7fd8 [34579.700713] ffff8800776f7fd8 ffff8800776f7fd8 ffff880077629f60 ffff880077629f60 [34579.702430] ffff88007769da80 ffff8800776ab880 ffff88007769dab0 ffff88007769da80 [34579.704115] Call Trace: [34579.705330] [] schedule+0x29/0x70 [34579.706805] [] rescuer_thread+0x359/0x460 [34579.708311] [] ? __schedule+0x3b0/0x990 [34579.709791] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.711363] [] kthread+0xcf/0xe0 [34579.712789] [] ? kthread_create_on_node+0x140/0x140 [34579.714364] [] ret_from_fork+0x58/0x90 [34579.715843] [] ? kthread_create_on_node+0x140/0x140 [34579.717402] deferwq S ffff88007c3da500 0 67 2 0x00000000 [34579.719014] ffff880077743e08 0000000000000046 ffff880077752f10 ffff880077743fd8 [34579.720736] ffff880077743fd8 ffff880077743fd8 ffff880077752f10 ffff880077752f10 [34579.722456] ffff88007c060800 ffff88007c361880 ffff88007c3da530 ffff88007c3da500 [34579.724140] Call Trace: [34579.725360] [] schedule+0x29/0x70 [34579.726827] [] rescuer_thread+0x359/0x460 [34579.728340] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.729922] [] kthread+0xcf/0xe0 [34579.731363] [] ? kthread_create_on_node+0x140/0x140 [34579.732930] [] ret_from_fork+0x58/0x90 [34579.734395] [] ? kthread_create_on_node+0x140/0x140 [34579.735974] kauditd S 0000000000000000 0 178 2 0x00000000 [34579.737599] ffff880036133e48 0000000000000046 ffff8800361d8fb0 ffff880036133fd8 [34579.739317] ffff880036133fd8 ffff880036133fd8 ffff8800361d8fb0 0000000000000000 [34579.740993] ffff8800361d8fb0 ffff8800361d8fb0 ffff8800361d8fb0 0000000000000000 [34579.742705] Call Trace: [34579.743920] [] schedule+0x29/0x70 [34579.745423] [] kauditd_thread+0x184/0x1e0 [34579.746925] [] ? wake_up_state+0x20/0x20 [34579.748443] [] ? audit_printk_skb+0x70/0x70 [34579.749949] [] kthread+0xcf/0xe0 [34579.751357] [] ? kthread_create_on_node+0x140/0x140 [34579.752911] [] ret_from_fork+0x58/0x90 [34579.754371] [] ? kthread_create_on_node+0x140/0x140 [34579.755935] ata_sff S ffff8800363cc580 0 234 2 0x00000000 [34579.757550] ffff880035c7fe08 0000000000000046 ffff8800361a8000 ffff880035c7ffd8 [34579.759233] ffff880035c7ffd8 ffff880035c7ffd8 ffff8800361a8000 ffff8800361a8000 [34579.760902] ffff8800363cc580 ffff88003617de80 ffff8800363cc5b0 ffff8800363cc580 [34579.762623] Call Trace: [34579.763829] [] schedule+0x29/0x70 [34579.765300] [] rescuer_thread+0x359/0x460 [34579.766803] [] ? __schedule+0x3b0/0x990 [34579.768276] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.769841] [] kthread+0xcf/0xe0 [34579.771273] [] ? kthread_create_on_node+0x140/0x140 [34579.772815] [] ret_from_fork+0x58/0x90 [34579.774275] [] ? kthread_create_on_node+0x140/0x140 [34579.775836] scsi_eh_0 S 0000000000000000 0 240 2 0x00000000 [34579.777446] ffff880035cbfe18 0000000000000046 ffff8800361daf10 ffff880035cbffd8 [34579.779138] ffff880035cbffd8 ffff880035cbffd8 ffff8800361daf10 ffff880035cb0800 [34579.780789] ffff8800361daf10 ffff8800361db588 0000000000000000 0000000000000000 [34579.782512] Call Trace: [34579.783727] [] schedule+0x29/0x70 [34579.785223] [] scsi_error_handler+0xa3/0x8b0 [34579.786738] [] ? scsi_eh_get_sense+0x250/0x250 [34579.788283] [] kthread+0xcf/0xe0 [34579.789718] [] ? kthread_create_on_node+0x140/0x140 [34579.791271] [] ret_from_fork+0x58/0x90 [34579.792732] [] ? kthread_create_on_node+0x140/0x140 [34579.794287] scsi_tmf_0 S ffff8800363cca80 0 241 2 0x00000000 [34579.795917] ffff880035cc7e08 0000000000000046 ffff8800361dbec0 ffff880035cc7fd8 [34579.797636] ffff880035cc7fd8 ffff880035cc7fd8 ffff8800361dbec0 ffff8800361dbec0 [34579.799355] ffff8800363cca80 ffff8800362efc80 ffff8800363ccab0 ffff8800363cca80 [34579.801038] Call Trace: [34579.802253] [] schedule+0x29/0x70 [34579.803710] [] rescuer_thread+0x359/0x460 [34579.805216] [] ? __schedule+0x3b0/0x990 [34579.806694] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.808265] [] kthread+0xcf/0xe0 [34579.809686] [] ? kthread_create_on_node+0x140/0x140 [34579.811247] [] ret_from_fork+0x58/0x90 [34579.812698] [] ? kthread_create_on_node+0x140/0x140 [34579.814274] scsi_eh_1 S 0000000000000000 0 243 2 0x00000000 [34579.815887] ffff880035d03e18 0000000000000046 ffff8800361ade20 ffff880035d03fd8 [34579.817601] ffff880035d03fd8 ffff880035d03fd8 ffff8800361ade20 ffff880035ca9800 [34579.819303] ffff8800361ade20 ffff8800361ae498 0000000000000000 0000000000000000 [34579.820974] Call Trace: [34579.822192] [] schedule+0x29/0x70 [34579.823652] [] scsi_error_handler+0xa3/0x8b0 [34579.825200] [] ? scsi_eh_get_sense+0x250/0x250 [34579.826748] [] kthread+0xcf/0xe0 [34579.828194] [] ? kthread_create_on_node+0x140/0x140 [34579.829770] [] ret_from_fork+0x58/0x90 [34579.831247] [] ? kthread_create_on_node+0x140/0x140 [34579.832819] scsi_tmf_1 S ffff880077789980 0 246 2 0x00000000 [34579.834458] ffff880036307e08 0000000000000046 ffff8800361abec0 ffff880036307fd8 [34579.836160] ffff880036307fd8 ffff880036307fd8 ffff8800361abec0 ffff8800361abec0 [34579.837845] ffff880077789980 ffff880035cc1c80 ffff8800777899b0 ffff880077789980 [34579.839563] Call Trace: [34579.840783] [] schedule+0x29/0x70 [34579.842301] [] rescuer_thread+0x359/0x460 [34579.843830] [] ? __schedule+0x3b0/0x990 [34579.845334] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.846936] [] kthread+0xcf/0xe0 [34579.848387] [] ? kthread_create_on_node+0x140/0x140 [34579.849965] [] ret_from_fork+0x58/0x90 [34579.851433] [] ? kthread_create_on_node+0x140/0x140 [34579.853013] ttm_swap S ffff880035e5f000 0 253 2 0x00000000 [34579.854642] ffff880035e67e08 0000000000000046 ffff88003611af10 ffff880035e67fd8 [34579.856377] ffff880035e67fd8 ffff880035e67fd8 ffff88003611af10 ffff88003611af10 [34579.858071] ffff880035e5f000 ffff8800362fa680 ffff880035e5f030 ffff880035e5f000 [34579.859758] Call Trace: [34579.861029] [] schedule+0x29/0x70 [34579.862515] [] rescuer_thread+0x359/0x460 [34579.864042] [] ? __schedule+0x3b0/0x990 [34579.865540] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.867133] [] kthread+0xcf/0xe0 [34579.868569] [] ? kthread_create_on_node+0x140/0x140 [34579.870130] [] ret_from_fork+0x58/0x90 [34579.871611] [] ? kthread_create_on_node+0x140/0x140 [34579.873201] jbd2/vda1-8 S ffff880035a19800 0 272 2 0x00000000 [34579.874824] ffff880035f13e30 0000000000000046 ffff8800360f6dd0 ffff880035f13fd8 [34579.876547] ffff880035f13fd8 ffff880035f13fd8 ffff8800360f6dd0 ffff880035a19828 [34579.878285] ffff880035a19be8 ffff880035a198a8 ffff880035f13e80 ffff880035a19800 [34579.879973] Call Trace: [34579.881201] [] schedule+0x29/0x70 [34579.882678] [] kjournald2+0x203/0x260 [jbd2] [34579.884190] [] ? wake_up_atomic_t+0x30/0x30 [34579.885752] [] ? commit_timeout+0x10/0x10 [jbd2] [34579.887290] [] kthread+0xcf/0xe0 [34579.888744] [] ? kthread_create_on_node+0x140/0x140 [34579.890314] [] ret_from_fork+0x58/0x90 [34579.891799] [] ? kthread_create_on_node+0x140/0x140 [34579.893381] ext4-rsv-conver S ffff880077789780 0 273 2 0x00000000 [34579.895013] ffff880035f27e08 0000000000000046 ffff8800360f2f10 ffff880035f27fd8 [34579.896719] ffff880035f27fd8 ffff880035f27fd8 ffff8800360f2f10 ffff8800360f2f10 [34579.898447] ffff880077789780 ffff880035cc1080 ffff8800777897b0 ffff880077789780 [34579.900149] Call Trace: [34579.901383] [] schedule+0x29/0x70 [34579.902861] [] rescuer_thread+0x359/0x460 [34579.904370] [] ? __schedule+0x3b0/0x990 [34579.905885] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.907461] [] kthread+0xcf/0xe0 [34579.908893] [] ? kthread_create_on_node+0x140/0x140 [34579.910472] [] ret_from_fork+0x58/0x90 [34579.911960] [] ? kthread_create_on_node+0x140/0x140 [34579.913525] rpciod S ffff88007c343180 0 350 2 0x00000000 [34579.915158] ffff88007b8ebe08 0000000000000046 ffff88007774edd0 ffff88007b8ebfd8 [34579.916881] ffff88007b8ebfd8 ffff88007b8ebfd8 ffff88007774edd0 ffff88007774edd0 [34579.918593] ffff88007c343180 ffff8800362f9a80 ffff88007c3431b0 ffff88007c343180 [34579.920318] Call Trace: [34579.921548] [] schedule+0x29/0x70 [34579.923038] [] rescuer_thread+0x359/0x460 [34579.924568] [] ? __schedule+0x3b0/0x990 [34579.926067] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34579.927643] [] kthread+0xcf/0xe0 [34579.929076] [] ? kthread_create_on_node+0x140/0x140 [34579.930651] [] ret_from_fork+0x58/0x90 [34579.932138] [] ? kthread_create_on_node+0x140/0x140 [34579.933710] systemd-journal R running task 0 358 1 0x00000000 [34579.935376] ffff880078783db8 0000000000000082 ffff88007c36af10 ffff880078783fd8 [34579.937080] ffff880078783fd8 ffff880078783fd8 ffff88007c36af10 0000000000000000 [34579.938771] 0000000000000000 ffff880035d3aba0 ffff88007c36af10 0000000000000000 [34579.940500] Call Trace: [34579.941725] [] schedule+0x29/0x70 [34579.943241] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34579.944890] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34579.946515] [] schedule_hrtimeout_range+0x13/0x20 [34579.948095] [] ep_poll+0x23e/0x360 [34579.949547] [] ? do_truncate+0x88/0xc0 [34579.951038] [] ? __sb_start_write+0x58/0x110 [34579.952572] [] ? wake_up_state+0x20/0x20 [34579.954078] [] SyS_epoll_wait+0xed/0x120 [34579.955554] [] system_call_fastpath+0x16/0x1b [34579.957088] lvmetad S 0000000000000004 0 367 1 0x00000000 [34579.958705] ffff8800790d38e8 0000000000000082 ffff880077748fb0 ffff8800790d3fd8 [34579.960431] ffff8800790d3fd8 ffff8800790d3fd8 ffff880077748fb0 0000000000000000 [34579.962122] ffff8800790d3ad0 00000000fffffffc 0000000000000004 0000000000000004 [34579.963792] Call Trace: [34579.965075] [] schedule+0x29/0x70 [34579.966535] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34579.968175] [] ? add_wait_queue+0x3c/0x50 [34579.969671] [] ? __pollwait+0x7f/0xf0 [34579.971141] [] schedule_hrtimeout_range+0x13/0x20 [34579.972699] [] poll_schedule_timeout+0x55/0xb0 [34579.974232] [] do_select+0x6d1/0x7c0 [34579.975679] [] ? poll_select_copy_remaining+0x150/0x150 [34579.977294] [] ? radix_tree_insert+0x41/0xf0 [34579.978814] [] ? __inc_zone_page_state+0x32/0x60 [34579.980366] [] ? shmem_add_to_page_cache.isra.19+0xf0/0x170 [34579.982011] [] ? zone_statistics+0x89/0xa0 [34579.983517] [] ? list_del+0xd/0x30 [34579.984941] [] ? list_del+0xd/0x30 [34579.986382] [] ? __rmqueue+0x8a/0x460 [34579.987824] [] ? zone_statistics+0x89/0xa0 [34579.989282] [] ? __slab_free+0x81/0x2f0 [34579.990737] [] ? kvm_clock_read+0x1f/0x30 [34579.992243] [] ? sched_clock+0x9/0x10 [34579.993650] [] ? sched_clock_local+0x1d/0x80 [34579.995122] [] core_sys_select+0x1db/0x300 [34579.996576] [] ? x2apic_send_IPI_mask+0x13/0x20 [34579.998061] [] ? native_smp_send_reschedule+0x4c/0x70 [34579.999581] [] ? resched_task+0xb8/0xd0 [34580.001009] [] ? wake_up_new_task+0x10c/0x170 [34580.002474] [] ? do_fork+0xe8/0x2c0 [34580.003845] [] SyS_select+0xba/0x110 [34580.005246] [] ? SyS_clone+0x16/0x20 [34580.006625] [] system_call_fastpath+0x16/0x1b [34580.008055] systemd-udevd S ffff8800361f7ea8 0 378 1 0x00000000 [34580.009585] ffff8800361f7db8 0000000000000086 ffff880036185e20 ffff8800361f7fd8 [34580.011222] ffff8800361f7fd8 ffff8800361f7fd8 ffff880036185e20 ffff8800361f7ea8 [34580.012812] 00000000002dc6bf 0000000000000000 ffff880036185e20 ffff8800361f7ea8 [34580.014456] Call Trace: [34580.015574] [] schedule+0x29/0x70 [34580.016981] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.018494] [] ? hrtimer_get_res+0x50/0x50 [34580.019919] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.021463] [] schedule_hrtimeout_range+0x13/0x20 [34580.022950] [] ep_poll+0x23e/0x360 [34580.024325] [] ? do_unlinkat+0xf1/0x2b0 [34580.025746] [] ? wake_up_state+0x20/0x20 [34580.027177] [] SyS_epoll_wait+0xed/0x120 [34580.028577] [] system_call_fastpath+0x16/0x1b [34580.030023] vballoon S 0000000000000000 0 398 2 0x00000000 [34580.031564] ffff880035967e20 0000000000000046 ffff8800360f3ec0 ffff880035967fd8 [34580.033208] ffff880035967fd8 ffff880035967fd8 ffff8800360f3ec0 ffff880035cb7800 [34580.034789] ffff880035967e64 ffff880035cb7820 ffff8800360f3ec0 0000000000000000 [34580.036405] Call Trace: [34580.037523] [] schedule+0x29/0x70 [34580.038911] [] balloon+0x1ea/0x2ac [virtio_balloon] [34580.040407] [] ? wake_up_atomic_t+0x30/0x30 [34580.041866] [] ? update_balloon_stats+0x130/0x130 [virtio_balloon] [34580.043469] [] kthread+0xcf/0xe0 [34580.044840] [] ? kthread_create_on_node+0x140/0x140 [34580.046342] [] ret_from_fork+0x58/0x90 [34580.047766] [] ? kthread_create_on_node+0x140/0x140 [34580.049274] auditd S ffff8800361b3ea8 0 462 1 0x00000000 [34580.050837] ffff8800361b3db8 0000000000000086 ffff880077755e20 ffff8800361b3fd8 [34580.052491] ffff8800361b3fd8 ffff8800361b3fd8 ffff880077755e20 ffff8800361b3ea8 [34580.054149] 00000000038f9b17 0000000000000000 ffff880077755e20 ffff8800361b3ea8 [34580.055777] Call Trace: [34580.056999] [] schedule+0x29/0x70 [34580.058417] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.060010] [] ? hrtimer_get_res+0x50/0x50 [34580.061494] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.063131] [] schedule_hrtimeout_range+0x13/0x20 [34580.064693] [] ep_poll+0x23e/0x360 [34580.066158] [] ? wake_up_state+0x20/0x20 [34580.067651] [] SyS_epoll_wait+0xed/0x120 [34580.069148] [] system_call_fastpath+0x16/0x1b [34580.070670] auditd S ffffc9000037f780 0 466 1 0x00000000 [34580.072342] ffff88007b43bcd0 0000000000000086 ffff88003606bec0 ffff88007b43bfd8 [34580.074049] ffff88007b43bfd8 ffff88007b43bfd8 ffff88003606bec0 ffff88003606bec0 [34580.075740] ffff88003606bec0 0000000000000000 ffff88007b43bde0 ffffc9000037f780 [34580.077486] Call Trace: [34580.078729] [] schedule+0x29/0x70 [34580.080244] [] futex_wait_queue_me+0xc4/0x120 [34580.081797] [] futex_wait+0x179/0x280 [34580.083299] [] ? set_next_entity+0x3c/0xe0 [34580.084837] [] ? __switch_to+0x15c/0x4c0 [34580.086330] [] do_futex+0xfe/0x5b0 [34580.087776] [] ? wake_up_atomic_t+0x30/0x30 [34580.089315] [] ? jbd2_complete_transaction+0x52/0xa0 [jbd2] [34580.090924] [] SyS_futex+0x80/0x180 [34580.092406] [] ? do_fsync+0x76/0xa0 [34580.093853] [] system_call_fastpath+0x16/0x1b [34580.095378] dbus-daemon S 0000000000000000 0 486 1 0x00000080 [34580.097016] ffff880035a37db8 0000000000000082 ffff88007774af10 ffff880035a37fd8 [34580.098736] ffff880035a37fd8 ffff880035a37fd8 ffff88007774af10 0000000000000000 [34580.100486] 0000000000000000 ffff88007b4564e0 ffff88007774af10 0000000000000000 [34580.102214] Call Trace: [34580.103429] [] schedule+0x29/0x70 [34580.104922] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.106561] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34580.108198] [] schedule_hrtimeout_range+0x13/0x20 [34580.109777] [] ep_poll+0x23e/0x360 [34580.111255] [] ? __fput+0x183/0x260 [34580.112718] [] ? wake_up_state+0x20/0x20 [34580.114228] [] SyS_epoll_wait+0xed/0x120 [34580.115710] [] system_call_fastpath+0x16/0x1b [34580.117221] NetworkManager S ffff880035a3fbdc 0 491 1 0x00000080 [34580.118832] ffff880035a3fa30 0000000000000086 ffff8800361dde20 ffff880035a3ffd8 [34580.120542] ffff880035a3ffd8 ffff880035a3ffd8 ffff8800361dde20 ffff880035a3fb88 [34580.122250] 0000000005f5e100 0000000000000000 0000000000000000 ffff880035a3fbdc [34580.123915] Call Trace: [34580.125108] [] schedule+0x29/0x70 [34580.126554] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.128174] [] ? hrtimer_get_res+0x50/0x50 [34580.129663] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.131416] [] schedule_hrtimeout_range+0x13/0x20 [34580.132991] [] poll_schedule_timeout+0x55/0xb0 [34580.134533] [] do_sys_poll+0x4cd/0x580 [34580.136010] [] ? dequeue_task_fair+0x41e/0x660 [34580.137532] [] ? set_next_entity+0x3c/0xe0 [34580.139003] [] ? __switch_to+0x15c/0x4c0 [34580.140498] [] ? __schedule+0x3b0/0x990 [34580.141950] [] ? poll_select_copy_remaining+0x150/0x150 [34580.143561] [] ? poll_select_copy_remaining+0x150/0x150 [34580.145172] [] ? poll_select_copy_remaining+0x150/0x150 [34580.146706] [] ? poll_select_copy_remaining+0x150/0x150 [34580.148239] [] ? poll_select_copy_remaining+0x150/0x150 [34580.149746] [] ? poll_select_copy_remaining+0x150/0x150 [34580.151238] [] ? poll_select_copy_remaining+0x150/0x150 [34580.152708] [] ? poll_select_copy_remaining+0x150/0x150 [34580.154152] [] ? kvm_clock_get_cycles+0x1f/0x30 [34580.155549] [] ? ktime_get_ts64+0x4c/0xf0 [34580.156916] [] SyS_poll+0x74/0x110 [34580.158231] [] system_call_fastpath+0x16/0x1b [34580.159609] gmain S ffff88007b8d3bac 0 518 1 0x00000080 [34580.161136] ffff88007b8d3a30 0000000000000086 ffff880077754e70 ffff88007b8d3fd8 [34580.162689] ffff88007b8d3fd8 ffff88007b8d3fd8 ffff880077754e70 0000000000000000 [34580.164291] ffff88007b8d3c90 00000000fffffffc 0000000000000000 ffff88007b8d3bac [34580.165848] Call Trace: [34580.166987] [] schedule+0x29/0x70 [34580.168321] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.169818] [] ? add_wait_queue+0x3c/0x50 [34580.171194] [] schedule_hrtimeout_range+0x13/0x20 [34580.172638] [] poll_schedule_timeout+0x55/0xb0 [34580.174060] [] do_sys_poll+0x4cd/0x580 [34580.175421] [] ? kvm_clock_read+0x1f/0x30 [34580.176797] [] ? dequeue_task_fair+0x41e/0x660 [34580.178220] [] ? set_next_entity+0x3c/0xe0 [34580.179585] [] ? __switch_to+0x15c/0x4c0 [34580.180936] [] ? __schedule+0x3b0/0x990 [34580.182275] [] ? poll_select_copy_remaining+0x150/0x150 [34580.183741] [] ? poll_select_copy_remaining+0x150/0x150 [34580.185207] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [34580.186656] [] ? get_futex_key+0x1c8/0x2c0 [34580.188007] [] ? futex_wake+0x80/0x160 [34580.189332] [] ? do_futex+0x122/0x5b0 [34580.190634] [] SyS_poll+0x74/0x110 [34580.191913] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.193311] [] system_call_fastpath+0x16/0x1b [34580.194659] gdbus S ffff88007850bbb4 0 526 1 0x00000080 [34580.196148] ffff88007850ba30 0000000000000086 ffff88003606de20 ffff88007850bfd8 [34580.197653] ffff88007850bfd8 ffff88007850bfd8 ffff88003606de20 0000000000000000 [34580.199203] ffff88007850bc90 00000000fffffffc 0000000000000000 ffff88007850bbb4 [34580.200694] Call Trace: [34580.201782] [] schedule+0x29/0x70 [34580.203076] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.204524] [] ? add_wait_queue+0x3c/0x50 [34580.205886] [] ? __pollwait+0x7f/0xf0 [34580.207197] [] ? add_wait_queue+0x3c/0x50 [34580.208557] [] schedule_hrtimeout_range+0x13/0x20 [34580.209970] [] poll_schedule_timeout+0x55/0xb0 [34580.211374] [] do_sys_poll+0x4cd/0x580 [34580.212739] [] ? unix_stream_recvmsg+0x54/0x70 [34580.214181] [] ? unix_state_double_unlock+0x50/0x50 [34580.215668] [] ? sock_recvmsg+0xbf/0x100 [34580.217098] [] ? finish_task_switch+0x53/0x180 [34580.218573] [] ? update_curr+0x104/0x190 [34580.220027] [] ? __enqueue_entity+0x78/0x80 [34580.221486] [] ? poll_select_copy_remaining+0x150/0x150 [34580.223040] [] ? poll_select_copy_remaining+0x150/0x150 [34580.224573] [] ? poll_select_copy_remaining+0x150/0x150 [34580.226102] [] ? try_to_wake_up+0x1c8/0x320 [34580.227534] [] ? default_wake_function+0x12/0x20 [34580.228990] [] ? pollwake+0x73/0x90 [34580.230352] [] ? wake_up_state+0x20/0x20 [34580.231744] [] ? eventfd_ctx_read+0x67/0x210 [34580.233154] [] ? vfs_read+0xf7/0x170 [34580.234502] [] SyS_poll+0x74/0x110 [34580.235834] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.237260] [] system_call_fastpath+0x16/0x1b [34580.238660] polkitd S ffff880036273bac 0 495 1 0x00000080 [34580.240205] ffff880036273a30 0000000000000086 ffff8800360a1f60 ffff880036273fd8 [34580.241769] ffff880036273fd8 ffff880036273fd8 ffff8800360a1f60 0000000000000000 [34580.243376] ffff880036273c90 00000000fffffffc 0000000000000000 ffff880036273bac [34580.244951] Call Trace: [34580.246067] [] schedule+0x29/0x70 [34580.247437] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.248982] [] ? _raw_spin_unlock_irqrestore+0x21/0x40 [34580.250497] [] ? add_wait_queue+0x3c/0x50 [34580.251937] [] schedule_hrtimeout_range+0x13/0x20 [34580.253426] [] poll_schedule_timeout+0x55/0xb0 [34580.254905] [] do_sys_poll+0x4cd/0x580 [34580.256325] [] ? __free_pages+0x25/0x30 [34580.257753] [] ? free_pages.part.80+0x40/0x50 [34580.259234] [] ? kprobe_flush_task+0xd0/0x170 [34580.260704] [] ? finish_task_switch+0x15f/0x180 [34580.262186] [] ? update_curr+0x104/0x190 [34580.263600] [] ? __enqueue_entity+0x78/0x80 [34580.265036] [] ? poll_select_copy_remaining+0x150/0x150 [34580.266555] [] ? poll_select_copy_remaining+0x150/0x150 [34580.268056] [] ? ttwu_do_wakeup+0x19/0xd0 [34580.269461] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34580.270961] [] ? inotify_free_event+0xe/0x10 [34580.272373] [] ? kfree+0x103/0x140 [34580.273701] [] ? eventfd_ctx_read+0x67/0x210 [34580.275106] [] ? vfs_read+0xf7/0x170 [34580.276459] [] SyS_poll+0x74/0x110 [34580.277787] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.279225] [] system_call_fastpath+0x16/0x1b [34580.280633] gmain S ffff88007b8cbbac 0 516 1 0x00000080 [34580.282173] ffff88007b8cba30 0000000000000086 ffff8800360a6dd0 ffff88007b8cbfd8 [34580.283749] ffff88007b8cbfd8 ffff88007b8cbfd8 ffff8800360a6dd0 0000000000000000 [34580.285358] ffff88007b8cbc90 00000000fffffffc 0000000000000000 ffff88007b8cbbac [34580.286950] Call Trace: [34580.288081] [] schedule+0x29/0x70 [34580.289458] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.291009] [] ? add_wait_queue+0x3c/0x50 [34580.292444] [] schedule_hrtimeout_range+0x13/0x20 [34580.293943] [] poll_schedule_timeout+0x55/0xb0 [34580.295461] [] do_sys_poll+0x4cd/0x580 [34580.296904] [] ? zone_statistics+0x89/0xa0 [34580.298372] [] ? get_page_from_freelist+0x51a/0x9f0 [34580.299913] [] ? poll_select_copy_remaining+0x150/0x150 [34580.301478] [] ? poll_select_copy_remaining+0x150/0x150 [34580.303041] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [34580.304596] [] ? lru_cache_add+0xe/0x10 [34580.306043] [] ? page_add_new_anon_rmap+0x91/0x150 [34580.307572] [] ? eventfd_ctx_read+0x67/0x210 [34580.309041] [] ? vfs_read+0xf7/0x170 [34580.310453] [] SyS_poll+0x74/0x110 [34580.311846] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.313456] [] system_call_fastpath+0x16/0x1b [34580.314868] gdbus S ffff88007b8d7bac 0 519 1 0x00000080 [34580.316504] ffff88007b8d7a30 0000000000000086 ffff88007774ce70 ffff88007b8d7fd8 [34580.318173] ffff88007b8d7fd8 ffff88007b8d7fd8 ffff88007774ce70 0000000000000000 [34580.319773] ffff88007b8d7c90 00000000fffffffc 0000000000000000 ffff88007b8d7bac [34580.321430] Call Trace: [34580.322579] [] schedule+0x29/0x70 [34580.324003] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.325569] [] ? add_wait_queue+0x3c/0x50 [34580.327011] [] ? add_wait_queue+0x3c/0x50 [34580.328470] [] schedule_hrtimeout_range+0x13/0x20 [34580.329976] [] poll_schedule_timeout+0x55/0xb0 [34580.331457] [] do_sys_poll+0x4cd/0x580 [34580.332888] [] ? unix_stream_recvmsg+0x54/0x70 [34580.334376] [] ? unix_state_double_unlock+0x50/0x50 [34580.335903] [] ? sock_recvmsg+0xbf/0x100 [34580.337369] [] ? update_curr+0x104/0x190 [34580.338812] [] ? __enqueue_entity+0x78/0x80 [34580.340286] [] ? poll_select_copy_remaining+0x150/0x150 [34580.341838] [] ? poll_select_copy_remaining+0x150/0x150 [34580.343374] [] ? ttwu_do_wakeup+0x19/0xd0 [34580.344823] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34580.346378] [] ? try_to_wake_up+0x1c8/0x320 [34580.347843] [] ? default_wake_function+0x12/0x20 [34580.349345] [] ? pollwake+0x73/0x90 [34580.350746] [] ? wake_up_state+0x20/0x20 [34580.352180] [] ? eventfd_ctx_read+0x67/0x210 [34580.353620] [] ? vfs_read+0xf7/0x170 [34580.355011] [] SyS_poll+0x74/0x110 [34580.356371] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.357805] [] system_call_fastpath+0x16/0x1b [34580.359237] JS GC Helper S ffffc9000037ffc0 0 523 1 0x00000080 [34580.360774] ffff88007b8dfcd0 0000000000000086 ffff8800361aaf10 ffff88007b8dffd8 [34580.362406] ffff88007b8dffd8 ffff88007b8dffd8 ffff8800361aaf10 ffff8800361aaf10 [34580.363992] ffff8800361aaf10 0000000000000000 ffff88007b8dfde0 ffffc9000037ffc0 [34580.365583] Call Trace: [34580.366707] [] schedule+0x29/0x70 [34580.368123] [] futex_wait_queue_me+0xc4/0x120 [34580.369571] [] futex_wait+0x179/0x280 [34580.370973] [] ? page_add_file_rmap+0x8c/0xc0 [34580.372446] [] ? unlock_page+0x2b/0x30 [34580.373863] [] do_futex+0xfe/0x5b0 [34580.375237] [] ? __do_page_fault+0x171/0x450 [34580.376677] [] ? strlcpy+0x42/0x60 [34580.378057] [] SyS_futex+0x80/0x180 [34580.379426] [] ? do_page_fault+0x35/0x90 [34580.380833] [] system_call_fastpath+0x16/0x1b [34580.382269] JS Sour~ Thread S ffffc900003825c0 0 525 1 0x00000080 [34580.383812] ffff88007b897cd0 0000000000000086 ffff880036068000 ffff88007b897fd8 [34580.385440] ffff88007b897fd8 ffff88007b897fd8 ffff880036068000 ffff880036068000 [34580.387035] ffff880036068000 0000000000000000 ffff88007b897de0 ffffc900003825c0 [34580.388608] Call Trace: [34580.389773] [] schedule+0x29/0x70 [34580.391119] [] futex_wait_queue_me+0xc4/0x120 [34580.392555] [] futex_wait+0x179/0x280 [34580.393927] [] ? wake_up_state+0x10/0x20 [34580.395315] [] ? wake_futex+0x66/0x80 [34580.396692] [] ? get_futex_key+0x1c8/0x2c0 [34580.398100] [] do_futex+0xfe/0x5b0 [34580.399439] [] ? __do_page_fault+0x171/0x450 [34580.400867] [] ? strlcpy+0x42/0x60 [34580.402203] [] SyS_futex+0x80/0x180 [34580.403542] [] system_call_fastpath+0x16/0x1b [34580.404956] runaway-killer- S ffff88007850fba4 0 532 1 0x00000080 [34580.406499] ffff88007850fa30 0000000000000086 ffff880036069f60 ffff88007850ffd8 [34580.408130] ffff88007850ffd8 ffff88007850ffd8 ffff880036069f60 0000000000000000 [34580.409707] ffff88007850fc90 00000000fffffffc 0000000000000000 ffff88007850fba4 [34580.411362] Call Trace: [34580.412504] [] schedule+0x29/0x70 [34580.413923] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.415483] [] ? dequeue_task_fair+0x41e/0x660 [34580.416965] [] ? set_next_entity+0x3c/0xe0 [34580.418413] [] ? __switch_to+0x15c/0x4c0 [34580.419850] [] ? add_wait_queue+0x3c/0x50 [34580.421300] [] schedule_hrtimeout_range+0x13/0x20 [34580.422813] [] poll_schedule_timeout+0x55/0xb0 [34580.424299] [] do_sys_poll+0x4cd/0x580 [34580.425721] [] ? kvm_clock_read+0x1f/0x30 [34580.427187] [] ? zone_statistics+0x89/0xa0 [34580.428636] [] ? zone_statistics+0x89/0xa0 [34580.430048] [] ? get_page_from_freelist+0x51a/0x9f0 [34580.431546] [] ? update_curr+0x104/0x190 [34580.432963] [] ? __enqueue_entity+0x78/0x80 [34580.434373] [] ? poll_select_copy_remaining+0x150/0x150 [34580.435877] [] ? check_preempt_curr+0x85/0xa0 [34580.437306] [] ? ttwu_do_wakeup+0x19/0xd0 [34580.438711] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34580.440222] [] ? try_to_wake_up+0x1c8/0x320 [34580.441646] [] ? wake_up_state+0x10/0x20 [34580.443050] [] ? wake_futex+0x66/0x80 [34580.444422] [] ? futex_wake+0x80/0x160 [34580.445804] [] ? eventfd_ctx_read+0x67/0x210 [34580.447236] [] ? vfs_read+0xf7/0x170 [34580.448606] [] SyS_poll+0x74/0x110 [34580.449973] [] ? __audit_syscall_exit+0x1e6/0x280 [34580.451429] [] system_call_fastpath+0x16/0x1b [34580.452840] irqbalance S 00007fff71c9c8a0 0 498 1 0x00000080 [34580.454361] ffff880078f5be68 0000000000000086 ffff8800360a4e70 ffff880078f5bfd8 [34580.455956] ffff880078f5bfd8 ffff880078f5bfd8 ffff8800360a4e70 ffff8800360a4e70 [34580.457528] ffff880078f5beb8 ffff8800360a4e70 ffff880078f5bf58 00007fff71c9c8a0 [34580.459137] Call Trace: [34580.460240] [] schedule+0x29/0x70 [34580.461629] [] do_nanosleep+0x86/0xf0 [34580.463013] [] hrtimer_nanosleep+0xbb/0x180 [34580.464431] [] ? hrtimer_get_res+0x50/0x50 [34580.465856] [] ? do_nanosleep+0x72/0xf0 [34580.467251] [] SyS_nanosleep+0x7e/0x90 [34580.468629] [] system_call_fastpath+0x16/0x1b [34580.470076] chronyd S 0000000000000007 0 504 1 0x00000080 [34580.471617] ffff880078f578e8 0000000000000086 ffff8800360a3ec0 ffff880078f57fd8 [34580.473256] ffff880078f57fd8 ffff880078f57fd8 ffff8800360a3ec0 ffff880078f57ac8 [34580.474842] 0000000005f5e100 0000000000000000 0000000000000007 0000000000000007 [34580.476474] Call Trace: [34580.477602] [] schedule+0x29/0x70 [34580.479011] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.480524] [] ? hrtimer_get_res+0x50/0x50 [34580.481962] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.483502] [] schedule_hrtimeout_range+0x13/0x20 [34580.484986] [] poll_schedule_timeout+0x55/0xb0 [34580.486440] [] do_select+0x6d1/0x7c0 [34580.487849] [] ? udp_send_skb+0x13c/0x380 [34580.489267] [] ? poll_select_copy_remaining+0x150/0x150 [34580.490793] [] ? poll_select_copy_remaining+0x150/0x150 [34580.492306] [] ? poll_select_copy_remaining+0x150/0x150 [34580.493822] [] ? poll_select_copy_remaining+0x150/0x150 [34580.495304] [] ? poll_select_copy_remaining+0x150/0x150 [34580.496766] [] ? unix_dgram_sendmsg+0x3a8/0x720 [34580.498204] [] core_sys_select+0x1db/0x300 [34580.499591] [] ? ip4_datagram_connect+0x269/0x300 [34580.501044] [] ? inet_dgram_connect+0x2e/0x80 [34580.502458] [] ? kvm_clock_get_cycles+0x1f/0x30 [34580.503897] [] ? ktime_get_ts64+0x4c/0xf0 [34580.505279] [] SyS_select+0xba/0x110 [34580.506620] [] system_call_fastpath+0x16/0x1b [34580.508035] gssproxy S ffff88007b4c3ea8 0 508 1 0x00000080 [34580.509552] ffff88007b4c3db8 0000000000000086 ffff880077628fb0 ffff88007b4c3fd8 [34580.511176] ffff88007b4c3fd8 ffff88007b4c3fd8 ffff880077628fb0 ffff88007b4c3ea8 [34580.512744] 0000000001c9c37f 0000000000000000 ffff880077628fb0 ffff88007b4c3ea8 [34580.514381] Call Trace: [34580.515509] [] schedule+0x29/0x70 [34580.516919] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.518455] [] ? hrtimer_get_res+0x50/0x50 [34580.519909] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.521471] [] schedule_hrtimeout_range+0x13/0x20 [34580.522959] [] ep_poll+0x23e/0x360 [34580.524324] [] ? unix_poll+0x29/0xc0 [34580.525715] [] ? wake_up_state+0x20/0x20 [34580.527129] [] SyS_epoll_wait+0xed/0x120 [34580.528520] [] system_call_fastpath+0x16/0x1b [34580.529961] gssproxy S ffffc90000380440 0 511 1 0x00000080 [34580.531510] ffff880077a1fcd0 0000000000000086 ffff880077750000 ffff880077a1ffd8 [34580.533130] ffff880077a1ffd8 ffff880077a1ffd8 ffff880077750000 ffff880077750000 [34580.534735] ffff880077750000 0000000000000000 ffff880077a1fde0 ffffc90000380440 [34580.536402] Call Trace: [34580.537548] [] schedule+0x29/0x70 [34580.538984] [] futex_wait_queue_me+0xc4/0x120 [34580.540447] [] futex_wait+0x179/0x280 [34580.541842] [] ? default_wake_function+0x12/0x20 [34580.543318] [] ? dequeue_entity+0x11c/0x5d0 [34580.544756] [] ? dequeue_task_fair+0x41e/0x660 [34580.546196] [] do_futex+0xfe/0x5b0 [34580.547562] [] ? schedule+0x29/0x70 [34580.548920] [] SyS_futex+0x80/0x180 [34580.550264] [] system_call_fastpath+0x16/0x1b [34580.551674] gssproxy S ffffc9000037e940 0 512 1 0x00000080 [34580.553236] ffff88007b4c7cd0 0000000000000086 ffff880077753ec0 ffff88007b4c7fd8 [34580.554806] ffff88007b4c7fd8 ffff88007b4c7fd8 ffff880077753ec0 ffff880077753ec0 [34580.556420] ffff880077753ec0 0000000000000000 ffff88007b4c7de0 ffffc9000037e940 [34580.557992] Call Trace: [34580.559115] [] schedule+0x29/0x70 [34580.560497] [] futex_wait_queue_me+0xc4/0x120 [34580.561976] [] futex_wait+0x179/0x280 [34580.563396] [] ? default_wake_function+0x12/0x20 [34580.564918] [] ? dequeue_entity+0x11c/0x5d0 [34580.566401] [] ? dequeue_task_fair+0x41e/0x660 [34580.567895] [] do_futex+0xfe/0x5b0 [34580.569302] [] ? schedule+0x29/0x70 [34580.570703] [] SyS_futex+0x80/0x180 [34580.572108] [] system_call_fastpath+0x16/0x1b [34580.573580] gssproxy S ffffc9000037e8c0 0 513 1 0x00000080 [34580.575159] ffff88007b893cd0 0000000000000086 ffff880077756dd0 ffff88007b893fd8 [34580.576828] ffff88007b893fd8 ffff88007b893fd8 ffff880077756dd0 ffff880077756dd0 [34580.578490] ffff880077756dd0 0000000000000000 ffff88007b893de0 ffffc9000037e8c0 [34580.580145] Call Trace: [34580.581306] [] schedule+0x29/0x70 [34580.582773] [] futex_wait_queue_me+0xc4/0x120 [34580.584258] [] futex_wait+0x179/0x280 [34580.585670] [] do_futex+0xfe/0x5b0 [34580.587066] [] SyS_futex+0x80/0x180 [34580.588456] [] system_call_fastpath+0x16/0x1b [34580.589916] gssproxy S ffffc90000380f80 0 514 1 0x00000080 [34580.591488] ffff88007b8c3cd0 0000000000000086 ffff880077751f60 ffff88007b8c3fd8 [34580.593139] ffff88007b8c3fd8 ffff88007b8c3fd8 ffff880077751f60 ffff880077751f60 [34580.594750] ffff880077751f60 0000000000000000 ffff88007b8c3de0 ffffc90000380f80 [34580.596421] Call Trace: [34580.597584] [] schedule+0x29/0x70 [34580.599014] [] futex_wait_queue_me+0xc4/0x120 [34580.600497] [] futex_wait+0x179/0x280 [34580.601906] [] do_futex+0xfe/0x5b0 [34580.603283] [] SyS_futex+0x80/0x180 [34580.604676] [] system_call_fastpath+0x16/0x1b [34580.606125] gssproxy S ffffc900003839c0 0 515 1 0x00000080 [34580.607671] ffff88007b8c7cd0 0000000000000086 ffff880077750fb0 ffff88007b8c7fd8 [34580.609319] ffff88007b8c7fd8 ffff88007b8c7fd8 ffff880077750fb0 ffff880077750fb0 [34580.610935] ffff880077750fb0 0000000000000000 ffff88007b8c7de0 ffffc900003839c0 [34580.612593] Call Trace: [34580.613750] [] schedule+0x29/0x70 [34580.615199] [] futex_wait_queue_me+0xc4/0x120 [34580.616681] [] futex_wait+0x179/0x280 [34580.618125] [] ? __sb_end_write+0x31/0x60 [34580.619586] [] ? pipe_write+0x46a/0x5b0 [34580.621031] [] do_futex+0xfe/0x5b0 [34580.622420] [] SyS_futex+0x80/0x180 [34580.623808] [] system_call_fastpath+0x16/0x1b [34580.625269] kworker/u5:0 S ffff88007baf2800 0 536 2 0x00000080 [34580.626848] ffff88007bbd7e50 0000000000000046 ffff8800360c0000 ffff88007bbd7fd8 [34580.628495] ffff88007bbd7fd8 ffff88007bbd7fd8 ffff8800360c0000 ffff88007baf2820 [34580.630149] ffff8800360cffb0 ffff8800360c0000 ffff8800360cff80 ffff88007baf2800 [34580.631753] Call Trace: [34580.632957] [] schedule+0x29/0x70 [34580.634352] [] worker_thread+0x1d9/0x410 [34580.635801] [] ? rescuer_thread+0x460/0x460 [34580.637271] [] kthread+0xcf/0xe0 [34580.638651] [] ? kthread_create_on_node+0x140/0x140 [34580.640196] [] ret_from_fork+0x58/0x90 [34580.641657] [] ? kthread_create_on_node+0x140/0x140 [34580.643194] ib-comp-wq S ffff8800360cff00 0 537 2 0x00000080 [34580.644775] ffff88007bbdbe08 0000000000000046 ffff8800360c5e20 ffff88007bbdbfd8 [34580.646444] ffff88007bbdbfd8 ffff88007bbdbfd8 ffff8800360c5e20 ffff8800360c5e20 [34580.648097] ffff8800360cff00 ffff880077db9080 ffff8800360cff30 ffff8800360cff00 [34580.649718] Call Trace: [34580.650903] [] schedule+0x29/0x70 [34580.652301] [] rescuer_thread+0x359/0x460 [34580.653749] [] ? __schedule+0x3b0/0x990 [34580.655204] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.656737] [] kthread+0xcf/0xe0 [34580.658141] [] ? kthread_create_on_node+0x140/0x140 [34580.659679] [] ret_from_fork+0x58/0x90 [34580.661134] [] ? kthread_create_on_node+0x140/0x140 [34580.662659] ib_addr S ffff8800360cf580 0 538 2 0x00000080 [34580.664289] ffff88007bbdfe08 0000000000000046 ffff8800360c1f60 ffff88007bbdffd8 [34580.665950] ffff88007bbdffd8 ffff88007bbdffd8 ffff8800360c1f60 ffff8800360c1f60 [34580.667618] ffff8800360cf580 ffff880077db8a80 ffff8800360cf5b0 ffff8800360cf580 [34580.669289] Call Trace: [34580.670493] [] schedule+0x29/0x70 [34580.671966] [] rescuer_thread+0x359/0x460 [34580.673444] [] ? __schedule+0x3b0/0x990 [34580.674914] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.676484] [] kthread+0xcf/0xe0 [34580.677906] [] ? kthread_create_on_node+0x140/0x140 [34580.679450] [] ret_from_fork+0x58/0x90 [34580.680913] [] ? kthread_create_on_node+0x140/0x140 [34580.682462] ib_mcast S ffff8800360cf600 0 539 2 0x00000080 [34580.684066] ffff88007b4e3e08 0000000000000046 ffff8800360c0fb0 ffff88007b4e3fd8 [34580.685752] ffff88007b4e3fd8 ffff88007b4e3fd8 ffff8800360c0fb0 ffff8800360c0fb0 [34580.687443] ffff8800360cf600 ffff880077db8880 ffff8800360cf630 ffff8800360cf600 [34580.689105] Call Trace: [34580.690303] [] schedule+0x29/0x70 [34580.691748] [] rescuer_thread+0x359/0x460 [34580.693260] [] ? __schedule+0x3b0/0x990 [34580.694749] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.696325] [] kthread+0xcf/0xe0 [34580.697746] [] ? kthread_create_on_node+0x140/0x140 [34580.699305] [] ret_from_fork+0x58/0x90 [34580.700781] [] ? kthread_create_on_node+0x140/0x140 [34580.702345] ib_nl_sa_wq S ffff8800360cf400 0 540 2 0x00000080 [34580.703960] ffff88007b4e7e08 0000000000000046 ffff8800360c2f10 ffff88007b4e7fd8 [34580.705664] ffff88007b4e7fd8 ffff88007b4e7fd8 ffff8800360c2f10 ffff8800360c2f10 [34580.707374] ffff8800360cf400 ffff880077db8680 ffff8800360cf430 ffff8800360cf400 [34580.709057] Call Trace: [34580.710271] [] schedule+0x29/0x70 [34580.711753] [] rescuer_thread+0x359/0x460 [34580.713263] [] ? __schedule+0x3b0/0x990 [34580.714738] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.716317] [] kthread+0xcf/0xe0 [34580.717740] [] ? kthread_create_on_node+0x140/0x140 [34580.719301] [] ret_from_fork+0x58/0x90 [34580.720790] [] ? kthread_create_on_node+0x140/0x140 [34580.722341] iw_cm_wq S ffff8800363cc180 0 552 2 0x00000080 [34580.723970] ffff88007bbcfe08 0000000000000046 ffff8800361a8fb0 ffff88007bbcffd8 [34580.725690] ffff88007bbcffd8 ffff88007bbcffd8 ffff8800361a8fb0 ffff8800361a8fb0 [34580.727425] ffff8800363cc180 ffff880077db8280 ffff8800363cc1b0 ffff8800363cc180 [34580.729113] Call Trace: [34580.730329] [] schedule+0x29/0x70 [34580.731798] [] rescuer_thread+0x359/0x460 [34580.733311] [] ? __schedule+0x3b0/0x990 [34580.734808] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.736371] [] kthread+0xcf/0xe0 [34580.737803] [] ? kthread_create_on_node+0x140/0x140 [34580.739366] [] ret_from_fork+0x58/0x90 [34580.740852] [] ? kthread_create_on_node+0x140/0x140 [34580.742417] ib_cm S ffff8800363cc880 0 557 2 0x00000080 [34580.744048] ffff880079fcbe08 0000000000000046 ffff8800361aedd0 ffff880079fcbfd8 [34580.745766] ffff880079fcbfd8 ffff880079fcbfd8 ffff8800361aedd0 ffff8800361aedd0 [34580.747495] ffff8800363cc880 ffff880077d8ba80 ffff8800363cc8b0 ffff8800363cc880 [34580.749224] Call Trace: [34580.750456] [] schedule+0x29/0x70 [34580.751946] [] rescuer_thread+0x359/0x460 [34580.753457] [] ? __schedule+0x3b0/0x990 [34580.754968] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.756544] [] kthread+0xcf/0xe0 [34580.757979] [] ? kthread_create_on_node+0x140/0x140 [34580.759540] [] ret_from_fork+0x58/0x90 [34580.761023] [] ? kthread_create_on_node+0x140/0x140 [34580.762589] rdma_cm S ffff8800363ccc80 0 562 2 0x00000080 [34580.764261] ffff880079877e08 0000000000000046 ffff880079fd0000 ffff880079877fd8 [34580.765961] ffff880079877fd8 ffff880079877fd8 ffff880079fd0000 ffff880079fd0000 [34580.767684] ffff8800363ccc80 ffff880077a03c80 ffff8800363cccb0 ffff8800363ccc80 [34580.769418] Call Trace: [34580.770650] [] schedule+0x29/0x70 [34580.772154] [] rescuer_thread+0x359/0x460 [34580.773670] [] ? __schedule+0x3b0/0x990 [34580.775173] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.776741] [] kthread+0xcf/0xe0 [34580.778175] [] ? kthread_create_on_node+0x140/0x140 [34580.779741] [] ret_from_fork+0x58/0x90 [34580.781221] [] ? kthread_create_on_node+0x140/0x140 [34580.782791] ipoib_flush S ffff88007b4ef700 0 583 2 0x00000080 [34580.784434] ffff88007977be08 0000000000000046 ffff880079fd1f60 ffff88007977bfd8 [34580.786135] ffff88007977bfd8 ffff88007977bfd8 ffff880079fd1f60 ffff880079fd1f60 [34580.787826] ffff88007b4ef700 ffff880077a02880 ffff88007b4ef730 ffff88007b4ef700 [34580.789540] Call Trace: [34580.790772] [] schedule+0x29/0x70 [34580.792277] [] rescuer_thread+0x359/0x460 [34580.793792] [] ? __schedule+0x3b0/0x990 [34580.795291] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.796874] [] kthread+0xcf/0xe0 [34580.798305] [] ? kthread_create_on_node+0x140/0x140 [34580.799870] [] ret_from_fork+0x58/0x90 [34580.801359] [] ? kthread_create_on_node+0x140/0x140 [34580.802933] srp_remove S ffff88007b4efc00 0 588 2 0x00000080 [34580.804574] ffff88007977fe08 0000000000000046 ffff880079fd0fb0 ffff88007977ffd8 [34580.806310] ffff88007977ffd8 ffff88007977ffd8 ffff880079fd0fb0 ffff880079fd0fb0 [34580.808010] ffff88007b4efc00 ffff880077a03680 ffff88007b4efc30 ffff88007b4efc00 [34580.809723] Call Trace: [34580.810991] [] schedule+0x29/0x70 [34580.812449] [] rescuer_thread+0x359/0x460 [34580.813977] [] ? __schedule+0x3b0/0x990 [34580.815475] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.817064] [] kthread+0xcf/0xe0 [34580.818505] [] ? kthread_create_on_node+0x140/0x140 [34580.820093] [] ret_from_fork+0x58/0x90 [34580.821571] [] ? kthread_create_on_node+0x140/0x140 [34580.823132] target_completi S ffff88007b4eff80 0 594 2 0x00000080 [34580.824754] ffff880078f67e08 0000000000000046 ffff880079fd4e70 ffff880078f67fd8 [34580.826493] ffff880078f67fd8 ffff880078f67fd8 ffff880079fd4e70 ffff880079fd4e70 [34580.828241] ffff88007b4eff80 ffff880077a03080 ffff88007b4effb0 ffff88007b4eff80 [34580.829922] Call Trace: [34580.831206] [] schedule+0x29/0x70 [34580.833230] [] rescuer_thread+0x359/0x460 [34580.834756] [] ? __schedule+0x3b0/0x990 [34580.836260] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.837842] [] kthread+0xcf/0xe0 [34580.839262] [] ? kthread_create_on_node+0x140/0x140 [34580.840847] [] ret_from_fork+0x58/0x90 [34580.842330] [] ? kthread_create_on_node+0x140/0x140 [34580.843905] tmr-rd_mcp S ffff88007b4ef500 0 595 2 0x00000080 [34580.845545] ffff88007b50be08 0000000000000046 ffff880079fd5e20 ffff88007b50bfd8 [34580.847284] ffff88007b50bfd8 ffff88007b50bfd8 ffff880079fd5e20 ffff880079fd5e20 [34580.848970] ffff88007b4ef500 ffff880077a03480 ffff88007b4ef530 ffff88007b4ef500 [34580.850690] Call Trace: [34580.851914] [] schedule+0x29/0x70 [34580.853412] [] rescuer_thread+0x359/0x460 [34580.854935] [] ? __schedule+0x3b0/0x990 [34580.856437] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.858009] [] kthread+0xcf/0xe0 [34580.859432] [] ? kthread_create_on_node+0x140/0x140 [34580.861009] [] ret_from_fork+0x58/0x90 [34580.862488] [] ? kthread_create_on_node+0x140/0x140 [34580.864057] xcopy_wq S ffff88007b4ef080 0 596 2 0x00000080 [34580.865674] ffff88007b513e08 0000000000000046 ffff880079fd3ec0 ffff88007b513fd8 [34580.867404] ffff88007b513fd8 ffff88007b513fd8 ffff880079fd3ec0 ffff880079fd3ec0 [34580.869098] ffff88007b4ef080 ffff880077a03e80 ffff88007b4ef0b0 ffff88007b4ef080 [34580.870774] Call Trace: [34580.872033] [] schedule+0x29/0x70 [34580.873467] [] rescuer_thread+0x359/0x460 [34580.875012] [] ? __schedule+0x3b0/0x990 [34580.876516] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.878085] [] kthread+0xcf/0xe0 [34580.879515] [] ? kthread_create_on_node+0x140/0x140 [34580.881095] [] ret_from_fork+0x58/0x90 [34580.882577] [] ? kthread_create_on_node+0x140/0x140 [34580.884148] iscsi_eh S ffff88007b4efd00 0 601 2 0x00000080 [34580.885775] ffff880078f77e08 0000000000000046 ffff880079fd6dd0 ffff880078f77fd8 [34580.887509] ffff880078f77fd8 ffff880078f77fd8 ffff880079fd6dd0 ffff880079fd6dd0 [34580.889237] ffff88007b4efd00 ffff8800777de880 ffff88007b4efd30 ffff88007b4efd00 [34580.890916] Call Trace: [34580.892172] [] schedule+0x29/0x70 [34580.893636] [] rescuer_thread+0x359/0x460 [34580.895156] [] ? __schedule+0x3b0/0x990 [34580.896653] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.898226] [] kthread+0xcf/0xe0 [34580.899653] [] ? kthread_create_on_node+0x140/0x140 [34580.901227] [] ret_from_fork+0x58/0x90 [34580.902700] [] ? kthread_create_on_node+0x140/0x140 [34580.904271] frwr_recovery S ffff880079000480 0 610 2 0x00000080 [34580.905909] ffff88007b517e08 0000000000000046 ffff880078f68fb0 ffff88007b517fd8 [34580.907629] ffff88007b517fd8 ffff88007b517fd8 ffff880078f68fb0 ffff880078f68fb0 [34580.909346] ffff880079000480 ffff8800777dee80 ffff8800790004b0 ffff880079000480 [34580.911037] Call Trace: [34580.912259] [] schedule+0x29/0x70 [34580.913728] [] rescuer_thread+0x359/0x460 [34580.915248] [] ? __schedule+0x3b0/0x990 [34580.916744] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.918317] [] kthread+0xcf/0xe0 [34580.919740] [] ? kthread_create_on_node+0x140/0x140 [34580.921313] [] ret_from_fork+0x58/0x90 [34580.922796] [] ? kthread_create_on_node+0x140/0x140 [34580.924359] xprtrdma_receiv S ffff880079000580 0 611 2 0x00000080 [34580.925984] ffff8800784c3e08 0000000000000046 ffff880078f6bec0 ffff8800784c3fd8 [34580.927699] ffff8800784c3fd8 ffff8800784c3fd8 ffff880078f6bec0 ffff880078f6bec0 [34580.929423] ffff880079000580 ffff8800777dea80 ffff8800790005b0 ffff880079000580 [34580.931124] Call Trace: [34580.932355] [] schedule+0x29/0x70 [34580.933828] [] rescuer_thread+0x359/0x460 [34580.935338] [] ? __schedule+0x3b0/0x990 [34580.936832] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34580.938411] [] kthread+0xcf/0xe0 [34580.939836] [] ? kthread_create_on_node+0x140/0x140 [34580.941407] [] ret_from_fork+0x58/0x90 [34580.942897] [] ? kthread_create_on_node+0x140/0x140 [34580.944459] rpc.gssd S 0000000000000000 0 622 1 0x00000080 [34580.946087] ffff88007772fdb8 0000000000000082 ffff880078f68000 ffff88007772ffd8 [34580.947806] ffff88007772ffd8 ffff88007772ffd8 ffff880078f68000 0000000000000000 [34580.949528] 0000000000000000 ffff880035c3e120 ffff880078f68000 0000000000000000 [34580.951250] Call Trace: [34580.952467] [] schedule+0x29/0x70 [34580.953970] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34580.955602] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34580.957228] [] schedule_hrtimeout_range+0x13/0x20 [34580.958805] [] ep_poll+0x23e/0x360 [34580.960271] [] ? wake_up_state+0x20/0x20 [34580.961768] [] SyS_epoll_wait+0xed/0x120 [34580.963313] [] system_call_fastpath+0x16/0x1b [34580.964837] sssd S ffff880079e57ea8 0 625 1 0x00000080 [34580.966421] ffff880079e57db8 0000000000000082 ffff880036180000 ffff880079e57fd8 [34580.968124] ffff880079e57fd8 ffff880079e57fd8 ffff880036180000 ffff880079e57ea8 [34580.969846] 0000000001c9c37f 0000000000000000 ffff880036180000 ffff880079e57ea8 [34580.971522] Call Trace: [34580.972782] [] schedule+0x29/0x70 [34580.974270] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.975903] [] ? hrtimer_get_res+0x50/0x50 [34580.977418] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34580.979059] [] schedule_hrtimeout_range+0x13/0x20 [34580.980613] [] ep_poll+0x23e/0x360 [34580.982067] [] ? wake_up_state+0x20/0x20 [34580.983589] [] SyS_epoll_wait+0xed/0x120 [34580.985081] [] system_call_fastpath+0x16/0x1b [34580.986603] sssd_be S ffff88003614fea8 0 628 625 0x00000080 [34580.988227] ffff88003614fdb8 0000000000000082 ffff880036186dd0 ffff88003614ffd8 [34580.989948] ffff88003614ffd8 ffff88003614ffd8 ffff880036186dd0 ffff88003614fea8 [34580.991647] 00000000005e0a37 0000000000000000 ffff880036186dd0 ffff88003614fea8 [34580.993375] Call Trace: [34580.994635] [] schedule+0x29/0x70 [34580.996143] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34580.997783] [] ? hrtimer_get_res+0x50/0x50 [34580.999323] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.000962] [] schedule_hrtimeout_range+0x13/0x20 [34581.002539] [] ep_poll+0x23e/0x360 [34581.004015] [] ? wake_up_state+0x20/0x20 [34581.005509] [] SyS_epoll_wait+0xed/0x120 [34581.007008] [] system_call_fastpath+0x16/0x1b [34581.008531] sssd_nss S ffff880000017ea8 0 641 625 0x00000080 [34581.010174] ffff880000017db8 0000000000000082 ffff88007774de20 ffff880000017fd8 [34581.011908] ffff880000017fd8 ffff880000017fd8 ffff88007774de20 ffff880000017ea8 [34581.013587] 000000000032a0ef 0000000000000000 ffff88007774de20 ffff880000017ea8 [34581.015304] Call Trace: [34581.016575] [] schedule+0x29/0x70 [34581.018075] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.019684] [] ? hrtimer_get_res+0x50/0x50 [34581.021199] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.022836] [] schedule_hrtimeout_range+0x13/0x20 [34581.024420] [] ep_poll+0x23e/0x360 [34581.025898] [] ? wake_up_state+0x20/0x20 [34581.027397] [] SyS_epoll_wait+0xed/0x120 [34581.028910] [] system_call_fastpath+0x16/0x1b [34581.030457] sssd_pam S ffff8800787c7ea8 0 642 625 0x00000080 [34581.032098] ffff8800787c7db8 0000000000000082 ffff8800360c4e70 ffff8800787c7fd8 [34581.033827] ffff8800787c7fd8 ffff8800787c7fd8 ffff8800360c4e70 ffff8800787c7ea8 [34581.035500] 000000000098967f 0000000000000000 ffff8800360c4e70 ffff8800787c7ea8 [34581.037236] Call Trace: [34581.038500] [] schedule+0x29/0x70 [34581.040007] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.041624] [] ? hrtimer_get_res+0x50/0x50 [34581.043155] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.044800] [] schedule_hrtimeout_range+0x13/0x20 [34581.046369] [] ep_poll+0x23e/0x360 [34581.047835] [] ? wake_up_state+0x20/0x20 [34581.049335] [] SyS_epoll_wait+0xed/0x120 [34581.050845] [] system_call_fastpath+0x16/0x1b [34581.052372] sssd_ssh S ffff880077d27ea8 0 643 625 0x00000080 [34581.053999] ffff880077d27db8 0000000000000086 ffff880079fd2f10 ffff880077d27fd8 [34581.055681] ffff880077d27fd8 ffff880077d27fd8 ffff880079fd2f10 ffff880077d27ea8 [34581.057407] 000000000098967f 0000000000000000 ffff880079fd2f10 ffff880077d27ea8 [34581.059118] Call Trace: [34581.060383] [] schedule+0x29/0x70 [34581.061863] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.063482] [] ? hrtimer_get_res+0x50/0x50 [34581.065012] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.066656] [] schedule_hrtimeout_range+0x13/0x20 [34581.068240] [] ep_poll+0x23e/0x360 [34581.069705] [] ? wake_up_state+0x20/0x20 [34581.071206] [] SyS_epoll_wait+0xed/0x120 [34581.072700] [] system_call_fastpath+0x16/0x1b [34581.074231] sssd_pac S ffff880077807ea8 0 644 625 0x00000080 [34581.075855] ffff880077807db8 0000000000000082 ffff8800360a2f10 ffff880077807fd8 [34581.077587] ffff880077807fd8 ffff880077807fd8 ffff8800360a2f10 ffff880077807ea8 [34581.079315] 000000000098967f 0000000000000000 ffff8800360a2f10 ffff880077807ea8 [34581.081002] Call Trace: [34581.082273] [] schedule+0x29/0x70 [34581.083709] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.085338] [] ? hrtimer_get_res+0x50/0x50 [34581.086867] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.088512] [] schedule_hrtimeout_range+0x13/0x20 [34581.090082] [] ep_poll+0x23e/0x360 [34581.091550] [] ? wake_up_state+0x20/0x20 [34581.093059] [] SyS_epoll_wait+0xed/0x120 [34581.094569] [] system_call_fastpath+0x16/0x1b [34581.096107] dhclient S 0000000000000007 0 646 491 0x00000080 [34581.097732] ffff88007969b8e8 0000000000000086 ffff880036181f60 ffff88007969bfd8 [34581.099450] ffff88007969bfd8 ffff88007969bfd8 ffff880036181f60 ffff88007969bac8 [34581.101153] 0000000005f5e100 0000000000000000 0000000000000007 0000000000000007 [34581.102857] Call Trace: [34581.104158] [] schedule+0x29/0x70 [34581.105621] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.107244] [] ? hrtimer_get_res+0x50/0x50 [34581.108777] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.110437] [] schedule_hrtimeout_range+0x13/0x20 [34581.112014] [] poll_schedule_timeout+0x55/0xb0 [34581.113565] [] do_select+0x6d1/0x7c0 [34581.115061] [] ? poll_select_copy_remaining+0x150/0x150 [34581.116680] [] ? poll_select_copy_remaining+0x150/0x150 [34581.118300] [] ? kmem_cache_free+0x1bb/0x1f0 [34581.119825] [] ? kfree_skbmem+0x37/0x90 [34581.121313] [] ? __kfree_skb+0x1a/0x20 [34581.122791] [] ? skb_free_datagram_locked+0x93/0xd0 [34581.124367] [] ? udp_recvmsg+0x29e/0x490 [34581.125866] [] ? inet_recvmsg+0x7b/0xa0 [34581.127332] [] ? sock_recvmsg+0xbf/0x100 [34581.128778] [] core_sys_select+0x1db/0x300 [34581.130260] [] ? handle_mm_fault+0x72c/0xfe0 [34581.131749] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.133256] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.134733] [] ? ktime_get_ts64+0x4c/0xf0 [34581.136173] [] SyS_select+0xba/0x110 [34581.137583] [] system_call_fastpath+0x16/0x1b [34581.139038] systemd-logind S 0000000000000000 0 660 1 0x00000080 [34581.140595] ffff8800778a3db8 0000000000000082 ffff8800360a5e20 ffff8800778a3fd8 [34581.142243] ffff8800778a3fd8 ffff8800778a3fd8 ffff8800360a5e20 0000000000000000 [34581.143864] 0000000000000000 ffff8800775097e0 ffff8800360a5e20 0000000000000000 [34581.145527] Call Trace: [34581.146679] [] schedule+0x29/0x70 [34581.148107] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.149651] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34581.151193] [] schedule_hrtimeout_range+0x13/0x20 [34581.152697] [] ep_poll+0x23e/0x360 [34581.154082] [] ? wake_up_state+0x20/0x20 [34581.155504] [] SyS_epoll_wait+0xed/0x120 [34581.156941] [] system_call_fastpath+0x16/0x1b [34581.158389] kworker/1:1H S ffff88007fd16840 0 696 2 0x00000080 [34581.159947] ffff880077513e50 0000000000000046 ffff880078f6af10 ffff880077513fd8 [34581.161594] ffff880077513fd8 ffff880077513fd8 ffff880078f6af10 ffff88007fd16860 [34581.163243] ffff880079ea83b0 ffff880078f6af10 ffff880079ea8380 ffff88007fd16840 [34581.164846] Call Trace: [34581.166032] [] schedule+0x29/0x70 [34581.167403] [] worker_thread+0x1d9/0x410 [34581.168858] [] ? rescuer_thread+0x460/0x460 [34581.170293] [] kthread+0xcf/0xe0 [34581.171663] [] ? kthread_create_on_node+0x140/0x140 [34581.173160] [] ret_from_fork+0x58/0x90 [34581.174555] [] ? kthread_create_on_node+0x140/0x140 [34581.176041] rsyslogd S 0000000000000000 0 892 1 0x00000080 [34581.177575] ffff88007851b8e8 0000000000000086 ffff8800361dce70 ffff88007851bfd8 [34581.179216] ffff88007851bfd8 ffff88007851bfd8 ffff8800361dce70 ffff88007851bac8 [34581.180799] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [34581.182454] Call Trace: [34581.183596] [] schedule+0x29/0x70 [34581.185012] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.186545] [] ? hrtimer_get_res+0x50/0x50 [34581.188013] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.189586] [] schedule_hrtimeout_range+0x13/0x20 [34581.191111] [] poll_schedule_timeout+0x55/0xb0 [34581.192611] [] do_select+0x6d1/0x7c0 [34581.194041] [] ? ep_poll_callback+0xee/0x210 [34581.195528] [] ? __wake_up_sync_key+0x4f/0x60 [34581.197030] [] ? sock_def_readable+0x3a/0x70 [34581.198522] [] ? unix_dgram_sendmsg+0x3a8/0x720 [34581.200050] [] ? sock_sendmsg+0xb0/0xf0 [34581.201507] [] ? __switch_to+0x15c/0x4c0 [34581.202969] [] ? __schedule+0x3b0/0x990 [34581.204450] [] core_sys_select+0x1db/0x300 [34581.205951] [] ? __d_free+0x35/0x40 [34581.207385] [] ? d_free+0x60/0x70 [34581.208802] [] ? dentry_kill+0x14e/0x1b0 [34581.210253] [] ? mntput_no_expire+0x3e/0x120 [34581.211726] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.213229] [] ? ktime_get_ts64+0x4c/0xf0 [34581.214684] [] SyS_select+0xba/0x110 [34581.216092] [] system_call_fastpath+0x16/0x1b [34581.217554] in:imjournal S ffff88007b9e3ba4 0 903 1 0x00000080 [34581.219133] ffff88007b9e3a30 0000000000000086 ffff8800361ace70 ffff88007b9e3fd8 [34581.220801] ffff88007b9e3fd8 ffff88007b9e3fd8 ffff8800361ace70 0000000000000000 [34581.222464] ffff88007b9e3c90 00000000fffffffc 0000000000000000 ffff88007b9e3ba4 [34581.224139] Call Trace: [34581.225311] [] schedule+0x29/0x70 [34581.226765] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.228373] [] ? ext4_free_blocks+0x67e/0xb80 [ext4] [34581.229900] [] ? add_wait_queue+0x3c/0x50 [34581.231373] [] schedule_hrtimeout_range+0x13/0x20 [34581.232905] [] poll_schedule_timeout+0x55/0xb0 [34581.234427] [] do_sys_poll+0x4cd/0x580 [34581.235884] [] ? jbd2_journal_release_jbd_inode+0x100/0x130 [jbd2] [34581.237537] [] ? __wake_up+0x44/0x50 [34581.238991] [] ? update_curr+0x104/0x190 [34581.240463] [] ? __enqueue_entity+0x78/0x80 [34581.241956] [] ? enqueue_entity+0x26c/0xb60 [34581.243439] [] ? kvm_clock_read+0x1f/0x30 [34581.244910] [] ? check_preempt_wakeup+0x11d/0x250 [34581.246431] [] ? poll_select_copy_remaining+0x150/0x150 [34581.248009] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34581.249560] [] ? call_rcu_sched+0x1d/0x20 [34581.251015] [] ? d_free+0x4d/0x70 [34581.252432] [] ? wake_up_state+0x10/0x20 [34581.253885] [] ? wake_futex+0x66/0x80 [34581.255284] [] ? futex_wake_op+0x332/0x570 [34581.256715] [] ? do_futex+0x172/0x5b0 [34581.258120] [] ? wake_up_atomic_t+0x30/0x30 [34581.259541] [] SyS_poll+0x74/0x110 [34581.260897] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.262368] [] system_call_fastpath+0x16/0x1b [34581.263796] rs:main Q:Reg S ffffc9000037ec00 0 904 1 0x00000080 [34581.265355] ffff88007b493cd0 0000000000000086 ffff880036184e70 ffff88007b493fd8 [34581.266958] ffff88007b493fd8 ffff88007b493fd8 ffff880036184e70 ffff880036184e70 [34581.268566] ffff880036184e70 0000000000000000 ffff88007b493de0 ffffc9000037ec00 [34581.270209] Call Trace: [34581.271360] [] schedule+0x29/0x70 [34581.272794] [] futex_wait_queue_me+0xc4/0x120 [34581.274288] [] futex_wait+0x179/0x280 [34581.275711] [] ? ext4_file_write+0xdb/0x470 [ext4] [34581.277231] [] do_futex+0xfe/0x5b0 [34581.278644] [] SyS_futex+0x80/0x180 [34581.280068] [] system_call_fastpath+0x16/0x1b [34581.281565] tuned S 0000000000000000 0 894 1 0x00000080 [34581.283161] ffff880079ebf8e8 0000000000000086 ffff8800360f1f60 ffff880079ebffd8 [34581.284854] ffff880079ebffd8 ffff880079ebffd8 ffff8800360f1f60 ffff880079ebfac8 [34581.286536] 0000000005f5e100 0000000000000000 0000000000000000 0000000000000000 [34581.288245] Call Trace: [34581.289440] [] schedule+0x29/0x70 [34581.290895] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.292489] [] ? hrtimer_get_res+0x50/0x50 [34581.293988] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.295582] [] schedule_hrtimeout_range+0x13/0x20 [34581.297118] [] poll_schedule_timeout+0x55/0xb0 [34581.298624] [] do_select+0x6d1/0x7c0 [34581.300061] [] ? __wake_up+0x44/0x50 [34581.301501] [] ? ext4_write_end+0x21a/0x340 [ext4] [34581.303021] [] ? __ext4_journal_stop+0x3c/0xb0 [ext4] [34581.304577] [] ? update_curr+0x104/0x190 [34581.306010] [] ? account_entity_dequeue+0xae/0xd0 [34581.307536] [] ? dequeue_entity+0x11c/0x5d0 [34581.309008] [] ? kvm_clock_read+0x1f/0x30 [34581.310451] [] ? dequeue_task_fair+0x41e/0x660 [34581.311947] [] ? set_next_entity+0x3c/0xe0 [34581.313405] [] ? __switch_to+0x15c/0x4c0 [34581.314855] [] ? unlock_page+0x2b/0x30 [34581.316278] [] ? do_wp_page+0x17b/0x530 [34581.317699] [] ? handle_mm_fault+0x72c/0xfe0 [34581.319156] [] core_sys_select+0x1db/0x300 [34581.320589] [] ? do_page_fault+0x35/0x90 [34581.322022] [] ? wake_futex+0x66/0x80 [34581.323425] [] ? handle_mm_fault+0x72c/0xfe0 [34581.324886] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.326370] [] ? ktime_get_ts64+0x4c/0xf0 [34581.327818] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.329306] [] ? ktime_get_ts64+0x4c/0xf0 [34581.330739] [] SyS_select+0xba/0x110 [34581.332156] [] system_call_fastpath+0x16/0x1b [34581.333627] gmain S ffff880077503ba4 0 965 1 0x00000080 [34581.335230] ffff880077503a30 0000000000000086 ffff880078f10fb0 ffff880077503fd8 [34581.336843] ffff880077503fd8 ffff880077503fd8 ffff880078f10fb0 0000000000000000 [34581.338487] ffff880077503c90 00000000fffffffc 0000000000000000 ffff880077503ba4 [34581.340111] Call Trace: [34581.341252] [] schedule+0x29/0x70 [34581.342694] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.344254] [] ? zone_statistics+0x89/0xa0 [34581.345707] [] ? add_wait_queue+0x3c/0x50 [34581.347148] [] schedule_hrtimeout_range+0x13/0x20 [34581.348652] [] poll_schedule_timeout+0x55/0xb0 [34581.350143] [] do_sys_poll+0x4cd/0x580 [34581.351565] [] ? zone_statistics+0x89/0xa0 [34581.353018] [] ? get_page_from_freelist+0x51a/0x9f0 [34581.354520] [] ? poll_select_copy_remaining+0x150/0x150 [34581.356071] [] ? filemap_fault+0x215/0x410 [34581.357511] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [34581.359024] [] ? unlock_page+0x2b/0x30 [34581.360439] [] ? do_read_fault.isra.42+0xe4/0x130 [34581.361925] [] ? eventfd_ctx_read+0x67/0x210 [34581.363379] [] ? vfs_read+0xf7/0x170 [34581.364764] [] SyS_poll+0x74/0x110 [34581.366128] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.367597] [] system_call_fastpath+0x16/0x1b [34581.369031] tuned S ffff8800787abbac 0 967 1 0x00000080 [34581.370583] ffff8800787aba30 0000000000000086 ffff880078f13ec0 ffff8800787abfd8 [34581.372241] ffff8800787abfd8 ffff8800787abfd8 ffff880078f13ec0 0000000000000000 [34581.373861] ffff8800787abc90 00000000fffffffc 0000000000000000 ffff8800787abbac [34581.375529] Call Trace: [34581.376680] [] schedule+0x29/0x70 [34581.378114] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.379678] [] ? add_wait_queue+0x3c/0x50 [34581.381148] [] ? add_wait_queue+0x3c/0x50 [34581.382590] [] schedule_hrtimeout_range+0x13/0x20 [34581.384107] [] poll_schedule_timeout+0x55/0xb0 [34581.385609] [] do_sys_poll+0x4cd/0x580 [34581.387058] [] ? kvm_clock_read+0x1f/0x30 [34581.388514] [] ? dequeue_task_fair+0x41e/0x660 [34581.390015] [] ? set_next_entity+0x3c/0xe0 [34581.391502] [] ? __switch_to+0x15c/0x4c0 [34581.392956] [] ? __schedule+0x3b0/0x990 [34581.394406] [] ? poll_select_copy_remaining+0x150/0x150 [34581.395978] [] ? poll_select_copy_remaining+0x150/0x150 [34581.397534] [] ? x2apic_send_IPI_mask+0x13/0x20 [34581.399037] [] ? try_to_wake_up+0x1c8/0x320 [34581.400506] [] ? wake_up_state+0x10/0x20 [34581.401965] [] ? wake_futex+0x66/0x80 [34581.403388] [] ? futex_wake+0x80/0x160 [34581.404799] [] ? do_futex+0x122/0x5b0 [34581.406188] [] ? wake_up_state+0x20/0x20 [34581.407602] [] SyS_poll+0x74/0x110 [34581.408962] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.410407] [] system_call_fastpath+0x16/0x1b [34581.411835] tuned S 0000000000000000 0 968 1 0x00000080 [34581.413361] ffff880035a4f8e8 0000000000000086 ffff880078f11f60 ffff880035a4ffd8 [34581.414950] ffff880035a4ffd8 ffff880035a4ffd8 ffff880078f11f60 ffff880035a4fac8 [34581.416542] 00000000000f423c 0000000000000000 0000000000000000 0000000000000000 [34581.418170] Call Trace: [34581.419294] [] schedule+0x29/0x70 [34581.420691] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.422230] [] ? hrtimer_get_res+0x50/0x50 [34581.423669] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.425307] [] schedule_hrtimeout_range+0x13/0x20 [34581.426795] [] poll_schedule_timeout+0x55/0xb0 [34581.428274] [] do_select+0x6d1/0x7c0 [34581.429665] [] ? ep_poll_callback+0xee/0x210 [34581.431125] [] ? __wake_up_common+0x58/0x90 [34581.432567] [] ? __wake_up_sync_key+0x4f/0x60 [34581.434019] [] ? sock_def_readable+0x3a/0x70 [34581.435462] [] ? unix_stream_sendmsg+0x3fd/0x430 [34581.436933] [] ? sock_sendmsg+0xb0/0xf0 [34581.438333] [] core_sys_select+0x1db/0x300 [34581.439722] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.441178] [] ? ktime_get_ts64+0x4c/0xf0 [34581.442572] [] ? kvm_clock_get_cycles+0x1f/0x30 [34581.444009] [] ? ktime_get_ts64+0x4c/0xf0 [34581.445382] [] SyS_select+0xba/0x110 [34581.446718] [] system_call_fastpath+0x16/0x1b [34581.448120] tuned S 0000000000000000 0 970 1 0x00000080 [34581.449614] ffff880078f27db8 0000000000000086 ffff880079a22f10 ffff880078f27fd8 [34581.451225] ffff880078f27fd8 ffff880078f27fd8 ffff880079a22f10 0000000000000000 [34581.452791] 0000000000000000 ffff88007789e360 ffff880079a22f10 0000000000000000 [34581.454404] Call Trace: [34581.455508] [] schedule+0x29/0x70 [34581.456908] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.458423] [] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0 [34581.459939] [] schedule_hrtimeout_range+0x13/0x20 [34581.461413] [] ep_poll+0x23e/0x360 [34581.462785] [] ? __fput+0x183/0x260 [34581.464164] [] ? wake_up_state+0x20/0x20 [34581.465588] [] SyS_epoll_wait+0xed/0x120 [34581.467014] [] system_call_fastpath+0x16/0x1b [34581.468463] xinetd S ffff88007b45bbb4 0 905 1 0x00000080 [34581.470018] ffff88007b45ba30 0000000000000082 ffff880036180fb0 ffff88007b45bfd8 [34581.471676] ffff88007b45bfd8 ffff88007b45bfd8 ffff880036180fb0 0000000000000000 [34581.473350] ffff88007b45bc90 00000000fffffffc 0000000000000000 ffff88007b45bbb4 [34581.474999] Call Trace: [34581.476218] [] schedule+0x29/0x70 [34581.477620] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.479251] [] ? add_wait_queue+0x3c/0x50 [34581.480742] [] ? add_wait_queue+0x3c/0x50 [34581.482212] [] schedule_hrtimeout_range+0x13/0x20 [34581.483725] [] poll_schedule_timeout+0x55/0xb0 [34581.485238] [] do_sys_poll+0x4cd/0x580 [34581.486667] [] ? __wake_up_common+0x58/0x90 [34581.488135] [] ? __wake_up_sync_key+0x4f/0x60 [34581.489600] [] ? sock_def_readable+0x3a/0x70 [34581.491063] [] ? unix_dgram_sendmsg+0x3a8/0x720 [34581.492537] [] ? poll_select_copy_remaining+0x150/0x150 [34581.494074] [] ? poll_select_copy_remaining+0x150/0x150 [34581.495589] [] ? poll_select_copy_remaining+0x150/0x150 [34581.497091] [] ? SYSC_sendto+0x121/0x1c0 [34581.498476] [] ? __do_page_fault+0x171/0x450 [34581.499897] [] SyS_poll+0x74/0x110 [34581.501242] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.502705] [] system_call_fastpath+0x16/0x1b [34581.504157] sshd S 0000000000000005 0 910 1 0x00000080 [34581.505696] ffff88003627b8e8 0000000000000086 ffff880036182f10 ffff88003627bfd8 [34581.507331] ffff88003627bfd8 ffff88003627bfd8 ffff880036182f10 0000000000000000 [34581.508932] ffff88003627bad0 00000000fffffffc 0000000000000005 0000000000000005 [34581.510537] Call Trace: [34581.511682] [] schedule+0x29/0x70 [34581.513108] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.514674] [] ? add_wait_queue+0x3c/0x50 [34581.516132] [] ? __pollwait+0x7f/0xf0 [34581.517552] [] schedule_hrtimeout_range+0x13/0x20 [34581.519053] [] poll_schedule_timeout+0x55/0xb0 [34581.520517] [] do_select+0x6d1/0x7c0 [34581.521919] [] ? poll_select_copy_remaining+0x150/0x150 [34581.523465] [] ? poll_select_copy_remaining+0x150/0x150 [34581.524992] [] ? poll_select_copy_remaining+0x150/0x150 [34581.526501] [] ? __alloc_skb+0xa1/0x2d0 [34581.527909] [] ? __alloc_pages_nodemask+0x176/0x420 [34581.529371] [] ? skb_copy_datagram_from_iovec+0x5f/0x290 [34581.530887] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [34581.532390] [] ? cpumask_any_but+0x39/0x50 [34581.533806] [] ? flush_tlb_page+0x39/0xa0 [34581.535218] [] ? lru_cache_add+0xe/0x10 [34581.536604] [] ? page_add_new_anon_rmap+0x91/0x150 [34581.538093] [] ? __slab_free+0x81/0x2f0 [34581.539476] [] ? wp_page_copy.isra.56+0x2e6/0x470 [34581.540921] [] core_sys_select+0x1db/0x300 [34581.542317] [] ? handle_mm_fault+0x72c/0xfe0 [34581.543724] [] ? d_free+0x60/0x70 [34581.545038] [] ? __do_page_fault+0x171/0x450 [34581.546425] [] SyS_select+0xba/0x110 [34581.547744] [] system_call_fastpath+0x16/0x1b [34581.549138] munged S ffff880036257e30 0 915 1 0x00000080 [34581.550635] ffff880036257cc0 0000000000000082 ffff88003611edd0 ffff880036257fd8 [34581.552248] ffff880036257fd8 ffff880036257fd8 ffff88003611edd0 ffff880077770890 [34581.553805] ffff880077770890 ffff880036257e34 ffff880077770800 ffff880036257e30 [34581.555421] Call Trace: [34581.556522] [] schedule+0x29/0x70 [34581.557925] [] schedule_timeout+0x239/0x2d0 [34581.559339] [] ? __d_alloc+0x25/0x180 [34581.560706] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34581.562220] [] __skb_recv_datagram+0x3ca/0x460 [34581.563683] [] ? datagram_poll+0x100/0x100 [34581.565135] [] skb_recv_datagram+0x41/0x60 [34581.566569] [] unix_accept+0x97/0x170 [34581.567989] [] SYSC_accept4+0x103/0x220 [34581.569398] [] ? SyS_futex+0x80/0x180 [34581.570790] [] SyS_accept4+0xe/0x10 [34581.572174] [] SyS_accept+0x10/0x20 [34581.573526] [] system_call_fastpath+0x16/0x1b [34581.574945] munged S ffffc90000380900 0 918 1 0x00000080 [34581.576477] ffff880078f07cd0 0000000000000082 ffff880036183ec0 ffff880078f07fd8 [34581.578096] ffff880078f07fd8 ffff880078f07fd8 ffff880036183ec0 ffff880036183ec0 [34581.579656] ffff880036183ec0 ffff880078f07d60 ffff880078f07de0 ffffc90000380900 [34581.581269] Call Trace: [34581.582371] [] schedule+0x29/0x70 [34581.583756] [] futex_wait_queue_me+0xc4/0x120 [34581.585188] [] futex_wait+0x179/0x280 [34581.586555] [] ? hrtimer_get_res+0x50/0x50 [34581.587961] [] ? futex_wait_queue_me+0xa2/0x120 [34581.589397] [] do_futex+0xfe/0x5b0 [34581.590733] [] ? mntput+0x24/0x40 [34581.592083] [] ? __fput+0x183/0x260 [34581.593444] [] SyS_futex+0x80/0x180 [34581.594795] [] system_call_fastpath+0x16/0x1b [34581.596213] munged S ffffc9000037d240 0 919 1 0x00000080 [34581.597737] ffff8800778bfcd0 0000000000000082 ffff8800360f0000 ffff8800778bffd8 [34581.599371] ffff8800778bffd8 ffff8800778bffd8 ffff8800360f0000 ffff8800360f0000 [34581.600961] ffff8800360f0000 0000000000000000 ffff8800778bfde0 ffffc9000037d240 [34581.602559] Call Trace: [34581.603699] [] schedule+0x29/0x70 [34581.605123] [] futex_wait_queue_me+0xc4/0x120 [34581.606569] [] futex_wait+0x179/0x280 [34581.607980] [] ? fsnotify_clear_marks_by_inode+0xa7/0x140 [34581.609514] [] do_futex+0xfe/0x5b0 [34581.610896] [] ? mntput+0x24/0x40 [34581.612256] [] ? __fput+0x183/0x260 [34581.613635] [] SyS_futex+0x80/0x180 [34581.615013] [] ? do_notify_resume+0x92/0xb0 [34581.616438] [] system_call_fastpath+0x16/0x1b [34581.617862] munged S ffffc9000037d240 0 920 1 0x00000080 [34581.619396] ffff88007753fcd0 0000000000000082 ffff8800360f0fb0 ffff88007753ffd8 [34581.620980] ffff88007753ffd8 ffff88007753ffd8 ffff8800360f0fb0 ffff8800360f0fb0 [34581.622578] ffff8800360f0fb0 0000000000000000 ffff88007753fde0 ffffc9000037d240 [34581.624202] Call Trace: [34581.625318] [] schedule+0x29/0x70 [34581.626714] [] futex_wait_queue_me+0xc4/0x120 [34581.628165] [] futex_wait+0x179/0x280 [34581.629556] [] ? fsnotify_clear_marks_by_inode+0xa7/0x140 [34581.631095] [] ? kmem_cache_free+0x1bb/0x1f0 [34581.632528] [] do_futex+0xfe/0x5b0 [34581.633916] [] ? mntput+0x24/0x40 [34581.635278] [] ? __fput+0x183/0x260 [34581.636653] [] SyS_futex+0x80/0x180 [34581.638015] [] ? do_notify_resume+0x92/0xb0 [34581.639438] [] system_call_fastpath+0x16/0x1b [34581.640892] nfsiod S ffff8800796bcb00 0 922 2 0x00000080 [34581.642455] ffff880079a13e08 0000000000000046 ffff8800360f4e70 ffff880079a13fd8 [34581.644074] ffff880079a13fd8 ffff880079a13fd8 ffff8800360f4e70 ffff8800360f4e70 [34581.645666] ffff8800796bcb00 ffff88007b85b280 ffff8800796bcb30 ffff8800796bcb00 [34581.647301] Call Trace: [34581.648439] [] schedule+0x29/0x70 [34581.649843] [] rescuer_thread+0x359/0x460 [34581.651267] [] ? __schedule+0x3b0/0x990 [34581.652677] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34581.654176] [] kthread+0xcf/0xe0 [34581.655546] [] ? kthread_create_on_node+0x140/0x140 [34581.657048] [] ret_from_fork+0x58/0x90 [34581.658471] [] ? kthread_create_on_node+0x140/0x140 [34581.659983] kworker/0:1H S ffff88007fc16840 0 933 2 0x00000080 [34581.661575] ffff880036233e50 0000000000000046 ffff880078f69f60 ffff880036233fd8 [34581.663274] ffff880036233fd8 ffff880036233fd8 ffff880078f69f60 ffff88007fc16860 [34581.664923] ffff88007bad3fb0 ffff880078f69f60 ffff88007bad3f80 ffff88007fc16840 [34581.666601] Call Trace: [34581.667762] [] schedule+0x29/0x70 [34581.669213] [] worker_thread+0x1d9/0x410 [34581.670664] [] ? rescuer_thread+0x460/0x460 [34581.672126] [] kthread+0xcf/0xe0 [34581.673502] [] ? kthread_create_on_node+0x140/0x140 [34581.675029] [] ret_from_fork+0x58/0x90 [34581.676454] [] ? kthread_create_on_node+0x140/0x140 [34581.677973] nfsv4.0-svc S ffff88007b430238 0 934 2 0x00000080 [34581.679549] ffff88007a143d80 0000000000000046 ffff8800360f5e20 ffff88007a143fd8 [34581.681229] ffff88007a143fd8 ffff88007a143fd8 ffff8800360f5e20 ffff8800360f5e20 [34581.682886] ffff88007b430000 ffff88007c0fbe80 0000000000002000 ffff88007b430238 [34581.684593] Call Trace: [34581.685788] [] schedule+0x29/0x70 [34581.687271] [] schedule_timeout+0x239/0x2d0 [34581.688826] [] ? svc_xprt_enqueue.part.3+0x17/0x20 [sunrpc] [34581.690417] [] svc_recv+0xa2a/0xb00 [sunrpc] [34581.691956] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [34581.693599] [] ? nfs_callback_authenticate+0x60/0x60 [nfsv4] [34581.695204] [] nfs4_callback_svc+0x33/0x60 [nfsv4] [34581.696743] [] kthread+0xcf/0xe0 [34581.698161] [] ? kthread_create_on_node+0x140/0x140 [34581.699705] [] ret_from_fork+0x58/0x90 [34581.701154] [] ? kthread_create_on_node+0x140/0x140 [34581.702698] master S ffff8800797a7ea8 0 1030 1 0x00000080 [34581.704349] ffff8800797a7db8 0000000000000082 ffff880078f16dd0 ffff8800797a7fd8 [34581.706034] ffff8800797a7fd8 ffff8800797a7fd8 ffff880078f16dd0 ffff8800797a7ea8 [34581.707706] 00000000039386ff 0000000000000000 ffff880078f16dd0 ffff8800797a7ea8 [34581.709428] Call Trace: [34581.710648] [] schedule+0x29/0x70 [34581.712159] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.713751] [] ? hrtimer_get_res+0x50/0x50 [34581.715277] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.716912] [] schedule_hrtimeout_range+0x13/0x20 [34581.718487] [] ep_poll+0x23e/0x360 [34581.719951] [] ? wake_up_state+0x20/0x20 [34581.721443] [] SyS_epoll_wait+0xed/0x120 [34581.722955] [] system_call_fastpath+0x16/0x1b [34581.724490] qmgr S ffff88007b5a7ea8 0 1032 1030 0x00000080 [34581.726133] ffff88007b5a7db8 0000000000000086 ffff880078f10000 ffff88007b5a7fd8 [34581.727865] ffff88007b5a7fd8 ffff88007b5a7fd8 ffff880078f10000 ffff88007b5a7ea8 [34581.729591] 0000000005f5e100 0000000000000000 ffff880078f10000 ffff88007b5a7ea8 [34581.731331] Call Trace: [34581.732554] [] schedule+0x29/0x70 [34581.734051] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34581.735655] [] ? hrtimer_get_res+0x50/0x50 [34581.737182] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34581.738821] [] schedule_hrtimeout_range+0x13/0x20 [34581.740391] [] ep_poll+0x23e/0x360 [34581.741858] [] ? do_vfs_ioctl+0x2d5/0x4b0 [34581.743366] [] ? wake_up_state+0x20/0x20 [34581.744868] [] SyS_epoll_wait+0xed/0x120 [34581.746375] [] system_call_fastpath+0x16/0x1b [34581.747918] atd S 0000000000000000 0 1045 1 0x00000080 [34581.749570] ffff88007918be68 0000000000000086 ffff880036118fb0 ffff88007918bfd8 [34581.751319] ffff88007918bfd8 ffff88007918bfd8 ffff880036118fb0 ffff880036118fb0 [34581.752993] ffff88007918beb8 ffff880036118fb0 ffff88007918bf58 0000000000000000 [34581.754691] Call Trace: [34581.755905] [] schedule+0x29/0x70 [34581.757394] [] do_nanosleep+0x86/0xf0 [34581.758862] [] hrtimer_nanosleep+0xbb/0x180 [34581.760367] [] ? hrtimer_get_res+0x50/0x50 [34581.761878] [] ? do_nanosleep+0x72/0xf0 [34581.763340] [] SyS_nanosleep+0x7e/0x90 [34581.764796] [] system_call_fastpath+0x16/0x1b [34581.766297] crond S 0000000000000000 0 1048 1 0x00000080 [34581.767911] ffff8800791cbe68 0000000000000082 ffff880036118000 ffff8800791cbfd8 [34581.769612] ffff8800791cbfd8 ffff8800791cbfd8 ffff880036118000 ffff880036118000 [34581.771321] ffff8800791cbeb8 ffff880036118000 ffff8800791cbf58 0000000000000000 [34581.772989] Call Trace: [34581.774186] [] schedule+0x29/0x70 [34581.775630] [] do_nanosleep+0x86/0xf0 [34581.777080] [] hrtimer_nanosleep+0xbb/0x180 [34581.778558] [] ? hrtimer_get_res+0x50/0x50 [34581.780043] [] ? do_nanosleep+0x72/0xf0 [34581.781501] [] SyS_nanosleep+0x7e/0x90 [34581.782940] [] system_call_fastpath+0x16/0x1b [34581.784452] automount S 0000000000000000 0 1051 1 0x00000080 [34581.786037] ffff8800791efd90 0000000000000086 ffff88007a148fb0 ffff8800791effd8 [34581.787714] ffff8800791effd8 ffff8800791effd8 ffff88007a148fb0 ffff88007a148fb0 [34581.789384] 7fffffffffffffff ffff8800791efed0 ffff88007a148fb0 0000000000000000 [34581.791020] Call Trace: [34581.792187] [] schedule+0x29/0x70 [34581.793600] [] schedule_timeout+0x239/0x2d0 [34581.795095] [] ? unlock_page+0x2b/0x30 [34581.796534] [] ? do_read_fault.isra.42+0xe4/0x130 [34581.798049] [] ? recalc_sigpending+0x1b/0x50 [34581.799508] [] schedule_timeout_interruptible+0x1e/0x20 [34581.801059] [] do_sigtimedwait+0x1c2/0x2a0 [34581.802526] [] SYSC_rt_sigtimedwait+0x85/0xf0 [34581.803993] [] ? __do_page_fault+0x171/0x450 [34581.805450] [] ? do_page_fault+0x35/0x90 [34581.806882] [] SyS_rt_sigtimedwait+0xe/0x10 [34581.808335] [] system_call_fastpath+0x16/0x1b [34581.809791] automount S ffffc900003823c0 0 1053 1 0x00000080 [34581.811360] ffff8800791fbcd0 0000000000000086 ffff88007a149f60 ffff8800791fbfd8 [34581.812976] ffff8800791fbfd8 ffff8800791fbfd8 ffff88007a149f60 ffff88007a149f60 [34581.814602] ffff88007a149f60 ffff8800791fbd60 ffff8800791fbde0 ffffc900003823c0 [34581.816250] Call Trace: [34581.817408] [] schedule+0x29/0x70 [34581.818842] [] futex_wait_queue_me+0xc4/0x120 [34581.820322] [] futex_wait+0x179/0x280 [34581.821729] [] ? hrtimer_get_res+0x50/0x50 [34581.823171] [] ? futex_wait_queue_me+0xa2/0x120 [34581.824653] [] do_futex+0xfe/0x5b0 [34581.826025] [] SyS_futex+0x80/0x180 [34581.827402] [] system_call_fastpath+0x16/0x1b [34581.828854] automount S ffffc90000380600 0 1054 1 0x00000080 [34581.830411] ffff8800791ffcd0 0000000000000086 ffff880079a25e20 ffff8800791fffd8 [34581.832047] ffff8800791fffd8 ffff8800791fffd8 ffff880079a25e20 ffff880079a25e20 [34581.833659] ffff880079a25e20 0000000000000000 ffff8800791ffde0 ffffc90000380600 [34581.835316] Call Trace: [34581.836482] [] schedule+0x29/0x70 [34581.837926] [] futex_wait_queue_me+0xc4/0x120 [34581.839409] [] futex_wait+0x179/0x280 [34581.840837] [] ? ttwu_do_activate.constprop.90+0x5d/0x70 [34581.842394] [] ? try_to_wake_up+0x1c8/0x320 [34581.843855] [] do_futex+0xfe/0x5b0 [34581.845250] [] ? wake_up_new_task+0x10c/0x170 [34581.846720] [] ? do_fork+0xe8/0x2c0 [34581.848115] [] SyS_futex+0x80/0x180 [34581.849510] [] system_call_fastpath+0x16/0x1b [34581.850980] automount S ffff88007a19fbb4 0 1077 1 0x00000080 [34581.852573] ffff88007a19fa30 0000000000000086 ffff88007a148000 ffff88007a19ffd8 [34581.854260] ffff88007a19ffd8 ffff88007a19ffd8 ffff88007a148000 0000000000000000 [34581.855888] ffff88007a19fc90 00000000fffffffc 0000000000000000 ffff88007a19fbb4 [34581.857553] Call Trace: [34581.858736] [] schedule+0x29/0x70 [34581.860194] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.861778] [] ? wake_up_bit+0x25/0x30 [34581.863247] [] ? zone_statistics+0x89/0xa0 [34581.864724] [] ? add_wait_queue+0x3c/0x50 [34581.866197] [] schedule_hrtimeout_range+0x13/0x20 [34581.867718] [] poll_schedule_timeout+0x55/0xb0 [34581.869236] [] do_sys_poll+0x4cd/0x580 [34581.870683] [] ? __d_alloc+0x25/0x180 [34581.872113] [] ? terminate_walk+0x51/0x60 [34581.873557] [] ? update_curr+0x104/0x190 [34581.875011] [] ? __enqueue_entity+0x78/0x80 [34581.876476] [] ? enqueue_entity+0x26c/0xb60 [34581.877951] [] ? kvm_clock_read+0x1f/0x30 [34581.879392] [] ? select_task_rq_fair+0x584/0x720 [34581.880880] [] ? poll_select_copy_remaining+0x150/0x150 [34581.882421] [] ? poll_select_copy_remaining+0x150/0x150 [34581.883947] [] ? poll_select_copy_remaining+0x150/0x150 [34581.885464] [] ? unlock_page+0x2b/0x30 [34581.886849] [] ? do_read_fault.isra.42+0xe4/0x130 [34581.888302] [] ? handle_mm_fault+0x6b1/0xfe0 [34581.889726] [] ? do_futex+0x172/0x5b0 [34581.891097] [] ? __do_page_fault+0x171/0x450 [34581.892532] [] SyS_poll+0x74/0x110 [34581.893888] [] ? page_fault+0x28/0x30 [34581.895250] [] system_call_fastpath+0x16/0x1b [34581.896675] automount S ffff88007a19bbb4 0 1088 1 0x00000080 [34581.898251] ffff88007a19ba30 0000000000000086 ffff880079f18000 ffff88007a19bfd8 [34581.899843] ffff88007a19bfd8 ffff88007a19bfd8 ffff880079f18000 0000000000000000 [34581.901461] ffff88007a19bc90 00000000fffffffc 0000000000000000 ffff88007a19bbb4 [34581.903063] Call Trace: [34581.904193] [] schedule+0x29/0x70 [34581.905577] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.907117] [] ? wake_up_bit+0x25/0x30 [34581.908507] [] ? zone_statistics+0x89/0xa0 [34581.909931] [] ? add_wait_queue+0x3c/0x50 [34581.911342] [] schedule_hrtimeout_range+0x13/0x20 [34581.912818] [] poll_schedule_timeout+0x55/0xb0 [34581.914269] [] do_sys_poll+0x4cd/0x580 [34581.915663] [] ? __d_lookup+0x120/0x160 [34581.917061] [] ? mntput+0x24/0x40 [34581.918414] [] ? terminate_walk+0x51/0x60 [34581.919852] [] ? update_curr+0x104/0x190 [34581.921243] [] ? __enqueue_entity+0x78/0x80 [34581.922644] [] ? enqueue_entity+0x26c/0xb60 [34581.924047] [] ? kvm_clock_read+0x1f/0x30 [34581.925422] [] ? select_task_rq_fair+0x584/0x720 [34581.926833] [] ? poll_select_copy_remaining+0x150/0x150 [34581.928291] [] ? poll_select_copy_remaining+0x150/0x150 [34581.929741] [] ? poll_select_copy_remaining+0x150/0x150 [34581.931182] [] ? wake_futex+0x66/0x80 [34581.932491] [] ? futex_wake_op+0x332/0x570 [34581.933828] [] ? do_futex+0x172/0x5b0 [34581.935139] [] ? autofs_dev_ioctl+0x13/0x20 [34581.936473] [] ? do_vfs_ioctl+0x2d5/0x4b0 [34581.937785] [] SyS_poll+0x74/0x110 [34581.939070] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.940473] [] system_call_fastpath+0x16/0x1b [34581.941844] automount S ffff8800791f7bb4 0 1092 1 0x00000080 [34581.943354] ffff8800791f7a30 0000000000000086 ffff880079f18fb0 ffff8800791f7fd8 [34581.944900] ffff8800791f7fd8 ffff8800791f7fd8 ffff880079f18fb0 0000000000000000 [34581.946447] ffff8800791f7c90 00000000fffffffc 0000000000000000 ffff8800791f7bb4 [34581.947992] Call Trace: [34581.949074] [] schedule+0x29/0x70 [34581.950430] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.951953] [] ? add_wait_queue+0x3c/0x50 [34581.953368] [] schedule_hrtimeout_range+0x13/0x20 [34581.954844] [] poll_schedule_timeout+0x55/0xb0 [34581.956304] [] do_sys_poll+0x4cd/0x580 [34581.957697] [] ? __d_lookup+0x120/0x160 [34581.959095] [] ? mntput+0x24/0x40 [34581.960462] [] ? terminate_walk+0x51/0x60 [34581.961873] [] ? update_curr+0x104/0x190 [34581.963273] [] ? __enqueue_entity+0x78/0x80 [34581.964672] [] ? enqueue_entity+0x26c/0xb60 [34581.966079] [] ? kvm_clock_read+0x1f/0x30 [34581.967462] [] ? check_preempt_wakeup+0x148/0x250 [34581.968900] [] ? poll_select_copy_remaining+0x150/0x150 [34581.970351] [] ? poll_select_copy_remaining+0x150/0x150 [34581.971797] [] ? poll_select_copy_remaining+0x150/0x150 [34581.973227] [] ? wake_futex+0x66/0x80 [34581.974523] [] ? futex_wake_op+0x332/0x570 [34581.975867] [] ? do_futex+0x172/0x5b0 [34581.977157] [] ? mntput+0x24/0x40 [34581.978423] [] ? __fput+0x183/0x260 [34581.979681] [] SyS_poll+0x74/0x110 [34581.980940] [] ? __audit_syscall_exit+0x1e6/0x280 [34581.982305] [] system_call_fastpath+0x16/0x1b [34581.983647] automount S ffff880077c9fbb4 0 1138 1 0x00000080 [34581.985143] ffff880077c9fa30 0000000000000086 ffff880079f1edd0 ffff880077c9ffd8 [34581.986654] ffff880077c9ffd8 ffff880077c9ffd8 ffff880079f1edd0 0000000000000000 [34581.988210] ffff880077c9fc90 00000000fffffffc 0000000000000000 ffff880077c9fbb4 [34581.989733] Call Trace: [34581.990887] [] schedule+0x29/0x70 [34581.992212] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34581.993702] [] ? add_wait_queue+0x3c/0x50 [34581.995088] [] schedule_hrtimeout_range+0x13/0x20 [34581.996557] [] poll_schedule_timeout+0x55/0xb0 [34581.998008] [] do_sys_poll+0x4cd/0x580 [34581.999394] [] ? kvm_clock_read+0x1f/0x30 [34582.000800] [] ? sched_clock+0x9/0x10 [34582.002182] [] ? sched_clock_local+0x1d/0x80 [34582.003610] [] ? sched_clock_cpu+0x98/0xc0 [34582.005029] [] ? __switch_to+0x15c/0x4c0 [34582.006423] [] ? timerqueue_del+0x24/0x70 [34582.007813] [] ? poll_select_copy_remaining+0x150/0x150 [34582.009300] [] ? poll_select_copy_remaining+0x150/0x150 [34582.010762] [] ? poll_select_copy_remaining+0x150/0x150 [34582.012214] [] ? hrtimer_get_res+0x50/0x50 [34582.013564] [] ? get_futex_key+0x1c8/0x2c0 [34582.014895] [] ? futex_wake+0x80/0x160 [34582.016203] [] ? do_futex+0x122/0x5b0 [34582.017485] [] ? wake_up_new_task+0x10c/0x170 [34582.018828] [] SyS_poll+0x74/0x110 [34582.020084] [] ? __audit_syscall_exit+0x1e6/0x280 [34582.021446] [] system_call_fastpath+0x16/0x1b [34582.022781] agetty S ffff8800791f3000 0 1078 1 0x00000080 [34582.024237] ffff8800791a3cf0 0000000000000086 ffff880079f08fb0 ffff8800791a3fd8 [34582.025745] ffff8800791a3fd8 ffff8800791a3fd8 ffff880079f08fb0 ffff88007bad1d00 [34582.027303] 7fffffffffffffff ffff8800791f2800 ffff8800791f2800 ffff8800791f3000 [34582.028818] Call Trace: [34582.029919] [] schedule+0x29/0x70 [34582.031216] [] schedule_timeout+0x239/0x2d0 [34582.032599] [] ? vt_console_print+0x275/0x420 [34582.033994] [] ? flush_work+0x15b/0x1c0 [34582.035343] [] ? set_room+0x68/0xc0 [34582.036666] [] n_tty_read+0x27c/0x990 [34582.038012] [] ? wake_up_state+0x20/0x20 [34582.039375] [] tty_read+0x8d/0x100 [34582.040698] [] vfs_read+0x9e/0x170 [34582.042011] [] SyS_read+0x7f/0xe0 [34582.043327] [] system_call_fastpath+0x16/0x1b [34582.044717] agetty S ffff8800791f3c00 0 1080 1 0x00000080 [34582.046256] ffff880077cd7cf0 0000000000000082 ffff88007a14ce70 ffff880077cd7fd8 [34582.047826] ffff880077cd7fd8 ffff880077cd7fd8 ffff88007a14ce70 ffff8800790da700 [34582.049422] 7fffffffffffffff ffff8800791f1800 ffff8800791f1800 ffff8800791f3c00 [34582.050972] Call Trace: [34582.052054] [] schedule+0x29/0x70 [34582.053387] [] schedule_timeout+0x239/0x2d0 [34582.054769] [] ? flush_work+0x15b/0x1c0 [34582.056118] [] ? _raw_spin_unlock_irqrestore+0x1b/0x40 [34582.057563] [] ? set_room+0x68/0xc0 [34582.058879] [] n_tty_read+0x27c/0x990 [34582.060193] [] ? wake_up_state+0x20/0x20 [34582.061531] [] tty_read+0x8d/0x100 [34582.062829] [] vfs_read+0x9e/0x170 [34582.064112] [] SyS_read+0x7f/0xe0 [34582.065378] [] system_call_fastpath+0x16/0x1b [34582.066729] kdmflush S ffff88007b4ef900 0 2537 2 0x00000080 [34582.068241] ffff88007bb3fe08 0000000000000046 ffff880079a20fb0 ffff88007bb3ffd8 [34582.069775] ffff88007bb3ffd8 ffff88007bb3ffd8 ffff880079a20fb0 ffff880079a20fb0 [34582.071345] ffff88007b4ef900 ffff88003617d880 ffff88007b4ef930 ffff88007b4ef900 [34582.072870] Call Trace: [34582.073943] [] schedule+0x29/0x70 [34582.075282] [] rescuer_thread+0x359/0x460 [34582.076672] [] ? __schedule+0x3b0/0x990 [34582.078053] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.079520] [] kthread+0xcf/0xe0 [34582.080873] [] ? kthread_create_on_node+0x140/0x140 [34582.082356] [] ret_from_fork+0x58/0x90 [34582.083739] [] ? kthread_create_on_node+0x140/0x140 [34582.085221] bioset S ffff88007c0fba00 0 2538 2 0x00000080 [34582.086761] ffff8800791dfe08 0000000000000046 ffff880079a21f60 ffff8800791dffd8 [34582.088413] ffff8800791dffd8 ffff8800791dffd8 ffff880079a21f60 ffff880079a21f60 [34582.090019] ffff88007c0fba00 ffff88003617c880 ffff88007c0fba30 ffff88007c0fba00 [34582.091587] Call Trace: [34582.092751] [] schedule+0x29/0x70 [34582.094116] [] rescuer_thread+0x359/0x460 [34582.095533] [] ? __schedule+0x3b0/0x990 [34582.096943] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.098425] [] kthread+0xcf/0xe0 [34582.099790] [] ? kthread_create_on_node+0x140/0x140 [34582.101276] [] ret_from_fork+0x58/0x90 [34582.102680] [] ? kthread_create_on_node+0x140/0x140 [34582.104193] kdmflush S ffff88007a194400 0 2584 2 0x00000080 [34582.105752] ffff88007765be08 0000000000000046 ffff880077748000 ffff88007765bfd8 [34582.107419] ffff88007765bfd8 ffff88007765bfd8 ffff880077748000 ffff880077748000 [34582.109071] ffff88007a194400 ffff880035f5c480 ffff88007a194430 ffff88007a194400 [34582.110704] Call Trace: [34582.111913] [] schedule+0x29/0x70 [34582.113331] [] rescuer_thread+0x359/0x460 [34582.114791] [] ? __schedule+0x3b0/0x990 [34582.116234] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.117772] [] kthread+0xcf/0xe0 [34582.119181] [] ? kthread_create_on_node+0x140/0x140 [34582.120703] [] ret_from_fork+0x58/0x90 [34582.122140] [] ? kthread_create_on_node+0x140/0x140 [34582.123669] bioset S ffff88007a194500 0 2585 2 0x00000080 [34582.125302] ffff880078f33e08 0000000000000046 ffff88007774bec0 ffff880078f33fd8 [34582.126958] ffff880078f33fd8 ffff880078f33fd8 ffff88007774bec0 ffff88007774bec0 [34582.128643] ffff88007a194500 ffff880078f40e80 ffff88007a194530 ffff88007a194500 [34582.130342] Call Trace: [34582.131540] [] schedule+0x29/0x70 [34582.133035] [] rescuer_thread+0x359/0x460 [34582.134541] [] ? __schedule+0x3b0/0x990 [34582.136042] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.137615] [] kthread+0xcf/0xe0 [34582.139041] [] ? kthread_create_on_node+0x140/0x140 [34582.140608] [] ret_from_fork+0x58/0x90 [34582.142110] [] ? kthread_create_on_node+0x140/0x140 [34582.143671] kdmflush S ffff88007a194600 0 2632 2 0x00000080 [34582.145328] ffff88003609be08 0000000000000046 ffff88007c075e20 ffff88003609bfd8 [34582.147016] ffff88003609bfd8 ffff88003609bfd8 ffff88007c075e20 ffff88007c075e20 [34582.148723] ffff88007a194600 ffff880035f5cc80 ffff88007a194630 ffff88007a194600 [34582.150443] Call Trace: [34582.151665] [] schedule+0x29/0x70 [34582.153181] [] rescuer_thread+0x359/0x460 [34582.154696] [] ? __schedule+0x3b0/0x990 [34582.156204] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.157768] [] kthread+0xcf/0xe0 [34582.159199] [] ? kthread_create_on_node+0x140/0x140 [34582.160774] [] ret_from_fork+0x58/0x90 [34582.162261] [] ? kthread_create_on_node+0x140/0x140 [34582.163845] bioset S ffff880079118f00 0 2633 2 0x00000080 [34582.165479] ffff88007c0d3e08 0000000000000046 ffff8800360c6dd0 ffff88007c0d3fd8 [34582.167208] ffff88007c0d3fd8 ffff88007c0d3fd8 ffff8800360c6dd0 ffff8800360c6dd0 [34582.168897] ffff880079118f00 ffff880035f5c880 ffff880079118f30 ffff880079118f00 [34582.170629] Call Trace: [34582.171858] [] schedule+0x29/0x70 [34582.173355] [] rescuer_thread+0x359/0x460 [34582.174873] [] ? __schedule+0x3b0/0x990 [34582.176368] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.177961] [] kthread+0xcf/0xe0 [34582.179388] [] ? kthread_create_on_node+0x140/0x140 [34582.181009] [] ret_from_fork+0x58/0x90 [34582.182486] [] ? kthread_create_on_node+0x140/0x140 [34582.184030] kdmflush S ffff880077789400 0 2677 2 0x00000080 [34582.185685] ffff88007908fe08 0000000000000046 ffff880077628000 ffff88007908ffd8 [34582.187421] ffff88007908ffd8 ffff88007908ffd8 ffff880077628000 ffff880077628000 [34582.189143] ffff880077789400 ffff8800362e5880 ffff880077789430 ffff880077789400 [34582.190823] Call Trace: [34582.192100] [] schedule+0x29/0x70 [34582.193560] [] rescuer_thread+0x359/0x460 [34582.195091] [] ? __schedule+0x3b0/0x990 [34582.196592] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.198178] [] kthread+0xcf/0xe0 [34582.199600] [] ? kthread_create_on_node+0x140/0x140 [34582.201171] [] ret_from_fork+0x58/0x90 [34582.202653] [] ? kthread_create_on_node+0x140/0x140 [34582.204230] bioset S ffff880036123d80 0 2678 2 0x00000080 [34582.205856] ffff880035d47e08 0000000000000046 ffff8800360a0000 ffff880035d47fd8 [34582.207583] ffff880035d47fd8 ffff880035d47fd8 ffff8800360a0000 ffff8800360a0000 [34582.209303] ffff880036123d80 ffff8800362e5a80 ffff880036123db0 ffff880036123d80 [34582.210987] Call Trace: [34582.212246] [] schedule+0x29/0x70 [34582.213718] [] rescuer_thread+0x359/0x460 [34582.215233] [] ? __schedule+0x3b0/0x990 [34582.216733] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.218314] [] kthread+0xcf/0xe0 [34582.219746] [] ? kthread_create_on_node+0x140/0x140 [34582.221319] [] ret_from_fork+0x58/0x90 [34582.222789] [] ? kthread_create_on_node+0x140/0x140 [34582.224354] kdmflush S ffff8800361f3880 0 2722 2 0x00000080 [34582.225990] ffff880078f2fe08 0000000000000046 ffff8800360a0fb0 ffff880078f2ffd8 [34582.227713] ffff880078f2ffd8 ffff880078f2ffd8 ffff8800360a0fb0 ffff8800360a0fb0 [34582.229436] ffff8800361f3880 ffff8800362e4880 ffff8800361f38b0 ffff8800361f3880 [34582.231132] Call Trace: [34582.232367] [] schedule+0x29/0x70 [34582.233854] [] rescuer_thread+0x359/0x460 [34582.235364] [] ? __schedule+0x3b0/0x990 [34582.236860] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.238422] [] kthread+0xcf/0xe0 [34582.239866] [] ? kthread_create_on_node+0x140/0x140 [34582.241430] [] ret_from_fork+0x58/0x90 [34582.242916] [] ? kthread_create_on_node+0x140/0x140 [34582.244489] bioset S ffff880077789e80 0 2723 2 0x00000080 [34582.246107] ffff88007968be08 0000000000000046 ffff880079f1ce70 ffff88007968bfd8 [34582.247829] ffff88007968bfd8 ffff88007968bfd8 ffff880079f1ce70 ffff880079f1ce70 [34582.249534] ffff880077789e80 ffff880035f5c080 ffff880077789eb0 ffff880077789e80 [34582.251226] Call Trace: [34582.252460] [] schedule+0x29/0x70 [34582.253955] [] rescuer_thread+0x359/0x460 [34582.255470] [] ? __schedule+0x3b0/0x990 [34582.256971] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.258539] [] kthread+0xcf/0xe0 [34582.259967] [] ? kthread_create_on_node+0x140/0x140 [34582.261523] [] ret_from_fork+0x58/0x90 [34582.263013] [] ? kthread_create_on_node+0x140/0x140 [34582.264569] kdmflush S ffff88007b58e700 0 2767 2 0x00000080 [34582.266194] ffff880077557e08 0000000000000046 ffff880079f19f60 ffff880077557fd8 [34582.267919] ffff880077557fd8 ffff880077557fd8 ffff880079f19f60 ffff880079f19f60 [34582.269620] ffff88007b58e700 ffff8800362fd080 ffff88007b58e730 ffff88007b58e700 [34582.271334] Call Trace: [34582.272572] [] schedule+0x29/0x70 [34582.274061] [] rescuer_thread+0x359/0x460 [34582.275579] [] ? __schedule+0x3b0/0x990 [34582.277077] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.278646] [] kthread+0xcf/0xe0 [34582.280071] [] ? kthread_create_on_node+0x140/0x140 [34582.281641] [] ret_from_fork+0x58/0x90 [34582.283132] [] ? kthread_create_on_node+0x140/0x140 [34582.284701] bioset S ffff88007b58e500 0 2768 2 0x00000080 [34582.286347] ffff88007b85fe08 0000000000000046 ffff880079f0bec0 ffff88007b85ffd8 [34582.288050] ffff88007b85ffd8 ffff88007b85ffd8 ffff880079f0bec0 ffff880079f0bec0 [34582.289748] ffff88007b58e500 ffff8800362fc280 ffff88007b58e530 ffff88007b58e500 [34582.291461] Call Trace: [34582.292702] [] schedule+0x29/0x70 [34582.294197] [] rescuer_thread+0x359/0x460 [34582.295715] [] ? __schedule+0x3b0/0x990 [34582.297212] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34582.298791] [] kthread+0xcf/0xe0 [34582.300221] [] ? kthread_create_on_node+0x140/0x140 [34582.301788] [] ret_from_fork+0x58/0x90 [34582.303278] [] ? kthread_create_on_node+0x140/0x140 [34582.304849] cfs_rh_00 S ffff880079fa7a30 0 3815 2 0x00000080 [34582.306464] ffff88000004be28 0000000000000046 ffff880079f0af10 ffff88000004bfd8 [34582.308201] ffff88000004bfd8 ffff88000004bfd8 ffff880079f0af10 ffff880079fa7a00 [34582.309902] ffff880079fa7a18 ffff880079fa7a10 0000000000000000 ffff880079fa7a30 [34582.311613] Call Trace: [34582.312854] [] schedule+0x29/0x70 [34582.314365] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [34582.315991] [] ? wake_up_atomic_t+0x30/0x30 [34582.317528] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [34582.319131] [] kthread+0xcf/0xe0 [34582.320567] [] ? kthread_create_on_node+0x140/0x140 [34582.322151] [] ret_from_fork+0x58/0x90 [34582.323639] [] ? kthread_create_on_node+0x140/0x140 [34582.325221] cfs_rh_01 S ffff880079fa7a30 0 3816 2 0x00000080 [34582.326867] ffff880078723e28 0000000000000046 ffff880079f0ce70 ffff880078723fd8 [34582.328588] ffff880078723fd8 ffff880078723fd8 ffff880079f0ce70 ffff880079fa7a00 [34582.330354] ffff880079fa7a18 ffff880079fa7a10 0000000000000000 ffff880079fa7a30 [34582.332077] Call Trace: [34582.333313] [] schedule+0x29/0x70 [34582.334825] [] cfs_wi_scheduler+0x2f8/0x440 [libcfs] [34582.336431] [] ? wake_up_atomic_t+0x30/0x30 [34582.338005] [] ? cfs_wi_schedule+0x190/0x190 [libcfs] [34582.339616] [] kthread+0xcf/0xe0 [34582.341041] [] ? kthread_create_on_node+0x140/0x140 [34582.342658] [] ret_from_fork+0x58/0x90 [34582.344174] [] ? kthread_create_on_node+0x140/0x140 [34582.345754] obd_zombid S 0000000000000000 0 5294 2 0x00000080 [34582.347420] ffff880078797e40 0000000000000046 ffff880078771f60 ffff880078797fd8 [34582.349154] ffff880078797fd8 ffff880078797fd8 ffff880078771f60 ffff880078771f60 [34582.350868] ffff880078771f60 ffff880078771f60 0000000000000000 0000000000000000 [34582.352610] Call Trace: [34582.353873] [] schedule+0x29/0x70 [34582.355419] [] obd_zombie_impexp_thread+0xd5/0x1c0 [obdclass] [34582.357113] [] ? wake_up_state+0x20/0x20 [34582.358663] [] ? obd_zombie_impexp_cull+0x930/0x930 [obdclass] [34582.360376] [] kthread+0xcf/0xe0 [34582.361857] [] ? kthread_create_on_node+0x140/0x140 [34582.363488] [] ret_from_fork+0x58/0x90 [34582.365033] [] ? kthread_create_on_node+0x140/0x140 [34582.366648] ptlrpc_hr00_000 S 00000000ffffffff 0 5300 2 0x00000080 [34582.368365] ffff880077d7fdc0 0000000000000046 ffff880078773ec0 ffff880077d7ffd8 [34582.370121] ffff880077d7ffd8 ffff880077d7ffd8 ffff880078773ec0 ffff880079fa7308 [34582.371860] 0000000000000000 ffff880078773ec0 ffff880079fa7300 00000000ffffffff [34582.373642] Call Trace: [34582.374927] [] schedule+0x29/0x70 [34582.376518] [] ptlrpc_hr_main+0x555/0x790 [ptlrpc] [34582.378170] [] ? wake_up_state+0x20/0x20 [34582.379769] [] ? ptlrpc_server_request_get+0x240/0x240 [ptlrpc] [34582.381507] [] kthread+0xcf/0xe0 [34582.383016] [] ? kthread_create_on_node+0x140/0x140 [34582.384666] [] ret_from_fork+0x58/0x90 [34582.386231] [] ? kthread_create_on_node+0x140/0x140 [34582.387878] ptlrpc_hr00_001 S 00000000ffffffff 0 5301 2 0x00000080 [34582.389584] ffff880078763dc0 0000000000000046 ffff880078772f10 ffff880078763fd8 [34582.391386] ffff880078763fd8 ffff880078763fd8 ffff880078772f10 ffff880079fa7340 [34582.393154] 0000000000000000 ffff880078772f10 ffff880079fa7338 00000000ffffffff [34582.394913] Call Trace: [34582.396246] [] schedule+0x29/0x70 [34582.397805] [] ptlrpc_hr_main+0x555/0x790 [ptlrpc] [34582.399461] [] ? wake_up_state+0x20/0x20 [34582.401063] [] ? ptlrpc_server_request_get+0x240/0x240 [ptlrpc] [34582.402802] [] kthread+0xcf/0xe0 [34582.404322] [] ? kthread_create_on_node+0x140/0x140 [34582.405998] [] ret_from_fork+0x58/0x90 [34582.407574] [] ? kthread_create_on_node+0x140/0x140 [34582.409239] socknal_cd00 S ffffffffa0a33348 0 5304 2 0x00000080 [34582.410960] ffff880078a97d50 0000000000000046 ffff880078774e70 ffff880078a97fd8 [34582.412778] ffff880078a97fd8 ffff880078a97fd8 ffff880078774e70 7fffffffffffffff [34582.414604] ffffffffa0a33348 0000000058e7b1cc ffff880078774e70 ffffffffa0a33348 [34582.416409] Call Trace: [34582.417716] [] schedule+0x29/0x70 [34582.419303] [] schedule_timeout+0x239/0x2d0 [34582.420926] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [34582.422597] [] ? wake_up_state+0x20/0x20 [34582.424181] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [34582.425900] [] kthread+0xcf/0xe0 [34582.427439] [] ? kthread_create_on_node+0x140/0x140 [34582.429127] [] ret_from_fork+0x58/0x90 [34582.430702] [] ? kthread_create_on_node+0x140/0x140 [34582.432366] socknal_cd01 S ffffffffa0a33348 0 5305 2 0x00000080 [34582.434082] ffff880077907d50 0000000000000046 ffff880078776dd0 ffff880077907fd8 [34582.435878] ffff880077907fd8 ffff880077907fd8 ffff880078776dd0 7fffffffffffffff [34582.437686] ffffffffa0a33348 0000000058e7b3e8 ffff880078776dd0 ffffffffa0a33348 [34582.439500] Call Trace: [34582.440795] [] schedule+0x29/0x70 [34582.442385] [] schedule_timeout+0x239/0x2d0 [34582.444005] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34582.445651] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [34582.447285] [] ? wake_up_state+0x20/0x20 [34582.448875] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [34582.450578] [] kthread+0xcf/0xe0 [34582.452086] [] ? kthread_create_on_node+0x140/0x140 [34582.453732] [] ret_from_fork+0x58/0x90 [34582.455296] [] ? kthread_create_on_node+0x140/0x140 [34582.456950] socknal_cd02 S ffffffffa0a33348 0 5306 2 0x00000080 [34582.458649] ffff880077853d50 0000000000000046 ffff8800361a9f60 ffff880077853fd8 [34582.460444] ffff880077853fd8 ffff880077853fd8 ffff8800361a9f60 7fffffffffffffff [34582.462207] ffffffffa0a33348 0000000058e7b3e8 ffff8800361a9f60 ffffffffa0a33348 [34582.463955] Call Trace: [34582.465301] [] schedule+0x29/0x70 [34582.466816] [] schedule_timeout+0x239/0x2d0 [34582.468402] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34582.470067] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [34582.471722] [] ? wake_up_state+0x20/0x20 [34582.473283] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [34582.474972] [] kthread+0xcf/0xe0 [34582.476472] [] ? kthread_create_on_node+0x140/0x140 [34582.478110] [] ret_from_fork+0x58/0x90 [34582.479644] [] ? kthread_create_on_node+0x140/0x140 [34582.481273] socknal_cd03 S ffffffffa0a33348 0 5307 2 0x00000080 [34582.482965] ffff88007c03bd50 0000000000000046 ffff88007c7b2f10 ffff88007c03bfd8 [34582.484751] ffff88007c03bfd8 ffff88007c03bfd8 ffff88007c7b2f10 7fffffffffffffff [34582.486547] ffffffffa0a33348 0000000058e7b3e8 ffff88007c7b2f10 ffffffffa0a33348 [34582.488323] Call Trace: [34582.489584] [] schedule+0x29/0x70 [34582.491132] [] schedule_timeout+0x239/0x2d0 [34582.492708] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34582.494341] [] ksocknal_connd+0x2f4/0xe10 [ksocklnd] [34582.495966] [] ? wake_up_state+0x20/0x20 [34582.497524] [] ? ksocknal_thread_fini+0x30/0x30 [ksocklnd] [34582.499212] [] kthread+0xcf/0xe0 [34582.500697] [] ? kthread_create_on_node+0x140/0x140 [34582.502324] [] ret_from_fork+0x58/0x90 [34582.503852] [] ? kthread_create_on_node+0x140/0x140 [34582.505482] socknal_reaper S 000000000000f888 0 5308 2 0x00000080 [34582.507162] ffff8800798efd70 0000000000000046 ffff880079f1af10 ffff8800798effd8 [34582.508920] ffff8800798effd8 ffff8800798effd8 ffff880079f1af10 ffff88007c0dc000 [34582.510670] ffff8800798efda0 00000001020b1f78 ffff88007c0dc000 000000000000f888 [34582.512416] Call Trace: [34582.513672] [] schedule+0x29/0x70 [34582.515212] [] schedule_timeout+0x175/0x2d0 [34582.516765] [] ? internal_add_timer+0x70/0x70 [34582.518329] [] ksocknal_reaper+0x36e/0x3b0 [ksocklnd] [34582.519976] [] ? wake_up_state+0x20/0x20 [34582.521528] [] ? ksocknal_recv_hello+0x830/0x830 [ksocklnd] [34582.523209] [] kthread+0xcf/0xe0 [34582.524689] [] ? kthread_create_on_node+0x140/0x140 [34582.526309] [] ret_from_fork+0x58/0x90 [34582.527826] [] ? kthread_create_on_node+0x140/0x140 [34582.529463] socknal_sd00_00 S ffffc9000073a008 0 5309 2 0x00000080 [34582.531143] ffff88007a1ebe18 0000000000000046 ffff880079f1de20 ffff88007a1ebfd8 [34582.532906] ffff88007a1ebfd8 ffff88007a1ebfd8 ffff880079f1de20 ffffc9000073a000 [34582.534666] ffffc9000073a038 ffff88007a1ebe68 ffffc9000073a008 ffffc9000073a008 [34582.536432] Call Trace: [34582.537682] [] schedule+0x29/0x70 [34582.539219] [] ksocknal_scheduler+0x617/0x670 [ksocklnd] [34582.540880] [] ? wake_up_atomic_t+0x30/0x30 [34582.542439] [] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] [34582.544076] [] kthread+0xcf/0xe0 [34582.545581] [] ? kthread_create_on_node+0x140/0x140 [34582.547224] [] ret_from_fork+0x58/0x90 [34582.548765] [] ? kthread_create_on_node+0x140/0x140 [34582.550386] socknal_sd00_01 S ffffc9000073b868 0 5310 2 0x00000080 [34582.552064] ffff8800798bfe18 0000000000000046 ffff880079f1bec0 ffff8800798bffd8 [34582.553823] ffff8800798bffd8 ffff8800798bffd8 ffff880079f1bec0 ffffc9000073b860 [34582.555585] ffffc9000073b898 ffff8800798bfe68 ffffc9000073b868 ffffc9000073b868 [34582.557354] Call Trace: [34582.558608] [] schedule+0x29/0x70 [34582.560126] [] ksocknal_scheduler+0x617/0x670 [ksocklnd] [34582.561797] [] ? wake_up_atomic_t+0x30/0x30 [34582.563379] [] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] [34582.565012] [] kthread+0xcf/0xe0 [34582.566511] [] ? kthread_create_on_node+0x140/0x140 [34582.568146] [] ret_from_fork+0x58/0x90 [34582.569682] [] ? kthread_create_on_node+0x140/0x140 [34582.571308] acceptor_000 S 0000000000000000 0 5311 2 0x00000080 [34582.572990] ffff880079a37db8 0000000000000046 ffff88007a14af10 ffff880079a37fd8 [34582.574755] ffff880079a37fd8 ffff880079a37fd8 ffff88007a14af10 ffff8800355f1400 [34582.576516] ffff88007a14af10 ffff880079a37e70 ffff880035c3a600 0000000000000000 [34582.578307] Call Trace: [34582.579585] [] schedule+0x29/0x70 [34582.581184] [] lnet_sock_accept+0x15d/0x190 [lnet] [34582.582801] [] ? wake_up_state+0x20/0x20 [34582.584365] [] lnet_acceptor+0x1be/0xc10 [lnet] [34582.585979] [] ? lnet_connect+0x170/0x170 [lnet] [34582.587590] [] kthread+0xcf/0xe0 [34582.589071] [] ? kthread_create_on_node+0x140/0x140 [34582.590686] [] ret_from_fork+0x58/0x90 [34582.592216] [] ? kthread_create_on_node+0x140/0x140 [34582.593837] router_checker S 0000000000000000 0 5312 2 0x00000080 [34582.595525] ffff880079803dd0 0000000000000046 ffff88007a14bec0 ffff880079803fd8 [34582.597303] ffff880079803fd8 ffff880079803fd8 ffff88007a14bec0 ffff88007a14bec0 [34582.599033] 0000000000000000 ffffffffa07cec80 0000000000000000 0000000000000000 [34582.600794] Call Trace: [34582.602099] [] ? lnet_notify_locked+0x240/0x240 [lnet] [34582.603776] [] schedule+0x29/0x70 [34582.605295] [] lnet_router_checker+0x6a2/0x890 [lnet] [34582.606959] [] ? __switch_to+0xd9/0x4c0 [34582.608506] [] ? __schedule+0x3b0/0x990 [34582.610055] [] ? wake_up_atomic_t+0x30/0x30 [34582.611629] [] ? lnet_notify_locked+0x240/0x240 [lnet] [34582.613271] [] kthread+0xcf/0xe0 [34582.614758] [] ? kthread_create_on_node+0x140/0x140 [34582.616375] [] ret_from_fork+0x58/0x90 [34582.617909] [] ? kthread_create_on_node+0x140/0x140 [34582.619522] ptlrpcd_rcv S 000000000000f888 0 5313 2 0x00000080 [34582.621200] ffff880078a57cf0 0000000000000046 ffff88007a14edd0 ffff880078a57fd8 [34582.622964] ffff880078a57fd8 ffff880078a57fd8 ffff88007a14edd0 ffff88007c0dc000 [34582.624727] ffff880078a57d20 00000001020b1cb8 ffff88007c0dc000 000000000000f888 [34582.626500] Call Trace: [34582.627763] [] schedule+0x29/0x70 [34582.629284] [] schedule_timeout+0x175/0x2d0 [34582.630850] [] ? internal_add_timer+0x70/0x70 [34582.632466] [] ptlrpcd+0x298/0x560 [ptlrpc] [34582.634028] [] ? wake_up_state+0x20/0x20 [34582.635598] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [34582.637218] [] kthread+0xcf/0xe0 [34582.638679] [] ? kthread_create_on_node+0x140/0x140 [34582.640277] [] ret_from_fork+0x58/0x90 [34582.641776] [] ? kthread_create_on_node+0x140/0x140 [34582.643387] ptlrpcd_00_00 S 000000000000f888 0 5314 2 0x00000080 [34582.645041] ffff88007b5e7cf0 0000000000000046 ffff880078f6de20 ffff88007b5e7fd8 [34582.646770] ffff88007b5e7fd8 ffff88007b5e7fd8 ffff880078f6de20 ffff88007c0dc000 [34582.648504] ffff88007b5e7d20 00000001020b1fa1 ffff88007c0dc000 000000000000f888 [34582.650240] Call Trace: [34582.651482] [] schedule+0x29/0x70 [34582.653007] [] schedule_timeout+0x175/0x2d0 [34582.654552] [] ? internal_add_timer+0x70/0x70 [34582.656143] [] ptlrpcd+0x298/0x560 [ptlrpc] [34582.657675] [] ? wake_up_state+0x20/0x20 [34582.659215] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [34582.660798] [] kthread+0xcf/0xe0 [34582.662251] [] ? kthread_create_on_node+0x140/0x140 [34582.663842] [] ret_from_fork+0x58/0x90 [34582.665331] [] ? kthread_create_on_node+0x140/0x140 [34582.666933] ptlrpcd_00_01 S 000000000000f888 0 5315 2 0x00000080 [34582.668587] ffff880079f07cf0 0000000000000046 ffff880078f6ce70 ffff880079f07fd8 [34582.670324] ffff880079f07fd8 ffff880079f07fd8 ffff880078f6ce70 ffff88007c0dc000 [34582.672013] ffff880079f07d20 00000001020b1f71 ffff88007c0dc000 000000000000f888 [34582.673750] Call Trace: [34582.675024] [] schedule+0x29/0x70 [34582.676473] [] schedule_timeout+0x175/0x2d0 [34582.677987] [] ? internal_add_timer+0x70/0x70 [34582.679530] [] ptlrpcd+0x298/0x560 [ptlrpc] [34582.681049] [] ? wake_up_state+0x20/0x20 [34582.682557] [] ? ptlrpcd_check+0x5d0/0x5d0 [ptlrpc] [34582.684128] [] kthread+0xcf/0xe0 [34582.685568] [] ? kthread_create_on_node+0x140/0x140 [34582.687156] [] ret_from_fork+0x58/0x90 [34582.688628] [] ? kthread_create_on_node+0x140/0x140 [34582.690206] ll_ping S 000000000000f888 0 5316 2 0x00000080 [34582.691833] ffff88007a1c3d20 0000000000000046 ffff880078f6edd0 ffff88007a1c3fd8 [34582.693543] ffff88007a1c3fd8 ffff88007a1c3fd8 ffff880078f6edd0 ffff88007c0dc000 [34582.695253] ffff88007a1c3d50 00000001020b2870 ffff88007c0dc000 000000000000f888 [34582.696916] Call Trace: [34582.698126] [] schedule+0x29/0x70 [34582.699584] [] schedule_timeout+0x175/0x2d0 [34582.701097] [] ? internal_add_timer+0x70/0x70 [34582.702651] [] ptlrpc_pinger_main+0x92c/0xbb0 [ptlrpc] [34582.704250] [] ? wake_up_state+0x20/0x20 [34582.705767] [] ? ptlrpc_obd_ping+0x130/0x130 [ptlrpc] [34582.707360] [] kthread+0xcf/0xe0 [34582.708800] [] ? kthread_create_on_node+0x140/0x140 [34582.710391] [] ret_from_fork+0x58/0x90 [34582.711880] [] ? kthread_create_on_node+0x140/0x140 [34582.713452] sptlrpc_gc S 000000000000f888 0 5317 2 0x00000080 [34582.715102] ffff880079f3fd70 0000000000000046 ffff880078775e20 ffff880079f3ffd8 [34582.716823] ffff880079f3ffd8 ffff880079f3ffd8 ffff880078775e20 ffff88007c0dc000 [34582.718543] ffff880079f3fda0 00000001020dbb7c ffff88007c0dc000 000000000000f888 [34582.720257] Call Trace: [34582.721476] [] schedule+0x29/0x70 [34582.722993] [] schedule_timeout+0x175/0x2d0 [34582.724505] [] ? internal_add_timer+0x70/0x70 [34582.726030] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [34582.727647] [] sec_gc_main+0x2e5/0x420 [ptlrpc] [34582.729228] [] ? wake_up_state+0x20/0x20 [34582.730764] [] ? sptlrpc_gc_add_ctx+0x130/0x130 [ptlrpc] [34582.732389] [] kthread+0xcf/0xe0 [34582.733842] [] ? kthread_create_on_node+0x140/0x140 [34582.735427] [] ret_from_fork+0x58/0x90 [34582.736942] [] ? kthread_create_on_node+0x140/0x140 [34582.738540] kworker/u4:1 S ffff88007c060800 0 16806 2 0x00000080 [34582.740189] ffff88007a337e50 0000000000000046 ffff88005f60ce70 ffff88007a337fd8 [34582.741922] ffff88007a337fd8 ffff88007a337fd8 ffff88005f60ce70 ffff88007c060820 [34582.743644] ffff88007b61fc30 ffff88005f60ce70 ffff88007b61fc00 ffff88007c060800 [34582.745382] Call Trace: [34582.746607] [] schedule+0x29/0x70 [34582.748130] [] worker_thread+0x1d9/0x410 [34582.749623] [] ? rescuer_thread+0x460/0x460 [34582.751149] [] kthread+0xcf/0xe0 [34582.752599] [] ? kthread_create_on_node+0x140/0x140 [34582.754192] [] ret_from_fork+0x58/0x90 [34582.755687] [] ? kthread_create_on_node+0x140/0x140 [34582.757280] kworker/1:1 S ffff88007fd16480 0 17793 2 0x00000080 [34582.758951] ffff880068c77e50 0000000000000046 ffff880035baaf10 ffff880068c77fd8 [34582.760703] ffff880068c77fd8 ffff880068c77fd8 ffff880035baaf10 ffff88007fd164a0 [34582.762446] ffff880036391bb0 ffff880035baaf10 ffff880036391b80 ffff88007fd16480 [34582.764152] Call Trace: [34582.765376] [] schedule+0x29/0x70 [34582.766857] [] worker_thread+0x1d9/0x410 [34582.768367] [] ? rescuer_thread+0x460/0x460 [34582.769895] [] kthread+0xcf/0xe0 [34582.771328] [] ? kthread_create_on_node+0x140/0x140 [34582.772906] [] ret_from_fork+0x58/0x90 [34582.774397] [] ? kthread_create_on_node+0x140/0x140 [34582.775987] ldlm_cb00_000 S ffff88004ea13400 0 18480 2 0x00000080 [34582.777630] ffff8800690ffd38 0000000000000046 ffff880035ba8fb0 ffff8800690fffd8 [34582.779372] ffff8800690fffd8 ffff8800690fffd8 ffff880035ba8fb0 0000000000000000 [34582.781076] ffff880035ba8fb0 ffff880036391200 ffff88004ea13468 ffff88004ea13400 [34582.782762] Call Trace: [34582.784037] [] schedule+0x29/0x70 [34582.785536] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34582.787154] [] ? wake_up_state+0x20/0x20 [34582.788694] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34582.790292] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34582.791971] [] kthread+0xcf/0xe0 [34582.793442] [] ? kthread_create_on_node+0x140/0x140 [34582.795052] [] ret_from_fork+0x58/0x90 [34582.796543] [] ? kthread_create_on_node+0x140/0x140 [34582.798141] ldlm_cb00_001 S ffff88004ea13400 0 18482 2 0x00000080 [34582.799791] ffff880061bcfd38 0000000000000046 ffff8800361d9f60 ffff880061bcffd8 [34582.801548] ffff880061bcffd8 ffff880061bcffd8 ffff8800361d9f60 0000000000000000 [34582.803304] ffff8800361d9f60 ffff88007a021700 ffff88004ea13468 ffff88004ea13400 [34582.805014] Call Trace: [34582.806265] [] schedule+0x29/0x70 [34582.807774] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34582.809409] [] ? wake_up_state+0x20/0x20 [34582.810980] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34582.812597] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34582.814299] [] kthread+0xcf/0xe0 [34582.815778] [] ? kthread_create_on_node+0x140/0x140 [34582.817400] [] ret_from_fork+0x58/0x90 [34582.818933] [] ? kthread_create_on_node+0x140/0x140 [34582.820541] ldlm_cn00_000 S ffff88004ea11400 0 18483 2 0x00000080 [34582.822210] ffff880064be3d38 0000000000000046 ffff8800361dedd0 ffff880064be3fd8 [34582.823975] ffff880064be3fd8 ffff880064be3fd8 ffff8800361dedd0 0000000000000000 [34582.825741] ffff8800361dedd0 ffff88007a021500 ffff88004ea11468 ffff88004ea11400 [34582.827517] Call Trace: [34582.828796] [] schedule+0x29/0x70 [34582.830391] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34582.832068] [] ? wake_up_state+0x20/0x20 [34582.833656] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34582.835310] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34582.837027] [] kthread+0xcf/0xe0 [34582.838515] [] ? kthread_create_on_node+0x140/0x140 [34582.840191] [] ret_from_fork+0x58/0x90 [34582.841745] [] ? kthread_create_on_node+0x140/0x140 [34582.843401] ldlm_cn00_001 S ffff88004ea11400 0 18484 2 0x00000080 [34582.845117] ffff880062bdfd38 0000000000000046 ffff880065c73ec0 ffff880062bdffd8 [34582.846927] ffff880062bdffd8 ffff880062bdffd8 ffff880065c73ec0 0000000000000000 [34582.848742] ffff880065c73ec0 ffff88007a021f00 ffff88004ea11468 ffff88004ea11400 [34582.850572] Call Trace: [34582.851900] [] schedule+0x29/0x70 [34582.853516] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34582.855205] [] ? wake_up_state+0x20/0x20 [34582.856833] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34582.858519] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34582.860262] [] kthread+0xcf/0xe0 [34582.861790] [] ? kthread_create_on_node+0x140/0x140 [34582.863466] [] ret_from_fork+0x58/0x90 [34582.865046] [] ? kthread_create_on_node+0x140/0x140 [34582.866709] ldlm_cn00_002 S ffff88004ea11400 0 18485 2 0x00000080 [34582.868473] ffff8800474f3d38 0000000000000046 ffff880065c70000 ffff8800474f3fd8 [34582.870262] ffff8800474f3fd8 ffff8800474f3fd8 ffff880065c70000 0000000000000000 [34582.872044] ffff880065c70000 ffff88007a021680 ffff88004ea11468 ffff88004ea11400 [34582.873867] Call Trace: [34582.875242] [] schedule+0x29/0x70 [34582.876824] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34582.878511] [] ? wake_up_state+0x20/0x20 [34582.880133] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34582.881807] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34582.883548] [] kthread+0xcf/0xe0 [34582.885092] [] ? kthread_create_on_node+0x140/0x140 [34582.886776] [] ret_from_fork+0x58/0x90 [34582.888364] [] ? kthread_create_on_node+0x140/0x140 [34582.890038] ldlm_bl_01 S ffff88007a021e28 0 18486 2 0x00000080 [34582.891750] ffff88006678be08 0000000000000046 ffff880065c75e20 ffff88006678bfd8 [34582.893566] ffff88006678bfd8 ffff88006678bfd8 ffff880065c75e20 ffff88007a021e00 [34582.895383] ffff880065c75e20 ffff88007a021e64 0000000000000000 ffff88007a021e28 [34582.897178] Call Trace: [34582.898498] [] schedule+0x29/0x70 [34582.900080] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [34582.901774] [] ? wake_up_state+0x20/0x20 [34582.903392] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [34582.905164] [] kthread+0xcf/0xe0 [34582.906698] [] ? kthread_create_on_node+0x140/0x140 [34582.908380] [] ret_from_fork+0x58/0x90 [34582.909966] [] ? kthread_create_on_node+0x140/0x140 [34582.911621] ldlm_bl_02 S ffff88007a021e28 0 18487 2 0x00000080 [34582.913390] ffff88005e74be08 0000000000000046 ffff880065c71f60 ffff88005e74bfd8 [34582.915190] ffff88005e74bfd8 ffff88005e74bfd8 ffff880065c71f60 ffff88007a021e00 [34582.916976] ffff880065c71f60 ffff88007a021e64 0000000000000000 ffff88007a021e28 [34582.918778] Call Trace: [34582.920089] [] schedule+0x29/0x70 [34582.921694] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [34582.923399] [] ? wake_up_state+0x20/0x20 [34582.925030] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [34582.926767] [] kthread+0xcf/0xe0 [34582.928284] [] ? kthread_create_on_node+0x140/0x140 [34582.929977] [] ret_from_fork+0x58/0x90 [34582.931547] [] ? kthread_create_on_node+0x140/0x140 [34582.933208] ldlm_elt S 0000000000000000 0 18488 2 0x00000080 [34582.934940] ffff8800617ffe00 0000000000000046 ffff880065c72f10 ffff8800617fffd8 [34582.936774] ffff8800617fffd8 ffff8800617fffd8 ffff880065c72f10 ffff880065c72f10 [34582.938576] 0000000000000000 ffffffffa0a8da00 ffff880065c72f10 0000000000000000 [34582.940356] Call Trace: [34582.941681] [] ? ldlm_bl_to_thread+0x490/0x510 [ptlrpc] [34582.943431] [] schedule+0x29/0x70 [34582.945021] [] expired_lock_main+0x1c2/0x910 [ptlrpc] [34582.946723] [] ? wake_up_state+0x20/0x20 [34582.948312] [] ? ldlm_bl_to_thread+0x510/0x510 [ptlrpc] [34582.950041] [] kthread+0xcf/0xe0 [34582.951590] [] ? kthread_create_on_node+0x140/0x140 [34582.953260] [] ret_from_fork+0x58/0x90 [34582.954840] [] ? kthread_create_on_node+0x140/0x140 [34582.956505] ldlm_poold S 000000000000f888 0 18489 2 0x00000080 [34582.958236] ffff880077a67d70 0000000000000046 ffff880065c76dd0 ffff880077a67fd8 [34582.960046] ffff880077a67fd8 ffff880077a67fd8 ffff880065c76dd0 ffff88007c0dc000 [34582.961840] ffff880077a67da0 00000001020b1ee3 ffff88007c0dc000 000000000000f888 [34582.963632] Call Trace: [34582.964943] [] schedule+0x29/0x70 [34582.966536] [] schedule_timeout+0x175/0x2d0 [34582.968153] [] ? internal_add_timer+0x70/0x70 [34582.969770] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [34582.971458] [] ldlm_pools_thread_main+0x23b/0x330 [ptlrpc] [34582.973173] [] ? wake_up_state+0x20/0x20 [34582.974801] [] ? ldlm_pools_recalc+0x3d0/0x3d0 [ptlrpc] [34582.976487] [] kthread+0xcf/0xe0 [34582.978012] [] ? kthread_create_on_node+0x140/0x140 [34582.979665] [] ret_from_fork+0x58/0x90 [34582.981235] [] ? kthread_create_on_node+0x140/0x140 [34582.982885] ll_evictor S ffff88007b67d8d0 0 18494 2 0x00000080 [34582.984596] ffff880060e6bdd0 0000000000000046 ffff880078770000 ffff880060e6bfd8 [34582.986410] ffff880060e6bfd8 ffff880060e6bfd8 ffff880078770000 0000000000000000 [34582.988181] ffff880078770000 0000000058e7b8c5 0000000058e7b499 ffff88007b67d8d0 [34582.989949] Call Trace: [34582.991294] [] schedule+0x29/0x70 [34582.992852] [] ping_evictor_main+0x476/0x6c0 [ptlrpc] [34582.994525] [] ? wake_up_state+0x20/0x20 [34582.996151] [] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc] [34582.997909] [] kthread+0xcf/0xe0 [34582.999439] [] ? kthread_create_on_node+0x140/0x140 [34583.001101] [] ret_from_fork+0x58/0x90 [34583.002660] [] ? kthread_create_on_node+0x140/0x140 [34583.004317] ll_cfg_requeue S ffff880078770fb0 0 18495 2 0x00000080 [34583.006035] ffff880036003ab0 0000000000000046 ffff880078770fb0 ffff880036003fd8 [34583.007850] ffff880036003fd8 ffff880036003fd8 ffff880078770fb0 ffff880066b93c00 [34583.009648] ffff880078770fb0 0000000000000000 ffffffffa0a85300 ffff880078770fb0 [34583.011459] Call Trace: [34583.012789] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [34583.014580] [] schedule+0x29/0x70 [34583.016154] [] ldlm_completion_ast+0x62d/0x910 [ptlrpc] [34583.017882] [] ? wake_up_state+0x20/0x20 [34583.019524] [] ldlm_cli_enqueue_fini+0x938/0xdb0 [ptlrpc] [34583.021259] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34583.022976] [] ldlm_cli_enqueue+0x6c2/0x810 [ptlrpc] [34583.024681] [] ? ldlm_expired_completion_wait+0x240/0x240 [ptlrpc] [34583.026503] [] ? mgc_setup+0x620/0x620 [mgc] [34583.028138] [] mgc_enqueue.isra.7.constprop.19+0x19a/0x320 [mgc] [34583.029920] [] ? mgc_setup+0x620/0x620 [mgc] [34583.031580] [] ? ldlm_expired_completion_wait+0x240/0x240 [ptlrpc] [34583.033389] [] mgc_process_log+0x183/0x890 [mgc] [34583.035082] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34583.036794] [] mgc_requeue_thread+0x2b8/0x880 [mgc] [34583.038495] [] ? wake_up_state+0x20/0x20 [34583.040110] [] ? mgc_process_config+0x1600/0x1600 [mgc] [34583.041837] [] kthread+0xcf/0xe0 [34583.043386] [] ? kthread_create_on_node+0x140/0x140 [34583.045086] [] ret_from_fork+0x58/0x90 [34583.046684] [] ? kthread_create_on_node+0x140/0x140 [34583.048377] mdt00_000 S ffff880063bea400 0 18496 2 0x00000080 [34583.050125] ffff880064e9fd38 0000000000000046 ffff880049b74e70 ffff880064e9ffd8 [34583.051963] ffff880064e9ffd8 ffff880064e9ffd8 ffff880049b74e70 0000000000000000 [34583.053797] ffff880049b74e70 ffff8800643b6880 ffff880063bea468 ffff880063bea400 [34583.055636] Call Trace: [34583.056976] [] schedule+0x29/0x70 [34583.058623] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.060353] [] ? wake_up_state+0x20/0x20 [34583.062012] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.063713] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.065456] [] kthread+0xcf/0xe0 [34583.067041] [] ? kthread_create_on_node+0x140/0x140 [34583.068731] [] ret_from_fork+0x58/0x90 [34583.070325] [] ? kthread_create_on_node+0x140/0x140 [34583.072012] mdt00_001 S ffff880063bea400 0 18497 2 0x00000080 [34583.073751] ffff88007a05fd38 0000000000000046 ffff880049b76dd0 ffff88007a05ffd8 [34583.075590] ffff88007a05ffd8 ffff88007a05ffd8 ffff880049b76dd0 0000000000000000 [34583.077430] ffff880049b76dd0 ffff8800643b6a00 ffff880063bea468 ffff880063bea400 [34583.079228] Call Trace: [34583.080556] [] schedule+0x29/0x70 [34583.082158] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.083879] [] ? wake_up_state+0x20/0x20 [34583.085518] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.087208] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.088990] [] kthread+0xcf/0xe0 [34583.090563] [] ? kthread_create_on_node+0x140/0x140 [34583.092250] [] ret_from_fork+0x58/0x90 [34583.093839] [] ? kthread_create_on_node+0x140/0x140 [34583.095505] mdt00_002 S ffff880063bea400 0 18498 2 0x00000080 [34583.097225] ffff8800607c3d38 0000000000000046 ffff880049b70000 ffff8800607c3fd8 [34583.099036] ffff8800607c3fd8 ffff8800607c3fd8 ffff880049b70000 0000000000000000 [34583.100858] ffff880049b70000 ffff8800643b6700 ffff880063bea468 ffff880063bea400 [34583.102658] Call Trace: [34583.103985] [] schedule+0x29/0x70 [34583.105585] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.107283] [] ? wake_up_state+0x20/0x20 [34583.108892] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.110564] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.112308] [] kthread+0xcf/0xe0 [34583.113839] [] ? kthread_create_on_node+0x140/0x140 [34583.115489] [] ret_from_fork+0x58/0x90 [34583.117058] [] ? kthread_create_on_node+0x140/0x140 [34583.118710] mdt_rdpg00_000 S ffff880064936000 0 18499 2 0x00000080 [34583.120468] ffff880007f33d38 0000000000000046 ffff880049b73ec0 ffff880007f33fd8 [34583.122254] ffff880007f33fd8 ffff880007f33fd8 ffff880049b73ec0 0000000000000000 [34583.124024] ffff880049b73ec0 ffff8800643b6200 ffff880064936068 ffff880064936000 [34583.125832] Call Trace: [34583.127189] [] schedule+0x29/0x70 [34583.128748] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.130436] [] ? wake_up_state+0x20/0x20 [34583.132053] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.133730] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.135468] [] kthread+0xcf/0xe0 [34583.137011] [] ? kthread_create_on_node+0x140/0x140 [34583.138666] [] ret_from_fork+0x58/0x90 [34583.140232] [] ? kthread_create_on_node+0x140/0x140 [34583.141905] mdt_rdpg00_001 S ffff880064936000 0 18500 2 0x00000080 [34583.143629] ffff880061323d38 0000000000000046 ffff880049b70fb0 ffff880061323fd8 [34583.145440] ffff880061323fd8 ffff880061323fd8 ffff880049b70fb0 0000000000000000 [34583.147227] ffff880049b70fb0 ffff8800643b6080 ffff880064936068 ffff880064936000 [34583.148997] Call Trace: [34583.150347] [] schedule+0x29/0x70 [34583.151906] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.153593] [] ? wake_up_state+0x20/0x20 [34583.155223] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.156904] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.158637] [] kthread+0xcf/0xe0 [34583.160165] [] ? kthread_create_on_node+0x140/0x140 [34583.161830] [] ret_from_fork+0x58/0x90 [34583.163411] [] ? kthread_create_on_node+0x140/0x140 [34583.165077] mdt_attr00_000 S ffff880077c6e800 0 18501 2 0x00000080 [34583.166784] ffff8800663f7d38 0000000000000046 ffff880049b71f60 ffff8800663f7fd8 [34583.168594] ffff8800663f7fd8 ffff8800663f7fd8 ffff880049b71f60 0000000000000000 [34583.170401] ffff880049b71f60 ffff8800643b6400 ffff880077c6e868 ffff880077c6e800 [34583.172178] Call Trace: [34583.173485] [] schedule+0x29/0x70 [34583.175088] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.176758] [] ? wake_up_state+0x20/0x20 [34583.178369] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.180020] [] ? __switch_to+0xd9/0x4c0 [34583.181586] [] ? __schedule+0x3b0/0x990 [34583.183184] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.184924] [] kthread+0xcf/0xe0 [34583.186450] [] ? kthread_create_on_node+0x140/0x140 [34583.188108] [] ret_from_fork+0x58/0x90 [34583.189663] [] ? kthread_create_on_node+0x140/0x140 [34583.191311] mdt_attr00_001 S ffff880077c6e800 0 18502 2 0x00000080 [34583.193026] ffff88005fe27d38 0000000000000046 ffff880049b72f10 ffff88005fe27fd8 [34583.194839] ffff88005fe27fd8 ffff88005fe27fd8 ffff880049b72f10 0000000000000000 [34583.196645] ffff880049b72f10 ffff8800643b6680 ffff880077c6e868 ffff880077c6e800 [34583.198434] Call Trace: [34583.199722] [] schedule+0x29/0x70 [34583.201310] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.203004] [] ? wake_up_state+0x20/0x20 [34583.204607] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.206222] [] ? __switch_to+0xd9/0x4c0 [34583.207792] [] ? __schedule+0x3b0/0x990 [34583.209373] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.211088] [] kthread+0xcf/0xe0 [34583.212586] [] ? kthread_create_on_node+0x140/0x140 [34583.214243] [] ret_from_fork+0x58/0x90 [34583.215797] [] ? kthread_create_on_node+0x140/0x140 [34583.217434] mdt_out00_000 S ffff880077c67400 0 18503 2 0x00000080 [34583.219138] ffff8800629bbd38 0000000000000046 ffff88005f608fb0 ffff8800629bbfd8 [34583.220916] ffff8800629bbfd8 ffff8800629bbfd8 ffff88005f608fb0 0000000000000000 [34583.222690] ffff88005f608fb0 ffff88005e64be00 ffff880077c67468 ffff880077c67400 [34583.224471] Call Trace: [34583.225759] [] schedule+0x29/0x70 [34583.227338] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.229003] [] ? wake_up_state+0x20/0x20 [34583.230602] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.232230] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.233975] [] kthread+0xcf/0xe0 [34583.235496] [] ? kthread_create_on_node+0x140/0x140 [34583.237148] [] ret_from_fork+0x58/0x90 [34583.238693] [] ? kthread_create_on_node+0x140/0x140 [34583.240336] mdt_out00_001 S ffff880077c67400 0 18504 2 0x00000080 [34583.242049] ffff880060a67d38 0000000000000046 ffff88005f608000 ffff880060a67fd8 [34583.243827] ffff880060a67fd8 ffff880060a67fd8 ffff88005f608000 0000000000000000 [34583.245597] ffff88005f608000 ffff88005e64b180 ffff880077c67468 ffff880077c67400 [34583.247380] Call Trace: [34583.248662] [] schedule+0x29/0x70 [34583.250249] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.251916] [] ? wake_up_state+0x20/0x20 [34583.253510] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.255154] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.256890] [] kthread+0xcf/0xe0 [34583.258405] [] ? kthread_create_on_node+0x140/0x140 [34583.260067] [] ret_from_fork+0x58/0x90 [34583.261623] [] ? kthread_create_on_node+0x140/0x140 [34583.263285] mdt_seqs_0000 S ffff880064774000 0 18505 2 0x00000080 [34583.265009] ffff8800666bfd38 0000000000000046 ffff88005f60af10 ffff8800666bffd8 [34583.266809] ffff8800666bffd8 ffff8800666bffd8 ffff88005f60af10 0000000000000000 [34583.268624] ffff88005f60af10 ffff88005e64b880 ffff880064774068 ffff880064774000 [34583.270410] Call Trace: [34583.271696] [] schedule+0x29/0x70 [34583.273284] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.274971] [] ? wake_up_state+0x20/0x20 [34583.276570] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.278207] [] ? __switch_to+0xd9/0x4c0 [34583.279769] [] ? __schedule+0x3b0/0x990 [34583.281344] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.283075] [] kthread+0xcf/0xe0 [34583.284595] [] ? kthread_create_on_node+0x140/0x140 [34583.286263] [] ret_from_fork+0x58/0x90 [34583.287833] [] ? kthread_create_on_node+0x140/0x140 [34583.289483] mdt_seqs_0001 S ffff880064774000 0 18506 2 0x00000080 [34583.291191] ffff88006156bd38 0000000000000046 ffff88005f60edd0 ffff88006156bfd8 [34583.292987] ffff88006156bfd8 ffff88006156bfd8 ffff88005f60edd0 0000000000000000 [34583.294785] ffff88005f60edd0 ffff88005e64bf00 ffff880064774068 ffff880064774000 [34583.296591] Call Trace: [34583.297891] [] schedule+0x29/0x70 [34583.299469] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.301148] [] ? wake_up_state+0x20/0x20 [34583.302748] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.304379] [] ? __switch_to+0xd9/0x4c0 [34583.305946] [] ? __schedule+0x3b0/0x990 [34583.307535] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.309261] [] kthread+0xcf/0xe0 [34583.310782] [] ? kthread_create_on_node+0x140/0x140 [34583.312674] [] ret_from_fork+0x58/0x90 [34583.314219] [] ? kthread_create_on_node+0x140/0x140 [34583.315843] mdt_seqm_0000 S ffff8800657c4c00 0 18507 2 0x00000080 [34583.317557] ffff880063d73d38 0000000000000046 ffff88005f609f60 ffff880063d73fd8 [34583.319331] ffff880063d73fd8 ffff880063d73fd8 ffff88005f609f60 0000000000000000 [34583.321085] ffff88005f609f60 ffff88005e64b200 ffff8800657c4c68 ffff8800657c4c00 [34583.322831] Call Trace: [34583.324158] [] schedule+0x29/0x70 [34583.325712] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.327583] [] ? wake_up_state+0x20/0x20 [34583.329171] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.330804] [] ? __switch_to+0xd9/0x4c0 [34583.332370] [] ? __schedule+0x3b0/0x990 [34583.333952] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.335662] [] kthread+0xcf/0xe0 [34583.337158] [] ? kthread_create_on_node+0x140/0x140 [34583.338793] [] ret_from_fork+0x58/0x90 [34583.340332] [] ? kthread_create_on_node+0x140/0x140 [34583.341961] mdt_seqm_0001 S ffff8800657c4c00 0 18508 2 0x00000080 [34583.343644] ffff88006566bd38 0000000000000046 ffff88005f60de20 ffff88006566bfd8 [34583.345428] ffff88006566bfd8 ffff88006566bfd8 ffff88005f60de20 0000000000000000 [34583.347190] ffff88005f60de20 ffff88005e64b300 ffff8800657c4c68 ffff8800657c4c00 [34583.348921] Call Trace: [34583.350221] [] schedule+0x29/0x70 [34583.351754] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.353411] [] ? wake_up_state+0x20/0x20 [34583.354994] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.356631] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.358353] [] kthread+0xcf/0xe0 [34583.359861] [] ? kthread_create_on_node+0x140/0x140 [34583.361504] [] ret_from_fork+0x58/0x90 [34583.363049] [] ? kthread_create_on_node+0x140/0x140 [34583.364687] mdt_fld_0000 S ffff88005f855800 0 18509 2 0x00000080 [34583.366428] ffff88000006bd38 0000000000000046 ffff880060d08000 ffff88000006bfd8 [34583.368197] ffff88000006bfd8 ffff88000006bfd8 ffff880060d08000 0000000000000000 [34583.369938] ffff880060d08000 ffff88005e64b080 ffff88005f855868 ffff88005f855800 [34583.371719] Call Trace: [34583.373010] [] schedule+0x29/0x70 [34583.374576] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.376210] [] ? wake_up_state+0x20/0x20 [34583.377826] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.379468] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.381188] [] kthread+0xcf/0xe0 [34583.382703] [] ? kthread_create_on_node+0x140/0x140 [34583.384362] [] ret_from_fork+0x58/0x90 [34583.385935] [] ? kthread_create_on_node+0x140/0x140 [34583.387592] mdt_fld_0001 S ffff88005f855800 0 18510 2 0x00000080 [34583.389351] ffff88000006fd38 0000000000000046 ffff880060d0bec0 ffff88000006ffd8 [34583.391146] ffff88000006ffd8 ffff88000006ffd8 ffff880060d0bec0 0000000000000000 [34583.392913] ffff880060d0bec0 ffff88005e64b100 ffff88005f855868 ffff88005f855800 [34583.394698] Call Trace: [34583.395983] [] schedule+0x29/0x70 [34583.397568] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.399245] [] ? wake_up_state+0x20/0x20 [34583.400848] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.402505] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.404230] [] kthread+0xcf/0xe0 [34583.405750] [] ? kthread_create_on_node+0x140/0x140 [34583.407394] [] ret_from_fork+0x58/0x90 [34583.408956] [] ? kthread_create_on_node+0x140/0x140 [34583.410612] mdt_out00_002 S ffff880077c67400 0 19436 2 0x00000080 [34583.412367] ffff880079b3bd38 0000000000000046 ffff880035b25e20 ffff880079b3bfd8 [34583.414155] ffff880079b3bfd8 ffff880079b3bfd8 ffff880035b25e20 0000000000000000 [34583.415926] ffff880035b25e20 ffff880062996400 ffff880077c67468 ffff880077c67400 [34583.417725] Call Trace: [34583.419047] [] schedule+0x29/0x70 [34583.420641] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34583.422321] [] ? wake_up_state+0x20/0x20 [34583.423938] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34583.425601] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34583.427348] [] kthread+0xcf/0xe0 [34583.428882] [] ? kthread_create_on_node+0x140/0x140 [34583.430548] [] ret_from_fork+0x58/0x90 [34583.432121] [] ? kthread_create_on_node+0x140/0x140 [34583.433771] jbd2/dm-2-8 S ffff880061aa9800 0 19811 2 0x00000080 [34583.435504] ffff88005e7d7e30 0000000000000046 ffff880035b20fb0 ffff88005e7d7fd8 [34583.437322] ffff88005e7d7fd8 ffff88005e7d7fd8 ffff880035b20fb0 ffff880061aa9828 [34583.439093] ffff880061aa9be8 ffff880061aa98a8 ffff88005e7d7e80 ffff880061aa9800 [34583.440896] Call Trace: [34583.442238] [] schedule+0x29/0x70 [34583.443780] [] kjournald2+0x203/0x260 [jbd2] [34583.445365] [] ? wake_up_atomic_t+0x30/0x30 [34583.446988] [] ? commit_timeout+0x10/0x10 [jbd2] [34583.448606] [] kthread+0xcf/0xe0 [34583.450131] [] ? kthread_create_on_node+0x140/0x140 [34583.451778] [] ret_from_fork+0x58/0x90 [34583.453322] [] ? kthread_create_on_node+0x140/0x140 [34583.454981] ldiskfs-rsv-con S ffff880062996180 0 19812 2 0x00000080 [34583.456684] ffff880063813e08 0000000000000046 ffff880035b23ec0 ffff880063813fd8 [34583.458480] ffff880063813fd8 ffff880063813fd8 ffff880035b23ec0 ffff880035b23ec0 [34583.460270] ffff880062996180 ffff88005f97f080 ffff8800629961b0 ffff880062996180 [34583.462010] Call Trace: [34583.463341] [] schedule+0x29/0x70 [34583.464864] [] rescuer_thread+0x359/0x460 [34583.466433] [] ? __schedule+0x3b0/0x990 [34583.467984] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34583.469613] [] kthread+0xcf/0xe0 [34583.471123] [] ? kthread_create_on_node+0x140/0x140 [34583.472741] [] ret_from_fork+0x58/0x90 [34583.474266] [] ? kthread_create_on_node+0x140/0x140 [34583.475888] osp_up0-2 S 0000000000000000 0 19814 2 0x00000080 [34583.477563] ffff880065f7fde0 0000000000000046 ffff8800361d8000 ffff880065f7ffd8 [34583.479331] ffff880065f7ffd8 ffff880065f7ffd8 ffff8800361d8000 ffff88005ff58960 [34583.481061] ffff8800361d8000 ffff88005f81d800 ffff88005ff58978 0000000000000000 [34583.482769] Call Trace: [34583.484058] [] schedule+0x29/0x70 [34583.485593] [] osp_send_update_thread+0x2f8/0x600 [osp] [34583.487259] [] ? wake_up_state+0x20/0x20 [34583.488805] [] ? osp_invalidate_request+0x390/0x390 [osp] [34583.490462] [] kthread+0xcf/0xe0 [34583.491950] [] ? kthread_create_on_node+0x140/0x140 [34583.493579] [] ret_from_fork+0x58/0x90 [34583.495101] [] ? kthread_create_on_node+0x140/0x140 [34583.496698] osp_up1-2 S 0000000000000000 0 19815 2 0x00000080 [34583.498398] ffff88004582bde0 0000000000000046 ffff88003611de20 ffff88004582bfd8 [34583.500132] ffff88004582bfd8 ffff88004582bfd8 ffff88003611de20 ffff88005fb94720 [34583.501846] ffff88003611de20 ffff8800779ae000 ffff88005fb94738 0000000000000000 [34583.503592] Call Trace: [34583.504838] [] schedule+0x29/0x70 [34583.506380] [] osp_send_update_thread+0x2f8/0x600 [osp] [34583.508037] [] ? wake_up_state+0x20/0x20 [34583.509572] [] ? osp_invalidate_request+0x390/0x390 [osp] [34583.511231] [] kthread+0xcf/0xe0 [34583.512698] [] ? kthread_create_on_node+0x140/0x140 [34583.514300] [] ret_from_fork+0x58/0x90 [34583.515830] [] ? kthread_create_on_node+0x140/0x140 [34583.517438] osp_up3-2 S 0000000000000000 0 19816 2 0x00000080 [34583.519098] ffff880066367de0 0000000000000046 ffff88003611bec0 ffff880066367fd8 [34583.520856] ffff880066367fd8 ffff880066367fd8 ffff88003611bec0 ffff88006252bcc0 [34583.522598] ffff88003611bec0 ffff880061aaf000 ffff88006252bcd8 0000000000000000 [34583.524366] Call Trace: [34583.525622] [] schedule+0x29/0x70 [34583.527156] [] osp_send_update_thread+0x2f8/0x600 [osp] [34583.528815] [] ? wake_up_state+0x20/0x20 [34583.530375] [] ? osp_invalidate_request+0x390/0x390 [osp] [34583.532046] [] kthread+0xcf/0xe0 [34583.533518] [] ? kthread_create_on_node+0x140/0x140 [34583.535146] [] ret_from_fork+0x58/0x90 [34583.536682] [] ? kthread_create_on_node+0x140/0x140 [34583.538297] osp-pre-0-2 S ffff880036119f60 0 19817 2 0x00000080 [34583.539969] ffff88005af3bd78 0000000000000046 ffff880036119f60 ffff88005af3bfd8 [34583.541742] ffff88005af3bfd8 ffff88005af3bfd8 ffff880036119f60 00000000ffffffff [34583.543517] ffffffff00000000 ffff880036119f60 ffff880061162000 ffff880036119f60 [34583.545306] Call Trace: [34583.546593] [] schedule+0x29/0x70 [34583.548164] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.549837] [] ? kvm_clock_read+0x1f/0x30 [34583.551425] [] ? wake_up_state+0x20/0x20 [34583.553004] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.554639] [] kthread+0xcf/0xe0 [34583.556120] [] ? kthread_create_on_node+0x140/0x140 [34583.557771] [] ret_from_fork+0x58/0x90 [34583.559315] [] ? kthread_create_on_node+0x140/0x140 [34583.560950] osp-syn-0-2 S 0000000000000000 0 19818 2 0x00000080 [34583.562637] ffff88005af3f9b8 0000000000000046 ffff88003611ce70 ffff88005af3ffd8 [34583.564427] ffff88005af3ffd8 ffff88005af3ffd8 ffff88003611ce70 ffff880061162000 [34583.566174] 0000000000000000 0000000000000001 ffff8800611622e8 0000000000000000 [34583.567921] Call Trace: [34583.569255] [] schedule+0x29/0x70 [34583.570773] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.572480] [] ? wake_up_state+0x20/0x20 [34583.574069] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.575775] [] ? llog_process_or_fork+0x56/0x450 [obdclass] [34583.577489] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.579157] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.580868] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.582563] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.584280] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.586020] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.587729] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.589473] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.591207] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.592861] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.594502] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.596150] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.597754] [] ? __switch_to+0xd9/0x4c0 [34583.599300] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.600996] [] kthread+0xcf/0xe0 [34583.602477] [] ? kthread_create_on_node+0x140/0x140 [34583.604110] [] ret_from_fork+0x58/0x90 [34583.605641] [] ? kthread_create_on_node+0x140/0x140 [34583.607276] osp-pre-1-2 S ffff8800611cce70 0 19819 2 0x00000080 [34583.608955] ffff88006535fd78 0000000000000046 ffff8800611cce70 ffff88006535ffd8 [34583.610724] ffff88006535ffd8 ffff88006535ffd8 ffff8800611cce70 00000000ffffffff [34583.612508] ffffffff00000000 ffff8800611cce70 ffff8800779ac800 ffff8800611cce70 [34583.614290] Call Trace: [34583.615566] [] schedule+0x29/0x70 [34583.617119] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.618758] [] ? kvm_clock_read+0x1f/0x30 [34583.620332] [] ? wake_up_state+0x20/0x20 [34583.621894] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.623519] [] kthread+0xcf/0xe0 [34583.625010] [] ? kthread_create_on_node+0x140/0x140 [34583.626612] [] ret_from_fork+0x58/0x90 [34583.628119] [] ? kthread_create_on_node+0x140/0x140 [34583.629755] osp-syn-1-2 S 0000000000000000 0 19820 2 0x00000080 [34583.631444] ffff8800675239b8 0000000000000046 ffff8800611cde20 ffff880067523fd8 [34583.633214] ffff880067523fd8 ffff880067523fd8 ffff8800611cde20 ffff8800779ac800 [34583.634947] 0000000000000000 0000000000000001 ffff8800779acae8 0000000000000000 [34583.636727] Call Trace: [34583.638015] [] schedule+0x29/0x70 [34583.639560] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.641241] [] ? wake_up_state+0x20/0x20 [34583.642821] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.644525] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.646161] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.647857] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.649541] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.651232] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.652955] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.654635] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.656364] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.658073] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.659694] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.661318] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.662946] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.664549] [] ? __switch_to+0xd9/0x4c0 [34583.666074] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.667742] [] kthread+0xcf/0xe0 [34583.669209] [] ? kthread_create_on_node+0x140/0x140 [34583.670831] [] ret_from_fork+0x58/0x90 [34583.672366] [] ? kthread_create_on_node+0x140/0x140 [34583.673987] osp-pre-2-2 S ffff8800611cbec0 0 19821 2 0x00000080 [34583.675668] ffff880061937d78 0000000000000046 ffff8800611cbec0 ffff880061937fd8 [34583.677456] ffff880061937fd8 ffff880061937fd8 ffff8800611cbec0 00000000ffffffff [34583.679210] ffffffff00000000 ffff8800611cbec0 ffff88003bc5f800 ffff8800611cbec0 [34583.680949] Call Trace: [34583.682264] [] schedule+0x29/0x70 [34583.683786] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.685454] [] ? kvm_clock_read+0x1f/0x30 [34583.687013] [] ? wake_up_state+0x20/0x20 [34583.688556] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.690183] [] kthread+0xcf/0xe0 [34583.691662] [] ? kthread_create_on_node+0x140/0x140 [34583.693279] [] ret_from_fork+0x58/0x90 [34583.694790] [] ? kthread_create_on_node+0x140/0x140 [34583.696406] osp-syn-2-2 S 0000000000000000 0 19822 2 0x00000080 [34583.698089] ffff880061a8b9b8 0000000000000046 ffff8800611caf10 ffff880061a8bfd8 [34583.699858] ffff880061a8bfd8 ffff880061a8bfd8 ffff8800611caf10 ffff88003bc5f800 [34583.701625] 0000000000000000 0000000000000001 ffff88003bc5fae8 0000000000000000 [34583.703401] Call Trace: [34583.704673] [] schedule+0x29/0x70 [34583.706217] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.707906] [] ? wake_up_state+0x20/0x20 [34583.709480] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.711189] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.712844] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.714544] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.716248] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.717947] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.719674] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.721368] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.723098] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.724813] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.726446] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.728078] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.729704] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.731325] [] ? __switch_to+0xd9/0x4c0 [34583.732860] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.734533] [] kthread+0xcf/0xe0 [34583.736012] [] ? kthread_create_on_node+0x140/0x140 [34583.737645] [] ret_from_fork+0x58/0x90 [34583.739192] [] ? kthread_create_on_node+0x140/0x140 [34583.740824] osp-pre-3-2 S ffff8800611c8fb0 0 19823 2 0x00000080 [34583.742525] ffff88007a493d78 0000000000000046 ffff8800611c8fb0 ffff88007a493fd8 [34583.744280] ffff88007a493fd8 ffff88007a493fd8 ffff8800611c8fb0 00000000ffffffff [34583.746014] ffffffff00000000 ffff8800611c8fb0 ffff880063ffe000 ffff8800611c8fb0 [34583.747790] Call Trace: [34583.749108] [] schedule+0x29/0x70 [34583.750635] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.752305] [] ? kvm_clock_read+0x1f/0x30 [34583.753876] [] ? wake_up_state+0x20/0x20 [34583.755428] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.757068] [] kthread+0xcf/0xe0 [34583.758581] [] ? kthread_create_on_node+0x140/0x140 [34583.760218] [] ret_from_fork+0x58/0x90 [34583.761745] [] ? kthread_create_on_node+0x140/0x140 [34583.763361] osp-syn-3-2 S 0000000000000000 0 19824 2 0x00000080 [34583.765034] ffff88007a4979b8 0000000000000046 ffff880060df4e70 ffff88007a497fd8 [34583.766789] ffff88007a497fd8 ffff88007a497fd8 ffff880060df4e70 ffff880063ffe000 [34583.768580] 0000000000000000 0000000000000001 ffff880063ffe2e8 0000000000000000 [34583.770353] Call Trace: [34583.771629] [] schedule+0x29/0x70 [34583.773187] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.774879] [] ? wake_up_state+0x20/0x20 [34583.776438] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.778148] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.779817] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.781514] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.783207] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.784907] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.786643] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.788334] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.790072] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.791785] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.793430] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.795061] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.796683] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.798275] [] ? __switch_to+0xd9/0x4c0 [34583.799780] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.801425] [] kthread+0xcf/0xe0 [34583.802920] [] ? kthread_create_on_node+0x140/0x140 [34583.804540] [] ret_from_fork+0x58/0x90 [34583.806071] [] ? kthread_create_on_node+0x140/0x140 [34583.807685] osp-pre-4-2 S ffff880060df5e20 0 19825 2 0x00000080 [34583.809412] ffff88007a283d78 0000000000000046 ffff880060df5e20 ffff88007a283fd8 [34583.811165] ffff88007a283fd8 ffff88007a283fd8 ffff880060df5e20 00000000ffffffff [34583.812913] ffffffff00000000 ffff880060df5e20 ffff88005f0c8000 ffff880060df5e20 [34583.814678] Call Trace: [34583.815957] [] schedule+0x29/0x70 [34583.817517] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.819178] [] ? kvm_clock_read+0x1f/0x30 [34583.820737] [] ? wake_up_state+0x20/0x20 [34583.822294] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.823922] [] kthread+0xcf/0xe0 [34583.825392] [] ? kthread_create_on_node+0x140/0x140 [34583.827014] [] ret_from_fork+0x58/0x90 [34583.828524] [] ? kthread_create_on_node+0x140/0x140 [34583.830151] osp-syn-4-2 S 0000000000000000 0 19826 2 0x00000080 [34583.831829] ffff88004447f9b8 0000000000000046 ffff880060df0fb0 ffff88004447ffd8 [34583.833599] ffff88004447ffd8 ffff88004447ffd8 ffff880060df0fb0 ffff88005f0c8000 [34583.835367] 0000000000000000 0000000000000001 ffff88005f0c82e8 0000000000000000 [34583.837113] Call Trace: [34583.838380] [] schedule+0x29/0x70 [34583.839912] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.841599] [] ? wake_up_state+0x20/0x20 [34583.843165] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.844862] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.846507] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.848195] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.849882] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.851563] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.853297] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.854974] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.856705] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.858421] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.860054] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.861685] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.863310] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.864925] [] ? __switch_to+0xd9/0x4c0 [34583.866452] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.868124] [] kthread+0xcf/0xe0 [34583.869600] [] ? kthread_create_on_node+0x140/0x140 [34583.871231] [] ret_from_fork+0x58/0x90 [34583.872767] [] ? kthread_create_on_node+0x140/0x140 [34583.874392] osp-pre-5-2 S ffff88006382bec0 0 19827 2 0x00000080 [34583.876074] ffff880064177d78 0000000000000046 ffff88006382bec0 ffff880064177fd8 [34583.877851] ffff880064177fd8 ffff880064177fd8 ffff88006382bec0 00000000ffffffff [34583.879626] ffffffff00000000 ffff88006382bec0 ffff88005f0ca000 ffff88006382bec0 [34583.881398] Call Trace: [34583.882674] [] schedule+0x29/0x70 [34583.884224] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.885887] [] ? kvm_clock_read+0x1f/0x30 [34583.887445] [] ? wake_up_state+0x20/0x20 [34583.889015] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.890647] [] kthread+0xcf/0xe0 [34583.892142] [] ? kthread_create_on_node+0x140/0x140 [34583.893759] [] ret_from_fork+0x58/0x90 [34583.895285] [] ? kthread_create_on_node+0x140/0x140 [34583.896905] osp-syn-5-2 S 0000000000000000 0 19828 2 0x00000080 [34583.898582] ffff8800506439b8 0000000000000046 ffff880063828000 ffff880050643fd8 [34583.900352] ffff880050643fd8 ffff880050643fd8 ffff880063828000 ffff88005f0ca000 [34583.902086] 0000000000000000 0000000000000001 ffff88005f0ca2e8 0000000000000000 [34583.903816] Call Trace: [34583.905106] [] schedule+0x29/0x70 [34583.906627] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.908319] [] ? wake_up_state+0x20/0x20 [34583.909887] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.911577] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.913229] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.914916] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.916600] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.918298] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.920027] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.921713] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.923432] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.925148] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.926777] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.928410] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.930039] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.931629] [] ? __switch_to+0xd9/0x4c0 [34583.933163] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34583.934835] [] kthread+0xcf/0xe0 [34583.936310] [] ? kthread_create_on_node+0x140/0x140 [34583.937933] [] ret_from_fork+0x58/0x90 [34583.939450] [] ? kthread_create_on_node+0x140/0x140 [34583.941083] osp-pre-6-2 S ffff88006382af10 0 19829 2 0x00000080 [34583.942751] ffff880062a23d78 0000000000000046 ffff88006382af10 ffff880062a23fd8 [34583.944584] ffff880062a23fd8 ffff880062a23fd8 ffff88006382af10 00000000ffffffff [34583.946373] ffffffff00000000 ffff88006382af10 ffff88004471b000 ffff88006382af10 [34583.948117] Call Trace: [34583.949388] [] schedule+0x29/0x70 [34583.950919] [] osp_precreate_thread+0xba4/0x1230 [osp] [34583.952581] [] ? kvm_clock_read+0x1f/0x30 [34583.954138] [] ? wake_up_state+0x20/0x20 [34583.955683] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34583.957311] [] kthread+0xcf/0xe0 [34583.958787] [] ? kthread_create_on_node+0x140/0x140 [34583.960405] [] ret_from_fork+0x58/0x90 [34583.961936] [] ? kthread_create_on_node+0x140/0x140 [34583.963542] osp-syn-6-2 S 0000000000000000 0 19830 2 0x00000080 [34583.965218] ffff880062a279b8 0000000000000046 ffff88006382edd0 ffff880062a27fd8 [34583.966980] ffff880062a27fd8 ffff880062a27fd8 ffff88006382edd0 ffff88004471b000 [34583.968746] 0000000000000000 0000000000000001 ffff88004471b2e8 0000000000000000 [34583.970528] Call Trace: [34583.971799] [] schedule+0x29/0x70 [34583.973293] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34583.974959] [] ? wake_up_state+0x20/0x20 [34583.976485] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.978154] [] ? llog_process_or_fork+0x56/0x450 [obdclass] [34583.979826] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.981429] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.983088] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34583.984738] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34583.986397] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.988100] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34583.989743] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34583.991464] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34583.993152] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.994756] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34583.996356] [] llog_cat_process+0x2e/0x30 [obdclass] [34583.997959] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34583.999515] [] ? __switch_to+0xd9/0x4c0 [34584.001022] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34584.002661] [] kthread+0xcf/0xe0 [34584.004136] [] ? kthread_create_on_node+0x140/0x140 [34584.005726] [] ret_from_fork+0x58/0x90 [34584.007230] [] ? kthread_create_on_node+0x140/0x140 [34584.008827] osp-pre-7-2 S ffff88006382de20 0 19831 2 0x00000080 [34584.010511] ffff88005f147d78 0000000000000046 ffff88006382de20 ffff88005f147fd8 [34584.012257] ffff88005f147fd8 ffff88005f147fd8 ffff88006382de20 00000000ffffffff [34584.013987] ffffffff00000000 ffff88006382de20 ffff88004471a800 ffff88006382de20 [34584.015723] Call Trace: [34584.017009] [] schedule+0x29/0x70 [34584.018488] [] osp_precreate_thread+0xba4/0x1230 [osp] [34584.020113] [] ? kvm_clock_read+0x1f/0x30 [34584.021661] [] ? wake_up_state+0x20/0x20 [34584.023443] [] ? osp_init_pre_fid+0x5f0/0x5f0 [osp] [34584.025065] [] kthread+0xcf/0xe0 [34584.026551] [] ? kthread_create_on_node+0x140/0x140 [34584.028170] [] ret_from_fork+0x58/0x90 [34584.029687] [] ? kthread_create_on_node+0x140/0x140 [34584.031322] osp-syn-7-2 S 0000000000000000 0 19832 2 0x00000080 [34584.033105] ffff880063b4b9b8 0000000000000046 ffff880063828fb0 ffff880063b4bfd8 [34584.034866] ffff880063b4bfd8 ffff880063b4bfd8 ffff880063828fb0 ffff88004471a800 [34584.036631] 0000000000000000 0000000000000001 ffff88004471aae8 0000000000000000 [34584.038404] Call Trace: [34584.039677] [] schedule+0x29/0x70 [34584.041228] [] osp_sync_process_queues+0x17c8/0x2340 [osp] [34584.042914] [] ? wake_up_state+0x20/0x20 [34584.044492] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34584.046203] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34584.047848] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34584.049550] [] llog_cat_process_cb+0x56d/0x620 [obdclass] [34584.051237] [] llog_process_thread+0x5a5/0x1240 [obdclass] [34584.052941] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34584.054674] [] llog_process_or_fork+0xbc/0x450 [obdclass] [34584.056364] [] ? llog_cat_cancel_records+0x350/0x350 [obdclass] [34584.058100] [] llog_cat_process_or_fork+0x1c9/0x2f0 [obdclass] [34584.059802] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34584.061440] [] ? osp_sync_thread+0x9d0/0x9d0 [osp] [34584.063066] [] llog_cat_process+0x2e/0x30 [obdclass] [34584.064756] [] osp_sync_thread+0x1fa/0x9d0 [osp] [34584.066353] [] ? __switch_to+0xd9/0x4c0 [34584.067904] [] ? osp_sync_add_rec.isra.19+0xa20/0xa20 [osp] [34584.069582] [] kthread+0xcf/0xe0 [34584.071062] [] ? kthread_create_on_node+0x140/0x140 [34584.072697] [] ret_from_fork+0x58/0x90 [34584.074237] [] ? kthread_create_on_node+0x140/0x140 [34584.075870] hsm_cdtr S 000000000000f888 0 19840 2 0x00000080 [34584.077567] ffff88007ae6bce0 0000000000000046 ffff88007ae60fb0 ffff88007ae6bfd8 [34584.079351] ffff88007ae6bfd8 ffff88007ae6bfd8 ffff88007ae60fb0 ffffffff81e380c0 [34584.081102] ffff88007ae6bd10 00000001020b2bd3 ffffffff81e380c0 000000000000f888 [34584.082840] Call Trace: [34584.084146] [] schedule+0x29/0x70 [34584.085652] [] schedule_timeout+0x175/0x2d0 [34584.087221] [] ? internal_add_timer+0x70/0x70 [34584.088808] [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] [34584.090501] [] mdt_coordinator+0x9d9/0xf30 [mdt] [34584.092101] [] ? wake_up_state+0x20/0x20 [34584.093629] [] ? mdt_hsm_cdt_stop+0x620/0x620 [mdt] [34584.095248] [] kthread+0xcf/0xe0 [34584.096715] [] ? kthread_create_on_node+0x140/0x140 [34584.098324] [] ret_from_fork+0x58/0x90 [34584.099851] [] ? kthread_create_on_node+0x140/0x140 [34584.101465] lquota_wb_lustr S 000000000000f888 0 19847 2 0x00000080 [34584.103136] ffff880061c2fd30 0000000000000046 ffff88005f764e70 ffff880061c2ffd8 [34584.104901] ffff880061c2ffd8 ffff880061c2ffd8 ffff88005f764e70 ffff88007c0dc000 [34584.106660] ffff880061c2fd60 00000001020befe5 ffff88007c0dc000 000000000000f888 [34584.108425] Call Trace: [34584.109675] [] schedule+0x29/0x70 [34584.111204] [] schedule_timeout+0x175/0x2d0 [34584.112761] [] ? internal_add_timer+0x70/0x70 [34584.114345] [] qsd_upd_thread+0x9e4/0xd30 [lquota] [34584.115941] [] ? wake_up_state+0x20/0x20 [34584.117458] [] ? qsd_upd_add+0xf0/0xf0 [lquota] [34584.119025] [] kthread+0xcf/0xe0 [34584.120470] [] ? kthread_create_on_node+0x140/0x140 [34584.122059] [] ret_from_fork+0x58/0x90 [34584.123538] [] ? kthread_create_on_node+0x140/0x140 [34584.125118] tdtd-2 S ffff88005f765e20 0 19848 2 0x00000080 [34584.126750] ffff8800450fbd70 0000000000000046 ffff88005f765e20 ffff8800450fbfd8 [34584.128501] ffff8800450fbfd8 ffff8800450fbfd8 ffff88005f765e20 ffff88007a291c40 [34584.130231] 0000000000000000 ffff8800450fbe30 ffff88007a291c00 ffff88005f765e20 [34584.131920] Call Trace: [34584.133144] [] schedule+0x29/0x70 [34584.134693] [] distribute_txn_commit_thread+0xec1/0x1410 [ptlrpc] [34584.136384] [] ? kvm_clock_read+0x1f/0x30 [34584.137928] [] ? set_next_entity+0x3c/0xe0 [34584.139474] [] ? wake_up_state+0x20/0x20 [34584.141038] [] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] [34584.142669] [] kthread+0xcf/0xe0 [34584.144118] [] ? kthread_create_on_node+0x140/0x140 [34584.145755] [] ret_from_fork+0x58/0x90 [34584.147289] [] ? kthread_create_on_node+0x140/0x140 [34584.148894] ldlm_bl_03 S ffff88007a021e28 0 20736 2 0x00000080 [34584.150560] ffff88003bc77e08 0000000000000046 ffff88007ae63ec0 ffff88003bc77fd8 [34584.152307] ffff88003bc77fd8 ffff88003bc77fd8 ffff88007ae63ec0 ffff88007a021e00 [34584.154009] ffff88007ae63ec0 ffff88007a021e64 0000000000000000 ffff88007a021e28 [34584.155750] Call Trace: [34584.157042] [] schedule+0x29/0x70 [34584.158527] [] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc] [34584.160180] [] ? wake_up_state+0x20/0x20 [34584.161742] [] ? ldlm_handle_bl_callback+0x410/0x410 [ptlrpc] [34584.163446] [] kthread+0xcf/0xe0 [34584.164939] [] ? kthread_create_on_node+0x140/0x140 [34584.166563] [] ret_from_fork+0x58/0x90 [34584.168096] [] ? kthread_create_on_node+0x140/0x140 [34584.169717] ldlm_cn00_003 S ffff88004ea11400 0 20737 2 0x00000080 [34584.171441] ffff880064ed7d38 0000000000000046 ffff88007ae62f10 ffff880064ed7fd8 [34584.173195] ffff880064ed7fd8 ffff880064ed7fd8 ffff88007ae62f10 0000000000000000 [34584.174935] ffff88007ae62f10 ffff880077c36280 ffff88004ea11468 ffff88004ea11400 [34584.176702] Call Trace: [34584.177983] [] schedule+0x29/0x70 [34584.179496] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.181147] [] ? wake_up_state+0x20/0x20 [34584.182720] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.184367] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.186084] [] kthread+0xcf/0xe0 [34584.187596] [] ? kthread_create_on_node+0x140/0x140 [34584.189250] [] ret_from_fork+0x58/0x90 [34584.190796] [] ? kthread_create_on_node+0x140/0x140 [34584.192438] mdt00_003 S ffff880063bea400 0 20957 2 0x00000080 [34584.194144] ffff8800657b7d38 0000000000000046 ffff880035b20000 ffff8800657b7fd8 [34584.195950] ffff8800657b7fd8 ffff8800657b7fd8 ffff880035b20000 0000000000000000 [34584.197758] ffff880035b20000 ffff88006849bd80 ffff880063bea468 ffff880063bea400 [34584.199569] Call Trace: [34584.200888] [] schedule+0x29/0x70 [34584.202496] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.204181] [] ? wake_up_state+0x20/0x20 [34584.205787] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.207464] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.209202] [] kthread+0xcf/0xe0 [34584.210720] [] ? kthread_create_on_node+0x140/0x140 [34584.212389] [] ret_from_fork+0x58/0x90 [34584.213962] [] ? kthread_create_on_node+0x140/0x140 [34584.215618] mdt_rdpg00_002 S ffff880064936000 0 20958 2 0x00000080 [34584.217373] ffff88005f97bd38 0000000000000046 ffff880035b21f60 ffff88005f97bfd8 [34584.219167] ffff88005f97bfd8 ffff88005f97bfd8 ffff880035b21f60 0000000000000000 [34584.220943] ffff880035b21f60 ffff88006849b300 ffff880064936068 ffff880064936000 [34584.222746] Call Trace: [34584.224071] [] schedule+0x29/0x70 [34584.225674] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.227363] [] ? wake_up_state+0x20/0x20 [34584.228984] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.230659] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.232417] [] kthread+0xcf/0xe0 [34584.233958] [] ? kthread_create_on_node+0x140/0x140 [34584.235629] [] ret_from_fork+0x58/0x90 [34584.237210] [] ? kthread_create_on_node+0x140/0x140 [34584.238875] ldlm_cb00_002 S ffff88004ea13400 0 21033 2 0x00000080 [34584.240600] ffff8800640efd38 0000000000000046 ffff88007ae64e70 ffff8800640effd8 [34584.242418] ffff8800640effd8 ffff8800640effd8 ffff88007ae64e70 0000000000000000 [34584.244202] ffff88007ae64e70 ffff8800639ea180 ffff88004ea13468 ffff88004ea13400 [34584.245978] Call Trace: [34584.247290] [] schedule+0x29/0x70 [34584.248862] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.250546] [] ? wake_up_state+0x20/0x20 [34584.252163] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.253843] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.255576] [] kthread+0xcf/0xe0 [34584.257118] [] ? kthread_create_on_node+0x140/0x140 [34584.258780] [] ret_from_fork+0x58/0x90 [34584.260360] [] ? kthread_create_on_node+0x140/0x140 [34584.262030] mdt_seqm_0002 S ffff8800657c4c00 0 24125 2 0x00000080 [34584.263747] ffff8800450c3d38 0000000000000046 ffff880078f15e20 ffff8800450c3fd8 [34584.265562] ffff8800450c3fd8 ffff8800450c3fd8 ffff880078f15e20 0000000000000000 [34584.267376] ffff880078f15e20 ffff88006849b900 ffff8800657c4c68 ffff8800657c4c00 [34584.269164] Call Trace: [34584.270475] [] schedule+0x29/0x70 [34584.272066] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.273752] [] ? wake_up_state+0x20/0x20 [34584.275374] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.277051] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.278786] [] kthread+0xcf/0xe0 [34584.280316] [] ? kthread_create_on_node+0x140/0x140 [34584.281985] [] ret_from_fork+0x58/0x90 [34584.283568] [] ? kthread_create_on_node+0x140/0x140 [34584.285237] mdt_rdpg00_003 S ffff880064936000 0 21146 2 0x00000080 [34584.286961] ffff88005ff0fd38 0000000000000046 ffff88005f763ec0 ffff88005ff0ffd8 [34584.288769] ffff88005ff0ffd8 ffff88005ff0ffd8 ffff88005f763ec0 0000000000000000 [34584.290588] ffff88005f763ec0 ffff88005eee3780 ffff880064936068 ffff880064936000 [34584.292398] Call Trace: [34584.293709] [] schedule+0x29/0x70 [34584.295312] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.296999] [] ? wake_up_state+0x20/0x20 [34584.298619] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.300302] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.302052] [] kthread+0xcf/0xe0 [34584.303589] [] ? kthread_create_on_node+0x140/0x140 [34584.305256] [] ret_from_fork+0x58/0x90 [34584.306827] [] ? kthread_create_on_node+0x140/0x140 [34584.308497] pickup S ffff88005fa53ea8 0 27592 1030 0x00000080 [34584.310220] ffff88005fa53db8 0000000000000082 ffff880078f14e70 ffff88005fa53fd8 [34584.312020] ffff88005fa53fd8 ffff88005fa53fd8 ffff880078f14e70 ffff88005fa53ea8 [34584.313812] 0000000005f5e0ff 0000000000000000 ffff880078f14e70 ffff88005fa53ea8 [34584.315609] Call Trace: [34584.316916] [] schedule+0x29/0x70 [34584.318476] [] schedule_hrtimeout_range_clock+0xb2/0x150 [34584.320179] [] ? hrtimer_get_res+0x50/0x50 [34584.321769] [] ? schedule_hrtimeout_range_clock+0xa6/0x150 [34584.323483] [] schedule_hrtimeout_range+0x13/0x20 [34584.325119] [] ep_poll+0x23e/0x360 [34584.326655] [] ? wake_up_state+0x20/0x20 [34584.328232] [] SyS_epoll_wait+0xed/0x120 [34584.329784] [] system_call_fastpath+0x16/0x1b [34584.331391] mdt00_004 S ffff880063bea400 0 28252 2 0x00000080 [34584.333083] ffff88005f433d38 0000000000000046 ffff880065f2bec0 ffff88005f433fd8 [34584.334856] ffff88005f433fd8 ffff88005f433fd8 ffff880065f2bec0 0000000000000000 [34584.336638] ffff880065f2bec0 ffff88005fc3a800 ffff880063bea468 ffff880063bea400 [34584.338431] Call Trace: [34584.339726] [] schedule+0x29/0x70 [34584.341301] [] ptlrpc_wait_event+0x325/0x340 [ptlrpc] [34584.342966] [] ? wake_up_state+0x20/0x20 [34584.344562] [] ptlrpc_main+0x80b/0x1de0 [ptlrpc] [34584.346219] [] ? ptlrpc_register_service+0xe60/0xe60 [ptlrpc] [34584.347945] [] kthread+0xcf/0xe0 [34584.349450] [] ? kthread_create_on_node+0x140/0x140 [34584.351098] [] ret_from_fork+0x58/0x90 [34584.352642] [] ? kthread_create_on_node+0x140/0x140 [34584.354289] kworker/1:2 S ffff88007fd16480 0 592 2 0x00000080 [34584.355983] ffff88005e43fe50 0000000000000046 ffff88005fc29f60 ffff88005e43ffd8 [34584.357774] ffff88005e43ffd8 ffff88005e43ffd8 ffff88005fc29f60 ffff88007fd164a0 [34584.359568] ffff88005f6eb2b0 ffff88005fc29f60 ffff88005f6eb280 ffff88007fd16480 [34584.361340] Call Trace: [34584.362626] [] schedule+0x29/0x70 [34584.364163] [] worker_thread+0x1d9/0x410 [34584.365704] [] ? rescuer_thread+0x460/0x460 [34584.367272] [] kthread+0xcf/0xe0 [34584.368767] [] ? kthread_create_on_node+0x140/0x140 [34584.370386] [] ret_from_fork+0x58/0x90 [34584.371906] [] ? kthread_create_on_node+0x140/0x140 [34584.373504] kworker/0:0 D 0000000000000000 0 8317 2 0x00000080 [34584.375186] Workqueue: events key_garbage_collector [34584.376683] ffff880066b33c40 0000000000000046 ffff88005f8b0000 ffff880066b33fd8 [34584.378453] ffff880066b33fd8 ffff880066b33fd8 ffff88005f8b0000 ffff880066b33d80 [34584.380185] ffff880066b33d88 7fffffffffffffff ffff88005f8b0000 0000000000000000 [34584.381900] Call Trace: [34584.383198] [] schedule+0x29/0x70 [34584.384681] [] schedule_timeout+0x239/0x2d0 [34584.386246] [] ? __slab_free+0x81/0x2f0 [34584.387761] [] ? insert_work+0x4c/0xa0 [34584.389282] [] wait_for_completion+0x116/0x170 [34584.390841] [] ? wake_up_state+0x20/0x20 [34584.392355] [] ? __call_rcu+0x2c0/0x2c0 [34584.393851] [] wait_rcu_gp+0x5b/0x80 [34584.395320] [] ? ftrace_raw_output_rcu_utilization+0x50/0x50 [34584.396963] [] synchronize_sched+0x3b/0x50 [34584.398487] [] key_garbage_collector+0x1cd/0x390 [34584.400047] [] process_one_work+0x17b/0x470 [34584.401557] [] worker_thread+0x126/0x410 [34584.403055] [] ? rescuer_thread+0x460/0x460 [34584.404565] [] kthread+0xcf/0xe0 [34584.406018] [] ? kthread_create_on_node+0x140/0x140 [34584.407576] [] ret_from_fork+0x58/0x90 [34584.409041] [] ? kthread_create_on_node+0x140/0x140 [34584.410608] kworker/0:1 S ffff88007fc16480 0 11255 2 0x00000080 [34584.412277] ffff88006750be50 0000000000000046 ffff880060d08fb0 ffff88006750bfd8 [34584.413955] ffff88006750bfd8 ffff88006750bfd8 ffff880060d08fb0 ffff88007fc164a0 [34584.415673] ffff88006901e8b0 ffff880060d08fb0 ffff88006901e880 ffff88007fc16480 [34584.417377] Call Trace: [34584.418570] [] schedule+0x29/0x70 [34584.420050] [] worker_thread+0x1d9/0x410 [34584.421518] [] ? rescuer_thread+0x460/0x460 [34584.423028] [] kthread+0xcf/0xe0 [34584.424434] [] ? kthread_create_on_node+0x140/0x140 [34584.425975] [] ret_from_fork+0x58/0x90 [34584.427412] [] ? kthread_create_on_node+0x140/0x140 [34584.428959] jbd2/dm-0-8 S ffff88007ae0d800 0 1918 2 0x00000080 [34584.430558] ffff88006298fe30 0000000000000046 ffff8800360c3ec0 ffff88006298ffd8 [34584.432251] ffff88006298ffd8 ffff88006298ffd8 ffff8800360c3ec0 ffff88007ae0d828 [34584.433900] ffff88007ae0dbe8 ffff88007ae0d8a8 ffff88006298fe80 ffff88007ae0d800 [34584.435579] Call Trace: [34584.436749] [] schedule+0x29/0x70 [34584.438215] [] kjournald2+0x203/0x260 [jbd2] [34584.439683] [] ? wake_up_atomic_t+0x30/0x30 [34584.441196] [] ? commit_timeout+0x10/0x10 [jbd2] [34584.442689] [] kthread+0xcf/0xe0 [34584.444129] [] ? kthread_create_on_node+0x140/0x140 [34584.445673] [] ret_from_fork+0x58/0x90 [34584.447118] [] ? kthread_create_on_node+0x140/0x140 [34584.448636] ldiskfs-rsv-con S ffff8800667b3280 0 1919 2 0x00000080 [34584.450261] ffff880079b33e08 0000000000000046 ffff88005f8b5e20 ffff880079b33fd8 [34584.451914] ffff880079b33fd8 ffff880079b33fd8 ffff88005f8b5e20 ffff88005f8b5e20 [34584.453593] ffff8800667b3280 ffff880060fd4e80 ffff8800667b32b0 ffff8800667b3280 [34584.455234] Call Trace: [34584.456411] [] schedule+0x29/0x70 [34584.457867] [] rescuer_thread+0x359/0x460 [34584.459347] [] ? __schedule+0x3b0/0x990 [34584.460812] [] ? manage_workers.isra.24+0x2d0/0x2d0 [34584.462358] [] kthread+0xcf/0xe0 [34584.463781] [] ? kthread_create_on_node+0x140/0x140 [34584.465343] [] ret_from_fork+0x58/0x90 [34584.466870] [] ? kthread_create_on_node+0x140/0x140 [34584.468429] mgs_lustre_noti S ffff88005f760000 0 1925 2 0x00000080 [34584.470045] ffff880078ed7ba8 0000000000000046 ffff88005f760000 ffff880078ed7fd8 [34584.471752] ffff880078ed7fd8 ffff880078ed7fd8 ffff88005f760000 ffff880049241600 [34584.473468] ffff88005f760000 0000000000000000 ffffffffa0a85300 ffff88005f760000 [34584.475191] Call Trace: [34584.476446] [] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc] [34584.478186] [] schedule+0x29/0x70 [34584.479690] [] ldlm_completion_ast+0x62d/0x910 [ptlrpc] [34584.481331] [] ? wake_up_state+0x20/0x20 [34584.482882] [] mgs_completion_ast_generic+0x9e/0x200 [mgs] [34584.484554] [] mgs_completion_ast_ir+0x13/0x20 [mgs] [34584.486189] [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] [34584.487853] [] ? mgs_completion_ast_barrier+0x20/0x20 [mgs] [34584.489555] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [34584.491271] [] mgs_revoke_lock+0xfc/0x370 [mgs] [34584.492884] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [34584.494596] [] ? mgs_completion_ast_barrier+0x20/0x20 [mgs] [34584.496295] [] mgs_ir_notify+0x148/0x2b0 [mgs] [34584.497901] [] ? wake_up_state+0x20/0x20 [34584.499459] [] ? lprocfs_ir_set_state+0x170/0x170 [mgs] [34584.501130] [] kthread+0xcf/0xe0 [34584.502653] [] ? kthread_create_on_node+0x140/0x140 [34584.504302] [] ret_from_fork+0x58/0x90 [34584.505836] [] ? kthread_create_on_node+0x140/0x140 [34584.507464] in.mrshd S 000000000000000b 0 5620 905 0x00000080 [34584.509159] ffff880035b038e8 0000000000000082 ffff88005f8b2f10 ffff880035b03fd8 [34584.510924] ffff880035b03fd8 ffff880035b03fd8 ffff88005f8b2f10 0000000000000000 [34584.512698] ffff880035b03ad0 00000000fffffffc 000000000000000b 000000000000000b [34584.514487] Call Trace: [34584.515770] [] schedule+0x29/0x70 [34584.517333] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34584.519045] [] ? add_wait_queue+0x3c/0x50 [34584.520637] [] ? add_wait_queue+0x3c/0x50 [34584.522218] [] schedule_hrtimeout_range+0x13/0x20 [34584.523847] [] poll_schedule_timeout+0x55/0xb0 [34584.525434] [] do_select+0x6d1/0x7c0 [34584.526939] [] ? poll_select_copy_remaining+0x150/0x150 [34584.528594] [] ? poll_select_copy_remaining+0x150/0x150 [34584.530250] [] ? __alloc_pages_nodemask+0x176/0x420 [34584.531935] [] ? copy_pte_range+0x345/0x5a0 [34584.533488] [] ? __mem_cgroup_commit_charge+0xe7/0x370 [34584.535125] [] ? cpumask_any_but+0x39/0x50 [34584.536674] [] ? flush_tlb_page+0x39/0xa0 [34584.538207] [] ? lru_cache_add+0xe/0x10 [34584.539722] [] ? page_add_new_anon_rmap+0x91/0x150 [34584.541314] [] ? wp_page_copy.isra.56+0x2e6/0x470 [34584.542905] [] core_sys_select+0x1db/0x300 [34584.544453] [] ? handle_mm_fault+0x72c/0xfe0 [34584.545990] [] ? native_smp_send_reschedule+0x4c/0x70 [34584.547581] [] ? do_vfs_ioctl+0x3c0/0x4b0 [34584.549099] [] ? __do_page_fault+0x171/0x450 [34584.550625] [] SyS_select+0xba/0x110 [34584.552091] [] system_call_fastpath+0x16/0x1b [34584.553602] bash S ffff880066ec5e10 0 5623 5620 0x00000080 [34584.555238] ffff88006636fe70 0000000000000082 ffff880066ec5e20 ffff88006636ffd8 [34584.556908] ffff88006636ffd8 ffff88006636ffd8 ffff880066ec5e20 ffff88006636fef8 [34584.558604] ffff880066ec6310 ffff880066ec5e20 ffff880066ec5e20 ffff880066ec5e10 [34584.560297] Call Trace: [34584.561499] [] schedule+0x29/0x70 [34584.562957] [] do_wait+0x1f3/0x250 [34584.564377] [] SyS_wait4+0x80/0x110 [34584.565801] [] ? task_stopped_code+0x60/0x60 [34584.567271] [] system_call_fastpath+0x16/0x1b [34584.568738] bash S ffff880065c70fa0 0 5654 5623 0x00000080 [34584.570348] ffff88005faa7e70 0000000000000082 ffff880065c70fb0 ffff88005faa7fd8 [34584.571981] ffff88005faa7fd8 ffff88005faa7fd8 ffff880065c70fb0 ffff88005faa7ef8 [34584.573618] ffff880065c714a0 ffff880065c70fb0 ffff880065c70fb0 ffff880065c70fa0 [34584.575288] Call Trace: [34584.576449] [] schedule+0x29/0x70 [34584.577883] [] do_wait+0x1f3/0x250 [34584.579276] [] SyS_wait4+0x80/0x110 [34584.580665] [] ? task_stopped_code+0x60/0x60 [34584.582107] [] system_call_fastpath+0x16/0x1b [34584.583539] umount D ffff88005e63c400 0 5655 5654 0x00000080 [34584.585088] ffff88005ec73a20 0000000000000082 ffff88003606ce70 ffff88005ec73fd8 [34584.586730] ffff88005ec73fd8 ffff88005ec73fd8 ffff88003606ce70 ffff88005e63c540 [34584.588356] ffff88005e63c548 7fffffffffffffff ffff88003606ce70 ffff88005e63c400 [34584.589946] Call Trace: [34584.591076] [] schedule+0x29/0x70 [34584.592460] [] schedule_timeout+0x239/0x2d0 [34584.593915] [] ? discard_slab+0x39/0x50 [34584.595338] [] ? __slab_free+0x2ca/0x2f0 [34584.596755] [] wait_for_completion+0x116/0x170 [34584.598221] [] ? wake_up_state+0x20/0x20 [34584.599661] [] mgs_ir_fini_fs+0x27e/0x2f0 [mgs] [34584.601142] [] mgs_put_fsdb+0x53/0x770 [mgs] [34584.602604] [] mgs_cleanup_fsdb_list+0x6c/0x90 [mgs] [34584.604122] [] mgs_device_fini+0x97/0x5b0 [mgs] [34584.605607] [] class_cleanup+0x7f4/0xd80 [obdclass] [34584.607128] [] class_process_config+0x1f84/0x2c30 [obdclass] [34584.608708] [] ? libcfs_debug_msg+0x57/0x80 [libcfs] [34584.610246] [] class_manual_cleanup+0xef/0x810 [obdclass] [34584.611830] [] server_put_super+0xb20/0xcd0 [obdclass] [34584.613381] [] generic_shutdown_super+0x72/0xf0 [34584.614876] [] kill_anon_super+0x12/0x20 [34584.616334] [] lustre_kill_super+0x32/0x50 [obdclass] [34584.617870] [] deactivate_locked_super+0x49/0x60 [34584.619382] [] deactivate_super+0x46/0x60 [34584.620838] [] mntput_no_expire+0xc5/0x120 [34584.622290] [] SyS_umount+0xa0/0x3b0 [34584.623694] [] system_call_fastpath+0x16/0x1b [34584.625164] kworker/u4:0 S ffff88007c060800 0 5832 2 0x00000080 [34584.626715] ffff88005fa5be50 0000000000000046 ffff880065f28fb0 ffff88005fa5bfd8 [34584.628377] ffff88005fa5bfd8 ffff88005fa5bfd8 ffff880065f28fb0 ffff88007c060820 [34584.630022] ffff88007b65b730 ffff880065f28fb0 ffff88007b65b700 ffff88007c060800 [34584.631631] Call Trace: [34584.632824] [] schedule+0x29/0x70 [34584.634225] [] worker_thread+0x1d9/0x410 [34584.635671] [] ? rescuer_thread+0x460/0x460 [34584.637164] [] kthread+0xcf/0xe0 [34584.638559] [] ? kthread_create_on_node+0x140/0x140 [34584.640078] [] ret_from_fork+0x58/0x90 [34584.641489] [] ? kthread_create_on_node+0x140/0x140 [34584.643021] kworker/u4:2 S ffff88007c060800 0 5856 2 0x00000080 [34584.644606] ffff88006195be50 0000000000000046 ffff880065f2ce70 ffff88006195bfd8 [34584.646287] ffff88006195bfd8 ffff88006195bfd8 ffff880065f2ce70 ffff88007c060820 [34584.647927] ffff88005efa17b0 ffff880065f2ce70 ffff88005efa1780 ffff88007c060800 [34584.649611] Call Trace: [34584.650789] [] schedule+0x29/0x70 [34584.652250] [] worker_thread+0x1d9/0x410 [34584.653727] [] ? rescuer_thread+0x460/0x460 [34584.655218] [] kthread+0xcf/0xe0 [34584.656598] [] ? kthread_create_on_node+0x140/0x140 [34584.658123] [] ret_from_fork+0x58/0x90 [34584.659573] [] ? kthread_create_on_node+0x140/0x140 [34584.661107] sshd S 0000000000000011 0 5869 910 0x00000080 [34584.662683] ffff88005f6938e8 0000000000000086 ffff88007ae66dd0 ffff88005f693fd8 [34584.664366] ffff88005f693fd8 ffff88005f693fd8 ffff88007ae66dd0 0000000000000000 [34584.666027] ffff88005f693ad0 00000000fffffffc 0000000000000011 0000000000000011 [34584.667675] Call Trace: [34584.668894] [] schedule+0x29/0x70 [34584.670397] [] schedule_hrtimeout_range_clock+0x12d/0x150 [34584.672010] [] ? add_wait_queue+0x3c/0x50 [34584.673503] [] ? add_wait_queue+0x3c/0x50 [34584.674993] [] schedule_hrtimeout_range+0x13/0x20 [34584.676662] [] poll_schedule_timeout+0x55/0xb0 [34584.678191] [] do_select+0x6d1/0x7c0 [34584.679638] [] ? ip_finish_output+0x268/0x750 [34584.681149] [] ? ip_output+0x73/0xe0 [34584.682575] [] ? poll_select_copy_remaining+0x150/0x150 [34584.684145] [] ? poll_select_copy_remaining+0x150/0x150 [34584.685697] [] ? poll_select_copy_remaining+0x150/0x150 [34584.687229] [] ? poll_select_copy_remaining+0x150/0x150 [34584.688740] [] ? _raw_spin_unlock_bh+0x1b/0x40 [34584.690182] [] ? release_sock+0x118/0x170 [34584.691591] [] ? tcp_sendmsg+0xd6/0xc40 [34584.692991] [] ? inet_sendmsg+0x64/0xb0 [34584.694391] [] core_sys_select+0x1db/0x300 [34584.695810] [] ? fsnotify+0x316/0x4b0 [34584.697176] [] ? do_sync_write+0x8d/0xd0 [34584.698556] [] SyS_select+0xba/0x110 [34584.699914] [] system_call_fastpath+0x16/0x1b [34584.701326] bash R running task 0 5871 5869 0x00000088 [34584.702837] ffff88007ae60000 00000000ef966893 ffff88005aeb7e40 ffffffff810c43a8 [34584.704438] ffff88007ae60000 0000000000000000 ffff88005aeb7e78 ffffffff810c4492 [34584.706018] ffff88007ae60430 00000000ef966893 ffffffff81a7be60 0000000000000074 [34584.707582] Call Trace: [34584.708737] [] sched_show_task+0xa8/0x110 [34584.710169] [] show_state_filter+0x82/0xd0 [34584.711579] [] sysrq_handle_showstate+0x10/0x20 [34584.713043] [] __handle_sysrq+0x107/0x170 [34584.714447] [] write_sysrq_trigger+0x2f/0x40 [34584.715860] [] proc_reg_write+0x3d/0x80 [34584.717225] [] vfs_write+0xbd/0x1e0 [34584.718567] [] SyS_write+0x7f/0xe0 [34584.719881] [] system_call_fastpath+0x16/0x1b [34584.721272] kworker/0:2 R running task 0 5920 2 0x00000080 [34584.722764] ffff880063b4fe50 0000000000000046 ffff880065f2edd0 ffff880063b4ffd8 [34584.724364] ffff880063b4ffd8 ffff880063b4ffd8 ffff880065f2edd0 ffff88007fc164a0 [34584.725927] ffff880064d96e30 ffff880065f2edd0 ffff880064d96e00 ffff88007fc16480 [34584.727493] Call Trace: [34584.728587] [] schedule+0x29/0x70 [34584.729952] [] worker_thread+0x1d9/0x410 [34584.731326] [] ? rescuer_thread+0x460/0x460 [34584.732704] [] kthread+0xcf/0xe0 [34584.734008] [] ? kthread_create_on_node+0x140/0x140 [34584.735439] [] ret_from_fork+0x58/0x90 [34584.736773] [] ? kthread_create_on_node+0x140/0x140 [34584.738209] Sched Debug Version: v0.11, 3.10.0-514.10.2.el7_lustre.x86_64 #1 [34584.739668] ktime : 34584738.204371 [34584.741138] sched_clk : 37515728.117488 [34584.742588] cpu_clk : 34584738.204522 [34584.744056] jiffies : 4329252041 [34584.745454] sched_clock_stable() : 0 [34584.746778] [34584.747783] sysctl_sched [34584.748874] .sysctl_sched_latency : 12.000000 [34584.750265] .sysctl_sched_min_granularity : 10.000000 [34584.751671] .sysctl_sched_wakeup_granularity : 15.000000 [34584.753079] .sysctl_sched_child_runs_first : 0 [34584.754436] .sysctl_sched_features : 77435 [34584.755829] .sysctl_sched_tunable_scaling : 1 (logaritmic) [34584.757278] [34584.758324] cpu#0, 2593.768 MHz [34584.759476] .nr_running : 5 [34584.760754] .load : 3072 [34584.762047] .nr_switches : 27425264 [34584.763372] .nr_load_updates : 8042924 [34584.764681] .nr_uninterruptible : -352 [34584.765956] .next_balance : 4329.252166 [34584.767275] .curr->pid : 5871 [34584.768528] .clock : 34579234.151272 [34584.769850] .cpu_load[0] : 3072 [34584.771091] .cpu_load[1] : 3072 [34584.772300] .cpu_load[2] : 3072 [34584.773499] .cpu_load[3] : 3072 [34584.774683] .cpu_load[4] : 3072 [34584.775872] .avg_idle : 502752 [34584.777049] .max_idle_balance_cost : 500000 [34584.778225] [34584.778225] cfs_rq[0]:/ [34584.779218] .exec_clock : 0.000000 [34584.779218] .MIN_vruntime : 10913710.743518 [34584.779218] .min_vruntime : 10913716.743518 [34584.779218] .max_vruntime : 10913710.743518 [34584.779218] .spread : 0.000000 [34584.779218] .spread0 : 0.000000 [34584.779218] .nr_spread_over : 0 [34584.779218] .nr_running : 3 [34584.779218] .load : 3072 [34584.779218] .runnable_load_avg : 971 [34584.779218] .blocked_load_avg : 16 [34584.779218] .tg_load_avg : 0 [34584.779218] .tg_load_contrib : 0 [34584.779218] .tg_runnable_contrib : 0 [34584.779218] .tg->runnable_avg : 0 [34584.779218] .tg->cfs_bandwidth.timer_active: 0 [34584.779218] .throttled : 0 [34584.779218] .throttle_count : 0 [34584.779218] .avg->runnable_avg_sum : 45379 [34584.779218] .avg->runnable_avg_period : 48023 [34584.779218] [34584.779218] rt_rq[0]:/ [34584.779218] .rt_nr_running : 1 [34584.779218] .rt_throttled : 0 [34584.779218] .rt_time : 0.000000 [34584.779218] .rt_runtime : 950.000000 [34584.779218] [34584.779218] runnable tasks: [34584.779218] task PID tree-key switches prio wait-time sum-exec sum-sleep [34584.779218] ---------------------------------------------------------------------------------------------------------- [34584.779218] systemd 1 10913588.203628 992705 120 0.000000 173734.832600 0.000000 0 / [34584.779218] kthreadd 2 10913586.773380 17915 120 0.000000 380.320942 0.000000 0 / [34584.779218] ksoftirqd/0 3 10913710.743518 123237 120 0.000000 1537.400927 0.000000 0 / [34584.779218] kworker/0:0H 5 3073.607163 7 100 0.000000 0.123711 0.000000 0 / [34584.779218] migration/0 7 0.000000 322394 0 0.000000 3587.136502 0.000000 0 / [34584.779218] rcu_bh 8 28.940280 2 120 0.000000 0.001839 0.000000 0 / [34584.779218] watchdog/0 10 0.000000 8647 0 0.000000 125.354555 0.000000 0 / [34584.779218] khungtaskd 19 10913213.061457 290 120 0.000000 296.077442 0.000000 0 / [34584.779218] kswapd0 31 133.776521 3 120 0.000000 0.072860 0.000000 0 / [34584.779218] ksmd 32 133.329162 2 125 0.000000 0.054730 0.000000 0 / [34584.779218] kthrotld 43 133.753210 2 100 0.000000 0.042141 0.000000 0 / [34584.779218] kmpath_rdacd 45 139.759425 2 100 0.000000 0.034940 0.000000 0 / [34584.779218] kpsmoused 46 145.896449 2 100 0.000000 0.019713 0.000000 0 / [34584.779218] kauditd 178 10913243.758427 180052 120 0.000000 1893.835157 0.000000 0 / [34584.779218] ata_sff 234 945.741601 2 100 0.000000 0.006188 0.000000 0 / [34584.779218] scsi_tmf_1 246 972.080171 2 100 0.000000 0.002788 0.000000 0 / [34584.779218] ext4-rsv-conver 273 1112.757066 2 100 0.000000 0.019119 0.000000 0 / [34584.779218] dbus-daemon 486 10913587.156940 1566749 120 0.000000 157321.476392 0.000000 0 / [34584.779218] NetworkManager 491 10913233.092807 450045 120 0.000000 17968.120466 0.000000 0 / [34584.779218] gmain 518 1645.654520 3 120 0.000000 0.044915 0.000000 0 / [34584.779218] gdbus 526 10913233.051382 147950 120 0.000000 15240.968122 0.000000 0 / [34584.779218] polkitd 495 10913233.056766 464812 120 0.000000 21035.842315 0.000000 0 / [34584.779218] gdbus 519 10913233.139886 456736 120 0.000000 26155.328467 0.000000 0 / [34584.779218] JS GC Helper 523 1647.965977 2 120 0.000000 0.066223 0.000000 0 / [34584.779218] JS Sour~ Thread 525 1653.999662 5 120 0.000000 0.064718 0.000000 0 / [34584.779218] runaway-killer- 532 1657.941743 4 120 0.000000 0.059612 0.000000 0 / [34584.779218] gssproxy 508 10913463.026512 1238 120 0.000000 30.585280 0.000000 0 / [34584.779218] gssproxy 511 1645.875580 1 120 0.000000 0.006309 0.000000 0 / [34584.779218] gssproxy 512 1645.474418 1 120 0.000000 0.005146 0.000000 0 / [34584.779218] gssproxy 514 1645.189986 1 120 0.000000 0.006428 0.000000 0 / [34584.779218] gssproxy 515 10913463.103876 42 120 0.000000 7.193021 0.000000 0 / [34584.779218] rdma_cm 562 1726.743806 2 100 0.000000 0.010032 0.000000 0 / [34584.779218] ipoib_flush 583 1827.535042 2 100 0.000000 0.026267 0.000000 0 / [34584.779218] tmr-rd_mcp 595 1873.136010 2 100 0.000000 0.027234 0.000000 0 / [34584.779218] xcopy_wq 596 1877.157374 2 100 0.000000 0.022604 0.000000 0 / [34584.779218] xprtrdma_receiv 611 1939.331146 2 100 0.000000 0.012174 0.000000 0 / [34584.779218] sssd 625 10913214.502591 1296 120 0.000000 1023.331977 0.000000 0 / [34584.779218] sssd_pam 642 10913215.273244 6934 120 0.000000 169.640455 0.000000 0 / [34584.779218] sssd_ssh 643 10913215.247581 6930 120 0.000000 160.086923 0.000000 0 / [34584.779218] sssd_pac 644 10913215.264118 6927 120 0.000000 167.881449 0.000000 0 / [34584.779218] dhclient 646 10510107.886817 80 120 0.000000 82.245545 0.000000 0 / [34584.779218] rsyslogd 892 3041.077159 36 120 0.000000 6.942804 0.000000 0 / [34584.779218] tuned 967 3240.230551 13 120 0.000000 1.084157 0.000000 0 / [34584.779218] xinetd 905 10912073.655281 78752 120 0.000000 15423.690488 0.000000 0 / [34584.779218] nfsiod 922 3068.707144 2 100 0.000000 0.009384 0.000000 0 / [34584.779218] kworker/0:1H 933 10913586.731673 3952 100 0.000000 52.716264 0.000000 0 / [34584.779218] automount 1053 10913215.250136 1865 120 0.000000 55.021171 0.000000 0 / [34584.779218] automount 1077 3685.229311 6 120 0.000000 1.369220 0.000000 0 / [34584.779218] automount 1088 3705.448990 6 120 0.000000 0.713333 0.000000 0 / [34584.779218] automount 1092 3835.649951 9 120 0.000000 5.649144 0.000000 0 / [34584.779218] automount 1138 10913378.073634 11 120 0.000000 0.829507 0.000000 0 / [34584.779218] bioset 2538 9209.542337 2 100 0.000000 0.005294 0.000000 0 / [34584.779218] bioset 2678 9806.102269 2 100 0.000000 0.005065 0.000000 0 / [34584.779218] kdmflush 2767 10236.805514 2 100 0.000000 0.006813 0.000000 0 / [34584.779218] bioset 2768 10246.807371 2 100 0.000000 0.003943 0.000000 0 / [34584.779218] cfs_rh_00 3815 10371992.265283 132 120 0.000000 18.604306 0.000000 0 / [34584.779218] cfs_rh_01 3816 10371993.633182 132 120 0.000000 60.922408 0.000000 0 / [34584.779218] ptlrpc_hr00_001 5301 10911568.960388 394026 120 0.000000 29801.850456 0.000000 0 / [34584.779218] socknal_cd02 5306 10528226.413577 14 120 0.000000 1.129727 0.000000 0 / [34584.779218] socknal_cd03 5307 10528226.487560 16 120 0.000000 0.982335 0.000000 0 / [34584.779218] acceptor_000 5311 10528226.465061 41 120 0.000000 4.032314 0.000000 0 / [34584.779218] router_checker 5312 22979.010024 2 120 0.000000 0.009254 0.000000 0 / [34584.779218] kworker/u4:1 16806 10913179.311768 3108 120 0.000000 40.610512 0.000000 0 / [34584.779218] ldlm_cb00_000 18480 10904551.023621 53 120 0.000000 1.464633 0.000000 0 / [34584.779218] ldlm_cn00_002 18485 10912265.197156 159 120 0.000000 19.324308 0.000000 0 / [34584.779218] ldlm_elt 18488 10514246.862686 2 120 0.000000 0.023573 0.000000 0 / [34584.779218] ll_evictor 18494 10757543.377251 6 120 0.000000 8.285741 0.000000 0 / [34584.779218] ll_cfg_requeue 18495 10912327.383605 1249 120 0.000000 152.527471 0.000000 0 / [34584.779218] mdt_attr00_000 18501 10514325.463936 2 120 0.000000 0.065108 0.000000 0 / [34584.779218] mdt_attr00_001 18502 10514331.515611 2 120 0.000000 0.051678 0.000000 0 / [34584.779218] mdt_seqs_0000 18505 10514349.701033 2 120 0.000000 0.058951 0.000000 0 / [34584.779218] mdt_seqs_0001 18506 10514355.735735 2 120 0.000000 0.034705 0.000000 0 / [34584.779218] mdt_seqm_0000 18507 10540914.597969 8 120 0.000000 0.092727 0.000000 0 / [34584.779218] mdt_seqm_0001 18508 10860572.065390 21 120 0.000000 0.810686 0.000000 0 / [34584.779218] jbd2/dm-2-8 19811 10912267.528352 301 120 0.000000 34.572623 0.000000 0 / [34584.779218] ldiskfs-rsv-con 19812 10520527.831165 2 100 0.000000 0.028970 0.000000 0 / [34584.779218] osp_up0-2 19814 10520749.081343 4 120 0.000000 0.163925 0.000000 0 / [34584.779218] osp_up3-2 19816 10520569.448191 2 120 0.000000 0.054623 0.000000 0 / [34584.779218] osp-syn-1-2 19820 10525917.056671 3 120 0.000000 0.134563 0.000000 0 / [34584.779218] osp-syn-2-2 19822 10525917.077711 3 120 0.000000 0.175843 0.000000 0 / [34584.779218] osp-syn-3-2 19824 10526601.915868 3 120 0.000000 0.158119 0.000000 0 / [34584.779218] osp-syn-4-2 19826 10526962.816139 3 120 0.000000 0.184481 0.000000 0 / [34584.779218] osp-syn-5-2 19828 10527373.720499 3 120 0.000000 0.174560 0.000000 0 / [34584.779218] osp-syn-6-2 19830 10527808.283862 3 120 0.000000 0.172521 0.000000 0 / [34584.779218] osp-syn-7-2 19832 10527808.281540 3 120 0.000000 0.173987 0.000000 0 / [34584.779218] hsm_cdtr 19840 10913215.739345 548 120 0.000000 122.634351 0.000000 0 / [34584.779218] ldlm_bl_03 20736 10912266.592569 1208 120 0.000000 28.585561 0.000000 0 / [34584.779218] mdt_rdpg00_002 20958 10908439.083125 372 120 0.000000 40.141861 0.000000 0 / [34584.779218] ldlm_cb00_002 21033 10904551.262275 24 120 0.000000 0.769761 0.000000 0 / [34584.779218] mdt_seqm_0002 24125 10862493.572089 28 120 0.000000 1.870443 0.000000 0 / [34584.779218] kworker/0:0 8317 10913586.754520 22821 120 0.000000 263.096311 0.000000 0 / [34584.779218] kworker/0:1 11255 10913586.755640 1607 120 0.000000 23.220476 0.000000 0 / [34584.779218] ldiskfs-rsv-con 1919 10894373.614659 2 100 0.000000 0.020704 0.000000 0 / [34584.779218] bash 5654 10912269.899965 1 120 0.000000 0.515923 0.000000 0 / [34584.779218] R bash 5871 10913716.743518 46 120 0.000000 138.560010 0.000000 0 / [34584.779218] kworker/0:2 5920 10913710.743518 2 120 0.000000 0.007570 0.000000 0 / [34584.970210] [34584.971029] cpu#1, 2593.768 MHz [34584.971969] .nr_running : 3 [34584.973046] .load : 4096 [34584.974148] .nr_switches : 26916978 [34584.975515] .nr_load_updates : 7993677 [34584.976838] .nr_uninterruptible : 354 [34584.978102] .next_balance : 4329.252330 [34584.979438] .curr->pid : 904 [34584.980723] .clock : 34584980.356698 [34584.981890] .cpu_load[0] : 3072 [34584.982978] .cpu_load[1] : 3006 [34584.984062] .cpu_load[2] : 2686 [34584.985133] .cpu_load[3] : 2050 [34584.986189] .cpu_load[4] : 1317 [34584.987224] .avg_idle : 1000000 [34584.988278] .max_idle_balance_cost : 500000 [34584.989321] [34584.989321] cfs_rq[1]:/ [34584.990315] .exec_clock : 0.000000 [34584.990315] .MIN_vruntime : 0.000001 [34584.990315] .min_vruntime : 10532226.893985 [34584.990315] .max_vruntime : 0.000001 [34584.990315] .spread : 0.000000 [34584.990315] .spread0 : -381489.849533 [34584.990315] .nr_spread_over : 0 [34584.990315] .nr_running : 0 [34584.990315] .load : 0 [34584.990315] .runnable_load_avg : 0 [34584.990315] .blocked_load_avg : 953 [34584.990315] .tg_load_avg : 0 [34584.990315] .tg_load_contrib : 0 [34584.990315] .tg_runnable_contrib : 0 [34584.990315] .tg->runnable_avg : 0 [34584.990315] .tg->cfs_bandwidth.timer_active: 0 [34584.990315] .throttled : 0 [34584.990315] .throttle_count : 0 [34584.990315] .avg->runnable_avg_sum : 13535 [34584.990315] .avg->runnable_avg_period : 46726 [34584.990315] [34584.990315] rt_rq[1]:/ [34584.990315] .rt_nr_running : 0 [34584.990315] .rt_throttled : 0 [34584.990315] .rt_time : 0.006105 [34584.990315] .rt_runtime : 950.000000 [34584.990315] [34584.990315] runnable tasks: [34584.990315] task PID tree-key switches prio wait-time sum-exec sum-sleep [34584.990315] ---------------------------------------------------------------------------------------------------------- [34584.990315] rcu_sched 9 10532222.709556 3443150 120 0.000000 21640.335761 0.000000 0 / [34584.990315] watchdog/1 11 -3.000000 8649 0 0.000000 151.457755 0.000000 0 / [34584.990315] migration/1 12 0.000000 308506 0 0.000000 4034.609446 0.000000 0 / [34584.990315] ksoftirqd/1 13 10532198.822017 673281 120 0.000000 181999.319709 0.000000 0 / [34584.990315] kworker/1:0H 15 3240.820592 8 100 0.000000 0.030254 0.000000 0 / [34584.990315] kdevtmpfs 17 22727.489735 141 120 0.000000 5.832469 0.000000 0 / [34584.990315] netns 18 10.979142 2 100 0.000000 0.009815 0.000000 0 / [34584.990315] writeback 20 18.988272 2 100 0.000000 0.000000 0.000000 0 / [34584.990315] kintegrityd 21 22.994549 2 100 0.000000 0.006278 0.000000 0 / [34584.990315] bioset 22 26.994548 2 100 0.000000 0.000000 0.000000 0 / [34584.990315] kblockd 23 30.994547 2 100 0.000000 0.000000 0.000000 0 / [34584.990315] md 24 35.002258 2 100 0.000000 0.007712 0.000000 0 / [34584.990315] khugepaged 33 10532083.208547 3464 139 0.000000 102.649174 0.000000 0 / [34584.990315] fsnotify_mark 34 10531848.061411 123 120 0.000000 0.771293 0.000000 0 / [34584.990315] crypto 35 526.029313 2 100 0.000000 0.043793 0.000000 0 / [34584.990315] ipv6_addrconf 48 663.822828 2 100 0.000000 0.023951 0.000000 0 / [34584.990315] deferwq 67 1151.348283 3 100 0.000000 0.036984 0.000000 0 / [34584.990315] scsi_eh_0 240 1185.315835 4 120 0.000000 0.350927 0.000000 0 / [34584.990315] scsi_tmf_0 241 1154.592036 2 100 0.000000 0.004784 0.000000 0 / [34584.990315] scsi_eh_1 243 1190.370198 4 120 0.000000 0.344051 0.000000 0 / [34584.990315] ttm_swap 253 1162.701217 2 100 0.000000 0.004885 0.000000 0 / [34584.990315] jbd2/vda1-8 272 10532102.179509 28789 120 0.000000 986.514576 0.000000 0 / [34584.990315] rpciod 350 1460.075184 2 100 0.000000 0.010785 0.000000 0 / [34584.990315] systemd-journal 358 10532216.144795 474115 120 0.000000 132420.043377 0.000000 0 / [34584.990315] lvmetad 367 13314.675024 21 120 0.000000 2.807734 0.000000 0 / [34584.990315] systemd-udevd 378 10532035.168350 1068 120 0.000000 110.880669 0.000000 0 / [34584.990315] vballoon 398 1583.870864 2 120 0.000000 0.057467 0.000000 0 / [34584.990315] auditd 462 10531577.671385 177530 116 0.000000 6396.555456 0.000000 0 / [34584.990315] auditd 466 10528995.579010 12153 116 0.000000 248.150073 0.000000 0 / [34584.990315] gmain 516 1839.541838 1 120 0.000000 0.032257 0.000000 0 / [34584.990315] irqbalance 498 10532049.786718 3474 120 0.000000 565.704232 0.000000 0 / [34584.990315] chronyd 504 10531563.951255 252 120 0.000000 37.202797 0.000000 0 / [34584.990315] gssproxy 513 1839.512359 1 120 0.000000 0.002778 0.000000 0 / [34584.990315] kworker/u5:0 536 2073.794879 4 100 0.000000 0.007817 0.000000 0 / [34584.990315] ib-comp-wq 537 1872.779906 2 100 0.000000 0.003122 0.000000 0 / [34584.990315] ib_addr 538 1876.780782 2 100 0.000000 0.002157 0.000000 0 / [34584.990315] ib_mcast 539 1880.781643 2 100 0.000000 0.002010 0.000000 0 / [34584.990315] ib_nl_sa_wq 540 1884.783733 2 100 0.000000 0.003741 0.000000 0 / [34584.990315] iw_cm_wq 552 1919.071319 2 100 0.000000 0.005121 0.000000 0 / [34584.990315] ib_cm 557 1933.211452 2 100 0.000000 0.004295 0.000000 0 / [34584.990315] srp_remove 588 2011.057145 2 100 0.000000 0.004022 0.000000 0 / [34584.990315] target_completi 594 2031.475595 2 100 0.000000 0.003398 0.000000 0 / [34584.990315] iscsi_eh 601 2051.942357 2 100 0.000000 0.004522 0.000000 0 / [34584.990315] frwr_recovery 610 2079.795926 2 100 0.000000 0.003332 0.000000 0 / [34584.990315] rpc.gssd 622 10531848.062865 63 120 0.000000 14.230074 0.000000 0 / [34584.990315] sssd_be 628 10532214.410138 7479 120 0.000000 413.525628 0.000000 0 / [34584.990315] sssd_nss 641 10532214.394905 363504 120 0.000000 12825.940007 0.000000 0 / [34584.990315] sssd_pam 642 10532222.679997 6935 120 0.000000 169.652605 0.000000 0 / [34584.990315] sssd_ssh 643 10532222.681036 6931 120 0.000000 160.100112 0.000000 0 / [34584.990315] sssd_pac 644 10532222.684988 6928 120 0.000000 167.898590 0.000000 0 / [34584.990315] systemd-logind 660 10531854.945272 1083243 120 0.000000 82688.378799 0.000000 0 / [34584.990315] kworker/1:1H 696 10531860.907133 3971 100 0.000000 179.264929 0.000000 0 / [34584.990315] in:imjournal 903 10532228.863503 456475 120 0.000000 35520.189356 0.000000 0 / [34584.990315] rs:main Q:Reg 904 10532227.558895 430575 120 0.000000 42293.096696 0.000000 0 / [34584.990315] tuned 894 10530325.205900 481 120 0.000000 141.510674 0.000000 0 / [34584.990315] gmain 965 4369.722296 1 120 0.000000 0.033132 0.000000 0 / [34584.990315] tuned 968 10532210.463478 34591 120 0.000000 2376.116581 0.000000 0 / [34584.990315] tuned 970 5923196.900819 106 120 0.000000 32.176851 0.000000 0 / [34584.990315] sshd 910 10531576.313016 449 120 0.000000 84.701178 0.000000 0 / [34584.990315] munged 915 10529309.085368 41523 120 0.000000 935.124032 0.000000 0 / [34584.990315] munged 918 10531564.024524 612 120 0.000000 161.409974 0.000000 0 / [34584.990315] munged 919 10529144.414989 21079 120 0.000000 1306.307307 0.000000 0 / [34584.990315] munged 920 10529309.132433 21067 120 0.000000 1306.998293 0.000000 0 / [34584.990315] nfsv4.0-svc 934 10531861.201242 59 120 0.000000 4.202715 0.000000 0 / [34584.990315] master 1030 10531535.279471 1431 120 0.000000 78.846902 0.000000 0 / [34584.990315] qmgr 1032 10531519.821798 218 120 0.000000 17.654658 0.000000 0 / [34584.990315] atd 1045 10530324.392676 14 120 0.000000 2.142358 0.000000 0 / [34584.990315] crond 1048 10531563.859800 595 120 0.000000 32.418867 0.000000 0 / [34584.990315] automount 1051 5073.962854 36 120 0.000000 9.694749 0.000000 0 / [34584.990315] automount 1054 10531854.053031 5541 120 0.000000 113.101434 0.000000 0 / [34584.990315] agetty 1078 9213.750151 4 120 0.000000 6.633879 0.000000 0 / [34584.990315] agetty 1080 9211.210690 6 120 0.000000 4.113389 0.000000 0 / [34584.990315] kdmflush 2537 10943.850340 2 100 0.000000 0.020983 0.000000 0 / [34584.990315] kdmflush 2584 11145.217875 2 100 0.000000 0.039970 0.000000 0 / [34584.990315] bioset 2585 11161.228643 2 100 0.000000 0.011908 0.000000 0 / [34584.990315] kdmflush 2632 11428.591492 2 100 0.000000 0.029634 0.000000 0 / [34584.990315] bioset 2633 11438.606543 2 100 0.000000 0.011649 0.000000 0 / [34584.990315] kdmflush 2677 11669.557022 2 100 0.000000 0.006522 0.000000 0 / [34584.990315] kdmflush 2722 11836.980091 2 100 0.000000 0.002617 0.000000 0 / [34584.990315] bioset 2723 11846.982433 2 100 0.000000 0.004151 0.000000 0 / [34584.990315] obd_zombid 5294 10529507.758643 12157 120 0.000000 1649.437759 0.000000 0 / [34584.990315] ptlrpc_hr00_000 5300 10528813.477077 394008 120 0.000000 30251.935518 0.000000 0 / [34584.990315] socknal_cd00 5304 10022050.698030 16 120 0.000000 1.252228 0.000000 0 / [34584.990315] socknal_cd01 5305 10165803.071373 17 120 0.000000 1.209474 0.000000 0 / [34584.990315] socknal_reaper 5308 10532190.457773 34041 120 0.000000 297.686633 0.000000 0 / [34584.990315] socknal_sd00_00 5309 10532207.141272 6130986 120 0.000000 388768.174391 0.000000 0 / [34584.990315] socknal_sd00_01 5310 10532223.170823 1507641 120 0.000000 289037.849310 0.000000 0 / [34584.990315] ptlrpcd_rcv 5313 10532207.715823 42361 120 0.000000 2111.796436 0.000000 0 / [34584.990315] ptlrpcd_00_00 5314 10532222.780512 323662 120 0.000000 22865.443250 0.000000 0 / [34584.990315] ptlrpcd_00_01 5315 10532222.684285 323321 120 0.000000 22531.888182 0.000000 0 / [34584.990315] ll_ping 5316 10531950.539256 10174 120 0.000000 571.098062 0.000000 0 / [34584.990315] sptlrpc_gc 5317 10530616.500290 20 120 0.000000 0.159929 0.000000 0 / [34584.990315] kworker/1:1 17793 10532209.205120 9796 120 0.000000 78.043816 0.000000 0 / [34584.990315] ldlm_cb00_001 18482 10522222.585425 99 120 0.000000 8.679477 0.000000 0 / [34584.990315] ldlm_cn00_000 18483 10521934.878126 148 120 0.000000 23.894367 0.000000 0 / [34584.990315] ldlm_cn00_001 18484 10528813.212890 121 120 0.000000 17.658746 0.000000 0 / [34584.990315] ldlm_bl_01 18486 10532182.629031 4598 120 0.000000 42.058244 0.000000 0 / [34584.990315] ldlm_bl_02 18487 10532182.630177 4681 120 0.000000 119.196540 0.000000 0 / [34584.990315] ldlm_poold 18489 10532182.638743 4906 120 0.000000 111.622787 0.000000 0 / [34584.990315] mdt00_000 18496 10532222.668655 8189 120 0.000000 1136.116738 0.000000 0 / [34584.990315] mdt00_001 18497 10532222.668855 7707 120 0.000000 1098.854945 0.000000 0 / [34584.990315] mdt00_002 18498 10532222.668481 8073 120 0.000000 1113.884224 0.000000 0 / [34584.990315] mdt_rdpg00_000 18499 10528814.324269 197 120 0.000000 17.723838 0.000000 0 / [34584.990315] mdt_rdpg00_001 18500 10529507.762678 543 120 0.000000 44.560548 0.000000 0 / [34584.990315] mdt_out00_000 18503 10532198.819463 3110 120 0.000000 354.566401 0.000000 0 / [34584.990315] mdt_out00_001 18504 10532198.820265 3339 120 0.000000 388.290047 0.000000 0 / [34584.990315] mdt_fld_0000 18509 10508858.564425 82 120 0.000000 4.727641 0.000000 0 / [34584.990315] mdt_fld_0001 18510 10507722.012251 74 120 0.000000 5.001501 0.000000 0 / [34584.990315] mdt_out00_002 19436 10532199.283440 3365 120 0.000000 423.182470 0.000000 0 / [34584.990315] osp_up1-2 19815 10158014.569698 2 120 0.000000 0.114263 0.000000 0 / [34584.990315] osp-pre-0-2 19817 10532176.161540 1077 120 0.000000 41.235268 0.000000 0 / [34584.990315] osp-syn-0-2 19818 10162310.299958 3 120 0.000000 0.151751 0.000000 0 / [34584.990315] osp-pre-1-2 19819 10531940.231956 1095 120 0.000000 37.415672 0.000000 0 / [34584.990315] osp-pre-2-2 19821 10531940.243315 1094 120 0.000000 37.033212 0.000000 0 / [34584.990315] osp-pre-3-2 19823 10531993.799452 1093 120 0.000000 35.856315 0.000000 0 / [34584.990315] osp-pre-4-2 19825 10531990.689922 1092 120 0.000000 39.460112 0.000000 0 / [34584.990315] osp-pre-5-2 19827 10532176.914747 1092 120 0.000000 38.112154 0.000000 0 / [34584.990315] osp-pre-6-2 19829 10532222.692736 1092 120 0.000000 37.060333 0.000000 0 / [34584.990315] osp-pre-7-2 19831 10532222.697231 1092 120 0.000000 39.257618 0.000000 0 / [34584.990315] lquota_wb_lustr 19847 10531563.818887 93 120 0.000000 0.807504 0.000000 0 / [34584.990315] tdtd-2 19848 10158173.781361 2 120 0.000000 0.040269 0.000000 0 / [34584.990315] ldlm_cn00_003 20737 10521934.871754 194 120 0.000000 23.492337 0.000000 0 / [34584.990315] mdt00_003 20957 10532223.133129 8225 120 0.000000 1194.538641 0.000000 0 / [34584.990315] mdt_rdpg00_003 21146 10525767.278075 123 120 0.000000 13.928538 0.000000 0 / [34584.990315] pickup 27592 10531535.246958 79 120 0.000000 9.572970 0.000000 0 / [34584.990315] mdt00_004 28252 10532222.672494 6109 120 0.000000 821.303141 0.000000 0 / [34584.990315] kworker/1:2 592 10529322.656620 1796 120 0.000000 10.604599 0.000000 0 / [34584.990315] jbd2/dm-0-8 1918 10529507.922717 46 120 0.000000 2.573373 0.000000 0 / [34584.990315] mgs_lustre_noti 1925 10529644.002953 9 118 0.000000 10.425966 0.000000 0 / [34584.990315] in.mrshd 5620 10529323.790648 9 120 0.000000 6.886625 0.000000 0 / [34584.990315] bash 5623 10529458.076061 26 120 0.000000 8.300807 0.000000 0 / [34584.990315] umount 5655 10529513.746301 40 120 0.000000 45.310328 0.000000 0 / [34584.990315] kworker/u4:0 5832 10531429.778401 37 120 0.000000 0.981512 0.000000 0 / [34584.990315] kworker/u4:2 5856 10532201.335500 39 120 0.000000 0.836205 0.000000 0 / [34584.990315] sshd 5869 10531583.961030 28 120 0.000000 17.910024 0.000000 0 / [34585.249059] [34675.679033] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491585376/real 1491585376] req@ffff88004592cf00 x1564004895023424/t0(0) o250->MGC10.9.5.243@tcp@0@lo:26/25 lens 520/544 e 0 to 1 dl 1491585401 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [34675.683797] Lustre: 5313:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [34737.679066] Lustre: lustre-OST0002-osc-MDT0002: Connection to lustre-OST0002 (at 10.9.5.244@tcp) was lost; in progress operations using this service will wait for recovery to complete [34737.682472] Lustre: Skipped 8 previous similar messages Console [trevis-41vm7] disconnected from at 04-07 17:17. Console [trevis-41vm7] connected to at 04-07 17:18. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fce000-0x367defff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=7af240cf-d87a-4e40-aa0e-730b1006c39f ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694540k/2097140k available (6765k kernel code, 400k absent, 402200k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008206] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.012552] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.015111] Mount-cache hash table entries: 4096 [ 0.016296] Initializing cgroup subsys memory [ 0.017011] Initializing cgroup subsys devices [ 0.019005] Initializing cgroup subsys freezer [ 0.020005] Initializing cgroup subsys net_cls [ 0.021005] Initializing cgroup subsys blkio [ 0.023004] Initializing cgroup subsys perf_event [ 0.024006] Initializing cgroup subsys hugetlb [ 0.025004] Initializing cgroup subsys pids [ 0.026005] Initializing cgroup subsys net_prio [ 0.028113] Disabled fast string operations [ 0.029019] mce: CPU supports 10 MCE banks [ 0.030056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] tlb_flushall_shift: 6 [ 0.034176] Freeing SMP alternatives: 28k freed [ 0.040253] ACPI: Core revision 20130517 [ 0.042342] ACPI: All ACPI Tables successfully acquired [ 0.043756] ftrace: allocating 25813 entries in 101 pages [ 0.053153] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.056003] smpboot: Max logical packages: 2 [ 0.057260] Enabling x2apic [ 0.058000] Enabled x2apic [ 0.058004] Switched APIC routing to physical x2apic. [ 0.060567] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.062002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.064025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.066588] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.069040] Brought up 2 CPUs [ 0.069030] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.071005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.072584] devtmpfs: initialized [ 0.074876] EVM: security.selinux [ 0.075005] EVM: security.ima [ 0.076004] EVM: security.capability [ 0.077761] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.078029] pinctrl core: initialized pinctrl subsystem [ 0.079167] NET: Registered protocol family 16 [ 0.080272] ACPI: bus type PCI registered [ 0.081006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.082087] PCI: Using configuration type 1 for base access [ 0.086086] ACPI: Added _OSI(Module Device) [ 0.087013] ACPI: Added _OSI(Processor Device) [ 0.088005] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.089005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.091685] ACPI: Interpreter enabled [ 0.092012] ACPI: (supports S0 S5) [ 0.093005] ACPI: Using IOAPIC for interrupt routing [ 0.094017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.097639] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.098010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.099010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.100018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.101249] acpiphp: Slot [1] registered [ 0.102032] acpiphp: Slot [2] registered [ 0.103022] acpiphp: Slot [3] registered [ 0.104022] acpiphp: Slot [4] registered [ 0.105027] acpiphp: Slot [5] registered [ 0.106033] acpiphp: Slot [6] registered [ 0.107022] acpiphp: Slot [7] registered [ 0.108022] acpiphp: Slot [8] registered [ 0.109030] acpiphp: Slot [9] registered [ 0.110022] acpiphp: Slot [10] registered [ 0.111022] acpiphp: Slot [11] registered [ 0.112022] acpiphp: Slot [12] registered [ 0.113022] acpiphp: Slot [13] registered [ 0.114022] acpiphp: Slot [14] registered [ 0.115022] acpiphp: Slot [15] registered [ 0.116030] acpiphp: Slot [16] registered [ 0.117030] acpiphp: Slot [17] registered [ 0.118022] acpiphp: Slot [18] registered [ 0.119022] acpiphp: Slot [19] registered [ 0.120022] acpiphp: Slot [20] registered [ 0.121022] acpiphp: Slot [21] registered [ 0.122022] acpiphp: Slot [22] registered [ 0.123029] acpiphp: Slot [23] registered [ 0.124023] acpiphp: Slot [24] registered [ 0.125022] acpiphp: Slot [25] registered [ 0.126037] acpiphp: Slot [26] registered [ 0.127022] acpiphp: Slot [27] registered [ 0.128032] acpiphp: Slot [28] registered [ 0.129023] acpiphp: Slot [29] registered [ 0.130022] acpiphp: Slot [30] registered [ 0.131030] acpiphp: Slot [31] registered [ 0.132258] PCI host bridge to bus 0000:00 [ 0.133007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.134006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.135006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.136007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.137005] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.138992] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.139006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.140005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.141005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.142826] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.143013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.160662] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.162077] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.163788] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.165071] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.166753] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.168245] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.169282] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.170007] vgaarb: loaded [ 0.171005] vgaarb: bridge control possible 0000:00:02.0 [ 0.172213] SCSI subsystem initialized [ 0.173053] ACPI: bus type USB registered [ 0.174038] usbcore: registered new interface driver usbfs [ 0.175018] usbcore: registered new interface driver hub [ 0.176055] usbcore: registered new device driver usb [ 0.177221] PCI: Using ACPI for IRQ routing [ 0.178342] NetLabel: Initializing [ 0.179007] NetLabel: domain hash size = 128 [ 0.180004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.181054] NetLabel: unlabeled traffic allowed by default [ 0.182159] Switched to clocksource kvm-clock [ 0.192585] pnp: PnP ACPI init [ 0.193689] ACPI: bus type PNP registered [ 0.195415] pnp: PnP ACPI: found 5 devices [ 0.196629] ACPI: bus type PNP unregistered [ 0.207652] NET: Registered protocol family 2 [ 0.209303] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.211679] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.213443] TCP: Hash tables configured (established 16384 bind 16384) [ 0.215107] TCP: reno registered [ 0.216222] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.217715] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.219392] NET: Registered protocol family 1 [ 0.220672] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.222096] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.223481] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.225179] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.226879] Unpacking initramfs... [ 0.657531] Freeing initrd memory: 24644k freed [ 0.673267] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.674958] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.676854] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.678376] Initialise system trusted keyring [ 0.679657] audit: initializing netlink socket (disabled) [ 0.681058] type=2000 audit(1491585511.649:1): initialized [ 0.702324] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.703843] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.706561] zpool: loaded [ 0.707621] zbud: loaded [ 0.708873] VFS: Disk quotas dquot_6.5.2 [ 0.710160] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.711891] msgmni has been set to 3357 [ 0.713178] Key type big_key registered [ 0.716004] NET: Registered protocol family 38 [ 0.717280] Key type asymmetric registered [ 0.718569] Asymmetric key parser 'x509' registered [ 0.719921] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.722260] io scheduler noop registered [ 0.723480] io scheduler deadline registered (default) [ 0.724847] io scheduler cfq registered [ 0.726152] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.727549] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.729230] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.731471] ACPI: Power Button [PWRF] [ 0.732801] GHES: HEST is not enabled! [ 0.734081] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.756902] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.758762] Non-volatile memory driver v1.3 [ 0.760013] Linux agpgart interface v0.103 [ 0.761430] crash memory driver: version 1.1 [ 0.762840] rdac: device handler registered [ 0.764248] hp_sw: device handler registered [ 0.780593] emc: device handler registered [ 0.781799] alua: device handler registered [ 0.783054] libphy: Fixed MDIO Bus: probed [ 0.784300] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.785776] ehci-pci: EHCI PCI platform driver [ 0.787023] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.788467] ohci-pci: OHCI PCI platform driver [ 0.789701] uhci_hcd: USB Universal Host Controller Interface driver [ 0.791389] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.792755] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.795010] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.796376] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.797870] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.799464] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.801738] usb usb1: Product: UHCI Host Controller [ 0.803132] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.805441] usb usb1: SerialNumber: 0000:00:01.2 [ 0.806832] hub 1-0:1.0: USB hub found [ 0.808073] hub 1-0:1.0: 2 ports detected [ 0.809451] usbcore: registered new interface driver usbserial [ 0.810925] usbcore: registered new interface driver usbserial_generic [ 0.812502] usbserial: USB Serial support registered for generic [ 0.814034] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.816999] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.818395] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.819868] mousedev: PS/2 mouse device common for all mice [ 0.821683] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.824470] rtc_cmos 00:00: RTC can wake from S4 [ 0.825280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.825708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.831756] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.833329] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.834842] cpuidle: using governor menu [ 0.836147] hidraw: raw HID events driver (C) Jiri Kosina [ 0.837625] usbcore: registered new interface driver usbhid [ 0.839034] usbhid: USB HID core driver [ 0.840291] drop_monitor: Initializing network drop monitor service [ 0.841829] TCP: cubic registered [ 0.843028] Initializing XFRM netlink socket [ 0.844372] NET: Registered protocol family 10 [ 0.846066] NET: Registered protocol family 17 [ 0.847586] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.849523] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.851095] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.853794] Loading compiled-in X.509 certificates [ 0.855179] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.857733] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.860849] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.863441] registered taskstats version 1 [ 0.867590] Key type trusted registered [ 0.871156] Key type encrypted registered [ 0.874567] IMA: No TPM chip found, activating TPM-bypass! [ 0.876336] rtc_cmos 00:00: setting system clock to 2017-04-07 17:18:30 UTC (1491585510) [ 0.879645] Freeing unused kernel memory: 1680k freed [ 0.885821] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.890389] systemd[1]: Detected virtualization kvm. [ 0.891819] systemd[1]: Detected architecture x86-64. [ 0.893273] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.895510] systemd[1]: Set hostname to . [ OK ] Reached target Swap. [ 0.929883] systemd[1]: Reached target Swap. [ 0.931192] systemd[1]: Starting Swap. [ OK ] Reached target Timers. [ 0.933045] systemd[1]: Reached target Timers. [ 0.934331] systemd[1]: Starting Timers. [ OK ] Reached target Local File Systems. [ 0.936184] systemd[1]: Reached target Local File Systems. [ 0.937616] systemd[1]: Starting Local File Systems. [ OK ] Created slice Root Slice. [ 0.939646] systemd[1]: Created slice Root Slice. [ 0.941023] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Kernel Socket. [ 0.942907] systemd[1]: Listening on udev Kernel Socket. [ 0.944309] systemd[1]: Starting udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ 0.946306] systemd[1]: Listening on udev Control Socket. [ 0.947732] systemd[1]: Starting udev Control Socket. [ OK ] Created slice System Slice. [ 0.949720] systemd[1]: Created slice System Slice. [ 0.951104] systemd[1]: Starting System Slice. [ OK ] Reached target Slices. [ 0.952955] systemd[1]: Reached target Slices. [ 0.954292] systemd[1]: Starting Slices. [ OK ] Listening on Journal Socket. [ 0.956282] systemd[1]: Listening on Journal Socket. [ 0.957636] systemd[1]: Starting Journal Socket. [ 0.959489] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 0.964339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 0.968191] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.970612] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 0.973142] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Sockets. [ 0.976112] systemd[1]: Reached target Sockets. [ 0.977436] systemd[1]: Starting Sockets. [ OK ] Started Journal Service. [ 1.060899] systemd[1]: Started Journal Service. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ OK ] Mounted Configuration File System. [ OK ] Reached target System Initialization. [ 1.148288] FDC 0 is a S82078B [ 1.152599] scsi host0: ata_piix [ 1.154708] scsi host1: ata_piix [ 1.155896] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.157412] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.159987] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.161849] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.164705] [drm] Initialized drm 1.1.0 20060810 [ 1.165708] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 10 [ 1.169658] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.175523] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.176210] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.176231] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.197826] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.199334] [TTM] Initializing pool allocator [ 1.200579] [TTM] Initializing DMA pool allocator [ 1.202523] [drm] fb mappable at 0xF0000000 [ 1.203820] [drm] vram aper at 0xF0000000 [ 1.205023] [drm] size 33554432 [ 1.206109] [drm] fb depth is 24 [ 1.207209] [drm] pitch is 3072 %G[ 1.208590] fbcon: cirrusdrmfb (fb0) is primary device [ 1.229657] vda: vda1 vda2 vda4 [ 1.251055] Console: switching to colour frame buffer device 128x48 [ 1.265048] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.280030] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 %G[ OK ] Found device /dev/disk/by-uuid/7af240cf-d87a-4e40-aa0e-730b1006c39f. Starting File System Check on /dev/...f-d87a-4e40-aa0e-730b1006c39f... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 1.675087] tsc: Refined TSC clocksource calibration: 2593.642 MHz [ OK ] Started File System Check on /dev/d...0cf-d87a-4e40-aa0e-730b1006c39f. Mounting /sysroot... [ 2.934387] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 2.935802] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Plymouth switch root service. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 3.172979] systemd-journald[94]: Received SIGTERM from PID 1 (systemd). [ 3.221795] SELinux: Disabled at runtime. [ 3.260027] type=1404 audit(1491585512.883:2): selinux=0 auid=4294967295 ses=4294967295 [ 3.268341] ip_tables: (C) 2000-2006 Netfilter Core Team [ 3.269754] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting NFSD configuration filesystem... Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Encrypted Volumes. [ OK ] Stopped target Switch Root. Starting Read and set NIS domainname from /etc/sysconfig/network... Mounting Debug File System... [ OK ] Listening on LVM2 poll daemon socket. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Mounting Huge Pages File System... [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on udev Kernel Socket. [ 3.354556] RPC: Registered named UNIX socket transport module. [ 3.354557] RPC: Registered udp transport module. [ 3.354557] RPC: Registered tcp transport module. [ 3.354557] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.375444] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st... nodes for the current kernel... [ OK ] Stopped Journal Service. Starting Journal Service... Starting Remount Root and Kernel File Systems... [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Mounted NFSD configuration filesystem. Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ 3.400041] EXT4-fs (vda1): re-mounted. Opts: (null) [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... Starting udev Kernel Device Manager... [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... [ 3.442863] systemd-journald[357]: Received request to flush runtime journal from PID 1 Starting Flush Journal to Persistent Storage... Starting udev Coldplug all Devices... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Configure read-only root support. [ OK ] Started udev Kernel Device Manager. [ 3.477664] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 3.485061] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Found device /dev/ttyS0. [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 253:4... [ OK ] Found device /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. [ 3.528492] AES CTR mode by8 optimization enabled [ 3.534155] ppdev: user-space parallel port driver [ 3.535175] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS Activating swap /dev/disk/by-uuid/d...6-54b7-4dec-ad18-326a879eb2d7... [ 3.544128] device-mapper: uevent: version 1.0.3 [ 3.545100] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com %G%G[ OK ] Activated swap /dev/disk/by-uuid/d6c032c6-54b7-4dec-ad18-326a879eb2d7. [ OK ] Reached target Swap. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... Starting Preprocess NFS configuration... [ OK ] Started Preprocess NFS configuration. [ OK ] Started Import network configu[ 3.589329] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) ration from initramfs. Starting Create Volatile Files and Directories... [ 3.636341] type=1305 audit(1491585513.259:3): audit_pid=457 old=0 auid=4294967295 ses=4294967295 res=1 [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... Mounting RPC Pipe File System... [ OK ] Mounted RPC Pipe File System. [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ 3.668229] alg: No test for crc32 (crc32-pclmul) [ OK ] Started LVM2 PV scan on device 253:4. [ 3.691841] intel_rapl: no valid rapl domains found in package 0 [ OK ] Found device /dev/lvm-Role_MDS/forcemnt. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ 3.721432] intel_rapl: no valid rapl domains found in package 0 [ OK ] Started Update UTMP about System Boot/Shutdown[ 3.727656] intel_powerclamp: No package C-state available. [ OK ] Reached target System Initialization. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ 3.743994] intel_powerclamp: No package C-state available[ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting Dump dmesg to /var/log/dmesg... Starting Authorization Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting NTP client/server... Starting Network Manager... Starting GSSAPI Proxy Daemon... Starting System Security Services Daemon... [ OK ] Started Dump dmesg to /var/log/dmesg. [ OK ] Started GSSAPI Proxy Daemon. Starting RPC security service for NFS client and server... [ OK ] Started NTP client/server. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Started Network Manager. Starting Network Manager Script Dispatcher Service... Starting Network Manager Wait Online... [ OK ] Started Authorization Manager. [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ 3.893612] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.897355] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ OK ] Started Hostname Service. [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ 4.154995] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 4.175071] Loading iSCSI transport class v2.0-870. [ 4.182991] iscsi: registered transport (iser) [ 4.212039] RPC: Registered rdma transport module. [ 4.213470] RPC: Registered rdma backchannel transport module. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting MUNGE authentication service... Starting Postfix Mail Transport Agent... Starting OpenSSH server daemon... Starting Xinetd A Powerful Replacement For Inetd... Starting Notify NFS peers of a restart... Starting System Logging Service... Starting PowerMan... [ OK ] Reached target Network is Online. [ 9.763678] EXT4-fs (dm-5): mounting ext3 file system using the ext4 subsystem Mounting /scratch... Mounting /mnt/forcemnt... [ OK ] Started Notify NFS peers of a restart. [FAILED] Failed to start PowerMan. See 'systemctl status powerman.service' for details. [ 9.793372] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started MUNGE authentication service. [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started System Logging Service. [ 9.806022] FS-Cache: Loaded [ 9.825536] FS-Cache: Netfs 'nfs' registered for caching [ OK ] Mounted /mnt/forcemnt. [ OK ] Started OpenSSH server daemon. [ 9.853708] Key type dns_resolver registered [ 9.879784] NFS: Registering the id_resolver key type [ 9.882150] Key type id_resolver registered [ 9.884789] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Postfix Mail Transport Agent. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Automounts filesystems on demand... Starting Crash recovery kernel arming... Starting Permit User Sessions... [ OK ] Started Permit User Sessions. Starting Wait for Plymouth Boot Screen to Quit... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Terminate Plymouth Boot Screen... [ OK ] Started Command Scheduler. Starting Command Scheduler... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: Console [trevis-41vm7] disconnected from at 04-07 17:27. Console [trevis-41vm7] connected to at 04-07 17:27. [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Nov 22 16:42:41 UTC 2016 [ 0.000000] Command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x7d547000-0x7fea4fff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 576MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007d200000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7d20f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: initrd=/images/CentOS-7.3-x86_64/initrd.img ksdevice=bootif lang= console=tty0 console=ttyS0,115200 text crashkernel=320M kssendmac ks=http://10.9.0.10/cblr/svc/op/ks/system/trevis-41vm7 BOOT_IMAGE=/images/CentOS-7.3-x86_64/vmlinuz BOOTIF=01-00-16-3e-5a-20-3f [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1676828k/2097140k available (6764k kernel code, 400k absent, 419912k reserved, 4433k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004006] pid_max: default: 32768 minimum: 301 [ 0.006031] Security Framework initialized [ 0.007010] SELinux: Initializing. [ 0.008202] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.012107] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.014569] Mount-cache hash table entries: 4096 [ 0.016163] Initializing cgroup subsys memory [ 0.017011] Initializing cgroup subsys devices [ 0.018006] Initializing cgroup subsys freezer [ 0.019005] Initializing cgroup subsys net_cls [ 0.021005] Initializing cgroup subsys blkio [ 0.022005] Initializing cgroup subsys perf_event [ 0.023006] Initializing cgroup subsys hugetlb [ 0.025004] Initializing cgroup subsys pids [ 0.026005] Initializing cgroup subsys net_prio [ 0.027114] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.030056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] tlb_flushall_shift: 6 [ 0.034175] Freeing SMP alternatives: 28k freed [ 0.039553] ACPI: Core revision 20130517 [ 0.041632] ACPI: All ACPI Tables successfully acquired [ 0.043028] ftrace: allocating 25811 entries in 101 pages [ 0.051266] smpboot: Max logical packages: 2 [ 0.053005] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.054005] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.056256] Enabling x2apic [ 0.057000] Enabled x2apic [ 0.057004] Switched APIC routing to physical x2apic. [ 0.060209] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.061003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.064025] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.066597] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.320045] Brought up 2 CPUs [ 0.320038] kvm-stealtime: cpu 1, msr 7d30f3c0 [ 0.322009] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.323336] devtmpfs: initialized [ 0.327005] EVM: security.selinux [ 0.328005] EVM: security.ima [ 0.329004] EVM: security.capability [ 0.330733] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.331037] pinctrl core: initialized pinctrl subsystem [ 0.332159] NET: Registered protocol family 16 [ 0.333273] ACPI: bus type PCI registered [ 0.334006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335094] PCI: Using configuration type 1 for base access [ 0.339084] ACPI: Added _OSI(Module Device) [ 0.340013] ACPI: Added _OSI(Processor Device) [ 0.341006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.344676] ACPI: Interpreter enabled [ 0.345013] ACPI: (supports S0 S5) [ 0.346005] ACPI: Using IOAPIC for interrupt routing [ 0.347030] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.350669] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351010] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.352010] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.353018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354250] acpiphp: Slot [1] registered [ 0.355049] acpiphp: Slot [2] registered [ 0.356023] acpiphp: Slot [3] registered [ 0.357023] acpiphp: Slot [4] registered [ 0.358023] acpiphp: Slot [5] registered [ 0.359022] acpiphp: Slot [6] registered [ 0.360022] acpiphp: Slot [7] registered [ 0.361023] acpiphp: Slot [8] registered [ 0.362030] acpiphp: Slot [9] registered [ 0.363023] acpiphp: Slot [10] registered [ 0.364022] acpiphp: Slot [11] registered [ 0.365022] acpiphp: Slot [12] registered [ 0.366022] acpiphp: Slot [13] registered [ 0.367022] acpiphp: Slot [14] registered [ 0.368022] acpiphp: Slot [15] registered [ 0.369030] acpiphp: Slot [16] registered [ 0.370029] acpiphp: Slot [17] registered [ 0.371023] acpiphp: Slot [18] registered [ 0.372022] acpiphp: Slot [19] registered [ 0.373022] acpiphp: Slot [20] registered [ 0.374023] acpiphp: Slot [21] registered [ 0.375022] acpiphp: Slot [22] registered [ 0.376030] acpiphp: Slot [23] registered [ 0.377022] acpiphp: Slot [24] registered [ 0.378021] acpiphp: Slot [25] registered [ 0.379023] acpiphp: Slot [26] registered [ 0.380029] acpiphp: Slot [27] registered [ 0.381023] acpiphp: Slot [28] registered [ 0.382022] acpiphp: Slot [29] registered [ 0.383031] acpiphp: Slot [30] registered [ 0.384023] acpiphp: Slot [31] registered [ 0.385261] PCI host bridge to bus 0000:00 [ 0.386007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.387006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.388006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.389006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.392016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.394005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.395005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396821] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.397013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.416681] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.418076] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.419752] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.421070] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.422811] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.424247] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.425277] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.426007] vgaarb: loaded [ 0.427005] vgaarb: bridge control possible 0000:00:02.0 [ 0.428213] SCSI subsystem initialized [ 0.429052] ACPI: bus type USB registered [ 0.430039] usbcore: registered new interface driver usbfs [ 0.431017] usbcore: registered new interface driver hub [ 0.432052] usbcore: registered new device driver usb [ 0.433208] PCI: Using ACPI for IRQ routing [ 0.434352] NetLabel: Initializing [ 0.435007] NetLabel: domain hash size = 128 [ 0.436005] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.437056] NetLabel: unlabeled traffic allowed by default [ 0.438163] Switched to clocksource kvm-clock [ 0.448480] pnp: PnP ACPI init [ 0.449572] ACPI: bus type PNP registered [ 0.451334] pnp: PnP ACPI: found 5 devices [ 0.452548] ACPI: bus type PNP unregistered [ 0.462337] NET: Registered protocol family 2 [ 0.463913] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.466257] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.467966] TCP: Hash tables configured (established 16384 bind 16384) [ 0.469569] TCP: reno registered [ 0.470654] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.472104] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.473679] NET: Registered protocol family 1 [ 0.474922] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.476349] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.477729] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.479457] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.481177] Unpacking initramfs... [ 4.628057] Freeing initrd memory: 42360k freed [ 4.647891] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 4.649554] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 4.651425] futex hash table entries: 512 (order: 3, 32768 bytes) [ 4.652886] Initialise system trusted keyring [ 4.654173] audit: initializing netlink socket (disabled) [ 4.655566] type=2000 audit(1491586084.623:1): initialized [ 4.676509] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 4.678024] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 4.680500] zpool: loaded [ 4.681544] zbud: loaded [ 4.682760] VFS: Disk quotas dquot_6.5.2 [ 4.683968] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 4.685656] msgmni has been set to 3357 [ 4.686873] Key type big_key registered [ 4.689587] NET: Registered protocol family 38 [ 4.690850] Key type asymmetric registered [ 4.692048] Asymmetric key parser 'x509' registered [ 4.693350] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 4.695603] io scheduler noop registered [ 4.696799] io scheduler deadline registered (default) [ 4.698142] io scheduler cfq registered [ 4.699405] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 4.700773] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 4.702432] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.704653] ACPI: Power Button [PWRF] [ 4.705997] GHES: HEST is not enabled! [ 4.707245] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.730030] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 4.731970] Non-volatile memory driver v1.3 [ 4.733186] Linux agpgart interface v0.103 [ 4.734510] crash memory driver: version 1.1 [ 4.735814] rdac: device handler registered [ 4.737083] hp_sw: device handler registered [ 4.738300] emc: device handler registered [ 4.739491] alua: device handler registered [ 4.740727] libphy: Fixed MDIO Bus: probed [ 4.741994] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.743494] ehci-pci: EHCI PCI platform driver [ 4.744732] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 4.746174] ohci-pci: OHCI PCI platform driver [ 4.747415] uhci_hcd: USB Universal Host Controller Interface driver [ 4.749094] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.750503] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.752728] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.754104] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 4.755636] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 4.757201] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 4.759440] usb usb1: Product: UHCI Host Controller [ 4.760770] usb usb1: Manufacturer: Linux 3.10.0-514.el7.x86_64 uhci_hcd [ 4.762332] usb usb1: SerialNumber: 0000:00:01.2 [ 4.763744] hub 1-0:1.0: USB hub found [ 4.764940] hub 1-0:1.0: 2 ports detected [ 4.766289] usbcore: registered new interface driver usbserial [ 4.767770] usbcore: registered new interface driver usbserial_generic [ 4.769307] usbserial: USB Serial support registered for generic [ 4.770804] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.773702] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.775073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.776533] mousedev: PS/2 mouse device common for all mice [ 4.778250] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.781278] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 4.783749] rtc_cmos 00:00: RTC can wake from S4 [ 4.785400] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.787853] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.789389] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 4.790913] cpuidle: using governor menu [ 4.792182] hidraw: raw HID events driver (C) Jiri Kosina [ 4.793655] usbcore: registered new interface driver usbhid [ 4.795090] usbhid: USB HID core driver [ 4.796385] drop_monitor: Initializing network drop monitor service [ 4.797968] TCP: cubic registered [ 4.799086] Initializing XFRM netlink socket [ 4.800399] NET: Registered protocol family 10 [ 4.801942] NET: Registered protocol family 17 [ 4.803417] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 4.804876] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 4.806444] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 4.809151] Loading compiled-in X.509 certificates [ 4.810506] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.813065] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.816184] Loaded X.509 cert 'CentOS Linux kernel signing key: d48863a7c16fcc274123e6298f74f057af19fc54' [ 4.818758] registered taskstats version 1 [ 4.822635] Key type trusted registered [ 4.825905] Key type encrypted registered [ 4.829088] IMA: No TPM chip found, activating TPM-bypass! [ 4.830783] rtc_cmos 00:00: setting system clock to 2017-04-07 17:28:03 UTC (1491586083) [ 4.833971] Freeing unused kernel memory: 1680k freed [ 4.839963] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 4.860023] systemd[1]: Detected virtualization kvm. [ 4.861428] systemd[1]: Detected architecture x86-64. [ 4.862825] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 4.865034] systemd[1]: No hostname configured. [ 4.866448] systemd[1]: Set hostname to . [ 4.867878] systemd[1]: Initializing machine ID from KVM UUID. [ 4.893877] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found. [ OK ] Created slice Root Slice. [ 4.897498] systemd[1]: Created slice Root Slice. [ 4.898802] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Control Socket. [ 4.900721] systemd[1]: Listening on udev Control Socket. [ 4.902140] systemd[1]: Starting udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ 4.904179] systemd[1]: Listening on udev Kernel Socket. [ 4.905530] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Encrypted Volumes. [ 4.907504] systemd[1]: Reached target Encrypted Volumes. [ 4.908868] systemd[1]: Starting Encrypted Volumes. [ OK ] Reached target Timers. [ 4.910754] systemd[1]: Reached target Timers. [ 4.911996] systemd[1]: Starting Timers. [ OK ] Listening on Journal Socket. [ 4.913849] systemd[1]: Listening on Journal Socket. [ 4.915204] systemd[1]: Starting Journal Socket. [ OK ] Reached target Swap. [ 4.917074] systemd[1]: Reached target Swap. [ 4.918317] systemd[1]: Starting Swap. [ OK ] Created slice System Slice. [ 4.920146] systemd[1]: Created slice System Slice. [ 4.921479] systemd[1]: Starting System Slice. [ 4.923381] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Reached target Slices. [ 4.928343] systemd[1]: Reached target Slices. [ 4.929622] systemd[1]: Starting Slices. [ 4.931406] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.935164] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 4.939062] systemd[1]: Starting Journal Service... Starting Journal Service... [ 4.942358] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ 4.945474] systemd[1]: Reached target Local File Systems. [ 4.946873] systemd[1]: Starting Local File Systems. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 4.951075] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ 4.957801] systemd[1]: Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ 4.959912] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Journal Service. [ 4.961984] systemd[1]: Started Journal Service. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ 5.011718] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found [ 5.084708] dracut-cmdline[89]: Warning: 'ksdevice=bootif' does nothing (BOOTIF is used by default if present) [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.215388] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 5.220082] Loading iSCSI transport class v2.0-870. [ 5.224587] iscsi: registered transport (tcp) [ 5.258321] FDC 0 is a S82078B [ 5.263381] iBFT detected. [ 5.264446] iBFT has incorrect checksum (0x54)! [ 5.272229] md: raid0 personality registered for level 0 [ 5.275947] md: raid1 personality registered for level 1 [ 5.279326] async_tx: api initialized (async) [ 5.281402] xor: automatically using best checksumming function: [ 5.292007] avx : 22404.000 MB/sec [ 5.312012] raid6: sse2x1 gen() 7929 MB/s [ 5.330009] raid6: sse2x2 gen() 9960 MB/s [ 5.348011] raid6: sse2x4 gen() 11546 MB/s [ 5.349252] raid6: using algorithm sse2x4 gen() (11546 MB/s) [ 5.350644] raid6: using ssse3x2 recovery algorithm [ 5.356401] md: raid6 personality registered for level 6 [ 5.357804] md: raid5 personality registered for level 5 [ 5.359174] md: raid4 personality registered for level 4 [ 5.365308] md: raid10 personality registered for level 10 [ 5.368869] md: linear personality registered for level -1 [ 5.374284] device-mapper: uevent: version 1.0.3 [ 5.375633] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 5.390325] device-mapper: multipath: version 1.11.0 loaded [ 5.393829] device-mapper: multipath round-robin: version 1.1.0 loaded [ 5.425639] RPC: Registered named UNIX socket transport module. [ 5.427149] RPC: Registered udp transport module. [ 5.428460] RPC: Registered tcp transport module. [ 5.429785] RPC: Registered tcp NFSv4.1 backchannel transport module. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ 5.649061] tsc: Refined TSC clocksource calibration: 2593.649 MHz [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... Starting Device-Mapper Multipath Device Controller... [ OK ] Reached target System Initialization. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Reached target Sockets. [ 5.681297] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 5.684369] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 5.690179] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 5.693650] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 5.699904] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 5.706312] AES CTR mode by8 optimization enabled [ 5.709296] [drm] Initialized drm 1.1.0 20060810 [ 5.710955] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 5.718120] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 5.724424] scsi host0: ata_piix [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 5.734879] scsi host1: ata_piix [ 5.734907] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 5.734908] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 5.737458] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 5.742132] [TTM] Zone kernel: Available graphics memory: 860448 kiB [ 5.742133] [TTM] Initializing pool allocator [ 5.742135] [TTM] Initializing DMA pool allocator [ 5.742398] [drm] fb mappable at 0xF0000000 [ 5.742399] [drm] vram aper at 0xF0000000 [ 5.742399] [drm] size 33554432 [ 5.742400] [drm] fb depth is 24 [ 5.742400] [drm] pitch is 3072 %G[ 5.752173] fbcon: cirrusdrmfb (fb0) is primary device [ 5.780840] 8021q: 802.1Q VLAN Support v1.8 [ 5.793076] Console: switching to colour frame buffer device 128x48 [ 5.799728] vda: vda1 vda2 vda4 [ 5.801332] alg: No test for crc32 (crc32-pclmul) [ 5.812028] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Started Device-Mapper Multipath Device Controller. %G Starting Open-iSCSI... [ 5.817381] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started Open-iSCSI. Starting dracut initqueue hook... [ 5.901742] 8139cp 0000:00:03.0 ens3f0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 8.622892] dracut-initqueue[606]: % Total % Received % Xferd Average Speed Time Time Time Current [ 8.623407] dracut-initqueue[606]: Dload Upload Total Spent Left Speed [ 8.729171] dracut-initqueue[606]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 14606 100 14606 0 0 134k 0 --:--:-- --:--:-- --:--:-- 134k [ 9.252526] dracut-initqueue[606]: % Total % Received % Xferd Average Speed Time Time Time Current [ 9.254207] dracut-initqueue[606]: Dload Upload Total Spent Left Speed [ 12.437046] dracut-initqueue[606]: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 23 317M 23 73.4M 0 0 98.1M 0 0:00:03 --:--:-- 0:00:03 98.0M 53 317M 53 170M 0 0 97.5M 0 0:00:03 0:00:01 0:00:02 97.5M 85 317M 85 270M 0 0 98.5M 0 0:00:03 0:00:02 0:00:01 98.5M 100 317M 100 317M 0 0 99.6M 0 0:00:03 0:00:03 --:--:-- 99.6M [ 12.535745] loop: module loaded [ OK ] Started dracut initqueue hook. Starting dracut pre-mount hook... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started dracut pre-mount hook. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. Starting dracut mount hook... [ OK ] Reached target Initrd File Systems. [ 12.694960] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Started dracut mount hook. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Starting Plymouth switch root service... [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut mount hook. Stopping dracut mount hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut pre-mount hook. Stopping dracut pre-mount hook... [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... Stopping Open-iSCSI... [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Paths. [ OK ] Stopped Open-iSCSI. Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped dracut pre-trigger hook. Stopping dracut pre-trigger hook... Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 14.751427] systemd-journald[90]: Received SIGTERM from PID 1 (systemd). [ 15.321758] type=1403 audit(1491586093.990:2): policy loaded auid=4294967295 ses=4294967295 [ 15.327512] systemd[1]: Successfully loaded SELinux policy in 283.540ms. [ 15.427780] systemd[1]: Relabelled /dev and /run in 19.308ms. Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Listening on LVM2 poll daemon socket. Mounting Debug File System... Mounting Temporary Directory... Starting Create list of required st... nodes for the current kernel... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Reached target Timers. [ OK ] Created slice system-anaconda\x2dtmux.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Debug File System. [ OK ] Mounted Temporary Directory. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Remount Root and Kernel File Systems. Starting Load/Save Random Seed... Starting Configure read-only root support... Starting Rebuild Hardware Database... [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Load/Save Random Seed. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... [ OK ] Started Apply Kernel Variables. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Configure read-only root support. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting Device-Mapper Multipath Device Controller... Starting udev Wait for Complete Device Initialization... [ OK ] Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device 252:4... %G%G[ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Started udev Wait for Complete Device Initialization. Starting Activation of DM RAID sets... [ OK ] Started Activation of DM RAID sets. [ OK ] Reached target Local File Systems. Starting Import network configuration from initramfs... Starting Tell Plymouth To Write Out Runtime Data... Starting Rebuild Journal Catalog... [ OK ] Reached target Encrypted Volumes. [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ OK ] Started Update is Completed. [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on Open-iSCSI iscsiuio Socket. [ OK ] Listening on Open-iSCSI iscsid Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Anaconda NetworkManager configuration. Starting Anaconda NetworkManager configuration... Starting firewalld - dynamic firewall daemon... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Hardware RNG Entropy Gatherer Daemon... Starting Wait for Plymouth Boot Screen to Quit... Starting Dump dmesg to /var/log/dmesg... Starting Service enabling compressing RAM with zRam... Starting Login Service... Starting Terminate Plymouth Boot Screen... Starting pre-anaconda logging service... [ OK ] Started pre-anaconda logging service. %G (BStarting installer, one moment...                       [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log (B [anaconda] 1:main* 2:shell 3:log 4:storage-lo> Switch tab: Alt+Tab | Help: F1 (B anaconda 21.48.22.93-1 for CentOS Linux 7 started.  * installation log files are stored in /tmp during the installation * shell is available on TTY2 * when reporting a bug add logs from /tmp as separate text/plain attachments17:28:20 Running pre-installation scripts 17:28:22 Not asking for VNC because of an automated install17:28:22 Not asking for VNC because text mode was explicitly asked for in kickstartStarting automated install.....................Checking software selectionGenerating updated storage configuration Checking storage configuration... ================================================================================ ================================================================================ Installation  1) [x] Language settings 2) [x] Time settings  (English (United States)) (Etc/UTC timezone)  3) [x] Installation source 4) [x] Software selection  (http://10.9.0.10/cblr/links/Ce (Custom software selected)  ntOS-7.3-x86_64) 6) [x] Kdump  5) [x] Installation Destination (Kdump is enabled)  (Custom partitioning selected) 8) [ ] User creation  7) [x] Network configuration (No user will be created)  (Wired (ens3f0) connected) ================================================================================ ================================================================================ Progress Setting up the installation environment . Creating disklabel on /dev/vda . Creating swap on /dev/vda2 . Creating ext3 on /dev/vda1 . Running pre-installation scripts . Starting package installation process Preparing transaction from installation source Installing libgcc (1/572) Installing centos-release (2/572) Installing setup (3/572) Installing filesystem (4/572) Installing kernel-headers (5/572) Installing basesystem (6/572) Installing poppler-data (7/572) Installing firewalld-filesystem (8/572) Installing kbd-legacy (9/572) Installing mailcap (10/572) Installing fontpackages-filesystem (11/572) Installing linux-firmware (12/572) Installing ncurses-base (13/572) Installing kbd-misc (14/572) Installing bind-license (15/572) Installing libX11-common (16/572) Installing emacs-filesystem (17/572) Installing tzdata (18/572) Installing nss-softokn-freebl (19/572) Installing glibc-common (20/572) Installing glibc (21/572) Installing libstdc++ (22/572) Installing ncurses-libs (23/572) Installing bash (24/572) Installing libsepol (25/572) Installing pcre (26/572) Installing libselinux (27/572) Installing zlib (28/572) Installing info (29/572) Installing freetype (30/572) Installing libuuid (31/572) Installing xz-libs (32/572) Installing libcom_err (33/572) Installing nspr (34/572) Installing sed (35/572) Installing bzip2-libs (36/572) Installing popt (37/572) Installing chkconfig (38/572) Installing grep (39/572) Installing readline (40/572) Installing expat (41/572) Installing libdb (42/572) Installing libjpeg-turbo (43/572) Installing nss-util (44/572) Installing libpng (45/572) Installing libxml2 (46/572) Installing dbus-libs (47/572) Installing libattr (48/572) Installing libacl (49/572) Installing libcap (50/572) Installing gawk (51/572) Installing libnl3 (52/572) Installing libcap-ng (53/572) Installing audit-libs (54/572) Installing elfutils-libelf (55/572) Installing gmp (56/572) Installing libffi (57/572) Installing pkgconfig (58/572) Installing glib2 (59/572) Installing shared-mime-info (60/572) Installing libICE (61/572) Installing libSM (62/572) Installing sqlite (63/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing findutils (64/572) Installing cpio (65/572) Installing tcl (66/572) Installing libgpg-error (67/572) Installing libgcrypt (68/572) Installing libidn (69/572) Installing diffutils (70/572) Installing gdbm (71/572) Installing p11-kit (72/572) Installing elfutils-libs (73/572) Installing lua (74/572) Installing libtool-ltdl (75/572) Installing jansson (76/572) Installing libcroco (77/572) Installing atk (78/572) Installing mpfr (79/572) Installing groff-base (80/572) Installing libquadmath (81/572) Installing which (82/572) Installing libgomp (83/572) Installing file-libs (84/572) Installing file (85/572) Installing sysvinit-tools (86/572) Installing pciutils-libs (87/572) Installing hostname (88/572) Installing libmnl (89/572) Installing libgfortran (90/572) Installing libmpc (91/572) Installing libassuan (92/572) Installing nss-softokn (93/572) Installing libnl3-cli (94/572) Installing vim-minimal (95/572) Installing tar (96/572) Installing avahi-libs (97/572) Installing libmng (98/572) Installing cyrus-sasl-lib (99/572) Installing e2fsprogs-libs (100/572) Installing kmod-libs (101/572) Installing xz (102/572) Installing make (103/572) Installing libunistring (104/572) Installing libfontenc (105/572) Installing mesa-libglapi (106/572) Installing libedit (107/572) Installing libtasn1 (108/572) Installing p11-kit-trust (109/572) Installing ca-certificates (110/572) Installing slang (111/572) Installing newt (112/572) Installing libnfnetlink (113/572) Installing numactl-libs (114/572) Installing hwloc-libs (115/572) Installing tcp_wrappers-libs (116/572) Installing lzo (117/572) Installing libxshmfence (118/572) Installing keyutils-libs (119/572) Installing ethtool (120/572) Installing libnetfilter_conntrack (121/572) Installing iptables (122/572) Installing iproute (123/572) Installing libXfont (124/572) Installing xorg-x11-font-utils (125/572) Installing gettext-libs (126/572) Installing libteam (127/572) Installing cpp (128/572) Installing ipset-libs (129/572) Installing ipset (130/572) Installing less (131/572) Installing perl-parent (132/572) Installing perl-HTTP-Tiny (133/572) Installing perl-podlators (134/572) Installing perl-Pod-Perldoc (135/572) Installing perl-Pod-Escapes (136/572) Installing perl-Text-ParseWords (137/572) Installing perl-Encode (138/572) Installing perl-Pod-Usage (139/572) Installing perl-macros (140/572) Installing perl-libs (141/572) Installing perl-threads (142/572) Installing perl-Storable (143/572) Installing perl-Exporter (144/572) Installing perl-Time-HiRes (145/572) Installing perl-Filter (146/572) Installing perl-Time-Local (147/572) Installing perl-constant (148/572) Installing perl-threads-shared (149/572) Installing perl-Scalar-List-Utils (150/572) Installing perl-Socket (151/572) Installing perl-Carp (152/572) Installing perl-File-Temp (153/572) Installing perl-File-Path (154/572) Installing perl-PathTools (155/572) Installing perl-Pod-Simple (156/572) Installing perl-Getopt-Long (157/572) Installing perl (158/572) Installing perl-Data-Dumper (159/572) Installing perl-Test-Harness (160/572) Installing perl-Pod-Parser (161/572) Installing perl-LWP-MediaTypes (162/572) Installing perl-Compress-Raw-Zlib (163/572) Installing perl-XML-NamespaceSupport (164/572) Installing perl-Encode-Locale (165/572) Installing perl-Digest (166/572) Installing perl-XML-SAX-Base (167/572) Installing perl-Digest-SHA (168/572) Installing perl-Digest-MD5 (169/572) Installing perl-Pod-Plainer (170/572) Installing perl-Pod-Checker (171/572) Installing perl-Pod-LaTeX (172/572) Installing perl-Test-Simple (173/572) Installing perl-Locale-Maketext (174/572) Installing perl-Sys-Syslog (175/572) Installing perl-Env (176/572) Installing perl-ExtUtils-Manifest (177/572) Installing perl-Compress-Raw-Bzip2 (178/572) Installing perl-IO-Compress (179/572) Installing perl-Locale-Codes (180/572) Installing perl-File-CheckTree (181/572) Installing perl-HTML-Tagset (182/572) Installing perl-FCGI (183/572) Installing perl-CGI (184/572) Installing perl-Class-ISA (185/572) Installing perl-Text-Unidecode (186/572) Installing perl-Text-Soundex (187/572) Installing perl-Net-LibIDN (188/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing perl-IO-Socket-IP (189/572) Installing perl-IO-HTML (190/572) Installing perl-Business-ISBN-Data (191/572) Installing perl-Business-ISBN (192/572) Installing perl-URI (193/572) Installing perl-WWW-RobotRules (194/572) Installing perl-autodie (195/572) Installing perl-local-lib (196/572) Installing perl-TimeDate (197/572) Installing perl-HTTP-Date (198/572) Installing perl-HTTP-Message (199/572) Installing perl-HTTP-Cookies (200/572) Installing perl-HTML-Parser (201/572) Installing perl-HTTP-Negotiate (202/572) Installing perl-HTTP-Daemon (203/572) Installing perl-File-Listing (204/572) Installing perl-Module-Pluggable (205/572) Installing perl-B-Lint (206/572) Installing gdbm-devel (207/572) Installing setools-libs (208/572) Installing desktop-file-utils (209/572) Installing gobject-introspection (210/572) Installing gsettings-desktop-schemas (211/572) Installing dbus-glib (212/572) Installing nettle (213/572) Installing acl (214/572) Installing spax (215/572) Installing patch (216/572) Installing foomatic-filters (217/572) Installing jasper-libs (218/572) Installing libdb-utils (219/572) Installing libdb-devel (220/572) Installing bc (221/572) Installing pinentry (222/572) Installing mozjs17 (223/572) Installing libss (224/572) Installing libpsm2 (225/572) Installing time (226/572) Installing m4 (227/572) Installing ed (228/572) Installing libpng12 (229/572) Installing GeoIP (230/572) Installing libselinux-utils (231/572) Installing psmisc (232/572) Installing swig (233/572) Installing glibc-headers (234/572) Installing glibc-devel (235/572) Installing ncurses (236/572) Installing graphite2 (237/572) Installing harfbuzz (238/572) Installing libmodman (239/572) Installing libproxy (240/572) Installing snappy (241/572) Installing libverto (242/572) Installing coreutils (243/572) Installing openssl-libs (244/572) Installing krb5-libs (245/572) Installing fontconfig (246/572) Installing python-libs (247/572) Installing python (248/572) Installing libblkid (249/572) Installing ruby-libs (250/572) Installing gzip (251/572) Installing cups-libs (252/572) Installing cracklib (253/572) Installing python-decorator (254/572) Installing gettext (255/572) Installing dbus-python (256/572) Installing binutils (257/572) Installing alsa-lib (258/572) Installing cracklib-dicts (259/572) Installing libpwquality (260/572) Installing pam (261/572) Installing systemd-libs (262/572) Installing libgudev1 (263/572) Installing procps-ng (264/572) Installing libmount (265/572) Installing libselinux-python (266/572) Installing pygobject3-base (267/572) Installing urw-fonts (268/572) Installing openssl (269/572) Installing centos-logos (270/572) Installing python-slip (271/572) Installing python-slip-dbus (272/572) Installing python-firewall (273/572) Installing plymouth-core-libs (274/572) Installing python-pyudev (275/572) Installing redhat-lsb-submod-multimedia (276/572) Installing cups-client (277/572) Installing grubby (278/572) Installing pyparsing (279/572) Installing systemtap-sdt-devel (280/572) Installing perl-ExtUtils-MakeMaker (281/572) Installing perl-ExtUtils-Install (282/572) Installing perl-ExtUtils-ParseXS (283/572) Installing perl-devel (284/572) Installing perl-CPAN (285/572) Installing pyliblzma (286/572) Installing yum-metadata-parser (287/572) Installing python-perf (288/572) Installing python-configobj (289/572) Installing pyxattr (290/572) Installing python-iniparse (291/572) Installing python-IPy (292/572) Installing newt-python (293/572) Installing audit-libs-python (294/572) Installing ghostscript-fonts (295/572) Installing bind-libs-lite (296/572) Installing mariadb-libs (297/572) Installing perl-Net-SSLeay (298/572) Installing perl-IO-Socket-SSL (299/572) Installing perl-Net-HTTP (300/572) Installing perl-libwww-perl (301/572) Installing perl-XML-SAX (302/572) Installing perl-XML-LibXML (303/572) Installing fipscheck-lib (304/572) Installing fipscheck (305/572) Installing libesmtp (306/572) Installing libssh2 (307/572) Installing logrotate (308/572) Installing nss-sysinit (309/572) Installing nss (310/572) Installing NetworkManager-libnm (311/572) Installing redhat-lsb-submod-security (312/572) Installing mailx (313/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing nss-tools (314/572) Installing libcurl (315/572) Installing curl (316/572) Installing rpm-libs (317/572) Installing rpm (318/572) Installing openldap (319/572) Installing dhcp-libs (320/572) Installing libuser (321/572) Installing passwd (322/572) Installing dhcp-common (323/572) Installing python-pycurl (324/572) Installing python-urlgrabber (325/572) Installing tcsh (326/572) Installing xdg-utils (327/572) Installing hicolor-icon-theme (328/572) Installing pixman (329/572) Installing hardlink (330/572) Installing pth (331/572) Installing gnupg2 (332/572) Installing gpgme (333/572) Installing pygpgme (334/572) Installing rpm-build-libs (335/572) Installing rpm-python (336/572) Installing yum-plugin-fastestmirror (337/572) Installing yum (338/572) Installing compat-glibc (339/572) Installing compat-glibc-headers (340/572) Installing dmidecode (341/572) Installing libyaml (342/572) Installing ruby-irb (343/572) Installing ruby (344/572) Installing rubygem-bigdecimal (345/572) Installing rubygem-psych (346/572) Installing rubygem-rdoc (347/572) Installing rubygem-json (348/572) Installing rubygems (349/572) Installing rubygem-io-console (350/572) Installing qrencode-libs (351/572) Installing lcms2 (352/572) Installing libndp (353/572) Installing libaio (354/572) Installing device-mapper-persistent-data (355/572) Installing libpipeline (356/572) Installing man-db (357/572) Installing lsscsi (358/572) Installing json-c (359/572) Installing libpcap (360/572) Installing libXau (361/572) Installing libxcb (362/572) Installing libX11 (363/572) Installing libXext (364/572) Installing libXrender (365/572) Installing libXi (366/572) Installing libXt (367/572) Installing libXfixes (368/572) Installing libXft (369/572) Installing libXpm (370/572) Installing libXcursor (371/572) Installing libXrandr (372/572) Installing libXinerama (373/572) Installing gd (374/572) Installing tk (375/572) Installing libXdamage (376/572) Installing libXmu (377/572) Installing environment-modules (378/572) Installing libXaw (379/572) Installing qt3 (380/572) Installing libXtst (381/572) Installing libXxf86vm (382/572) Installing libXcomposite (383/572) Installing ustr (384/572) Installing libsemanage (385/572) Installing shadow-utils (386/572) Installing libutempter (387/572) Installing util-linux (388/572) Installing policycoreutils (389/572) Installing kpartx (390/572) Installing device-mapper (391/572) Installing device-mapper-libs (392/572) Installing cryptsetup-libs (393/572) Installing dracut (394/572) Installing kmod (395/572) Installing systemd (396/572) Installing dbus (397/572) Installing systemd-sysv (398/572) Installing polkit (399/572) Installing polkit-pkla-compat (400/572) Installing infinipath-psm (401/572) Installing crontabs (402/572) Installing cronie-anacron (403/572) Installing cronie (404/572) Installing iputils (405/572) Installing initscripts (406/572) Installing device-mapper-event-libs (407/572) Installing dhclient (408/572) Installing wpa_supplicant (409/572) Installing os-prober (410/572) Installing hwdata (411/572) Installing pciutils (412/572) Installing rdma (413/572) Installing libibverbs (414/572) Installing libibumad (415/572) Installing opensm-libs (416/572) Installing librdmacm (417/572) Installing compat-openmpi16 (418/572) Installing openssh (419/572) Installing openssh-clients (420/572) Installing pdsh-rcmd-ssh (421/572) Installing pdsh (422/572) Installing libfabric (423/572) Installing openmpi (424/572) Installing ibutils-libs (425/572) Installing libibmad (426/572) Installing libcxgb3 (427/572) Installing qt-settings (428/572) Installing qt (429/572) Installing libpciaccess (430/572) Installing libdrm (431/572) Installing mesa-libGL (432/572) Installing mesa-libGLU (433/572) Installing plymouth-scripts (434/572) Installing plymouth (435/572) Installing mesa-libgbm (436/572) Installing mesa-libEGL (437/572) Installing cairo (438/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing grub2-tools (439/572) Installing dracut-network (440/572) Installing device-mapper-event (441/572) Installing lvm2-libs (442/572) Installing dnsmasq (443/572) Installing postfix (444/572) Installing ebtables (445/572) Installing trousers (446/572) Installing gnutls (447/572) Installing glib-networking (448/572) Installing libsoup (449/572) Installing ppp (450/572) Installing NetworkManager (451/572) Installing libcgroup (452/572) Installing at (453/572) Installing redhat-lsb-core (454/572) Installing redhat-lsb-languages (455/572) Installing redhat-lsb-cxx (456/572) Installing fxload (457/572) Installing alsa-firmware (458/572) Installing alsa-tools-firmware (459/572) Installing selinux-policy (460/572) Installing virt-what (461/572) Installing libsemanage-python (462/572) Installing checkpolicy (463/572) Installing munge-libs (464/572) Installing kernel-tools-libs (465/572) Installing libdaemon (466/572) Installing teamd (467/572) Installing libestr (468/572) Installing libthai (469/572) Installing pango (470/572) Installing jbigkit-libs (471/572) Installing libtiff (472/572) Installing gdk-pixbuf2 (473/572) Installing gtk2 (474/572) Installing ghostscript (475/572) Installing redhat-lsb-printing (476/572) Installing librsvg2 (477/572) Installing graphviz (478/572) Installing graphviz-tcl (479/572) Installing qt-x11 (480/572) Installing redhat-lsb-desktop (481/572) Installing redhat-lsb (482/572) Installing ibutils (483/572) Installing rsyslog (484/572) Installing NetworkManager-team (485/572) Installing kernel-tools (486/572) Installing pdsh-rcmd-mrsh (487/572) Installing policycoreutils-python (488/572) Installing tuned (489/572) Installing selinux-policy-targeted (490/572) Installing NetworkManager-tui (491/572) Installing NetworkManager-wifi (492/572) Installing firewalld (493/572) Installing lvm2 (494/572) Installing kexec-tools (495/572) Installing grub2 (496/572) Installing libibverbs-utils (497/572) Installing infiniband-diags (498/572) Installing simul (499/572) Installing IOR (500/572) Installing metabench (501/572) Installing mdtest (502/572) Installing pdsh-rcmd-rsh (503/572) Installing openssh-server (504/572) Installing librdmacm-utils (505/572) Installing dapl (506/572) Installing ibacm (507/572) Installing libnes (508/572) Installing libmlx5 (509/572) Installing libi40iw (510/572) Installing libmthca (511/572) Installing libibcm (512/572) Installing libcxgb4 (513/572) Installing libmlx4 (514/572) Installing libipathverbs (515/572) Installing kernel (516/572) Installing kbd (517/572) Installing audit (518/572) Installing irqbalance (519/572) Installing chrony (520/572) Installing cvs (521/572) Installing net-tools (522/572) Installing aic94xx-firmware (523/572) Installing rsync (524/572) Installing biosdevname (525/572) Installing microcode_ctl (526/572) Installing dracut-config-rescue (527/572) Installing parted (528/572) Installing authconfig (529/572) Installing iprutils (530/572) Installing ruby-devel (531/572) Installing sudo (532/572) Installing gcc (533/572) Installing xfsprogs (534/572) Installing e2fsprogs (535/572) Installing btrfs-progs (536/572) Installing gdb (537/572) Installing wget (538/572) Installing compilebench (539/572) Installing posix (540/572) Installing kernel-devel (541/572) Installing bonnie++ (542/572) Installing expect (543/572) Installing attr (544/572) Installing dbench (545/572) Installing bzip2 (546/572) Installing lsof (547/572) Installing iozone (548/572) Installing fdtree (549/572) Installing byacc (550/572) Installing fsx (551/572) Installing libsysfs (552/572) Installing connectathon (553/572) Installing iwl1000-firmware (554/572) Installing iwl2000-firmware (555/572) Installing iwl5150-firmware (556/572) Installing iwl6000-firmware (557/572) Installing iwl3160-firmware (558/572) Installing iwl135-firmware (559/572) Installing iwl7260-firmware (560/572) Installing iwl3945-firmware (561/572) Installing ivtv-firmware (562/572) Installing iwl6050-firmware (563/572) Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons Installing rootfiles (564/572) Installing iwl100-firmware (565/572) Installing iwl7265-firmware (566/572) Installing iwl6000g2b-firmware (567/572) Installing iwl2030-firmware (568/572) Installing iwl6000g2a-firmware (569/572) Installing iwl5000-firmware (570/572) Installing iwl4965-firmware (571/572) Installing iwl105-firmware (572/572) Performing post-installation setup tasks Installing boot loader . Performing post-installation setup tasks .  Configuring installed system . Writing network configuration . Creating users . Configuring addons . Generating initramfs . Running post-installation scripts . (B[terminated] [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped System Logging Service. [ OK ] Stopped Service enabling compressing RAM with zRam. [ OK ] Stopped Availability of block devices. Stopping Logout off all iSCSI sessions on shutdown... [ OK ] Stopped Logout off all iSCSI sessions on shutdown. [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Closed Open-iSCSI iscsid Socket. [ OK ] Closed Open-iSCSI iscsiuio Socket. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... Stopping Load/Save Random Seed... [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Swap. [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped Create Volatile Files and Directories. Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. Stopping Import network configuration from initramfs... [ OK ] Stopped target Local File Systems. Unmounting Temporary Directory... Unmounting /mnt/sysimage/dev/shm... Unmounting /mnt/sysimage/proc... Unmounting /mnt/sysimage/run... Unmounting Configuration File System... [ OK ] Stopped Configure read-only root support. Stopping Configure read-only root support... Unmounting /mnt/sysimage/dev/pts... Unmounting /mnt/sysimage/sys/fs/selinux... [ OK ] Stopped Load/Save Random Seed. [ OK ] Unmounted Temporary Directory. [ OK ] Unmounted /mnt/sysimage/proc. [ OK ] Unmounted /mnt/sysimage/dev/pts. [ OK ] Unmounted /mnt/sysimage/dev/shm. [ OK ] Unmounted /mnt/sysimage/run. [ OK ] Unmounted Configuration File System. [ OK ] Unmounted /mnt/sysimage/sys/fs/selinux. Unmounting /mnt/sysimage/sys... Unmounting /mnt/sysimage/dev... [ OK ] Unmounted /mnt/sysimage/sys. [ OK ] Unmounted /mnt/sysimage/dev. Unmounting /mnt/sysimage... [ OK ] Deactivated swap /dev/zram1. [ OK ] Deactivated swap /dev/zram0. [ OK ] Unmounted /mnt/sysimage. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. Stopping LVM2 metadata daemon... [ OK ] Started Restore /run/initramfs. [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes Rebooting. [ 983.323327] Restarting system. [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=416566bc-0301-4b51-b3fe-88b582e6c791 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fca000-0x367dcfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=416566bc-0301-4b51-b3fe-88b582e6c791 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694532k/2097140k available (6765k kernel code, 400k absent, 402208k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004006] pid_max: default: 32768 minimum: 301 [ 0.006030] Security Framework initialized [ 0.007009] SELinux: Initializing. [ 0.008194] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011757] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013498] Mount-cache hash table entries: 4096 [ 0.015235] Initializing cgroup subsys memory [ 0.016012] Initializing cgroup subsys devices [ 0.017006] Initializing cgroup subsys freezer [ 0.019004] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.023005] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.025005] Initializing cgroup subsys net_prio [ 0.026105] Disabled fast string operations [ 0.028018] mce: CPU supports 10 MCE banks [ 0.029056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] tlb_flushall_shift: 6 [ 0.033174] Freeing SMP alternatives: 28k freed [ 0.038855] ACPI: Core revision 20130517 [ 0.040845] ACPI: All ACPI Tables successfully acquired [ 0.042015] ftrace: allocating 25813 entries in 101 pages [ 0.050186] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.052004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.053004] smpboot: Max logical packages: 2 [ 0.055112] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056004] Switched APIC routing to physical x2apic. [ 0.058916] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.060002] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.062901] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.064432] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.068036] Brought up 2 CPUs [ 0.068032] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.070006] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.071509] devtmpfs: initialized [ 0.073860] EVM: security.selinux [ 0.074005] EVM: security.ima [ 0.075004] EVM: security.capability [ 0.076728] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.077029] pinctrl core: initialized pinctrl subsystem [ 0.078138] NET: Registered protocol family 16 [ 0.079269] ACPI: bus type PCI registered [ 0.080006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.081085] PCI: Using configuration type 1 for base access [ 0.085078] ACPI: Added _OSI(Module Device) [ 0.086013] ACPI: Added _OSI(Processor Device) [ 0.087006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.088005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.090666] ACPI: Interpreter enabled [ 0.091013] ACPI: (supports S0 S5) [ 0.092005] ACPI: Using IOAPIC for interrupt routing [ 0.093017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.096549] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.097009] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.098009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.099025] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.100245] acpiphp: Slot [1] registered [ 0.101030] acpiphp: Slot [2] registered [ 0.102022] acpiphp: Slot [3] registered [ 0.103022] acpiphp: Slot [4] registered [ 0.104022] acpiphp: Slot [5] registered [ 0.105022] acpiphp: Slot [6] registered [ 0.106022] acpiphp: Slot [7] registered [ 0.107022] acpiphp: Slot [8] registered [ 0.108028] acpiphp: Slot [9] registered [ 0.109022] acpiphp: Slot [10] registered [ 0.110022] acpiphp: Slot [11] registered [ 0.111021] acpiphp: Slot [12] registered [ 0.112021] acpiphp: Slot [13] registered [ 0.113022] acpiphp: Slot [14] registered [ 0.114021] acpiphp: Slot [15] registered [ 0.115028] acpiphp: Slot [16] registered [ 0.116028] acpiphp: Slot [17] registered [ 0.117022] acpiphp: Slot [18] registered [ 0.118022] acpiphp: Slot [19] registered [ 0.119022] acpiphp: Slot [20] registered [ 0.120022] acpiphp: Slot [21] registered [ 0.121022] acpiphp: Slot [22] registered [ 0.122028] acpiphp: Slot [23] registered [ 0.123022] acpiphp: Slot [24] registered [ 0.124021] acpiphp: Slot [25] registered [ 0.125022] acpiphp: Slot [26] registered [ 0.126022] acpiphp: Slot [27] registered [ 0.127028] acpiphp: Slot [28] registered [ 0.128022] acpiphp: Slot [29] registered [ 0.129022] acpiphp: Slot [30] registered [ 0.130031] acpiphp: Slot [31] registered [ 0.131257] PCI host bridge to bus 0000:00 [ 0.132007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.133006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.134006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.135006] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.136006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.137997] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.138006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.139006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.140005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.141817] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.142013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.159763] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.161078] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.162888] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.164071] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.165827] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.167241] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.168272] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.169006] vgaarb: loaded [ 0.170005] vgaarb: bridge control possible 0000:00:02.0 [ 0.171201] SCSI subsystem initialized [ 0.172050] ACPI: bus type USB registered [ 0.173037] usbcore: registered new interface driver usbfs [ 0.174017] usbcore: registered new interface driver hub [ 0.175049] usbcore: registered new device driver usb [ 0.176177] PCI: Using ACPI for IRQ routing [ 0.177338] NetLabel: Initializing [ 0.178007] NetLabel: domain hash size = 128 [ 0.179004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.180039] NetLabel: unlabeled traffic allowed by default [ 0.181140] Switched to clocksource kvm-clock [ 0.190334] pnp: PnP ACPI init [ 0.191473] ACPI: bus type PNP registered [ 0.193165] pnp: PnP ACPI: found 5 devices [ 0.194391] ACPI: bus type PNP unregistered [ 0.204076] NET: Registered protocol family 2 [ 0.205654] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.207998] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.209641] TCP: Hash tables configured (established 16384 bind 16384) [ 0.211235] TCP: reno registered [ 0.212345] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.213817] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.215440] NET: Registered protocol family 1 [ 0.216771] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.218258] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.219805] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.221550] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.223317] Unpacking initramfs... [ 0.620505] Freeing initrd memory: 24652k freed [ 0.636128] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.637841] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.639691] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.641232] Initialise system trusted keyring [ 0.642536] audit: initializing netlink socket (disabled) [ 0.643959] type=2000 audit(1491587091.552:1): initialized [ 0.664847] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.666409] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.668970] zpool: loaded [ 0.670035] zbud: loaded [ 0.671270] VFS: Disk quotas dquot_6.5.2 [ 0.672494] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674227] msgmni has been set to 3357 [ 0.675479] Key type big_key registered [ 0.678151] NET: Registered protocol family 38 [ 0.679418] Key type asymmetric registered [ 0.680626] Asymmetric key parser 'x509' registered [ 0.681958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.684301] io scheduler noop registered [ 0.685497] io scheduler deadline registered (default) [ 0.686841] io scheduler cfq registered [ 0.688128] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.689513] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.691199] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.693455] ACPI: Power Button [PWRF] [ 0.694820] GHES: HEST is not enabled! [ 0.696093] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.719072] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.721100] Non-volatile memory driver v1.3 [ 0.722327] Linux agpgart interface v0.103 [ 0.723671] crash memory driver: version 1.1 [ 0.724987] rdac: device handler registered [ 0.726297] hp_sw: device handler registered [ 0.742563] emc: device handler registered [ 0.743790] alua: device handler registered [ 0.745053] libphy: Fixed MDIO Bus: probed [ 0.746347] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.747861] ehci-pci: EHCI PCI platform driver [ 0.749117] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.750570] ohci-pci: OHCI PCI platform driver [ 0.751848] uhci_hcd: USB Universal Host Controller Interface driver [ 0.753547] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.754982] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.757252] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.758625] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.760143] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.761797] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.764183] usb usb1: Product: UHCI Host Controller [ 0.765612] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.768041] usb usb1: SerialNumber: 0000:00:01.2 [ 0.769599] hub 1-0:1.0: USB hub found [ 0.770856] hub 1-0:1.0: 2 ports detected [ 0.772299] usbcore: registered new interface driver usbserial [ 0.773821] usbcore: registered new interface driver usbserial_generic [ 0.775480] usbserial: USB Serial support registered for generic [ 0.777031] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.780150] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.781589] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.783123] mousedev: PS/2 mouse device common for all mice [ 0.784916] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.787673] rtc_cmos 00:00: RTC can wake from S4 [ 0.895175] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.899074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.899079] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.899179] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.899219] cpuidle: using governor menu [ 0.899293] hidraw: raw HID events driver (C) Jiri Kosina [ 0.899392] usbcore: registered new interface driver usbhid [ 0.899394] usbhid: USB HID core driver [ 0.899432] drop_monitor: Initializing network drop monitor service [ 0.899503] TCP: cubic registered [ 0.899505] Initializing XFRM netlink socket [ 0.899599] NET: Registered protocol family 10 [ 0.901607] NET: Registered protocol family 17 [ 0.902464] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.902944] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.902979] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.903099] Loading compiled-in X.509 certificates [ 0.903133] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.903146] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.903607] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.903631] registered taskstats version 1 [ 0.908273] Key type trusted registered [ 0.910180] Key type encrypted registered [ 0.912067] IMA: No TPM chip found, activating TPM-bypass! [ 0.913309] rtc_cmos 00:00: setting system clock to 2017-04-07 17:44:50 UTC (1491587090) [ 0.938477] Freeing unused kernel memory: 1680k freed [ 0.944707] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.948964] systemd[1]: Detected virtualization kvm. [ 0.950358] systemd[1]: Detected architecture x86-64. [ 0.951723] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.953857] systemd[1]: Set hostname to . [ OK ] Reached target Local File Systems. [ 0.986587] systemd[1]: Reached target Local File Systems. [ 0.988022] systemd[1]: Starting Local File Systems. [ OK ] Created slice Root Slice. [ 0.990057] systemd[1]: Created slice Root Slice. [ 0.991349] systemd[1]: Starting Root Slice. [ OK ] Listening on Journal Socket. [ 0.993271] systemd[1]: Listening on Journal Socket. [ 0.994599] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Kernel Socket. [ 0.996547] systemd[1]: Listening on udev Kernel Socket. [ 0.997916] systemd[1]: Starting udev Kernel Socket. [ OK ] Created slice System Slice. [ 0.999898] systemd[1]: Created slice System Slice. [ 1.001225] systemd[1]: Starting System Slice. [ 1.002922] systemd[1]: Starting Journal Service... Starting Journal Service... [ 1.005261] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ 1.007854] systemd[1]: Reached target Slices. [ 1.009137] systemd[1]: Starting Slices. [ 1.010954] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Timers. [ 1.014998] systemd[1]: Reached target Timers. [ 1.016272] systemd[1]: Starting Timers. [ 1.017863] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ 1.021136] systemd[1]: Reached target Swap. [ 1.022399] systemd[1]: Starting Swap. [ OK ] Listening on udev Control Socket. [ 1.104432] systemd[1]: Listening on udev Control Socket. [ 1.105825] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Sockets. [ 1.107795] systemd[1]: Reached target Sockets. [ 1.109105] systemd[1]: Starting Sockets. [ 1.110824] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ OK ] Started Journal Service. [ 1.115060] systemd[1]: Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting Show Plymouth Boot Screen... [ OK ] Reached target System Initialization. Starting dracut initqueue hook... [ 1.526032] scsi host0: ata_piix [ 1.527484] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.550620] [drm] Initialized drm 1.1.0 20060810 %G[ 1.555091] scsi host1: ata_piix [ 1.555119] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.555120] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.555337] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.555943] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.555987] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.556449] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.558699] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc90000302000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.559901] 8139too: 8139too Fast Ethernet driver 0.9.28 [ 1.587697] vda: vda1 vda2 [ 1.591114] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.591115] [TTM] Initializing pool allocator [ 1.591118] [TTM] Initializing DMA pool allocator [ 1.592036] [drm] fb mappable at 0xF0000000 [ 1.592037] [drm] vram aper at 0xF0000000 [ 1.592037] [drm] size 33554432 [ 1.592038] [drm] fb depth is 24 [ 1.592038] [drm] pitch is 3072 [ 1.596072] FDC 0 is a S82078B [ 1.644951] tsc: Refined TSC clocksource calibration: 2593.644 MHz [ 1.651789] fbcon: cirrusdrmfb (fb0) is primary device [ 1.678931] Console: switching to colour frame buffer device 128x48 [ 1.690404] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ 1.694251] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [%G OK ] Found device /dev/disk/by-uuid/416566bc-0301-4b51-b3fe-88b582e6c791. [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. Starting File System Check on /dev/...c-0301-4b51-b3fe-88b582e6c791... [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/d...6bc-0301-4b51-b3fe-88b582e6c791. Mounting /sysroot... [ 1.792118] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.794751] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 2.171397] systemd-journald[93]: Received SIGTERM from PID 1 (systemd). [ 2.386085] SELinux: Disabled at runtime. [ 2.424026] type=1404 audit(1491587092.011:2): selinux=0 auid=4294967295 ses=4294967295 [ 2.431965] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.433443] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Reached target Remote File Systems. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on LVM2 metadata daemon socket. Mounting Debug File System... [ OK ] Listening on LVM2 poll daemon socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 2.920049] EXT4-fs (vda1): re-mounted. Opts: (null) Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Device-mapper event daemon FIFOs. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st... nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... [ OK ] Started Remount Root and Kernel File Systems. Starting Rebuild Hardware Database... Starting Flush Journal to Persistent Storage... Starting Load/Save Random Seed... Starting Configure read-only root support... [ 2.975386] systemd-journald[341]: Received request to flush runtime journal from PID 1 [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Load/Save Random Seed. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started Configure read-only root support. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ OK ] Started Rebuild Hardware Database. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. [ 3.466496] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 3.470515] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/2099f471-7c6f-4c85-a7b0-d5769905a720. Activating swap /dev/disk/by-uuid/2...1-7c6f-4c85-a7b0-d5769905a720... %G[ 3.509298] AES CTR mode by8 optimization enabled %G[ 3.511226] ppdev: user-space parallel port driver [ 3.526075] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 3.531414] alg: No test for crc32 (crc32-pclmul) [ 3.551049] intel_rapl: no valid rapl domains found in package 0 [ 3.554029] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS [ OK ] Activated swap /dev/disk/by-uuid/2099f471-7c6f-4c85-a7b0-d5769905a720. [ OK ] Reached target Swap. [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Tell Plymouth To Write Out Runtime Data... Starting Mark the need to relabel after reboot... Starting Import network configuration from initramfs... Starting Rebuild Journal Catalog... [ 3.594368] intel_rapl: no valid rapl domains found in package 0 [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ 3.605247] intel_powerclamp: No package C-state available[ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Security Auditing Service... [ OK ] Started Rebuild Journal Catalog. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ 3.862692] type=1305 audit(1491587093.449:3): audit_pid=449 old=0 auid=4294967295 ses=4294967295 res=1 Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting Dump dmesg to /var/log/dmesg... Starting OpenSSH Server Key Generation... Starting NTP client/server... Starting Authorization Manager... Starting Permit User Sessions... Starting Login Service... [ OK ] Reached target Timers. [ OK ] Started Permit User Sessions. [ OK ] Started Dump dmesg to /var/log/dmesg. Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... [ OK ] Started Job spooling tools. Starting Job spooling tools... [ OK ] Started Command Scheduler. Starting Command Scheduler... [ 4.594423] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.596479] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [2017-04-07T17:45:05+00:00] WARN: ***************************************** [2017-04-07T17:45:05+00:00] WARN: Did not find config file: /etc/chef/solo.rb, using command line options. [2017-04-07T17:45:05+00:00] WARN: ***************************************** [2017-04-07T17:45:09+00:00] INFO: Forking chef instance to converge... [2017-04-07T17:45:10+00:00] INFO: *** Chef 12.4.3 *** [2017-04-07T17:45:10+00:00] INFO: Chef-client pid: 7948 [2017-04-07T17:45:12+00:00] INFO: Setting the run_list to ["role[trevis_config]", "role[test_node]"] from CLI options [2017-04-07T17:45:12+00:00] INFO: Run List is [role[trevis_config], role[test_node]] [2017-04-07T17:45:12+00:00] INFO: Run List expands to [yum, local_repos::local_repos_only, solo_data_bags, resolver, timezone, sudo, freeipa_or_sssd, ntp, autofs, mail::client, munge, mrsh::server, ssh::server, install_packages, rsyslog::client, rsh::server, test_node, mcelog, openmpi, openmpi::test_node] [2017-04-07T17:45:12+00:00] INFO: Starting Chef Run for trevis-41vm7.trevis.hpdd.intel.com [2017-04-07T17:45:12+00:00] INFO: Running start handlers [2017-04-07T17:45:12+00:00] INFO: Start handlers complete. [2017-04-07T17:45:13+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action enable (test_node::default line 288) [2017-04-07T17:45:13+00:00] INFO: Disabling Chef::Handler::JsonFile as a report handler. [2017-04-07T17:45:13+00:00] INFO: Disabling Chef::Handler::JsonFile as a exception handler. [2017-04-07T17:45:13+00:00] INFO: Enabling Chef::Handler::JsonFile as a report handler. [2017-04-07T17:45:13+00:00] INFO: Enabling Chef::Handler::JsonFile as a exception handler. [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-CR.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-CR.repo.chef-20170407174513.101409 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-CR.repo] deleted file at /etc/yum.repos.d/CentOS-CR.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Base.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Base.repo.chef-20170407174513.103468 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Base.repo] deleted file at /etc/yum.repos.d/CentOS-Base.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Debuginfo.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Debuginfo.repo.chef-20170407174513.107245 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Debuginfo.repo] deleted file at /etc/yum.repos.d/CentOS-Debuginfo.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-fasttrack.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-fasttrack.repo.chef-20170407174513.111170 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-fasttrack.repo] deleted file at /etc/yum.repos.d/CentOS-fasttrack.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Vault.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Vault.repo.chef-20170407174513.115066 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Vault.repo] deleted file at /etc/yum.repos.d/CentOS-Vault.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Media.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Media.repo.chef-20170407174513.119063 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Media.repo] deleted file at /etc/yum.repos.d/CentOS-Media.repo [2017-04-07T17:45:13+00:00] INFO: Processing file[/etc/yum.repos.d/CentOS-Sources.repo] action delete (local_repos::local_repos_only line 21) [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] backed up to /var/chef/backup/etc/yum.repos.d/CentOS-Sources.repo.chef-20170407174513.122947 [2017-04-07T17:45:13+00:00] INFO: file[/etc/yum.repos.d/CentOS-Sources.repo] deleted file at /etc/yum.repos.d/CentOS-Sources.repo [2017-04-07T17:45:13+00:00] INFO: Processing template[/etc/resolv.conf] action create (resolver::default line 23) [2017-04-07T17:45:13+00:00] INFO: template[/etc/resolv.conf] backed up to /var/chef/backup/etc/resolv.conf.chef-20170407174513.131727 [2017-04-07T17:45:13+00:00] INFO: template[/etc/resolv.conf] updated file contents /etc/resolv.conf [2017-04-07T17:45:13+00:00] INFO: Processing yum_package[sudo] action install (sudo::default line 20) [2017-04-07T17:45:20+00:00] INFO: Processing template[/etc/sudoers] action create (sudo::default line 24) [2017-04-07T17:45:20+00:00] INFO: template[/etc/sudoers] backed up to /var/chef/backup/etc/sudoers.chef-20170407174520.646808 [2017-04-07T17:45:20+00:00] INFO: template[/etc/sudoers] updated file contents /etc/sudoers [2017-04-07T17:45:20+00:00] INFO: Processing yum_package[ipa-client] action install (freeipa::client line 9) [2017-04-07T17:45:20+00:00] INFO: yum_package[ipa-client] installing ipa-client-4.4.0-14.el7.centos.6 from updates-centos7.3-x86_64 repository [2017-04-07T17:45:57+00:00] INFO: yum_package[ipa-client] installed ipa-client at 4.4.0-14.el7.centos.6 [2017-04-07T17:45:57+00:00] INFO: Processing bash[Uninstall_freeipa_client] action run (freeipa::client line 25) [2017-04-07T17:45:57+00:00] INFO: Processing bash[configure_freeipa_client] action run (freeipa::client line 33) [2017-04-07T17:46:12+00:00] INFO: bash[configure_freeipa_client] ran successfully [2017-04-07T17:46:12+00:00] INFO: Processing directory[/scratch] action create (freeipa::client line 49) [2017-04-07T17:46:12+00:00] INFO: directory[/scratch] created directory /scratch [2017-04-07T17:46:12+00:00] INFO: Processing directory[/shared_test] action create (freeipa::client line 49) [2017-04-07T17:46:13+00:00] INFO: directory[/shared_test] created directory /shared_test [2017-04-07T17:46:13+00:00] INFO: Processing service[nscd] action disable (sssd::default line 10) [2017-04-07T17:46:13+00:00] INFO: Processing service[nscd] action stop (sssd::default line 10) [2017-04-07T17:46:13+00:00] INFO: Processing yum_package[sssd] action install (sssd::default line 20) [2017-04-07T17:46:13+00:00] INFO: Processing template[/etc/sssd/sssd.conf] action create (sssd::default line 31) [2017-04-07T17:46:13+00:00] INFO: template[/etc/sssd/sssd.conf] backed up to /var/chef/backup/etc/sssd/sssd.conf.chef-20170407174613.801865 [2017-04-07T17:46:13+00:00] INFO: template[/etc/sssd/sssd.conf] updated file contents /etc/sssd/sssd.conf [2017-04-07T17:46:13+00:00] INFO: template[/etc/sssd/sssd.conf] sending restart action to service[sssd] (immediate) [2017-04-07T17:46:13+00:00] INFO: Processing service[sssd] action restart (sssd::default line 92) [2017-04-07T17:46:14+00:00] INFO: service[sssd] restarted [2017-04-07T17:46:14+00:00] INFO: Processing execute[authconfig-tui-sssd] action run (sssd::default line 54) [2017-04-07T17:46:14+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:14+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:14+00:00] INFO: Processing service[sssd] action enable (sssd::default line 92) [2017-04-07T17:46:14+00:00] INFO: Processing service[sssd] action start (sssd::default line 92) [2017-04-07T17:46:14+00:00] INFO: Processing yum_package[ntp] action install (ntp::default line 25) [2017-04-07T17:46:14+00:00] INFO: Processing service[ntpd] action start (ntp::default line 27) [2017-04-07T17:46:14+00:00] INFO: Processing service[ntpd] action enable (ntp::default line 27) [2017-04-07T17:46:15+00:00] INFO: service[ntpd] enabled [2017-04-07T17:46:15+00:00] INFO: Processing cookbook_file[/etc/sysconfig/ntpd] action create (ntp::default line 40) [2017-04-07T17:46:15+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] backed up to /var/chef/backup/etc/sysconfig/ntpd.chef-20170407174615.289341 [2017-04-07T17:46:15+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] updated file contents /etc/sysconfig/ntpd [2017-04-07T17:46:15+00:00] INFO: cookbook_file[/etc/sysconfig/ntpd] sending restart action to service[ntpd] (immediate) [2017-04-07T17:46:15+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-07T17:46:15+00:00] INFO: service[ntpd] restarted [2017-04-07T17:46:15+00:00] INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 47) [2017-04-07T17:46:15+00:00] INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/ntp.conf.chef-20170407174615.591586 [2017-04-07T17:46:15+00:00] INFO: template[/etc/ntp.conf] updated file contents /etc/ntp.conf [2017-04-07T17:46:15+00:00] INFO: template[/etc/ntp.conf] sending restart action to service[ntpd] (immediate) [2017-04-07T17:46:15+00:00] INFO: Processing service[ntpd] action restart (ntp::default line 27) [2017-04-07T17:46:15+00:00] INFO: service[ntpd] restarted [2017-04-07T17:46:15+00:00] INFO: Processing service[nfs-server] action nothing (nfs::default line 29) [2017-04-07T17:46:15+00:00] INFO: Processing yum_package[nfs-utils] action install (nfs::default line 35) [2017-04-07T17:46:15+00:00] INFO: Processing service[rpcbind] action enable (nfs::default line 39) [2017-04-07T17:46:16+00:00] INFO: Processing service[rpcbind] action start (nfs::default line 39) [2017-04-07T17:46:16+00:00] INFO: Processing template[/etc/idmapd.conf] action create (nfs::default line 45) [2017-04-07T17:46:16+00:00] INFO: template[/etc/idmapd.conf] backed up to /var/chef/backup/etc/idmapd.conf.chef-20170407174616.399739 [2017-04-07T17:46:16+00:00] INFO: template[/etc/idmapd.conf] updated file contents /etc/idmapd.conf [2017-04-07T17:46:16+00:00] INFO: Processing yum_package[autofs] action install (autofs::default line 17) [2017-04-07T17:46:16+00:00] INFO: Processing template[/etc/sysconfig/autofs] action create (autofs::default line 33) [2017-04-07T17:46:16+00:00] INFO: template[/etc/sysconfig/autofs] backed up to /var/chef/backup/etc/sysconfig/autofs.chef-20170407174616.406560 [2017-04-07T17:46:16+00:00] INFO: template[/etc/sysconfig/autofs] updated file contents /etc/sysconfig/autofs [2017-04-07T17:46:16+00:00] INFO: template[/etc/sysconfig/autofs] sending restart action to service[autofs] (immediate) [2017-04-07T17:46:16+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T17:46:16+00:00] INFO: service[autofs] restarted [2017-04-07T17:46:16+00:00] INFO: Processing cookbook_file[/etc/auto.master] action create (autofs::default line 40) [2017-04-07T17:46:16+00:00] INFO: cookbook_file[/etc/auto.master] backed up to /var/chef/backup/etc/auto.master.chef-20170407174616.738268 [2017-04-07T17:46:16+00:00] INFO: cookbook_file[/etc/auto.master] updated file contents /etc/auto.master [2017-04-07T17:46:16+00:00] INFO: cookbook_file[/etc/auto.master] sending restart action to service[autofs] (immediate) [2017-04-07T17:46:16+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T17:46:17+00:00] INFO: service[autofs] restarted [2017-04-07T17:46:17+00:00] INFO: Processing bash[nsswitch-automount] action run (autofs::default line 47) [2017-04-07T17:46:17+00:00] INFO: bash[nsswitch-automount] ran successfully [2017-04-07T17:46:17+00:00] INFO: bash[nsswitch-automount] sending restart action to service[autofs] (immediate) [2017-04-07T17:46:17+00:00] INFO: Processing service[autofs] action restart (autofs::default line 63) [2017-04-07T17:46:18+00:00] INFO: service[autofs] restarted [2017-04-07T17:46:18+00:00] INFO: Processing service[autofs] action enable (autofs::default line 63) [2017-04-07T17:46:18+00:00] INFO: service[autofs] enabled [2017-04-07T17:46:18+00:00] INFO: Processing service[autofs] action start (autofs::default line 63) [2017-04-07T17:46:18+00:00] INFO: Processing yum_package[sendmail] action remove (mail::default line 16) [2017-04-07T17:46:18+00:00] INFO: Processing yum_package[postfix] action install (postfix::default line 21) [2017-04-07T17:46:18+00:00] INFO: Processing service[postfix] action enable (postfix::default line 25) [2017-04-07T17:46:18+00:00] INFO: Processing template[/etc/postfix/main.cf] action create (postfix::default line 30) [2017-04-07T17:46:19+00:00] INFO: template[/etc/postfix/main.cf] backed up to /var/chef/backup/etc/postfix/main.cf.chef-20170407174619.012017 [2017-04-07T17:46:19+00:00] INFO: template[/etc/postfix/main.cf] updated file contents /etc/postfix/main.cf [2017-04-07T17:46:19+00:00] INFO: Processing template[/etc/postfix/master.cf] action create (postfix::default line 30) [2017-04-07T17:46:19+00:00] INFO: template[/etc/postfix/master.cf] backed up to /var/chef/backup/etc/postfix/master.cf.chef-20170407174619.018266 [2017-04-07T17:46:19+00:00] INFO: template[/etc/postfix/master.cf] updated file contents /etc/postfix/master.cf [2017-04-07T17:46:19+00:00] INFO: template[/etc/postfix/master.cf] not queuing delayed action restart on service[postfix] (delayed), as it's already been queued [2017-04-07T17:46:19+00:00] INFO: Processing yum_package[mailx] action install (mail::default line 27) [2017-04-07T17:46:19+00:00] INFO: Processing execute[newaliases] action nothing (mail::default line 30) [2017-04-07T17:46:19+00:00] INFO: Processing template[/etc/aliases] action create (mail::default line 35) [2017-04-07T17:46:19+00:00] INFO: template[/etc/aliases] backed up to /var/chef/backup/etc/aliases.chef-20170407174619.026143 [2017-04-07T17:46:19+00:00] INFO: template[/etc/aliases] updated file contents /etc/aliases [2017-04-07T17:46:19+00:00] INFO: Processing yum_package[munge] action install (munge::default line 20) [2017-04-07T17:46:19+00:00] INFO: yum_package[munge] installing munge-0.5.11-1.el7 from addon-epel7-x86_64 repository [2017-04-07T17:46:21+00:00] INFO: yum_package[munge] installed munge at 0.5.11-1.el7 [2017-04-07T17:46:21+00:00] INFO: Processing cookbook_file[/etc/munge/munge.key] action create (munge::default line 47) [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] created file /etc/munge/munge.key [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] updated file contents /etc/munge/munge.key [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] owner changed to 995 [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] group changed to 992 [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] mode changed to 400 [2017-04-07T17:46:21+00:00] INFO: cookbook_file[/etc/munge/munge.key] sending restart action to service[munge] (immediate) [2017-04-07T17:46:21+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-07T17:46:21+00:00] INFO: service[munge] restarted [2017-04-07T17:46:21+00:00] INFO: Processing template[/etc/sysconfig/munge] action create (munge::default line 54) [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] created file /etc/sysconfig/munge [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] updated file contents /etc/sysconfig/munge [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] owner changed to 0 [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] group changed to 0 [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] mode changed to 644 [2017-04-07T17:46:21+00:00] INFO: template[/etc/sysconfig/munge] sending restart action to service[munge] (immediate) [2017-04-07T17:46:21+00:00] INFO: Processing service[munge] action restart (munge::default line 63) [2017-04-07T17:46:22+00:00] INFO: service[munge] restarted [2017-04-07T17:46:22+00:00] INFO: Processing service[munge] action enable (munge::default line 63) [2017-04-07T17:46:22+00:00] INFO: service[munge] enabled [2017-04-07T17:46:22+00:00] INFO: Processing service[munge] action start (munge::default line 63) [2017-04-07T17:46:22+00:00] INFO: Processing yum_package[mrsh-server] action install (mrsh::default line 10) [2017-04-07T17:46:22+00:00] INFO: yum_package[mrsh-server] installing mrsh-server-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T17:46:26+00:00] INFO: yum_package[mrsh-server] installed mrsh-server at 2.6-6.wc1 [2017-04-07T17:46:26+00:00] INFO: Processing yum_package[mrsh] action install (mrsh::default line 11) [2017-04-07T17:46:26+00:00] INFO: yum_package[mrsh] installing mrsh-2.6-6.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T17:46:28+00:00] INFO: yum_package[mrsh] installed mrsh at 2.6-6.wc1 [2017-04-07T17:46:28+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrlogind] action create (mrsh::server line 14) [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] backed up to /var/chef/backup/etc/xinetd.d/mrlogind.chef-20170407174628.418920 [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] updated file contents /etc/xinetd.d/mrlogind [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrlogind] mode changed to 644 [2017-04-07T17:46:28+00:00] INFO: Processing cookbook_file[/etc/xinetd.d/mrshd] action create (mrsh::server line 21) [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] backed up to /var/chef/backup/etc/xinetd.d/mrshd.chef-20170407174628.422884 [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] updated file contents /etc/xinetd.d/mrshd [2017-04-07T17:46:28+00:00] INFO: cookbook_file[/etc/xinetd.d/mrshd] mode changed to 644 [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_mrsh_to_securetty] action run (mrsh::server line 32) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_mrsh_to_securetty] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_mrlogin_to_securetty] action run (mrsh::server line 32) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_mrlogin_to_securetty] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[enable mrshd] action run (mrsh::server line 47) [2017-04-07T17:46:28+00:00] INFO: execute[enable mrshd] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[enable mrlogind] action run (mrsh::server line 47) [2017-04-07T17:46:28+00:00] INFO: execute[enable mrlogind] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts0_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts0_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts1_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts1_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts2_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts2_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts3_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts3_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts4_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts4_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts5_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts5_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts6_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts6_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing execute[add_pts7_to_securetty_mrsh] action run (mrsh::server line 60) [2017-04-07T17:46:28+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:28+00:00] INFO: execute[add_pts7_to_securetty_mrsh] ran successfully [2017-04-07T17:46:28+00:00] INFO: Processing service[xinetd] action start (mrsh::server line 66) [2017-04-07T17:46:29+00:00] INFO: service[xinetd] started [2017-04-07T17:46:29+00:00] INFO: Processing service[xinetd] action enable (mrsh::server line 66) [2017-04-07T17:46:29+00:00] INFO: Processing yum_package[openssh-clients] action install (ssh::default line 9) [2017-04-07T17:46:30+00:00] INFO: Processing yum_package[openssh-server] action install (ssh::server line 14) [2017-04-07T17:46:30+00:00] INFO: Processing service[sshd] action start (ssh::server line 19) [2017-04-07T17:46:30+00:00] INFO: Processing service[sshd] action enable (ssh::server line 19) [2017-04-07T17:46:30+00:00] INFO: Processing template[/etc/ssh/sshd_config] action create (ssh::server line 29) [2017-04-07T17:46:30+00:00] INFO: template[/etc/ssh/sshd_config] backed up to /var/chef/backup/etc/ssh/sshd_config.chef-20170407174630.608956 [2017-04-07T17:46:30+00:00] INFO: template[/etc/ssh/sshd_config] updated file contents /etc/ssh/sshd_config [2017-04-07T17:46:30+00:00] INFO: Processing directory[/root/.ssh/] action create (ssh::server line 36) [2017-04-07T17:46:30+00:00] INFO: directory[/root/.ssh/] created directory /root/.ssh/ [2017-04-07T17:46:30+00:00] INFO: directory[/root/.ssh/] owner changed to 0 [2017-04-07T17:46:30+00:00] INFO: directory[/root/.ssh/] group changed to 0 [2017-04-07T17:46:30+00:00] INFO: directory[/root/.ssh/] mode changed to 700 [2017-04-07T17:46:30+00:00] INFO: Processing template[/root/.ssh/authorized_keys] action create (ssh::server line 52) [2017-04-07T17:46:30+00:00] INFO: template[/root/.ssh/authorized_keys] created file /root/.ssh/authorized_keys [2017-04-07T17:46:30+00:00] INFO: template[/root/.ssh/authorized_keys] updated file contents /root/.ssh/authorized_keys [2017-04-07T17:46:30+00:00] INFO: template[/root/.ssh/authorized_keys] owner changed to 0 [2017-04-07T17:46:30+00:00] INFO: template[/root/.ssh/authorized_keys] group changed to 0 [2017-04-07T17:46:30+00:00] INFO: template[/root/.ssh/authorized_keys] mode changed to 600 [2017-04-07T17:46:30+00:00] INFO: Processing yum_package[rsyslog] action install (rsyslog::client line 12) [2017-04-07T17:46:30+00:00] INFO: Processing yum_package[ed] action install (rsyslog::client line 13) [2017-04-07T17:46:30+00:00] INFO: Processing service[rsyslog] action nothing (rsyslog::client line 22) [2017-04-07T17:46:30+00:00] INFO: Processing execute[configure_remote_syslog] action run (rsyslog::client line 26) [2017-04-07T17:46:30+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:30+00:00] INFO: execute[configure_remote_syslog] ran successfully [2017-04-07T17:46:30+00:00] INFO: Processing yum_package[rsh] action install (rsh::default line 9) [2017-04-07T17:46:30+00:00] INFO: yum_package[rsh] installing rsh-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-07T17:46:33+00:00] INFO: yum_package[rsh] installed rsh at 0.17-76.el7_1.1 [2017-04-07T17:46:33+00:00] INFO: Processing yum_package[rsh-server] action install (rsh::server line 11) [2017-04-07T17:46:33+00:00] INFO: yum_package[rsh-server] installing rsh-server-0.17-76.el7_1.1 from updates-centos7.1-x86_64 repository [2017-04-07T17:46:35+00:00] INFO: yum_package[rsh-server] installed rsh-server at 0.17-76.el7_1.1 [2017-04-07T17:46:35+00:00] INFO: Processing service[rsh.socket] action enable (rsh::server line 17) [2017-04-07T17:46:35+00:00] INFO: service[rsh.socket] enabled [2017-04-07T17:46:35+00:00] INFO: Processing service[rsh.socket] action start (rsh::server line 17) [2017-04-07T17:46:35+00:00] INFO: service[rsh.socket] started [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_rsh_to_securetty] action run (rsh::server line 35) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[add_rsh_to_securetty] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rsh] action run (rsh::server line 40) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[enable_promiscuous_pam_for_rsh] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing service[rlogin.socket] action enable (rsh::server line 17) [2017-04-07T17:46:35+00:00] INFO: service[rlogin.socket] enabled [2017-04-07T17:46:35+00:00] INFO: Processing service[rlogin.socket] action start (rsh::server line 17) [2017-04-07T17:46:35+00:00] INFO: service[rlogin.socket] started [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_rlogin_to_securetty] action run (rsh::server line 35) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[add_rlogin_to_securetty] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[enable_promiscuous_pam_for_rlogin] action run (rsh::server line 40) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[enable_promiscuous_pam_for_rlogin] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts0_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts1_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts2_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts3_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts4_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts5_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts6_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing execute[add_pts7_to_securetty_rsh] action run (rsh::server line 48) [2017-04-07T17:46:35+00:00] INFO: Processing execute[Guard resource] action run (dynamically defined) [2017-04-07T17:46:35+00:00] INFO: execute[Guard resource] ran successfully [2017-04-07T17:46:35+00:00] INFO: Processing cookbook_file[/root/.rhosts] action create (rsh::server line 54) [2017-04-07T17:46:35+00:00] INFO: cookbook_file[/root/.rhosts] created file /root/.rhosts [2017-04-07T17:46:35+00:00] INFO: cookbook_file[/root/.rhosts] updated file contents /root/.rhosts [2017-04-07T17:46:35+00:00] INFO: cookbook_file[/root/.rhosts] mode changed to 644 [2017-04-07T17:46:35+00:00] INFO: Processing yum_package[openmpi] action install (openmpi::default line 20) [2017-04-07T17:46:36+00:00] INFO: Processing cookbook_file[/etc/profile.d/openmpi.sh] action create (openmpi::default line 26) [2017-04-07T17:46:36+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] created file /etc/profile.d/openmpi.sh [2017-04-07T17:46:36+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] updated file contents /etc/profile.d/openmpi.sh [2017-04-07T17:46:36+00:00] INFO: cookbook_file[/etc/profile.d/openmpi.sh] mode changed to 644 [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[cvs] action install (test_node::default line 12) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[bc] action install (test_node::default line 13) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[expect] action install (test_node::default line 14) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[make] action install (test_node::default line 15) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[gcc] action install (test_node::default line 16) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[sg3_utils] action install (test_node::default line 17) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[patch] action install (test_node::default line 18) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[kexec-tools] action install (test_node::default line 19) [2017-04-07T17:46:36+00:00] INFO: Processing yum_package[setools-console] action install (test_node::default line 23) [2017-04-07T17:46:36+00:00] INFO: yum_package[setools-console] installing setools-console-3.3.8-1.1.el7 from core-0 repository [2017-04-07T17:46:38+00:00] INFO: yum_package[setools-console] installed setools-console at 3.3.8-1.1.el7 [2017-04-07T17:46:38+00:00] INFO: Processing yum_package[logwatch] action remove (test_node::default line 25) [2017-04-07T17:46:38+00:00] INFO: Processing yum_package[mlocate] action remove (test_node::default line 25) [2017-04-07T17:46:38+00:00] INFO: Processing yum_package[libyaml] action install (test_node::default line 31) [2017-04-07T17:46:38+00:00] INFO: Processing yum_package[libyaml-devel] action install (test_node::default line 32) [2017-04-07T17:46:38+00:00] INFO: yum_package[libyaml-devel] installing libyaml-devel-0.1.4-11.el7_0 from core-0 repository [2017-04-07T17:46:40+00:00] INFO: yum_package[libyaml-devel] installed libyaml-devel at 0.1.4-11.el7_0 [2017-04-07T17:46:40+00:00] INFO: Processing yum_package[powerman] action install (test_node::default line 34) [2017-04-07T17:46:41+00:00] INFO: yum_package[powerman] installing powerman-2.3.24-3.01.wc1 from local-toolkit_el7-x86_64 repository [2017-04-07T17:46:44+00:00] INFO: yum_package[powerman] installed powerman at 2.3.24-3.01.wc1 [2017-04-07T17:46:44+00:00] INFO: Processing yum_package[compat-glibc-headers] action install (test_node::default line 38) [2017-04-07T17:46:44+00:00] INFO: Processing service[abrtd] action disable (test_node::default line 50) [2017-04-07T17:46:44+00:00] INFO: Processing service[abrtd] action stop (test_node::default line 50) [2017-04-07T17:46:45+00:00] INFO: Processing service[rdma] action enable (test_node::default line 58) [2017-04-07T17:46:45+00:00] INFO: service[rdma] enabled [2017-04-07T17:46:45+00:00] INFO: Processing service[rdma] action start (test_node::default line 58) [ 114.639654] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 114.648754] Loading iSCSI transport class v2.0-870. [ 114.652470] iscsi: registered transport (iser) [ 114.721095] RPC: Registered named UNIX socket transport module. [ 114.721675] RPC: Registered udp transport module. [ 114.722133] RPC: Registered tcp transport module. [ 114.722555] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 114.725113] RPC: Registered rdma transport module. [ 114.725554] RPC: Registered rdma backchannel transport module. [2017-04-07T17:46:46+00:00] INFO: service[rdma] started [2017-04-07T17:46:46+00:00] INFO: Processing bash[add filesystem to /etc/fstab for kdump] action run (test_node::default line 74) [2017-04-07T17:46:46+00:00] INFO: bash[add filesystem to /etc/fstab for kdump] ran successfully [2017-04-07T17:46:46+00:00] INFO: Processing bash[ensure filesystem is mounted for kdumps] action run (test_node::default line 80) [ 114.811192] FS-Cache: Loaded [ 114.820408] FS-Cache: Netfs 'nfs' registered for caching [ 114.823487] Key type dns_resolver registered [ 114.832122] NFS: Registering the id_resolver key type [ 114.832610] Key type id_resolver registered [ 114.833120] Key type id_legacy registered [2017-04-07T17:46:46+00:00] INFO: bash[ensure filesystem is mounted for kdumps] ran successfully [2017-04-07T17:46:46+00:00] INFO: Processing template[/etc/kdump.conf] action create (test_node::default line 86) [2017-04-07T17:46:46+00:00] INFO: template[/etc/kdump.conf] backed up to /var/chef/backup/etc/kdump.conf.chef-20170407174646.153484 [2017-04-07T17:46:46+00:00] INFO: template[/etc/kdump.conf] updated file contents /etc/kdump.conf [2017-04-07T17:46:46+00:00] INFO: Processing directory[/scratch/dumps/trevis-41vm7.trevis.hpdd.intel.com] action create (test_node::default line 91) [2017-04-07T17:46:46+00:00] INFO: Processing service[kdump] action restart (test_node::default line 96) [2017-04-07T17:47:25+00:00] INFO: service[kdump] restarted [2017-04-07T17:47:25+00:00] INFO: Processing directory[/mnt/lustre] action create (test_node::default line 150) [2017-04-07T17:47:25+00:00] INFO: directory[/mnt/lustre] created directory /mnt/lustre [2017-04-07T17:47:25+00:00] INFO: directory[/mnt/lustre] mode changed to 777 [2017-04-07T17:47:25+00:00] INFO: Processing group[quota_usr] action create (test_node::default line 154) [2017-04-07T17:47:25+00:00] INFO: group[quota_usr] created [2017-04-07T17:47:25+00:00] INFO: Processing user[quota_usr] action create (test_node::default line 158) [2017-04-07T17:47:25+00:00] INFO: user[quota_usr] created [2017-04-07T17:47:25+00:00] INFO: Processing group[quota_2usr] action create (test_node::default line 164) [2017-04-07T17:47:25+00:00] INFO: group[quota_2usr] created [2017-04-07T17:47:25+00:00] INFO: Processing user[quota_2usr] action create (test_node::default line 168) [2017-04-07T17:47:25+00:00] INFO: user[quota_2usr] created [2017-04-07T17:47:25+00:00] INFO: Processing group[sanityusr] action create (test_node::default line 174) [2017-04-07T17:47:26+00:00] INFO: group[sanityusr] created [2017-04-07T17:47:26+00:00] INFO: Processing user[sanityusr] action create (test_node::default line 178) [2017-04-07T17:47:26+00:00] INFO: user[sanityusr] created [2017-04-07T17:47:26+00:00] INFO: Processing group[sanityusr1] action create (test_node::default line 184) [2017-04-07T17:47:26+00:00] INFO: group[sanityusr1] created [2017-04-07T17:47:26+00:00] INFO: Processing user[sanityusr1] action create (test_node::default line 188) [2017-04-07T17:47:26+00:00] INFO: user[sanityusr1] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_1] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_1] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_2] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_2] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_3] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_3] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_4] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_4] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_5] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_5] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_6] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_6] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_7] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_7] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_8] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_8] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_9] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_9] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_10] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_10] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_11] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_11] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_12] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_12] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_13] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_13] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_14] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_14] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_15] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_15] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_16] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_16] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_17] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_17] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_18] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_18] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_19] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_19] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_20] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_20] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_21] action create (test_node::default line 195) [2017-04-07T17:47:26+00:00] INFO: user[quota15_21] created [2017-04-07T17:47:26+00:00] INFO: Processing user[quota15_22] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_22] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_23] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_23] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_24] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_24] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_25] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_25] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_26] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_26] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_27] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_27] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_28] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_28] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_29] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_29] created [2017-04-07T17:47:27+00:00] INFO: Processing user[quota15_30] action create (test_node::default line 195) [2017-04-07T17:47:27+00:00] INFO: user[quota15_30] created [2017-04-07T17:47:27+00:00] INFO: Processing directory[/localhome] action create (test_node::default line 202) [2017-04-07T17:47:27+00:00] INFO: directory[/localhome] created directory /localhome [2017-04-07T17:47:27+00:00] INFO: directory[/localhome] owner changed to 0 [2017-04-07T17:47:27+00:00] INFO: directory[/localhome] group changed to 0 [2017-04-07T17:47:27+00:00] INFO: directory[/localhome] mode changed to 755 [2017-04-07T17:47:27+00:00] INFO: Processing bash[set_hostid] action run (test_node::default line 208) [2017-04-07T17:47:27+00:00] INFO: bash[set_hostid] ran successfully [2017-04-07T17:47:27+00:00] INFO: Processing bash[increase_memlock_limits] action run (test_node::default line 217) [2017-04-07T17:47:27+00:00] INFO: bash[increase_memlock_limits] ran successfully [2017-04-07T17:47:27+00:00] INFO: Processing bash[install_autotest_ssh_key] action run (test_node::default line 230) [2017-04-07T17:47:27+00:00] INFO: bash[install_autotest_ssh_key] ran successfully [2017-04-07T17:47:27+00:00] INFO: Processing directory[/etc/sysctl.d] action create (test_node::default line 270) [2017-04-07T17:47:27+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] action create (test_node::default line 275) [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] created file /etc/sysctl.d/90-panic_on_oom.conf [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] updated file contents /etc/sysctl.d/90-panic_on_oom.conf [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_oom.conf] mode changed to 1204 [2017-04-07T17:47:27+00:00] INFO: Processing cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] action create (test_node::default line 281) [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] created file /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] updated file contents /etc/sysctl.d/90-panic_on_soft_lockup.conf [2017-04-07T17:47:27+00:00] INFO: cookbook_file[/etc/sysctl.d/90-panic_on_soft_lockup.conf] mode changed to 1204 [2017-04-07T17:47:27+00:00] INFO: Processing chef_handler[Chef::Handler::JsonFile] action nothing (test_node::default line 288) [2017-04-07T17:47:27+00:00] INFO: Processing execute[testnode disable systemd readahead] action run (test_node::default line 300) [2017-04-07T17:47:27+00:00] INFO: execute[testnode disable systemd readahead] ran successfully [2017-04-07T17:47:27+00:00] INFO: Processing yum_package[mcelog] action install (mcelog::default line 9) [2017-04-07T17:47:27+00:00] INFO: Processing service[mcelog.service] action start (mcelog::default line 21) [2017-04-07T17:47:27+00:00] INFO: Processing user[mpiuser] action create (openmpi::test_node line 10) [2017-04-07T17:47:27+00:00] INFO: user[mpiuser] created [2017-04-07T17:47:27+00:00] INFO: Processing group[mpiuser] action create (openmpi::test_node line 29) [2017-04-07T17:47:27+00:00] INFO: group[mpiuser] created [2017-04-07T17:47:27+00:00] INFO: Processing directory[/localhome/mpiuser] action create (openmpi::test_node line 39) [2017-04-07T17:47:27+00:00] INFO: directory[/localhome/mpiuser] group changed to 502 [2017-04-07T17:47:27+00:00] INFO: Processing execute[add_rhosts_to_mpiuser] action run (openmpi::test_node line 44) [2017-04-07T17:47:27+00:00] INFO: execute[add_rhosts_to_mpiuser] ran successfully [2017-04-07T17:47:27+00:00] INFO: Processing bash[use_rsh_for_mpi] action run (openmpi::test_node line 49) [2017-04-07T17:47:27+00:00] INFO: bash[use_rsh_for_mpi] ran successfully [2017-04-07T17:47:27+00:00] INFO: template[/etc/postfix/main.cf] sending restart action to service[postfix] (delayed) [2017-04-07T17:47:27+00:00] INFO: Processing service[postfix] action restart (postfix::default line 25) [2017-04-07T17:47:28+00:00] INFO: service[postfix] restarted [2017-04-07T17:47:28+00:00] INFO: template[/etc/aliases] sending run action to execute[newaliases] (delayed) [2017-04-07T17:47:28+00:00] INFO: Processing execute[newaliases] action run (mail::default line 30) [2017-04-07T17:47:28+00:00] INFO: execute[newaliases] ran successfully [2017-04-07T17:47:28+00:00] INFO: template[/etc/ssh/sshd_config] sending restart action to service[sshd] (delayed) [2017-04-07T17:47:28+00:00] INFO: Processing service[sshd] action restart (ssh::server line 19) [2017-04-07T17:47:28+00:00] INFO: service[sshd] restarted [2017-04-07T17:47:28+00:00] INFO: execute[configure_remote_syslog] sending restart action to service[rsyslog] (delayed) [2017-04-07T17:47:28+00:00] INFO: Processing service[rsyslog] action restart (rsyslog::client line 22) [2017-04-07T17:47:29+00:00] INFO: service[rsyslog] restarted [2017-04-07T17:47:29+00:00] INFO: Chef Run complete in 136.422405319 seconds [2017-04-07T17:47:29+00:00] INFO: Skipping removal of unused files from the cache [2017-04-07T17:47:29+00:00] INFO: Running report handlers [2017-04-07T17:47:29+00:00] INFO: Creating JSON run report [2017-04-07T17:47:29+00:00] INFO: Report handlers complete [ OK ] Started Show Plymouth Reboot Screen. [ OK ] Stopped Dynamic System Tuning Daemon. [ OK ] Unmounted /scratch. [ OK ] Stopped Automounts filesystems on demand. Stopping System Security Services Daemon... [ OK ] Stopped target Remote File Systems. Unmounting /home/autotest2... [ OK ] Stopped System Security Services Daemon. [ OK ] Unmounted /home/autotest2. [ OK ] Stopped target Network is Online. [ OK ] Stopped Network Manager Wait Online. Stopping Network Manager Wait Online... [ OK ] Stopped target Network. Stopping Network Manager... [ OK ] Stopped Network Manager. Stopping D-Bus System Message Bus... [ OK ] Stopped D-Bus System Message Bus. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Closed Remote Shell Facilities Activation Socket. [ OK ] Closed RPCbind Server Activation Socket. [ OK ] Closed Remote Login Facilities Activation Socket. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Removed slice User and Session Slice. [ OK ] Closed D-Bus System Message Bus Socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... Stopping Load/Save Random Seed... Stopping Update UTMP about System Boot/Shutdown... [ OK ] Stopped target Swap. [ OK ] Stopped Mark the need to relabel after reboot. Stopping Mark the need to relabel after reboot... [ OK ] Stopped Update is Completed. Stopping Update is Completed... [ OK ] Stopped Rebuild Journal Catalog. Stopping Rebuild Journal Catalog... [ OK ] Stopped Setup Virtual Console. Stopping Setup Virtual Console... [ OK ] Stopped Rebuild Hardware Database. Stopping Rebuild Hardware Database... [ OK ] Stopped target Encrypted Volumes. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Update UTMP about System Boot/Shutdown. Stopping Security Auditing Service... [ 2082.489423] type=1305 audit(1491589173.759:1083): audit_pid=0 old=449 auid=4294967295 ses=4294967295 res=1 [ OK ] Stopped Security Auditing Service. [ 2082.491812] type=1130 audit(1491589173.761:1084): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2082.491824] type=1131 audit(1491589173.761:1085): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped Create Volatile Files and Directories. [ 2082.499644] type=1130 audit(1491589173.769:1086): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2082.499654] type=1131 audit(1491589173.769:1087): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Create Volatile Files and Directories... [ OK ] Stopped Import network configuration from initramfs. [ 2082.510248] type=1130 audit(1491589173.780:1088): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2082.510259] type=1131 audit(1491589173.780:1089): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-import-state comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Import network configuration from initramfs... [ 2082.521548] type=1130 audit(1491589173.791:1090): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 2082.526100] type=1131 audit(1491589173.796:1091): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=rhel-readonly comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ OK ] Stopped target Local File Systems. Unmounting /run/user/0... [ OK ] Stopped Configure read-only root support. [ 2082.538352] type=1130 audit(1491589173.808:1092): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Stopping Configure read-only root support... [ OK ] Unmounted /run/user/0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped target Local File Systems (Pre). [ OK ] Stopped Remount Root and Kernel File Systems. Stopping Remount Root and Kernel File Systems... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Reached target Shutdown. [ 2082.560726] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 2082.567217] systemd-journald[341]: Received SIGTERM from PID 1 (systemd-shutdow). [ 2082.573306] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 2082.578488] systemd-shutdown[1]: Unmounting file systems. [ 2082.580823] systemd-shutdown[1]: Unmounting /home. [ 2082.674605] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2082.691722] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2082.693938] EXT4-fs (vda1): re-mounted. Opts: (null) [ 2082.696082] systemd-shutdown[1]: All filesystems unmounted. [ 2082.698275] systemd-shutdown[1]: Deactivating swaps. [ 2082.700492] systemd-shutdown[1]: All swaps deactivated. [ 2082.702644] systemd-shutdown[1]: Detaching loop devices. [ 2082.705389] systemd-shutdown[1]: All loop devices detached. [ 2082.707567] systemd-shutdown[1]: Detaching DM devices. [ 2082.709696] systemd-shutdown[1]: All DM devices detached. [ 2082.712576] systemd-shutdown[1]: Rebooting. [ 2082.715264] Restarting system. [ 2082.717013] reboot: machine restart [?25lUse the ^ and v keys to change the selection. Press 'e' to edit the selected item, or 'c' for a command prompt.   CentOS Linux (3.10.0-514.10.2.el7_lustre.x86_64) 7 (Core)  CentOS Linux (0-rescue-83db70eef13f144d7a4b592f50325d07) 7 (Core)                The selected entry will be started automatically in 5s.  The selected entry will be started automatically in 4s.  The selected entry will be started automatically in 3s.  The selected entry will be started automatically in 2s.  The selected entry will be started automatically in 1s.  The selected entry will be started automatically in 0s. [?25h[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-514.10.2.el7_lustre.x86_64 (jenkins@trevis-308.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 03:07:17 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=416566bc-0301-4b51-b3fe-88b582e6c791 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] Disabled fast string operations [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffcfff] usable [ 0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0x7fffd max_arch_pfn = 0x400000000 [ 0.000000] x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106 [ 0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30] [ 0.000000] iBFT found at 0x9aff0. [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0x34fca000-0x367dcfff] [ 0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01 BXPC BXDSDT 00000001 INTL 20090123) [ 0.000000] ACPI: FACS 000000007ffffdc0 00040 [ 0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: SSDT 000000007fffd610 0010F (v01 BXPC BXSSDTPC 00000001 INTL 20090123) [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffcfff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x7fffcfff] [ 0.000000] NODE_DATA [mem 0x7ffd6000-0x7fffcfff] [ 0.000000] Reserving 320MB of memory at 512MB for crashkernel (System RAM: 2047MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 0:7ff86001, primary cpu clock [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x7fffcfff] [ 0.000000] ACPI: PM-Timer IO Port: 0xb008 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] e820: [mem 0x80000000-0xfffbbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88007fc00000 s96728 r8192 d30248 u1048576 [ 0.000000] kvm-stealtime: cpu 0, msr 7fc0f3c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 515970 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.10.2.el7_lustre.x86_64 root=UUID=416566bc-0301-4b51-b3fe-88b582e6c791 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 1694532k/2097140k available (6765k kernel code, 400k absent, 402208k reserved, 4431k data, 1680k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2. [ 0.000000] NR_IRQS:327936 nr_irqs:440 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 8388608 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2593.768 MHz processor [ 0.003000] Calibrating delay loop (skipped) preset value.. 5187.53 BogoMIPS (lpj=2593768) [ 0.004005] pid_max: default: 32768 minimum: 301 [ 0.006028] Security Framework initialized [ 0.007014] SELinux: Initializing. [ 0.008189] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.011737] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.013502] Mount-cache hash table entries: 4096 [ 0.015242] Initializing cgroup subsys memory [ 0.016012] Initializing cgroup subsys devices [ 0.017006] Initializing cgroup subsys freezer [ 0.019004] Initializing cgroup subsys net_cls [ 0.020005] Initializing cgroup subsys blkio [ 0.021005] Initializing cgroup subsys perf_event [ 0.023005] Initializing cgroup subsys hugetlb [ 0.024005] Initializing cgroup subsys pids [ 0.025005] Initializing cgroup subsys net_prio [ 0.026105] Disabled fast string operations [ 0.028019] mce: CPU supports 10 MCE banks [ 0.029056] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] tlb_flushall_shift: 6 [ 0.033176] Freeing SMP alternatives: 28k freed [ 0.038956] ACPI: Core revision 20130517 [ 0.041056] ACPI: All ACPI Tables successfully acquired [ 0.042015] ftrace: allocating 25813 entries in 101 pages [ 0.050186] smpboot: APIC(0) Converting physical 0 to logical package 0 [ 0.052004] smpboot: APIC(1) Converting physical 1 to logical package 1 [ 0.053004] smpboot: Max logical packages: 2 [ 0.055253] Enabling x2apic [ 0.056000] Enabled x2apic [ 0.056005] Switched APIC routing to physical x2apic. [ 0.059218] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.060003] smpboot: CPU0: Intel Xeon E312xx (Sandy Bridge) (fam: 06, model: 2a, stepping: 01) [ 0.063024] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. [ 0.065431] smpboot: Booting Node 0, Processors #1 OK [ 0.003000] kvm-clock: cpu 1, msr 0:7ff86041, secondary cpu clock [ 0.003000] Disabled fast string operations [ 0.069035] Brought up 2 CPUs [ 0.069031] kvm-stealtime: cpu 1, msr 7fd0f3c0 [ 0.071005] smpboot: Total of 2 processors activated (10375.07 BogoMIPS) [ 0.072507] devtmpfs: initialized [ 0.074859] EVM: security.selinux [ 0.075005] EVM: security.ima [ 0.076004] EVM: security.capability [ 0.077743] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.078029] pinctrl core: initialized pinctrl subsystem [ 0.079139] NET: Registered protocol family 16 [ 0.080263] ACPI: bus type PCI registered [ 0.081006] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.082086] PCI: Using configuration type 1 for base access [ 0.086078] ACPI: Added _OSI(Module Device) [ 0.087013] ACPI: Added _OSI(Processor Device) [ 0.088005] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.089005] ACPI: Added _OSI(Processor Aggregator Device) [ 0.091661] ACPI: Interpreter enabled [ 0.092013] ACPI: (supports S0 S5) [ 0.093005] ACPI: Using IOAPIC for interrupt routing [ 0.094017] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.097547] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.098009] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.099009] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.100018] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.101238] acpiphp: Slot [1] registered [ 0.102030] acpiphp: Slot [2] registered [ 0.103022] acpiphp: Slot [3] registered [ 0.104022] acpiphp: Slot [4] registered [ 0.105022] acpiphp: Slot [5] registered [ 0.106022] acpiphp: Slot [6] registered [ 0.107022] acpiphp: Slot [7] registered [ 0.108022] acpiphp: Slot [8] registered [ 0.109028] acpiphp: Slot [9] registered [ 0.110022] acpiphp: Slot [10] registered [ 0.111022] acpiphp: Slot [11] registered [ 0.112021] acpiphp: Slot [12] registered [ 0.113021] acpiphp: Slot [13] registered [ 0.114021] acpiphp: Slot [14] registered [ 0.115022] acpiphp: Slot [15] registered [ 0.116029] acpiphp: Slot [16] registered [ 0.117028] acpiphp: Slot [17] registered [ 0.118028] acpiphp: Slot [18] registered [ 0.119036] acpiphp: Slot [19] registered [ 0.120022] acpiphp: Slot [20] registered [ 0.121022] acpiphp: Slot [21] registered [ 0.122022] acpiphp: Slot [22] registered [ 0.123028] acpiphp: Slot [23] registered [ 0.124022] acpiphp: Slot [24] registered [ 0.125021] acpiphp: Slot [25] registered [ 0.126022] acpiphp: Slot [26] registered [ 0.127021] acpiphp: Slot [27] registered [ 0.128029] acpiphp: Slot [28] registered [ 0.129022] acpiphp: Slot [29] registered [ 0.130022] acpiphp: Slot [30] registered [ 0.131028] acpiphp: Slot [31] registered [ 0.132256] PCI host bridge to bus 0000:00 [ 0.133007] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.134006] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.135006] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.136005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.137006] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfebfffff window] [ 0.138994] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.139006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.140005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.141005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.142819] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.143013] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.162712] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.164080] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.165874] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.167073] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.168828] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.170271] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.171279] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.172014] vgaarb: loaded [ 0.173005] vgaarb: bridge control possible 0000:00:02.0 [ 0.174207] SCSI subsystem initialized [ 0.175051] ACPI: bus type USB registered [ 0.176037] usbcore: registered new interface driver usbfs [ 0.177018] usbcore: registered new interface driver hub [ 0.178057] usbcore: registered new device driver usb [ 0.179192] PCI: Using ACPI for IRQ routing [ 0.180356] NetLabel: Initializing [ 0.181007] NetLabel: domain hash size = 128 [ 0.182004] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.183040] NetLabel: unlabeled traffic allowed by default [ 0.184148] Switched to clocksource kvm-clock [ 0.193043] pnp: PnP ACPI init [ 0.194158] ACPI: bus type PNP registered [ 0.195830] pnp: PnP ACPI: found 5 devices [ 0.197053] ACPI: bus type PNP unregistered [ 0.208106] NET: Registered protocol family 2 [ 0.209704] TCP established hash table entries: 16384 (order: 5, 131072 bytes) [ 0.212073] TCP bind hash table entries: 16384 (order: 6, 262144 bytes) [ 0.213754] TCP: Hash tables configured (established 16384 bind 16384) [ 0.215390] TCP: reno registered [ 0.216531] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 0.218057] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 0.219727] NET: Registered protocol family 1 [ 0.221052] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.222553] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.224027] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.225790] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.227575] Unpacking initramfs... [ 0.628862] Freeing initrd memory: 24652k freed [ 0.636458] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.638171] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.640120] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.641701] Initialise system trusted keyring [ 0.643065] audit: initializing netlink socket (disabled) [ 0.644510] type=2000 audit(1491589201.685:1): initialized [ 0.665673] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.667275] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.669978] zpool: loaded [ 0.671105] zbud: loaded [ 0.672408] VFS: Disk quotas dquot_6.5.2 [ 0.673728] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.675524] msgmni has been set to 3357 [ 0.676857] Key type big_key registered [ 0.679705] NET: Registered protocol family 38 [ 0.681041] Key type asymmetric registered [ 0.682331] Asymmetric key parser 'x509' registered [ 0.683733] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.686186] io scheduler noop registered [ 0.687466] io scheduler deadline registered (default) [ 0.688906] io scheduler cfq registered [ 0.690286] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.691749] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.693517] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.695885] ACPI: Power Button [PWRF] [ 0.697296] GHES: HEST is not enabled! [ 0.698617] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.721702] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.723892] Non-volatile memory driver v1.3 [ 0.725395] Linux agpgart interface v0.103 [ 0.727059] crash memory driver: version 1.1 [ 0.728668] rdac: device handler registered [ 0.730267] hp_sw: device handler registered [ 0.748506] emc: device handler registered [ 0.749783] alua: device handler registered [ 0.751077] libphy: Fixed MDIO Bus: probed [ 0.752376] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.753938] ehci-pci: EHCI PCI platform driver [ 0.755264] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.756810] ohci-pci: OHCI PCI platform driver [ 0.758110] uhci_hcd: USB Universal Host Controller Interface driver [ 0.759860] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 0.761306] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 0.763664] uhci_hcd 0000:00:01.2: detected 2 ports [ 0.765099] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 [ 0.766648] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 0.768355] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.770832] usb usb1: Product: UHCI Host Controller [ 0.772293] usb usb1: Manufacturer: Linux 3.10.0-514.10.2.el7_lustre.x86_64 uhci_hcd [ 0.774808] usb usb1: SerialNumber: 0000:00:01.2 [ 0.776347] hub 1-0:1.0: USB hub found [ 0.777630] hub 1-0:1.0: 2 ports detected [ 0.779136] usbcore: registered new interface driver usbserial [ 0.780713] usbcore: registered new interface driver usbserial_generic [ 0.782376] usbserial: USB Serial support registered for generic [ 0.784036] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.787187] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.788552] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.790103] mousedev: PS/2 mouse device common for all mice [ 0.791805] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.794467] rtc_cmos 00:00: RTC can wake from S4 [ 0.795080] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.796462] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.801792] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 0.803497] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 0.805218] cpuidle: using governor menu [ 0.806613] hidraw: raw HID events driver (C) Jiri Kosina [ 0.808294] usbcore: registered new interface driver usbhid [ 0.809885] usbhid: USB HID core driver [ 0.811269] drop_monitor: Initializing network drop monitor service [ 0.813016] TCP: cubic registered [ 0.814258] Initializing XFRM netlink socket [ 0.815711] NET: Registered protocol family 10 [ 0.817412] NET: Registered protocol family 17 [ 0.819055] microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 [ 0.820498] microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 [ 0.822123] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 0.824968] Loading compiled-in X.509 certificates [ 0.826415] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.829189] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.832500] Loaded X.509 cert 'CentOS Linux kernel signing key: 7f5179e8726bacf8d425e789afb8ca5d9c01d803' [ 0.835302] registered taskstats version 1 [ 0.839285] Key type trusted registered [ 0.842601] Key type encrypted registered [ 0.845949] IMA: No TPM chip found, activating TPM-bypass! [ 0.847723] rtc_cmos 00:00: setting system clock to 2017-04-07 18:20:02 UTC (1491589202) [ 0.851114] Freeing unused kernel memory: 1680k freed [ 0.857145] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 0.861529] systemd[1]: Detected virtualization kvm. [ 0.863002] systemd[1]: Detected architecture x86-64. [ 0.864430] systemd[1]: Running in initial RAM disk. Welcome to CentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)! [ 0.866783] systemd[1]: Set hostname to . [ OK ] Reached target Timers. [ 0.901909] systemd[1]: Reached target Timers. [ 0.903224] systemd[1]: Starting Timers. [ OK ] Created slice Root Slice. [ 0.905200] systemd[1]: Created slice Root Slice. [ 0.906521] systemd[1]: Starting Root Slice. [ OK ] Listening on udev Control Socket. [ 0.908502] systemd[1]: Listening on udev Control Socket. [ 0.909883] systemd[1]: Starting udev Control Socket. [ OK ] Reached target Local File Systems. [ 0.911886] systemd[1]: Reached target Local File Systems. [ 0.913292] systemd[1]: Starting Local File Systems. [ OK ] Listening on Journal Socket. [ 0.915297] systemd[1]: Listening on Journal Socket. [ 0.916644] systemd[1]: Starting Journal Socket. [ OK ] Listening on udev Kernel Socket. [ 0.918577] systemd[1]: Listening on udev Kernel Socket. [ 0.919958] systemd[1]: Starting udev Kernel Socket. [ OK ] Reached target Sockets. [ 0.921851] systemd[1]: Reached target Sockets. [ 0.923172] systemd[1]: Starting Sockets. [ OK ] Reached target Swap. [ 0.924957] systemd[1]: Reached target Swap. [ 0.926248] systemd[1]: Starting Swap. [ OK ] Created slice System Slice. [ 0.928112] systemd[1]: Created slice System Slice. [ 0.929456] systemd[1]: Starting System Slice. [ 0.931345] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 0.934983] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 0.939686] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st... nodes for the current kernel... [ 1.025685] systemd[1]: Starting Journal Service... Starting Journal Service... [ 1.029612] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ OK ] Reached target Slices. [ 1.033654] systemd[1]: Reached target Slices. [ 1.035652] systemd[1]: Starting Slices. [ OK ] Started Apply Kernel Variables. [ 1.041042] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Create list of required sta...ce nodes for the current kernel. [ 1.051163] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 1.054314] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ 1.058681] systemd[1]: Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Setup Virtual Console. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting udev Coldplug all Devices... Mounting Configuration File System... [ OK ] Mounted Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. Starting dracut initqueue hook... Starting Show Plymouth Boot Screen... [ 1.203853] FDC 0 is a S82078B [ 1.204249] scsi host0: ata_piix [ 1.205599] scsi host1: ata_piix [ 1.205629] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 [ 1.205629] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 [ 1.212324] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.214644] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.216178] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 1.218312] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ OK ] Started Show Plymouth Boot Screen. [ OK ] Reached target Paths. [ OK ] Reached target Basic System. [ 1.227459] [drm] Initialized drm 1.1.0 20060810 [ 1.233381] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.240152] 8139cp 0000:00:03.0 eth0: RTL-8139C+ at 0xffffc900003ac000, 00:16:3e:5a:20:3f, IRQ 11 [ 1.252349] 8139too: 8139too Fast Ethernet driver 0.9.28 %G[ 1.275990] [TTM] Zone kernel: Available graphics memory: 860446 kiB [ 1.275990] [TTM] Initializing pool allocator [ 1.275992] [TTM] Initializing DMA pool allocator [ 1.276238] [drm] fb mappable at 0xF0000000 [ 1.276239] [drm] vram aper at 0xF0000000 [ 1.276239] [drm] size 33554432 [ 1.276240] [drm] fb depth is 24 [ 1.276240] [drm] pitch is 3072 [ 1.277038] vda: vda1 vda2 vda4 [ 1.350394] fbcon: cirrusdrmfb (fb0) is primary device [ 1.377766] Console: switching to colour frame buffer device 128x48 [ 1.389448] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device [ OK ] Found device /dev/disk/by-uuid/416566bc-0301-4b51-b3fe-88b582e6c791. %G Starting File System Check on /dev/...c-0301-4b51-b3fe-88b582e6c791... [ 1.395030] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0 [ OK ] Started File System Check on /dev/d...6bc-0301-4b51-b3fe-88b582e6c791. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 1.485206] EXT4-fs (vda1): mounting ext3 file system using the ext4 subsystem [ 1.486744] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 1.636058] tsc: Refined TSC clocksource calibration: 2593.643 MHz [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Starting Plymouth switch root service... [ OK ] Stopped target Timers. [ OK ] Stopped Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping dracut pre-pivot and cleanup hook... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping Apply Kernel Variables... [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ OK ] Stopped udev Coldplug all Devices. Stopping udev Coldplug all Devices... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ OK ] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ OK ] Stopped Create Static Device Nodes in /dev. Stopping Create Static Device Nodes in /dev... [ OK ] Stopped Create list of required sta...ce nodes for the current kernel. Stopping Create list of required st... nodes for the current kernel... [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. [ 1.719690] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ OK ] Started Plymouth switch root service. Starting Switch Root... [ 1.744889] SELinux: Disabled at runtime. [ 1.781024] type=1404 audit(1491589203.433:2): selinux=0 auid=4294967295 ses=4294967295 [ 1.837076] ip_tables: (C) 2000-2006 Netfilter Core Team [ 1.838527] systemd[1]: Inserted module 'ip_tables' Welcome to CentOS Linux 7 (Core)! [ OK ] Stopped Switch Root. [ OK ] Reached target Encrypted Volumes. [ OK ] Set up automount Arbitrary Executab...ats File System Automount Point. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on Delayed Shutdown Socket. [ OK ] Listening on udev Kernel Socket. Starting Read and set NIS domainname from /etc/sysconfig/network... Starting Kernel Module supporting RPCSEC_GSS... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on LVM2 poll daemon socket. Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... [ OK ] Listening on LVM2 metadata daemon socket. Starting Monitoring of LVM2 mirrors... dmeventd or progress polling... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st... nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Mounting Debug File System... [ OK ] Created slice system-selinux\x2dpol...grate\x2dlocal\x2dchanges.slice. Mounting NFSD configuration filesystem... [ 1.946692] RPC: Registered named UNIX socket transport module. [ 1.948256] RPC: Registered udp transport module. [ 1.948256] RPC: Registered tcp transport module. [ 1.948256] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.972301] EXT4-fs (vda1): re-mounted. Opts: (null) [ 1.978043] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped Journal Service. Starting Journal Service... [ OK ] Mounted Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Read and set NIS domainname from /etc/sysconfig/network. [ OK ] Started Create list of required sta...ce nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Kernel Module supporting RPCSEC_GSS. [ OK ] Started Apply Kernel Variables. [ OK ] Started LVM2 metadata daemon. Starting LVM2 metadata daemon... [ OK ] Started Journal Service. [ OK ] Mounted NFSD configuration filesystem. [ 2.008966] systemd-journald[362]: Received request to flush runtime journal from PID 1 [ OK ] Started Remount Root and Kernel File Systems. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... Starting udev Coldplug all Devices... Starting Configure read-only root support... Starting Load/Save Random Seed... Starting Flush Journal to Persistent Storage... [ OK ] Started Load/Save Random Seed. [ OK ] Started udev Kernel Device Manager. [ OK ] Started Flush Journal to Persistent Storage. [ 2.058001] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 2.066213] input: PC Speaker as /devices/platform/pcspkr/input/input4 [ OK ] Started Configure read-only root support. [ OK ] Started udev Coldplug all Devices. [ OK ] Found device /dev/ttyS0. [ OK ] Found device /dev/disk/by-uuid/2099f471-7c6f-4c85-a7b0-d5769905a720. [ 2.104175] Adding 3671036k swap on /dev/vda2. Priority:-1 extents:1 across:3671036k FS Activating swap /dev/disk/by-uuid/2...1-7c6f-4c85-a7b0-d5769905a720... [ OK ] Started Monitoring of LVM2 mirrors,...ng dmeventd or progress polling. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Preprocess NFS configuration... Starting Tell Plymouth To Write Out Runtime Data... Starting Import network configuration from initramfs... [ OK ] Activated swap /dev/disk/by-uuid/2099f471-7c6f-4c85-a7b0-d5769905a720. [ OK ] Reached target Swap. [ 2.124740] ppdev: user-space parallel port driver [ 2.130776] AES CTR mode by8 optimization enabled %G%G[ OK ] Started Preprocess NFS configuration. [ 2.161275] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 2.166485] alg: No test for crc32 (crc32-pclmul) [ OK ] Started Import network configuration from initramfs. Starting Create Volatile Files and Directories... [ OK ] Started Tell Plymouth To Write Out Runtime Data. [ 2.190328] intel_rapl: no valid rapl domains found in package 0 [ 2.213479] intel_rapl: no valid rapl domains found in package 0 [ OK ] Started Create Volatile Files and Directories. Mounting RPC Pipe File System... Starting Security Auditing Service... [ 2.230273] intel_powerclamp: No package C-state available[ OK ] Mounted RPC Pipe File System. [ 2.247898] type=1305 audit(1491589203.899:3): audit_pid=490 old=0 auid=4294967295 ses=4294967295 res=1 [ 2.260062] intel_powerclamp: No package C-state available[ OK ] Started Security Auditing Service. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Remote Shell Facilities Activation Socket. [ OK ] Listening on Remote Login Facilities Activation Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting System Security Services Daemon... [ OK ] Started irqbalance daemon. Starting irqbalance daemon... Starting GSSAPI Proxy Daemon... Starting NTP client/server... Starting Dump dmesg to /var/log/dmesg... Starting Authorization Manager... [ OK ] Started D-Bus System Message Bus. Starting D-Bus System Message Bus... Starting Network Manager... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Dump dmesg to /var/log/dmesg. Starting RPC security service for NFS client and server... [ OK ] Started NTP client/server. [ OK ] Started RPC security service for NFS client and server. [ OK ] Reached target NFS client services. [ OK ] Started Authorization Manager. [ OK ] Started Network Manager. Starting Network Manager Wait Online... Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Hostname Service... [ 2.443562] Rounding down aligned max_sectors from 4294967295 to 4294967288 [ 2.461566] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.465465] 8139cp 0000:00:03.0 eth0: link up, 100Mbps, full-duplex, lpa 0x05E1 [ 2.470200] Loading iSCSI transport class v2.0-870. [[ 2.479361] iscsi: registered transport (iser)  OK ] Started Hostname Service. [ 2.534101] RPC: Registered rdma transport module. [ 2.535681] RPC: Registered rdma backchannel transport module. [ OK ] Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started System Security Services Daemon. [ OK ] Reached target User and Group Name Lookups. Starting Login Service... [ OK ] Started Login Service. [ OK ] Started Network Manager Wait Online. Starting LSB: Bring up/down networking... [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Reached target Network. Starting Xinetd A Powerful Replacement For Inetd... Starting Postfix Mail Transport Agent... Starting Dynamic System Tuning Daemon... Starting MUNGE authentication service... Starting Notify NFS peers of a restart... Starting System Logging Service... Starting PowerMan... [ OK ] Reached target Network is Online. Mounting /scratch... Starting OpenSSH server daemon... [ OK ] Started Xinetd A Powerful Replacement For Inetd. [ OK ] Started Notify NFS peers of a restart. [FAILED] Failed to start PowerMan. See 'systemctl status powerman.service' for details. [ OK ] Started System Logging Service. [ OK ] Started MUNGE authentication service. [ OK ] Started OpenSSH server daemon. [ 9.897860] FS-Cache: Loaded [ 9.914142] FS-Cache: Netfs 'nfs' registered for caching [ 9.926881] Key type dns_resolver registered [ 9.950128] NFS: Registering the id_resolver key type [ 9.951584] Key type id_resolver registered [ 9.953193] Key type id_legacy registered [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Postfix Mail Transport Agent. [ OK ] Mounted /scratch. [ OK ] Reached target Remote File Systems. Starting Crash recovery kernel arming... Starting Automounts filesystems on demand... Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. Starting Command Scheduler... [ OK ] Started Job spooling tools. Starting Job spooling tools... Starting Wait for Plymouth Boot Screen to Quit... Starting Terminate Plymouth Boot Screen... CentOS Linux 7 (Core) Kernel 3.10.0-514.10.2.el7_lustre.x86_64 on an x86_64 trevis-41vm7 login: [ 242.903559] device-mapper: uevent: version 1.0.3 [ 242.904450] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com [ 386.845520] libcfs: loading out-of-tree module taints kernel. [ 386.846351] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 386.848994] LNet: HW nodes: 1, HW CPU cores: 2, npartitions: 1 [ 386.851794] alg: No test for adler32 (adler32-zlib) [ 386.852363] alg: No test for crc32 (crc32-table) [ 387.988706] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 388.040359] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 388.057286] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 388.203157] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 388.234657] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_hostid [ 388.274376] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_hostid [ 389.109110] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 389.167708] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 389.295089] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_hostid [ 389.423279] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_hostid [ 391.279136] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 391.517965] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 391.786093] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reformat /dev/lvm-Role_MDS/P [ 392.237284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.687715] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 393.928669] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 394.172674] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-41vm7@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --reforma [ 394.456791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.783020] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 410.783020] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 410.783020] if [ $running -ne $mpts ]; then [ 410.783020] echo $(hostname) env are INSANE!; [ 410.783020] exit 1; [ 410.783020] fi [ 411.315099] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); [ 411.315099] mpts=$(mount | grep -c /mnt/lustre-mds3' '); [ 411.315099] if [ $running -ne $mpts ]; then [ 411.315099] echo $(hostname) env are INSANE!; [ 411.315099] exit 1; [ 411.315099] fi [ 411.849377] Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); [ 411.849377] mpts=$(mount | grep -c /mnt/lustre-mds1' '); [ 411.849377] if [ $running -ne $mpts ]; then [ 411.849377] echo $(hostname) env are INSANE!; [ 411.849377] exit 1; [ 411.849377] fi [ 412.723357] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 [ 412.971049] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 [ 413.212859] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 [ 413.450680] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/lustre-mds1 [ 413.633385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 413.660607] Lustre: Lustre: Build Version: 2.9.54_77_g2fc9ae7 [ 413.679299] LNet: Added LNI 10.9.5.243@tcp [8/256/0/180] [ 413.681122] LNet: Accept all, port 7988 [ 413.692489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.715073] Lustre: MGS: Connection restored to ad7139ab-9cba-001f-dd0b-7243f02259aa (at 0@lo) [ 413.764978] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 [ 413.803943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space [ 413.831349] Lustre: lustre-MDT0000: new disk, initializing [ 413.866463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.877338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 414.150901] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 414.413124] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 414.906416] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 414.908400] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 415.067552] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 415.068672] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 415.208397] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 415.454264] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 415.702812] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 415.949672] Lustre: DEBUG MARKER: sync; sync; sync [ 418.855291] Lustre: lustre-MDT0000: Connection restored to ad7139ab-9cba-001f-dd0b-7243f02259aa (at 0@lo) [ 421.222864] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null [ 421.470527] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 422.750198] Lustre: MGS: Connection restored to df8ac74e-680c-7bca-10fd-b71b3bcc3a20 (at 10.9.5.239@tcp) [ 422.803413] Lustre: Setting parameter lustre-MDT0001-mdtlov.lov.stripesize in log lustre-MDT0001 [ 422.805836] Lustre: Skipped 4 previous similar messages [ 422.954426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.880236] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 424.040911] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 430.715741] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 [ 430.971361] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P3 [ 431.220587] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 [ 431.470644] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre /dev/lvm-Role_MDS/P3 /mnt/lustre-mds3 [ 431.743018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 431.760568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.779299] Lustre: Setting parameter lustre-MDT0002-mdtlov.lov.stripesize in log lustre-MDT0002 [ 431.782086] Lustre: Skipped 4 previous similar messages [ 431.850025] LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.857191] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space [ 431.859784] Lustre: Skipped 1 previous similar message [ 431.888306] Lustre: lustre-MDT0002: new disk, initializing [ 431.902254] Lustre: lustre-MDT0000: Connection restored to ad7139ab-9cba-001f-dd0b-7243f02259aa (at 0@lo) [ 431.905112] Lustre: Skipped 2 previous similar messages [ 431.913312] Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 [ 431.929644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt [ 431.933633] Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] [ 432.113704] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [ 432.364853] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li [ 432.849309] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 432.850873] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 433.008205] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 433.010413] Lustre: DEBUG MARKER: trevis-41vm7.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 433.153710] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 433.419970] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 433.681599] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' [ 433.929950] Lustre: DEBUG MARKER: sync; sync; sync [ 436.911388] Lustre: lustre-MDT0002: Connection restored to ad7139ab-9cba-001f-dd0b-7243f02259aa (at 0@lo) [ 436.914227] Lustre: Skipped 2 previous similar messages [ 439.256581] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P3 2>/dev/null [ 439.509869] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 [ 440.767688] Lustre: Setting parameter lustre-MDT0003-mdtlov.lov.stripesize in log lustre-MDT0003 [ 440.770408] Lustre: Skipped 4 previous similar messages [ 441.190760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3:mdt [ 442.117542] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 442.272147] Lustre: DEBUG MARKER: trevis-41vm3.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 450.337329] Lustre: MGS: Connection restored to ec80f8f5-db08-b7b8-bc55-1391be83d107 (at 10.9.5.244@tcp) [ 450.340595] Lustre: Skipped 3 previous similar messages [ 451.308870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:ost [ 452.042175] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 452.199182] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 460.534801] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 460.763599] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 461.275253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:1:ost [ 468.150173] Lustre: lustre-MDT0000: Connection restored to ec80f8f5-db08-b7b8-bc55-1391be83d107 (at 10.9.5.244@tcp) [ 468.153357] Lustre: Skipped 4 previous similar messages [ 469.117493] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 469.272128] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 477.538254] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 477.698137] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 477.955149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000003c0000400-0x0000000400000400]:3:ost [ 477.958391] Lustre: Skipped 1 previous similar message [ 486.353084] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 486.505547] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 494.747585] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 494.908797] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 502.208425] Lustre: lustre-MDT0000: Connection restored to ec80f8f5-db08-b7b8-bc55-1391be83d107 (at 10.9.5.244@tcp) [ 502.211761] Lustre: Skipped 6 previous similar messages [ 503.329135] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 503.492686] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 512.053890] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 512.222155] Lustre: DEBUG MARKER: trevis-41vm8.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 513.526079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000004c0000400-0x0000000500000400]:7:ost [ 513.530377] Lustre: Skipped 3 previous similar messages [ 519.730462] Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 519.890301] Lustre: DEBUG MARKER: trevis-41vm2.trevis.hpdd.intel.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 521.787032] Lustre: DEBUG MARKER: lctl get_param -n timeout [ 522.097745] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 [ 522.262465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 522.396659] Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l [ 522.686746] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.jobid_var='procname_uid' [ 530.287215] Lustre: DEBUG MARKER: /usr/sbin/lctl mark Client: Lustre version: 2.9.54_77_g2fc9ae7 [ 530.445197] Lustre: DEBUG MARKER: Client: Lustre version: 2.9.54_77_g2fc9ae7 [ 530.577099] Lustre: DEBUG MARKER: /usr/sbin/lctl lustre_build_version [ 530.884621] Lustre: DEBUG MARKER: /usr/sbin/lctl mark MDS: Lustre version: 2.9.54_77_g2fc9ae7 [ 531.045097] Lustre: DEBUG MARKER: MDS: Lustre version: 2.9.54_77_g2fc9ae7 [ 531.501031] Lustre: DEBUG MARKER: /usr/sbin/lctl mark OSS: Lustre version: 2.9.54_77_g2fc9ae7 [ 531.658553] Lustre: DEBUG MARKER: OSS: Lustre version: 2.9.54_77_g2fc9ae7 [ 532.684355] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts [ 532.965508] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 [ 533.248110] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005fa38600 x1564045072154112/t0(0) o13->lustre-OST0006-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 533.254130] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 533.535664] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 533.538523] Lustre: Skipped 1 previous similar message [ 534.848094] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005fa38600 x1564045072154160/t0(0) o13->lustre-OST0000-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 535.513241] LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation obd_ping to node 0@lo failed: rc = -107 [ 535.516426] Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.521907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.524747] Lustre: Skipped 3 previous similar messages [ 536.352074] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005f4d7600 x1564045072154336/t0(0) o13->lustre-OST0001-osc-MDT0000@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 536.357869] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 538.535058] Lustre: lustre-MDT0000: Not available for connect from 10.9.5.239@tcp (stopping) [ 538.538040] Lustre: Skipped 9 previous similar messages [ 539.219984] Lustre: server umount lustre-MDT0000 complete [ 539.348030] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST ' [ 540.513563] LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation obd_ping to node 10.9.5.239@tcp failed: rc = -107 [ 540.516451] LustreError: Skipped 1 previous similar message [ 540.518845] Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.9.5.239@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 540.523988] Lustre: Skipped 1 previous similar message [ 543.535031] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.540318] LustreError: Skipped 1 previous similar message [ 545.513227] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.518529] LustreError: Skipped 1 previous similar message [ 546.487495] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts [ 546.737957] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 [ 547.696099] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800639ecc00 x1564045072154960/t0(0) o13->lustre-OST0004-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 547.701595] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 548.535198] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.239@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.540438] LustreError: Skipped 9 previous similar messages [ 548.549025] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.239@tcp (stopping) [ 549.288061] Lustre: 5292:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1491589742/real 1491589742] req@ffff88005f4d7600 x1564045072154512/t0(0) o400->MGC10.9.5.243@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1491589749 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 549.296077] LustreError: 166-1: MGC10.9.5.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.712066] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880063a2e400 x1564045072155088/t0(0) o13->lustre-OST0001-osc-MDT0002@10.9.5.244@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [ 551.717975] LustreError: 5292:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 552.846903] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.9.5.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.852949] LustreError: Skipped 7 previous similar messages [ 552.861614] Lustre: lustre-MDT0002: Not available for connect from 10.9.5.244@tcp (stopping) [ 552.864538] Lustre: Skipped 1 previous similar message [ 552.948973] Lustre: server umount lustre-MDT0002 complete [ 553.074309] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '